Compare commits
4
Commits
f4e614acce
...
50e459ed79
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
50e459ed79 | ||
|
|
c422deee7c | ||
|
|
23dc906da0 | ||
|
|
810529d502 |
@@ -21,6 +21,7 @@ Nachbarschaft betreibt ihre eigene Instanz nach eigenen Regeln.
|
|||||||
| [[Ethos]] | Werte: Gemeingut, Fairness, Leechen erlaubt, legale Basis |
|
| [[Ethos]] | Werte: Gemeingut, Fairness, Leechen erlaubt, legale Basis |
|
||||||
| [[Idee]] | Kernidee + ehrlicher Realitätscheck |
|
| [[Idee]] | Kernidee + ehrlicher Realitätscheck |
|
||||||
| [[Architektur]] | Schichten: Physik → Routing → Pooling → Freenet-Inhalte |
|
| [[Architektur]] | Schichten: Physik → Routing → Pooling → Freenet-Inhalte |
|
||||||
|
| [[Overlay]] | Yggdrasil als Overlay: Auswahl, Deployment, Grenzen |
|
||||||
| [[Recht]] | Deutsche Rechtslage + Pflichtregeln für die Software |
|
| [[Recht]] | Deutsche Rechtslage + Pflichtregeln für die Software |
|
||||||
| [[Splitting]] | Was sich splitten lässt: Ranges, Flow-Verteilung, Realtime |
|
| [[Splitting]] | Was sich splitten lässt: Ranges, Flow-Verteilung, Realtime |
|
||||||
| [[Lizenz]] | Lizenz-Optionen (Code & Doku) |
|
| [[Lizenz]] | Lizenz-Optionen (Code & Doku) |
|
||||||
@@ -41,6 +42,8 @@ Nachbarschaft betreibt ihre eigene Instanz nach eigenen Regeln.
|
|||||||
- [[Recht]] §5 (Pflichtregeln) ↔ [[Server-README]] (Haftungsausschluss)
|
- [[Recht]] §5 (Pflichtregeln) ↔ [[Server-README]] (Haftungsausschluss)
|
||||||
- [[Idee]] (Realitätscheck) ↔ [[Architektur]] (link-agnostische Physik, `sn-fetch`)
|
- [[Idee]] (Realitätscheck) ↔ [[Architektur]] (link-agnostische Physik, `sn-fetch`)
|
||||||
- [[Splitting]] ↔ [[Architektur]] (Schicht 2) — was der Exit-Pool tatsächlich nutzen kann
|
- [[Splitting]] ↔ [[Architektur]] (Schicht 2) — was der Exit-Pool tatsächlich nutzen kann
|
||||||
|
- [[Overlay]] ↔ [[Architektur]] (Schicht 1) — Routing-Entscheidung und deren Grenzen
|
||||||
|
- [[Overlay]] ↔ [[Server-README]] — `sn-node` generiert die Yggdrasil-Config, `sn-fetch` nutzt `Egress::Proxy`
|
||||||
- [[Lizenz]] ↔ [[Ethos]] §6 (Transparenz) und `freenet-core` (AGPL)
|
- [[Lizenz]] ↔ [[Ethos]] §6 (Transparenz) und `freenet-core` (AGPL)
|
||||||
|
|
||||||
## Außerhalb der Doku
|
## Außerhalb der Doku
|
||||||
|
|||||||
@@ -53,7 +53,8 @@ mehr Pooling-Gewinn.
|
|||||||
|
|
||||||
## Schicht 1 — Routing (Overlay über alle Haushalte)
|
## Schicht 1 — Routing (Overlay über alle Haushalte)
|
||||||
|
|
||||||
Kandidaten (Auswahl offen, Kriterium: simpel, wartbar, verschlüsselt):
|
Auswahl, Deployment und Grenzen stehen in [[Overlay]] — kurz: **Yggdrasil
|
||||||
|
ist gewählt** (erste Wahl, bewusst austauschbar gehalten):
|
||||||
|
|
||||||
| Kandidat | Pro | Con |
|
| Kandidat | Pro | Con |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
# Overlay: Yggdrasil
|
||||||
|
|
||||||
|
Wie die Häuser untereinander erreichbar werden, bevor irgendwas gepoolt
|
||||||
|
wird. Stand: konzeptionell entschieden (erste Wahl), `sn-node`-Anbindung
|
||||||
|
begonnen (siehe [[Server-README]]).
|
||||||
|
|
||||||
|
## Warum Yggdrasil?
|
||||||
|
|
||||||
|
[Yggdrasil](https://yggdrasilnetwork.github.io/) ist ein
|
||||||
|
**self-contained Userspace-Router**: Jeder Node bekommt eine stabilen
|
||||||
|
IPv6-Adresse im `200::/7`-Raum, Nodes verbinden sich zu einem
|
||||||
|
end-to-end-verschlüsselten Overlay (Routing über einen
|
||||||
|
Spanning-Tree), egal ob Glasfaser, LAN, WLAN oder Internet dazwischen
|
||||||
|
liegt.
|
||||||
|
|
||||||
|
Pro:
|
||||||
|
|
||||||
|
- **Out-of-the-box verschlüsselt** — Peering-Verbindungen sind
|
||||||
|
kryptographisch gesichert, ohne extra WireGuard-Stapel
|
||||||
|
- **Null Konfiguration fürs Peering**: auf demselben LAN (also über die
|
||||||
|
Nachbarschafts-Faser oder den Keller-Switch) finden sich Nodes per
|
||||||
|
Multicast automatisch
|
||||||
|
- **Link-agnostisch** wie SpiderNet selbst: tcp://, tls://, quic://
|
||||||
|
Listener und Peer-URIs, Funk/LAN/Faser egal
|
||||||
|
- **Feste Adresse pro Keypair** — die Node-Adresse bleibt, solange der
|
||||||
|
Schlüssel bleibt (wichtig für Exit-Zuteilung und Nachbarschafts-ACL)
|
||||||
|
- **`AllowedPublicKeys`**: nur gelistete Nachbarn dürfen peeren —
|
||||||
|
das Nachbarschafts-Vertrauen wird direkt vom Overlay erzwungen
|
||||||
|
- Single Rust/CPU-Binary, läuft auf Mini-PCs und Einplatinen
|
||||||
|
|
||||||
|
Contra (ehrlich benannt):
|
||||||
|
|
||||||
|
- Experimenteller als Babel; Performance-Obergrenze liegt deutlich
|
||||||
|
unter der, die dichte Glasfaser theoretisch bieten könnte — für
|
||||||
|
Phase 1 (2–3 Haushalte) völlig ausreichend, Messwerte dokumentieren
|
||||||
|
- Wenn das Overlay zum Flaschenhals wird: Babel + WireGuard als
|
||||||
|
Ausweich-Kandidat (siehe [[Architektur]], Schicht 1)
|
||||||
|
|
||||||
|
## Rollen im SpiderNet-Node
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph Haushalt
|
||||||
|
DEVICES[Geraete] -->|own uplink| ROUTER[Router/ISP]
|
||||||
|
NODE[sn-node] --- YG[yggdrasil daemon]
|
||||||
|
YG <-->|mesh links| NACHBAR[yggdrasil der Nachbarn]
|
||||||
|
NODE -->|getSelf / getPeers| YG
|
||||||
|
NODE -->|sn-exit: HTTP proxy| YG
|
||||||
|
end
|
||||||
|
NACHBAR -->| Exit requests| ORIGIN((Origin))
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **yggdrasil-Daemon** (Prozess/Sidecar, nicht von uns implementiert):
|
||||||
|
bildet das Overlay, jede Faser wird zur Overlay-Verbindung.
|
||||||
|
2. **sn-node** (unser Daemon): generiert die yggdrasil.conf aus
|
||||||
|
validierten Einstellungen, fragt den Admin-Socket ab (Node-Adresse,
|
||||||
|
Peers), orchestriert Exit und Fetch.
|
||||||
|
3. **sn-exit** (geplant): HTTP-Proxy, der auf der Yggdrasil-Adresse des
|
||||||
|
Haushalts lauscht — Nachbarn senden ihre Segment-Requests dorthin,
|
||||||
|
der Haushalt holt sie über seinen Uplink.
|
||||||
|
4. **sn-fetch**: kennt pro Exit einen `Egress` (`Direct` = eigener
|
||||||
|
Uplink, `Proxy` = HTTP-Proxy eines Nachbarn) und plant Segmente
|
||||||
|
darauf.
|
||||||
|
|
||||||
|
## Admin-Socket (technische Fakten)
|
||||||
|
|
||||||
|
Quelle: [Admin-API-Doku](https://yggdrasil-network.github.io/admin.html)
|
||||||
|
|
||||||
|
- Default `localhost:9001` (TCP), alternativ Unix-Socket
|
||||||
|
(`AdminListen` in der Konfig)
|
||||||
|
- Protokoll: JSON-Stanza + `\n`; Anfrage `{"request": "getself"}`;
|
||||||
|
Antwort enthält immer `"status"` (`"success"`/`"error"`), optional
|
||||||
|
`"response"` und `"error"`
|
||||||
|
- `getSelf` → eigene Adresse, routed `/64`-Subnetz (`300:…::/64`),
|
||||||
|
Public Key, Build-Version
|
||||||
|
- `getPeers` → aktive Peer-Sessions inkl. Peering-URI
|
||||||
|
- `sn-node status` kapselt genau das (siehe unten)
|
||||||
|
|
||||||
|
## Deployment: zwei Häuser verkabeln
|
||||||
|
|
||||||
|
Schritt-für-Schritt für Phase 1 (Prototyp-Scope, **keine Garantie**,
|
||||||
|
siehe [[Recht]]):
|
||||||
|
|
||||||
|
1. **Hardware:** je Haus ein Mini-PC mit zwei NICs (eine zum
|
||||||
|
Heim-Router, eine zur Nachbarschafts-Faser bzw. zum Switch im
|
||||||
|
Keller) — oder übergangsweise eine NIC am Mesh-Switch.
|
||||||
|
2. **Yggdrasil installieren** (Debian-Paket oder Binary).
|
||||||
|
3. **Config generieren:**
|
||||||
|
`sn-node genconf --peer "tcp://[<Nachbar-Adresse>]:1337" --listen "tcp://[::]:1337" --allow-key <Nachbar-Public-Key> --out /etc/yggdrasil.conf`
|
||||||
|
- im selben LAN reicht Multicast-Auto-Peering ohne `--peer`
|
||||||
|
- `--no-multicast`, wenn Peering nur über die explizite Liste laufen
|
||||||
|
soll (z. B. gemischte Nachbarschaft)
|
||||||
|
- die `allowed_public_keys`-ACL: nur die gelisteten Nachbarn können
|
||||||
|
peeren — für ein Viertel die persönliche Freischaltung
|
||||||
|
4. **Start:** `yggdrasil -useconffile /etc/yggdrasil.conf`
|
||||||
|
5. **Status:** `sn-node status` → eigene Overlay-Adresse, Subnetz,
|
||||||
|
Version, verbundene Nachbarn
|
||||||
|
6. **Test:** von Haus A `ping <Adresse von Haus B>` (IPv6 im Overlay).
|
||||||
|
7. **Exit anbieten** (geplanter `sn-exit`): HTTP-Proxy auf der
|
||||||
|
Overlay-Adresse binden; beim Nachbarn dann
|
||||||
|
`sn-fetch <url> -e "1=1,2=1@http://<Nachbar-Adresse>:8080"`.
|
||||||
|
|
||||||
|
Die Config-Datei enthält den **privaten** Schlüssel des Nodes:
|
||||||
|
`chmod 600` (macht `sn-node genconf` selbst), niemals committen, nie
|
||||||
|
teilen. Private Adressen realer Nachbarn gehören nicht ins Repo
|
||||||
|
(Regeln in `AGENTS.md`).
|
||||||
|
|
||||||
|
## Grenzen & Sicherheit
|
||||||
|
|
||||||
|
- **Yggdrasil ist keine Anonymisierung.** Die Overlay-Adresse ist ein
|
||||||
|
stabiler Pseudonym; wer Peering-Verbindungen beobachtet, sieht, wer
|
||||||
|
mit wem verbunden ist. SpiderNet verspricht deshalb nirgends
|
||||||
|
Anonymität (konsistent mit Freenet-Position in [[Ethos]]).
|
||||||
|
- **Exit = Vertrauensstellung:** Der Exit-Haushalt sieht die URLs der
|
||||||
|
Segmente, die er für Nachbarn holt. Standard-Kontingente, jederzeit
|
||||||
|
widerruflich (Kill-Switch, siehe [[Architektur]]).
|
||||||
|
- **Port-Forwarding:** Eingehende Peerings über das Internet (z. B.
|
||||||
|
wenn zwei Häuser nicht direkt verbunden sind und den Weg übers
|
||||||
|
normale Internet nehmen) brauchen ein Forward des `Listen`-Ports;
|
||||||
|
innerhalb der Faser/Multicast nicht nötig.
|
||||||
|
- **Wenn die Faser später schneller als Yggdrasil wird:** Messwerte
|
||||||
|
sammeln; ggf. pro Link WireGuard-Punkt-zu-Punkt direkt nutzen und
|
||||||
|
Yggdrasil nur als Control-Plane — Architektur-Schicht 1 ist bewusst
|
||||||
|
austauschbar gehalten.
|
||||||
|
|
||||||
|
## Quellen
|
||||||
|
|
||||||
|
- Yggdrasil-Website & Docs: <https://yggdrasilnetwork.github.io/>
|
||||||
|
- Admin API: <https://yggdrasil-network.github.io/admin.html>
|
||||||
|
- Configuration: <https://yggdrasil-network.github.io/configuration.html>
|
||||||
|
(genconf/useconf, HJSON/JSON, `Peers`, `Listen`, `MulticastInterfaces`,
|
||||||
|
`AllowedPublicKeys`, `IfName: none` für router-only)
|
||||||
|
- Quellcode (Go, AGPL-3.0 — wir rufen nur das Binary auf, keine
|
||||||
|
Derivate): <https://github.com/yggdrasil-network/yggdrasil-go>
|
||||||
@@ -18,8 +18,14 @@ Ziel: Beweis, dass Pooling über zwei Anschlüsse real funktioniert.
|
|||||||
- [x] `sn-fetch` PoC (lokal): Range-Probe, gewichteter Planner, parallele
|
- [x] `sn-fetch` PoC (lokal): Range-Probe, gewichteter Planner, parallele
|
||||||
Segment-Downloads, Reassembly + SHA-256 — verifiziert gegen einen
|
Segment-Downloads, Reassembly + SHA-256 — verifiziert gegen einen
|
||||||
echten Server (Debian-Mirror); Multi-Exit-Routing folgt mit dem Mesh
|
echten Server (Debian-Mirror); Multi-Exit-Routing folgt mit dem Mesh
|
||||||
- [ ] SpiderNet-Node auf Mini-PC (Raspberry Pi / N100-Box), Yggdrasil-Overlay
|
- [x] `sn-fetch`: Segment-Retry mit Exit-Reassignment, Progress-Sinks,
|
||||||
zwischen 2 Haushalten (Kabel oder Kurzstrecke WLAN/60-GHz)
|
per-Exit-Egress (`Direct`/`Proxy`) — Nachbar-Exit über HTTP-Proxy
|
||||||
|
end-to-end getestet
|
||||||
|
- [x] `sn-node` v0: Yggdrasil-Config-Generierung (`genconf`) + Admin-Socket-
|
||||||
|
Client (`status`) — die Overlay-Anbindung beginnt
|
||||||
|
- [ ] Yggdrasil-Overlay zwischen 2 Haushalten (Kabel oder Kurzstrecke
|
||||||
|
WLAN/60-GHz), `sn-node`-Daemon orchestriert den Node
|
||||||
|
- [ ] `sn-exit` v0: HTTP-Proxy auf der Yggdrasil-Adresse (Kontingente)
|
||||||
- [ ] `sn-fetch` PoC über 2 echte Exits, plus Baseline-Messung
|
- [ ] `sn-fetch` PoC über 2 echte Exits, plus Baseline-Messung
|
||||||
(1-Exit vs. 2-Exit, gleiche Datei)
|
(1-Exit vs. 2-Exit, gleiche Datei)
|
||||||
- [ ] Fair-Share-Scheduler v0: zwei gleichzeitige Downloads, faire Aufteilung
|
- [ ] Fair-Share-Scheduler v0: zwei gleichzeitige Downloads, faire Aufteilung
|
||||||
@@ -29,8 +35,8 @@ Ziel: Beweis, dass Pooling über zwei Anschlüsse real funktioniert.
|
|||||||
|
|
||||||
Ziel: ein Paket, das 3–10 Haushalte installieren können.
|
Ziel: ein Paket, das 3–10 Haushalte installieren können.
|
||||||
|
|
||||||
- [ ] `sn-node` Daemon: Discovery, Peering, Metriken (per-Exit Bandbreite/
|
- [ ] `sn-node` Daemon v1: Metriken aus Yggdrasil-Peering (per-Exit
|
||||||
Latenz), Onboarding-Flow (QR/Knopfdruck)
|
Bandbreite/Latenz), Onboarding-Flow (QR/Knopfdruck)
|
||||||
- [ ] Exit-Pool: Kontingente, opt-in/opt-out, Leech-Modus, Kill-Switch
|
- [ ] Exit-Pool: Kontingente, opt-in/opt-out, Leech-Modus, Kill-Switch
|
||||||
- [ ] `sn-fetch`: API + CLI + einfache Browser-Integration
|
- [ ] `sn-fetch`: API + CLI + einfache Browser-Integration
|
||||||
- [ ] `sn-cache`: lancache-artiger transparenter Cache (DNS-basiert)
|
- [ ] `sn-cache`: lancache-artiger transparenter Cache (DNS-basiert)
|
||||||
|
|||||||
+3
-2
@@ -1,6 +1,6 @@
|
|||||||
[workspace]
|
[workspace]
|
||||||
resolver = "2"
|
resolver = "2"
|
||||||
members = ["crates/sn-fetch"]
|
members = ["crates/sn-fetch", "crates/sn-node"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.1.0"
|
version = "0.1.0"
|
||||||
@@ -18,7 +18,8 @@ panic = "deny"
|
|||||||
unreachable = "deny"
|
unreachable = "deny"
|
||||||
|
|
||||||
[workspace.dependencies]
|
[workspace.dependencies]
|
||||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "fs", "io-util"] }
|
tokio = { version = "1", features = ["rt-multi-thread", "macros", "fs", "io-util", "net", "time"] }
|
||||||
|
serde_json = "1"
|
||||||
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
|
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
|
||||||
sha2 = "0.10"
|
sha2 = "0.10"
|
||||||
hex = "0.4"
|
hex = "0.4"
|
||||||
|
|||||||
+30
-8
@@ -12,17 +12,39 @@ macht. Sprache: **Rust**.
|
|||||||
Exit, Mindest-Segmentgröße mit Merge-Fix, validated: lückenlose,
|
Exit, Mindest-Segmentgröße mit Merge-Fix, validated: lückenlose,
|
||||||
vollständige Abdeckung)
|
vollständige Abdeckung)
|
||||||
- Parallele Segment-Downloads (Streaming, eigener reqwest-Client pro Exit)
|
- Parallele Segment-Downloads (Streaming, eigener reqwest-Client pro Exit)
|
||||||
|
- **Retry mit Exit-Reassignment:** fehlgeschlagene Segmente werden (wieder-
|
||||||
|
holbar, Standard 3 Versuche) bevorzugt auf einem anderen Exit wiederholt;
|
||||||
|
nicht-retrybare Fehler (4xx, korrupte Parts) brechen sofort ab — ein
|
||||||
|
halbfertiges File wird nie als Erfolg ausgegeben
|
||||||
|
- **Progress:** `ProgressSink`-Events (Segment started/finished/failed,
|
||||||
|
`BytesTracker`) + `tracing`-Logging
|
||||||
|
- **Per-Exit-Egress:** `Direct` (eigener Uplink) oder `Proxy` (HTTP-Proxy
|
||||||
|
eines Nachbarn — der Weg für `sn-exit` über das Overlay); end-to-end
|
||||||
|
getestet gegen einen echten Forward-Proxy
|
||||||
- Reassembly in Index-Reihenfolge + Streaming-SHA-256 + Längen-Check
|
- Reassembly in Index-Reihenfolge + Streaming-SHA-256 + Längen-Check
|
||||||
- Fallback: range-hostile Server werden als single-stream geladen
|
- Fallback: range-hostile Server werden als single-stream geladen
|
||||||
- CLI: `sn-fetch <url> [-o out] [-e "1=2,2=1"] [-s n] [--sha256 hex] --probe-only`
|
- CLI: `sn-fetch <url> [-o out] [-e "1=2,2=1@http://[ygg]:8080"] [-s n]
|
||||||
|
[--max-attempts n] [--sha256 hex] --probe-only`
|
||||||
|
|
||||||
Verifiziert gegen einen echten Server (Debian-CD-Mirror, 756 MB, 8 Segmente,
|
Verifiziert gegen einen echten Server (Debian-CD-Mirror, 756 MB, 8 Segmente,
|
||||||
SHA-256 entspricht der offiziellen `SHA256SUMS`).
|
SHA-256 entspricht der offiziellen `SHA256SUMS`).
|
||||||
|
|
||||||
**Noch nicht drin:** Multi-Exit-Routing über das Mesh (aktuell laufen alle
|
**`sn-node` v0 ist implementiert** (siehe `crates/sn-node/`):
|
||||||
Exits als lokale Verbindungen — die Struktur [`HttpExit`]-Trait,
|
|
||||||
Exit-Registry] ist dafür schon bereit), Retry bei Segment-Fehlern,
|
- `sn-node genconf`: erzeugt eine Yggdrasil-Config aus validierten
|
||||||
Progress-Anzeige während des Laufs.
|
Einstellungen — Basis ist immer der echte `yggdrasil -genconf -json`
|
||||||
|
Output (keine hartcodierten Defaults), gepatcht mit `Peers`, `Listen`,
|
||||||
|
`MulticastInterfaces`, `AdminListen`, `AllowedPublicKeys` (Nachbarschafts-
|
||||||
|
ACL), `IfName`, `PrivateKeyPath`; Output-Datei wird auf 600 gesetzt
|
||||||
|
- `sn-node status`: fragt den Admin-Socket ab (`getSelf`/`getPeers` über
|
||||||
|
TCP `localhost:9001` oder Unix-Socket) — Overlay-Adresse, Subnetz,
|
||||||
|
Version, aktive Peer-Verbindungen
|
||||||
|
|
||||||
|
**Noch nicht drin:** `sn-exit` (der Proxy-Dienst, der auf der Yggdrasil-
|
||||||
|
Adresse lauscht und Segmente für Nachbarn holt), `sn-node`-Daemon (Supervisor,
|
||||||
|
der yggdrasil + sn-exit + sn-fetch orchestriert), Fairness-Scheduler.
|
||||||
|
|
||||||
|
Details zum Overlay: [[Overlay]].
|
||||||
|
|
||||||
## Haftungsausschluss (muss im Produkt sichtbar sein)
|
## Haftungsausschluss (muss im Produkt sichtbar sein)
|
||||||
|
|
||||||
@@ -52,14 +74,14 @@ Einplatinenrechner), der:
|
|||||||
|
|
||||||
| Crate/Binary | Zweck |
|
| Crate/Binary | Zweck |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `sn-node` | Daemon: Discovery, Peering, Metriken, Onboarding |
|
| `sn-node` | Node-Glue: Yggdrasil-Config (`genconf`), Overlay-Status (`status`), später Daemon/Supervisor |
|
||||||
| `sn-exit` | Exit-Dienst: kontingentierter Uplink-Proxy für Nachbarn (verschlüsselt) |
|
| `sn-exit` | Exit-Dienst: kontingentierter Uplink-Proxy auf der Yggdrasil-Adresse (geplant) |
|
||||||
| `sn-fetch` | Segmentierter Multi-Exit-Downloader (HTTP-API + CLI) |
|
| `sn-fetch` | Segmentierter Multi-Exit-Downloader (HTTP-API + CLI) |
|
||||||
| `sn-fair` | Fairness-Scheduler (max-min, Exit-Kontingente, Karma, Leech-Modus) |
|
| `sn-fair` | Fairness-Scheduler (max-min, Exit-Kontingente, Karma, Leech-Modus) |
|
||||||
| `sn-cache` | Transparenter Nachbarschafts-Cache (DNS-basiert) |
|
| `sn-cache` | Transparenter Nachbarschafts-Cache (DNS-basiert) |
|
||||||
| `sn-freenet` | Integration von `freenet-core` (Contracts für Nachbarschafts-Inhalte) |
|
| `sn-freenet` | Integration von `freenet-core` (Contracts für Nachbarschafts-Inhalte) |
|
||||||
|
|
||||||
## MVP (Phase 2 der [[Roadmap]])
|
## MVP (Phase 1→2, siehe [[Roadmap]])
|
||||||
|
|
||||||
Kleinster brauchbarer Schnitt:
|
Kleinster brauchbarer Schnitt:
|
||||||
|
|
||||||
|
|||||||
@@ -47,4 +47,12 @@ pub enum FetchError {
|
|||||||
|
|
||||||
#[error("invalid sha256 hex string")]
|
#[error("invalid sha256 hex string")]
|
||||||
InvalidDigest,
|
InvalidDigest,
|
||||||
|
|
||||||
|
#[error("segment {index}: all {attempts} attempt(s) failed; last error: {last_error}")]
|
||||||
|
SegmentAttemptsExhausted {
|
||||||
|
index: SegmentIndex,
|
||||||
|
attempts: u32,
|
||||||
|
#[source]
|
||||||
|
last_error: Box<FetchError>,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,18 @@
|
|||||||
//! Execution: download planned segments in parallel and reassemble the
|
//! Execution: download planned segments in parallel (with per-segment
|
||||||
//! target file.
|
//! retry and exit reassignment) and reassemble the target file.
|
||||||
|
//!
|
||||||
|
//! API note: [`SegmentStats`] keeps its original shape and `run_segments`
|
||||||
|
//! keeps its signature, so existing callers (runner.rs constructs
|
||||||
|
//! `SegmentStats` literals directly) stay source-compatible. The
|
||||||
|
//! retry-aware path [`run_segments_with_policy`] returns
|
||||||
|
//! [`SegmentAttemptStats`] instead, which additionally records the exit
|
||||||
|
//! that served the successful attempt and how many attempts (retries
|
||||||
|
//! included) the segment needed; `run_segments` converts via `From`.
|
||||||
|
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
use std::num::NonZeroU32;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::{Arc, Mutex, MutexGuard};
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
use sha2::{Digest, Sha256};
|
use sha2::{Digest, Sha256};
|
||||||
@@ -10,6 +21,7 @@ use tokio::io::AsyncWriteExt;
|
|||||||
use crate::error::FetchError;
|
use crate::error::FetchError;
|
||||||
use crate::exit::ExitRegistry;
|
use crate::exit::ExitRegistry;
|
||||||
use crate::plan::FetchPlan;
|
use crate::plan::FetchPlan;
|
||||||
|
use crate::progress::{ProgressEvent, ProgressSink};
|
||||||
use crate::types::{ByteRange, ExitId, FileSize, SegmentIndex, Sha256Digest, TargetUrl};
|
use crate::types::{ByteRange, ExitId, FileSize, SegmentIndex, Sha256Digest, TargetUrl};
|
||||||
|
|
||||||
/// Statistics for one finished segment.
|
/// Statistics for one finished segment.
|
||||||
@@ -20,6 +32,50 @@ pub struct SegmentStats {
|
|||||||
pub duration: Duration,
|
pub duration: Duration,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Retry behaviour for segment downloads.
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
pub struct RetryPolicy {
|
||||||
|
/// Total tries per segment; attempt 1 is the initial try.
|
||||||
|
pub max_attempts: NonZeroU32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RetryPolicy {
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(max_attempts: NonZeroU32) -> Self {
|
||||||
|
Self { max_attempts }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for RetryPolicy {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self {
|
||||||
|
max_attempts: NonZeroU32::new(3).unwrap_or(NonZeroU32::MIN),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Statistics for one finished segment, including retry information.
|
||||||
|
/// `duration` spans all attempts (retries included); `exit` is the exit
|
||||||
|
/// that served the successful attempt.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct SegmentAttemptStats {
|
||||||
|
pub index: SegmentIndex,
|
||||||
|
pub exit: ExitId,
|
||||||
|
pub bytes: u64,
|
||||||
|
pub duration: Duration,
|
||||||
|
pub attempts: u32,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl From<SegmentAttemptStats> for SegmentStats {
|
||||||
|
fn from(stats: SegmentAttemptStats) -> Self {
|
||||||
|
SegmentStats {
|
||||||
|
index: stats.index,
|
||||||
|
bytes: stats.bytes,
|
||||||
|
duration: stats.duration,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Result of a complete fetch.
|
/// Result of a complete fetch.
|
||||||
#[derive(Debug, Clone)]
|
#[derive(Debug, Clone)]
|
||||||
pub struct FetchReport {
|
pub struct FetchReport {
|
||||||
@@ -41,19 +97,63 @@ pub async fn run_segments(
|
|||||||
exits: &ExitRegistry,
|
exits: &ExitRegistry,
|
||||||
parts_dir: &Path,
|
parts_dir: &Path,
|
||||||
) -> Result<Vec<SegmentStats>, FetchError> {
|
) -> Result<Vec<SegmentStats>, FetchError> {
|
||||||
|
let stats = run_segments_with_policy(
|
||||||
|
plan,
|
||||||
|
exits,
|
||||||
|
parts_dir,
|
||||||
|
&RetryPolicy::default(),
|
||||||
|
&ProgressSink::noop(),
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
Ok(stats.into_iter().map(SegmentStats::from).collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Download every planned segment in parallel, retrying failed segments
|
||||||
|
/// according to `policy` — preferably on a different exit — and report
|
||||||
|
/// progress to `progress`. Each segment is written to its own part file
|
||||||
|
/// inside `parts_dir` (named `seg-NNNNNN.part`); the part file of a
|
||||||
|
/// failed attempt is removed before the next attempt.
|
||||||
|
///
|
||||||
|
/// Failures abort the whole fetch: a segment that exhausts all attempts
|
||||||
|
/// fails the fetch with [`FetchError::SegmentAttemptsExhausted`], a
|
||||||
|
/// non-retryable failure aborts immediately. A partially assembled file
|
||||||
|
/// is never presented as success.
|
||||||
|
pub async fn run_segments_with_policy(
|
||||||
|
plan: &FetchPlan,
|
||||||
|
exits: &ExitRegistry,
|
||||||
|
parts_dir: &Path,
|
||||||
|
policy: &RetryPolicy,
|
||||||
|
progress: &ProgressSink,
|
||||||
|
) -> Result<Vec<SegmentAttemptStats>, FetchError> {
|
||||||
tokio::fs::create_dir_all(parts_dir).await?;
|
tokio::fs::create_dir_all(parts_dir).await?;
|
||||||
|
|
||||||
|
// Validate that every planned exit exists, then snapshot the
|
||||||
|
// clients of all registered exits: tasks must fetch clients by id
|
||||||
|
// without borrowing the registry ('static spawn requirement).
|
||||||
|
for seg in &plan.segments {
|
||||||
|
exits.get(seg.exit)?;
|
||||||
|
}
|
||||||
|
let mut clients: BTreeMap<ExitId, reqwest::Client> = BTreeMap::new();
|
||||||
|
for id in exits.ids() {
|
||||||
|
clients.insert(id, exits.get(id)?.client().clone());
|
||||||
|
}
|
||||||
|
let clients = Arc::new(clients);
|
||||||
|
let selector = Arc::new(ExitSelector::new(exits));
|
||||||
|
|
||||||
let mut handles = Vec::with_capacity(plan.segments.len());
|
let mut handles = Vec::with_capacity(plan.segments.len());
|
||||||
for seg in &plan.segments {
|
for seg in &plan.segments {
|
||||||
let client = exits.get(seg.exit)?.client().clone();
|
let task = SegmentTask {
|
||||||
handles.push(tokio::spawn(download_segment(
|
url: plan.url.clone(),
|
||||||
plan.url.clone(),
|
index: seg.index,
|
||||||
seg.exit,
|
range: seg.range,
|
||||||
seg.index,
|
planned_exit: seg.exit,
|
||||||
seg.range,
|
clients: Arc::clone(&clients),
|
||||||
client,
|
selector: Arc::clone(&selector),
|
||||||
parts_dir.to_path_buf(),
|
parts_dir: parts_dir.to_path_buf(),
|
||||||
)));
|
policy: *policy,
|
||||||
|
progress: progress.clone(),
|
||||||
|
};
|
||||||
|
handles.push(tokio::spawn(download_with_retry(task)));
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut stats = Vec::with_capacity(handles.len());
|
let mut stats = Vec::with_capacity(handles.len());
|
||||||
@@ -67,18 +167,158 @@ pub async fn run_segments(
|
|||||||
Ok(stats)
|
Ok(stats)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Download one byte range and write it to `parts_dir/seg-NNNNNN.part`.
|
/// Everything one segment task needs, bundled to keep the task
|
||||||
async fn download_segment(
|
/// signature small.
|
||||||
|
struct SegmentTask {
|
||||||
url: TargetUrl,
|
url: TargetUrl,
|
||||||
|
index: SegmentIndex,
|
||||||
|
range: ByteRange,
|
||||||
|
planned_exit: ExitId,
|
||||||
|
clients: Arc<BTreeMap<ExitId, reqwest::Client>>,
|
||||||
|
selector: Arc<ExitSelector>,
|
||||||
|
parts_dir: PathBuf,
|
||||||
|
policy: RetryPolicy,
|
||||||
|
progress: ProgressSink,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Download one segment with up to `policy.max_attempts` attempts,
|
||||||
|
/// reassigning to a different exit after a retryable failure (while the
|
||||||
|
/// registry offers more than one exit).
|
||||||
|
async fn download_with_retry(task: SegmentTask) -> Result<SegmentAttemptStats, FetchError> {
|
||||||
|
let SegmentTask {
|
||||||
|
url,
|
||||||
|
index,
|
||||||
|
range,
|
||||||
|
planned_exit,
|
||||||
|
clients,
|
||||||
|
selector,
|
||||||
|
parts_dir,
|
||||||
|
policy,
|
||||||
|
progress,
|
||||||
|
} = task;
|
||||||
|
let max_attempts = policy.max_attempts.get();
|
||||||
|
let started = Instant::now();
|
||||||
|
let mut exit = planned_exit;
|
||||||
|
let mut attempt: u32 = 0;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
attempt += 1;
|
||||||
|
let client = clients
|
||||||
|
.get(&exit)
|
||||||
|
.ok_or(FetchError::UnknownExit(exit))?
|
||||||
|
.clone();
|
||||||
|
progress.on_event(&ProgressEvent::SegmentStarted {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
attempt,
|
||||||
|
});
|
||||||
|
tracing::debug!(segment = index.value(), exit = %exit, attempt, "segment attempt started");
|
||||||
|
|
||||||
|
match download_attempt(&url, exit, index, range, client, &parts_dir).await {
|
||||||
|
Ok(bytes) => {
|
||||||
|
let duration = started.elapsed();
|
||||||
|
progress.on_event(&ProgressEvent::SegmentFinished {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
bytes,
|
||||||
|
duration,
|
||||||
|
});
|
||||||
|
tracing::info!(
|
||||||
|
segment = index.value(),
|
||||||
|
exit = %exit,
|
||||||
|
attempts = attempt,
|
||||||
|
bytes,
|
||||||
|
"segment finished"
|
||||||
|
);
|
||||||
|
return Ok(SegmentAttemptStats {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
bytes,
|
||||||
|
duration,
|
||||||
|
attempts: attempt,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Err(err) => {
|
||||||
|
let retryable = is_retryable(&err);
|
||||||
|
let err_text = err.to_string();
|
||||||
|
progress.on_event(&ProgressEvent::SegmentFailed {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
attempt,
|
||||||
|
error: err_text,
|
||||||
|
});
|
||||||
|
remove_part_file(&parts_dir, index).await;
|
||||||
|
|
||||||
|
if !retryable {
|
||||||
|
tracing::warn!(
|
||||||
|
segment = index.value(),
|
||||||
|
exit = %exit,
|
||||||
|
attempt,
|
||||||
|
"segment failed with non-retryable error: {err}"
|
||||||
|
);
|
||||||
|
return Err(err);
|
||||||
|
}
|
||||||
|
selector.mark_failed(exit);
|
||||||
|
if attempt >= max_attempts {
|
||||||
|
tracing::warn!(
|
||||||
|
segment = index.value(),
|
||||||
|
attempts = attempt,
|
||||||
|
"segment failed after all attempts were exhausted: {err}"
|
||||||
|
);
|
||||||
|
return Err(FetchError::SegmentAttemptsExhausted {
|
||||||
|
index,
|
||||||
|
attempts: attempt,
|
||||||
|
last_error: Box::new(err),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
tracing::warn!(
|
||||||
|
segment = index.value(),
|
||||||
|
exit = %exit,
|
||||||
|
attempt,
|
||||||
|
"segment attempt failed; retrying: {err}"
|
||||||
|
);
|
||||||
|
exit = selector.pick_retry(exit);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Retryable: transport-level failures (network, timeouts), 5xx
|
||||||
|
/// responses and length mismatches. Non-retryable: 4xx range rejections
|
||||||
|
/// (e.g. 416), corrupt parts, plan or digest errors — retrying those on
|
||||||
|
/// another exit cannot succeed.
|
||||||
|
fn is_retryable(error: &FetchError) -> bool {
|
||||||
|
match error {
|
||||||
|
FetchError::Http(err) => {
|
||||||
|
!err.is_status() || err.status().is_some_and(|status| status.is_server_error())
|
||||||
|
}
|
||||||
|
FetchError::Io(_) | FetchError::SegmentLengthMismatch { .. } => true,
|
||||||
|
FetchError::RangeRejected { status, .. } => *status >= 500,
|
||||||
|
FetchError::InvalidUrl(_)
|
||||||
|
| FetchError::ZeroFileSize
|
||||||
|
| FetchError::TotalLengthMismatch { .. }
|
||||||
|
| FetchError::HashMismatch { .. }
|
||||||
|
| FetchError::InvalidPlan(_)
|
||||||
|
| FetchError::UnknownExit(_)
|
||||||
|
| FetchError::CorruptPart(_)
|
||||||
|
| FetchError::InvalidDigest
|
||||||
|
| FetchError::SegmentAttemptsExhausted { .. } => false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Download one byte range and write it to `parts_dir/seg-NNNNNN.part`.
|
||||||
|
/// Returns the number of bytes written. A failed attempt can leave a
|
||||||
|
/// partial part file behind; the retry loop removes it before the next
|
||||||
|
/// attempt.
|
||||||
|
async fn download_attempt(
|
||||||
|
url: &TargetUrl,
|
||||||
exit: ExitId,
|
exit: ExitId,
|
||||||
index: SegmentIndex,
|
index: SegmentIndex,
|
||||||
range: ByteRange,
|
range: ByteRange,
|
||||||
client: reqwest::Client,
|
client: reqwest::Client,
|
||||||
parts_dir: PathBuf,
|
parts_dir: &Path,
|
||||||
) -> Result<SegmentStats, FetchError> {
|
) -> Result<u64, FetchError> {
|
||||||
let started = Instant::now();
|
let part_path = part_file(parts_dir, index);
|
||||||
|
|
||||||
let part_path = part_file(&parts_dir, index);
|
|
||||||
let mut response = client
|
let mut response = client
|
||||||
.get(url.inner().clone())
|
.get(url.inner().clone())
|
||||||
.header(reqwest::header::RANGE, range.http_value())
|
.header(reqwest::header::RANGE, range.http_value())
|
||||||
@@ -106,7 +346,7 @@ async fn download_segment(
|
|||||||
file.flush().await?;
|
file.flush().await?;
|
||||||
|
|
||||||
if written != range.len() {
|
if written != range.len() {
|
||||||
tokio::fs::remove_file(&part_path).await.ok();
|
let _ = tokio::fs::remove_file(&part_path).await;
|
||||||
return Err(FetchError::SegmentLengthMismatch {
|
return Err(FetchError::SegmentLengthMismatch {
|
||||||
index,
|
index,
|
||||||
got: written,
|
got: written,
|
||||||
@@ -114,11 +354,68 @@ async fn download_segment(
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(SegmentStats {
|
Ok(written)
|
||||||
index,
|
}
|
||||||
bytes: written,
|
|
||||||
duration: started.elapsed(),
|
/// Chooses exits for retry attempts across all segment tasks of one
|
||||||
})
|
/// run. Preference: the exit that failed least recently (exits that
|
||||||
|
/// never failed first, ties broken by ascending id). The exit an
|
||||||
|
/// attempt just failed on is avoided while at least one other exit
|
||||||
|
/// exists; with a single-exit registry, retries stay on that exit.
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct ExitSelector {
|
||||||
|
ids: Vec<ExitId>,
|
||||||
|
last_failed: Mutex<BTreeMap<ExitId, Instant>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ExitSelector {
|
||||||
|
fn new(exits: &ExitRegistry) -> Self {
|
||||||
|
Self {
|
||||||
|
ids: exits.ids(),
|
||||||
|
last_failed: Mutex::new(BTreeMap::new()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn mark_failed(&self, id: ExitId) {
|
||||||
|
let mut guard = self.lock();
|
||||||
|
guard.insert(id, Instant::now());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn pick_retry(&self, avoid: ExitId) -> ExitId {
|
||||||
|
let guard = self.lock();
|
||||||
|
let mut best: Option<ExitId> = None;
|
||||||
|
for &id in &self.ids {
|
||||||
|
if id == avoid {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
best = Some(match best {
|
||||||
|
None => id,
|
||||||
|
Some(current) => {
|
||||||
|
let prefer_id = match (guard.get(¤t), guard.get(&id)) {
|
||||||
|
// tie or current never failed: keep the lower id
|
||||||
|
(None, None | Some(_)) => false,
|
||||||
|
(Some(_), None) => true, // id never failed: better
|
||||||
|
(Some(current_failed), Some(id_failed)) => *id_failed < *current_failed,
|
||||||
|
};
|
||||||
|
if prefer_id {
|
||||||
|
id
|
||||||
|
} else {
|
||||||
|
current
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
best.unwrap_or(avoid) // single-exit registry: retry on the same exit
|
||||||
|
}
|
||||||
|
|
||||||
|
/// std mutexes can be poisoned by a panicking holder; since the
|
||||||
|
/// map is only advisory (exit preference), keep using its contents.
|
||||||
|
fn lock(&self) -> MutexGuard<'_, BTreeMap<ExitId, Instant>> {
|
||||||
|
match self.last_failed.lock() {
|
||||||
|
Ok(guard) => guard,
|
||||||
|
Err(poisoned) => poisoned.into_inner(),
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Reassemble part files (in index order) into the output file while
|
/// Reassemble part files (in index order) into the output file while
|
||||||
@@ -199,6 +496,11 @@ async fn collect_part_files(parts_dir: &Path) -> Result<Vec<PathBuf>, FetchError
|
|||||||
Ok(out)
|
Ok(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Best-effort removal of a failed attempt's part file.
|
||||||
|
async fn remove_part_file(parts_dir: &Path, index: SegmentIndex) {
|
||||||
|
let _ = tokio::fs::remove_file(part_file(parts_dir, index)).await;
|
||||||
|
}
|
||||||
|
|
||||||
fn part_file(parts_dir: &Path, index: SegmentIndex) -> PathBuf {
|
fn part_file(parts_dir: &Path, index: SegmentIndex) -> PathBuf {
|
||||||
parts_dir.join(format!("seg-{:06}.part", index.value()))
|
parts_dir.join(format!("seg-{:06}.part", index.value()))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -48,6 +48,37 @@ impl HttpExit for LocalExit {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A neighbor's uplink, reached through an HTTP proxy over the mesh.
|
||||||
|
/// The proxy is expected to listen on the neighbor's Yggdrasil address,
|
||||||
|
/// so the transport between the households is encrypted end-to-end by
|
||||||
|
/// the overlay.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct MeshExit {
|
||||||
|
id: ExitId,
|
||||||
|
client: reqwest::Client,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MeshExit {
|
||||||
|
pub fn new(id: ExitId, proxy: reqwest::Url) -> Result<Self, FetchError> {
|
||||||
|
let proxy = reqwest::Proxy::all(proxy)?;
|
||||||
|
let client = reqwest::Client::builder()
|
||||||
|
.connect_timeout(Duration::from_secs(30))
|
||||||
|
.proxy(proxy)
|
||||||
|
.build()?;
|
||||||
|
Ok(Self { id, client })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HttpExit for MeshExit {
|
||||||
|
fn id(&self) -> ExitId {
|
||||||
|
self.id
|
||||||
|
}
|
||||||
|
|
||||||
|
fn client(&self) -> &reqwest::Client {
|
||||||
|
&self.client
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// All exits available for a fetch, keyed by id.
|
/// All exits available for a fetch, keyed by id.
|
||||||
#[derive(Default, Clone)]
|
#[derive(Default, Clone)]
|
||||||
pub struct ExitRegistry {
|
pub struct ExitRegistry {
|
||||||
|
|||||||
@@ -18,5 +18,6 @@ pub mod exec;
|
|||||||
pub mod exit;
|
pub mod exit;
|
||||||
pub mod plan;
|
pub mod plan;
|
||||||
pub mod probe;
|
pub mod probe;
|
||||||
|
pub mod progress;
|
||||||
pub mod runner;
|
pub mod runner;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ use std::path::PathBuf;
|
|||||||
use clap::Parser;
|
use clap::Parser;
|
||||||
use sn_fetch::runner::{self, FetchOptions};
|
use sn_fetch::runner::{self, FetchOptions};
|
||||||
use sn_fetch::types::{
|
use sn_fetch::types::{
|
||||||
ExitId, ExitWeight, MinSegmentBytes, SegmentsPerExit, Sha256Digest, TargetUrl,
|
Egress, ExitId, ExitWeight, MinSegmentBytes, SegmentsPerExit, Sha256Digest, TargetUrl,
|
||||||
};
|
};
|
||||||
|
|
||||||
#[derive(Debug, Parser)]
|
#[derive(Debug, Parser)]
|
||||||
@@ -28,12 +28,20 @@ struct Args {
|
|||||||
#[arg(short, long)]
|
#[arg(short, long)]
|
||||||
output: Option<PathBuf>,
|
output: Option<PathBuf>,
|
||||||
|
|
||||||
/// Exits as `id=weight` pairs, comma separated, e.g. `local=1,local2=2`.
|
/// Exits as `id=weight` pairs, comma separated, e.g. `1=2,2=1`.
|
||||||
/// `PoC`: all ids map to plain local connections; the weights only shape
|
/// A proxy route turns an exit into a neighbor's uplink over the
|
||||||
/// the plan. Multi-exit routing comes with the mesh integration.
|
/// mesh overlay: `id=weight@http://[ygg-addr]:8080`. Without a
|
||||||
#[arg(short = 'e', long = "exit", default_value = "local=1")]
|
/// proxy, the exit is this machine's own connection.
|
||||||
|
/// `PoC`: weights only shape the plan; a real proxy service
|
||||||
|
/// (`sn-exit`) comes with the mesh integration.
|
||||||
|
#[arg(short = 'e', long = "exit", default_value = "1=1")]
|
||||||
exits: String,
|
exits: String,
|
||||||
|
|
||||||
|
/// Total tries per segment (initial try + retries; failed segments
|
||||||
|
/// are retried on a different exit when available).
|
||||||
|
#[arg(long, default_value_t = 3)]
|
||||||
|
max_attempts: u32,
|
||||||
|
|
||||||
/// Parallel sub-segments per exit.
|
/// Parallel sub-segments per exit.
|
||||||
#[arg(short = 's', long, default_value_t = 1)]
|
#[arg(short = 's', long, default_value_t = 1)]
|
||||||
segments_per_exit: usize,
|
segments_per_exit: usize,
|
||||||
@@ -51,10 +59,10 @@ struct Args {
|
|||||||
probe_only: bool,
|
probe_only: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
fn parse_exits(spec: &str) -> Result<Vec<(ExitId, ExitWeight)>, String> {
|
fn parse_exits(spec: &str) -> Result<Vec<(ExitId, ExitWeight, Egress)>, String> {
|
||||||
spec.split(',')
|
spec.split(',')
|
||||||
.map(|pair| {
|
.map(|pair| {
|
||||||
let (id, weight) = pair
|
let (id, rest) = pair
|
||||||
.split_once('=')
|
.split_once('=')
|
||||||
.ok_or_else(|| format!("invalid exit spec `{pair}`, expected `id=weight`"))?;
|
.ok_or_else(|| format!("invalid exit spec `{pair}`, expected `id=weight`"))?;
|
||||||
let id = id
|
let id = id
|
||||||
@@ -63,13 +71,21 @@ fn parse_exits(spec: &str) -> Result<Vec<(ExitId, ExitWeight)>, String> {
|
|||||||
.ok()
|
.ok()
|
||||||
.and_then(NonZeroU8::new)
|
.and_then(NonZeroU8::new)
|
||||||
.ok_or_else(|| format!("invalid exit id `{id}` (must be a non-zero number)"))?;
|
.ok_or_else(|| format!("invalid exit id `{id}` (must be a non-zero number)"))?;
|
||||||
|
let (weight, egress) = match rest.split_once('@') {
|
||||||
|
Some((weight, proxy)) => {
|
||||||
|
let egress = Egress::proxy(proxy.trim())
|
||||||
|
.map_err(|err| format!("invalid proxy in exit spec `{pair}`: {err}"))?;
|
||||||
|
(weight, egress)
|
||||||
|
}
|
||||||
|
None => (rest, Egress::Direct),
|
||||||
|
};
|
||||||
let weight = weight
|
let weight = weight
|
||||||
.trim()
|
.trim()
|
||||||
.parse::<u32>()
|
.parse::<u32>()
|
||||||
.ok()
|
.ok()
|
||||||
.and_then(NonZeroU32::new)
|
.and_then(NonZeroU32::new)
|
||||||
.ok_or_else(|| format!("invalid exit weight `{weight}` (must be non-zero)"))?;
|
.ok_or_else(|| format!("invalid exit weight `{weight}` (must be non-zero)"))?;
|
||||||
Ok((ExitId::new(id), ExitWeight::new(weight)))
|
Ok((ExitId::new(id), ExitWeight::new(weight), egress))
|
||||||
})
|
})
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
@@ -131,6 +147,8 @@ async fn run() -> Result<(), String> {
|
|||||||
NonZeroU64::new(args.min_segment_mib.saturating_mul(1024 * 1024))
|
NonZeroU64::new(args.min_segment_mib.saturating_mul(1024 * 1024))
|
||||||
.ok_or("min_segment_mib must be at least 1")?,
|
.ok_or("min_segment_mib must be at least 1")?,
|
||||||
);
|
);
|
||||||
|
let max_attempts =
|
||||||
|
NonZeroU32::new(args.max_attempts).ok_or("max_attempts must be at least 1")?;
|
||||||
|
|
||||||
let expected = match &args.sha256 {
|
let expected = match &args.sha256 {
|
||||||
Some(hex) => Some(Sha256Digest::from_hex(hex).map_err(|err| err.to_string())?),
|
Some(hex) => Some(Sha256Digest::from_hex(hex).map_err(|err| err.to_string())?),
|
||||||
@@ -146,9 +164,10 @@ async fn run() -> Result<(), String> {
|
|||||||
segments_per_exit: SegmentsPerExit::new(spe),
|
segments_per_exit: SegmentsPerExit::new(spe),
|
||||||
min_segment,
|
min_segment,
|
||||||
expected_sha256: expected,
|
expected_sha256: expected,
|
||||||
|
retries: sn_fetch::exec::RetryPolicy::new(max_attempts),
|
||||||
};
|
};
|
||||||
|
|
||||||
let registry = runner::local_registry(&exit_specs).map_err(|err| err.to_string())?;
|
let registry = runner::registry_from_specs(&exit_specs).map_err(|err| err.to_string())?;
|
||||||
let parts_dir = std::env::temp_dir().join("sn-fetch-parts");
|
let parts_dir = std::env::temp_dir().join("sn-fetch-parts");
|
||||||
|
|
||||||
let report = runner::run(&target, &options, ®istry, &parts_dir)
|
let report = runner::run(&target, &options, ®istry, &parts_dir)
|
||||||
|
|||||||
@@ -2,8 +2,8 @@
|
|||||||
|
|
||||||
use crate::error::FetchError;
|
use crate::error::FetchError;
|
||||||
use crate::types::{
|
use crate::types::{
|
||||||
ByteRange, ExitId, ExitWeight, FileSize, MinSegmentBytes, SegmentIndex, SegmentsPerExit,
|
ByteRange, Egress, ExitId, ExitWeight, FileSize, MinSegmentBytes, SegmentIndex,
|
||||||
TargetUrl,
|
SegmentsPerExit, TargetUrl,
|
||||||
};
|
};
|
||||||
use std::num::NonZeroUsize;
|
use std::num::NonZeroUsize;
|
||||||
|
|
||||||
@@ -23,8 +23,9 @@ pub struct FetchPlan {
|
|||||||
pub segments: Vec<SegmentPlan>,
|
pub segments: Vec<SegmentPlan>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Exits with their capacity weights.
|
/// exit description, including how traffic leaves (`Direct` = own
|
||||||
pub type ExitSpecs = Vec<(ExitId, ExitWeight)>;
|
/// uplink, `Proxy` = a neighbor's exit over the mesh).
|
||||||
|
pub type ExitSpecs = Vec<(ExitId, ExitWeight, Egress)>;
|
||||||
|
|
||||||
impl FetchPlan {
|
impl FetchPlan {
|
||||||
/// Build a validated plan.
|
/// Build a validated plan.
|
||||||
@@ -56,12 +57,12 @@ impl FetchPlan {
|
|||||||
let weakest = exits
|
let weakest = exits
|
||||||
.iter()
|
.iter()
|
||||||
.enumerate()
|
.enumerate()
|
||||||
.min_by_key(|(_, (id, weight))| (weight.value(), std::cmp::Reverse(id.value())))
|
.min_by_key(|(_, (id, weight, _))| (weight.value(), std::cmp::Reverse(id.value())))
|
||||||
.map_or(0, |(pos, _)| pos);
|
.map_or(0, |(pos, _)| pos);
|
||||||
exits.remove(weakest);
|
exits.remove(weakest);
|
||||||
}
|
}
|
||||||
// Deterministic boundary assignment order.
|
// Deterministic boundary assignment order.
|
||||||
exits.sort_by_key(|(id, _)| *id);
|
exits.sort_by_key(|(id, _, _)| *id);
|
||||||
|
|
||||||
// If even one segment per exit would be too small, reduce to one
|
// If even one segment per exit would be too small, reduce to one
|
||||||
// segment per exit.
|
// segment per exit.
|
||||||
@@ -85,7 +86,7 @@ impl FetchPlan {
|
|||||||
// Weighted split across exits, then even sub-split per exit.
|
// Weighted split across exits, then even sub-split per exit.
|
||||||
let units: Vec<u128> = exits
|
let units: Vec<u128> = exits
|
||||||
.iter()
|
.iter()
|
||||||
.map(|(_, w)| u128::from(w.value()) * spe.value() as u128)
|
.map(|(_, w, _)| u128::from(w.value()) * spe.value() as u128)
|
||||||
.collect();
|
.collect();
|
||||||
let total_units: u128 = units.iter().sum();
|
let total_units: u128 = units.iter().sum();
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,171 @@
|
|||||||
|
//! Progress reporting for segmented downloads.
|
||||||
|
//!
|
||||||
|
//! [`ProgressSink`] is a cloneable callback handle rather than a trait:
|
||||||
|
//! spawned segment tasks need an owned, `'static` reporter, and a
|
||||||
|
//! callback handle can be cloned cheaply from a plain `&ProgressSink`
|
||||||
|
//! parameter without extra plumbing. [`ProgressEvent`] carries one
|
||||||
|
//! event per segment attempt; [`BytesTracker`] aggregates the bytes
|
||||||
|
//! finished across all segments.
|
||||||
|
|
||||||
|
#![allow(clippy::module_name_repetitions)] // `Progress*` types read naturally inside `mod progress`
|
||||||
|
|
||||||
|
use std::fmt;
|
||||||
|
use std::sync::atomic::{AtomicU64, Ordering};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crate::types::{ExitId, SegmentIndex};
|
||||||
|
|
||||||
|
/// One event describing the state of a segment download attempt.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub enum ProgressEvent {
|
||||||
|
/// A segment attempt started (attempt 1 is the initial try).
|
||||||
|
SegmentStarted {
|
||||||
|
index: SegmentIndex,
|
||||||
|
exit: ExitId,
|
||||||
|
attempt: u32,
|
||||||
|
},
|
||||||
|
/// A segment finished successfully. `duration` spans the whole
|
||||||
|
/// segment, all attempts (retries included).
|
||||||
|
SegmentFinished {
|
||||||
|
index: SegmentIndex,
|
||||||
|
exit: ExitId,
|
||||||
|
bytes: u64,
|
||||||
|
duration: Duration,
|
||||||
|
},
|
||||||
|
/// A segment attempt failed. `error` is the formatted failure
|
||||||
|
/// reason; the typed error surfaces through the fetch's result.
|
||||||
|
SegmentFailed {
|
||||||
|
index: SegmentIndex,
|
||||||
|
exit: ExitId,
|
||||||
|
attempt: u32,
|
||||||
|
error: String,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for ProgressEvent {
|
||||||
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::SegmentStarted {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
attempt,
|
||||||
|
} => {
|
||||||
|
write!(f, "segment {index} on {exit}: attempt {attempt} started")
|
||||||
|
}
|
||||||
|
Self::SegmentFinished {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
bytes,
|
||||||
|
duration,
|
||||||
|
} => {
|
||||||
|
write!(
|
||||||
|
f,
|
||||||
|
"segment {index} on {exit}: {bytes} bytes in {duration:.2?}"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
Self::SegmentFailed {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
attempt,
|
||||||
|
error,
|
||||||
|
} => {
|
||||||
|
write!(
|
||||||
|
f,
|
||||||
|
"segment {index} on {exit}: attempt {attempt} failed: {error}"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable handle to a progress callback; `Send + Sync`, so it can be
|
||||||
|
/// shared across tokio tasks. All clones call the same callback.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct ProgressSink {
|
||||||
|
on_event: Arc<dyn Fn(&ProgressEvent) + Send + Sync>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ProgressSink {
|
||||||
|
/// Wrap any event callback into a sink.
|
||||||
|
pub fn new<F>(on_event: F) -> Self
|
||||||
|
where
|
||||||
|
F: Fn(&ProgressEvent) + Send + Sync + 'static,
|
||||||
|
{
|
||||||
|
Self {
|
||||||
|
on_event: Arc::new(on_event),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A sink that discards every event.
|
||||||
|
#[must_use]
|
||||||
|
pub fn noop() -> Self {
|
||||||
|
Self::new(|_| ())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A sink that logs every event through `tracing`: started attempts
|
||||||
|
/// at debug, finished segments at info, failures at warn.
|
||||||
|
#[must_use]
|
||||||
|
pub fn tracing_log() -> Self {
|
||||||
|
Self::new(|event| match event {
|
||||||
|
ProgressEvent::SegmentStarted { .. } => tracing::debug!("{event}"),
|
||||||
|
ProgressEvent::SegmentFinished { .. } => tracing::info!("{event}"),
|
||||||
|
ProgressEvent::SegmentFailed { .. } => tracing::warn!("{event}"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Combine two sinks: both receive every event.
|
||||||
|
#[must_use]
|
||||||
|
pub fn fanout(first: &Self, second: &Self) -> Self {
|
||||||
|
let first = first.clone();
|
||||||
|
let second = second.clone();
|
||||||
|
Self::new(move |event| {
|
||||||
|
first.on_event(event);
|
||||||
|
second.on_event(event);
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Deliver one event to the underlying callback.
|
||||||
|
pub fn on_event(&self, event: &ProgressEvent) {
|
||||||
|
(self.on_event)(event);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Debug for ProgressSink {
|
||||||
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
f.debug_struct("ProgressSink").finish_non_exhaustive()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Counts the bytes of successfully finished segments across the whole
|
||||||
|
/// fetch. Cheap to clone: every clone observes the same total.
|
||||||
|
#[derive(Debug, Default, Clone)]
|
||||||
|
pub struct BytesTracker {
|
||||||
|
bytes: Arc<AtomicU64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BytesTracker {
|
||||||
|
#[must_use]
|
||||||
|
pub fn new() -> Self {
|
||||||
|
Self::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Total bytes reported as finished so far.
|
||||||
|
#[must_use]
|
||||||
|
pub fn total(&self) -> u64 {
|
||||||
|
self.bytes.load(Ordering::Relaxed)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A sink that feeds this tracker; the tracker stays queryable.
|
||||||
|
#[must_use]
|
||||||
|
pub fn sink(&self) -> ProgressSink {
|
||||||
|
let tracker = self.clone();
|
||||||
|
ProgressSink::new(move |event| tracker.record(event))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn record(&self, event: &ProgressEvent) {
|
||||||
|
if let ProgressEvent::SegmentFinished { bytes, .. } = event {
|
||||||
|
self.bytes.fetch_add(*bytes, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,12 +10,13 @@ use sha2::Digest;
|
|||||||
use tokio::io::AsyncWriteExt;
|
use tokio::io::AsyncWriteExt;
|
||||||
|
|
||||||
use crate::error::FetchError;
|
use crate::error::FetchError;
|
||||||
use crate::exec::{self, FetchReport, SegmentStats};
|
use crate::exec::{self, FetchReport, RetryPolicy, SegmentStats};
|
||||||
use crate::exit::{ExitRegistry, LocalExit};
|
use crate::exit::{ExitRegistry, HttpExit, LocalExit, MeshExit};
|
||||||
use crate::plan::ExitSpecs;
|
use crate::plan::ExitSpecs;
|
||||||
use crate::probe::{self, ProbeInfo};
|
use crate::probe::{self, ProbeInfo};
|
||||||
|
use crate::progress::ProgressSink;
|
||||||
use crate::types::{
|
use crate::types::{
|
||||||
FileSize, MinSegmentBytes, RangeSupport, SegmentsPerExit, Sha256Digest, TargetUrl,
|
Egress, FileSize, MinSegmentBytes, RangeSupport, SegmentsPerExit, Sha256Digest, TargetUrl,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// All knobs for one fetch.
|
/// All knobs for one fetch.
|
||||||
@@ -26,6 +27,8 @@ pub struct FetchOptions {
|
|||||||
pub segments_per_exit: SegmentsPerExit,
|
pub segments_per_exit: SegmentsPerExit,
|
||||||
pub min_segment: MinSegmentBytes,
|
pub min_segment: MinSegmentBytes,
|
||||||
pub expected_sha256: Option<Sha256Digest>,
|
pub expected_sha256: Option<Sha256Digest>,
|
||||||
|
/// How often a segment is retried (preferably on another exit).
|
||||||
|
pub retries: RetryPolicy,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Run a full fetch and produce a report.
|
/// Run a full fetch and produce a report.
|
||||||
@@ -74,7 +77,17 @@ async fn run_ranged(
|
|||||||
tracing::info!("plan: {} segment(s)", plan.segments.len());
|
tracing::info!("plan: {} segment(s)", plan.segments.len());
|
||||||
|
|
||||||
let started = Instant::now();
|
let started = Instant::now();
|
||||||
let segment_stats: Vec<SegmentStats> = exec::run_segments(&plan, exits, parts_dir).await?;
|
let segment_stats: Vec<SegmentStats> = exec::run_segments_with_policy(
|
||||||
|
&plan,
|
||||||
|
exits,
|
||||||
|
parts_dir,
|
||||||
|
&options.retries,
|
||||||
|
&ProgressSink::tracing_log(),
|
||||||
|
)
|
||||||
|
.await?
|
||||||
|
.into_iter()
|
||||||
|
.map(std::convert::From::from)
|
||||||
|
.collect();
|
||||||
let digest = exec::assemble(
|
let digest = exec::assemble(
|
||||||
parts_dir,
|
parts_dir,
|
||||||
&options.output,
|
&options.output,
|
||||||
@@ -105,7 +118,7 @@ async fn run_whole(
|
|||||||
let exit_id = options
|
let exit_id = options
|
||||||
.exits
|
.exits
|
||||||
.first()
|
.first()
|
||||||
.map(|(id, _)| *id)
|
.map(|(id, _, _)| *id)
|
||||||
.ok_or_else(|| FetchError::InvalidPlan("no exits given".to_string()))?;
|
.ok_or_else(|| FetchError::InvalidPlan("no exits given".to_string()))?;
|
||||||
let client = exits.get(exit_id)?.client().clone();
|
let client = exits.get(exit_id)?.client().clone();
|
||||||
|
|
||||||
@@ -163,11 +176,18 @@ async fn run_whole(
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Convenience: build a registry with one [`LocalExit`] per spec id.
|
/// Build a registry from exit specs: `Egress::Direct` exits connect
|
||||||
pub fn local_registry(specs: &ExitSpecs) -> Result<ExitRegistry, FetchError> {
|
/// locally (own uplink), `Egress::Proxy` exits route through the given
|
||||||
|
/// proxy — typically a neighbor's exit service over the Yggdrasil
|
||||||
|
/// overlay.
|
||||||
|
pub fn registry_from_specs(specs: &ExitSpecs) -> Result<ExitRegistry, FetchError> {
|
||||||
let mut registry = ExitRegistry::empty();
|
let mut registry = ExitRegistry::empty();
|
||||||
for (id, _) in specs {
|
for (id, _, egress) in specs {
|
||||||
registry.register(std::sync::Arc::new(LocalExit::new(*id)?));
|
let exit: std::sync::Arc<dyn HttpExit> = match egress {
|
||||||
|
Egress::Direct => std::sync::Arc::new(LocalExit::new(*id)?),
|
||||||
|
Egress::Proxy(url) => std::sync::Arc::new(MeshExit::new(*id, url.clone())?),
|
||||||
|
};
|
||||||
|
registry.register(exit);
|
||||||
}
|
}
|
||||||
Ok(registry)
|
Ok(registry)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -230,3 +230,36 @@ pub enum RangeSupport {
|
|||||||
/// Probe was inconclusive; treat as unsupported (safe fallback).
|
/// Probe was inconclusive; treat as unsupported (safe fallback).
|
||||||
Unknown,
|
Unknown,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// How a request physically leaves through an exit.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum Egress {
|
||||||
|
/// The household's own uplink: plain outgoing HTTP, no detour.
|
||||||
|
Direct,
|
||||||
|
/// Through an HTTP proxy — e.g. a neighbor's exit service reachable
|
||||||
|
/// over the mesh overlay (typically bound to their Yggdrasil
|
||||||
|
/// address).
|
||||||
|
Proxy(reqwest::Url),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Egress {
|
||||||
|
/// Parse a proxy URL for [`Egress::Proxy`]; only `http`/`https`
|
||||||
|
/// upstreams are accepted.
|
||||||
|
pub fn proxy(input: &str) -> Result<Self, FetchError> {
|
||||||
|
let url =
|
||||||
|
reqwest::Url::parse(input).map_err(|_| FetchError::InvalidUrl(input.to_string()))?;
|
||||||
|
match url.scheme() {
|
||||||
|
"http" | "https" => Ok(Self::Proxy(url)),
|
||||||
|
_ => Err(FetchError::InvalidUrl(input.to_string())),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for Egress {
|
||||||
|
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::Direct => write!(f, "direct"),
|
||||||
|
Self::Proxy(url) => write!(f, "proxy:{url}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
//! A minimal HTTP server for integration tests. Supports HEAD, GET with
|
//! A minimal HTTP server for integration tests. Supports HEAD, GET with
|
||||||
//! optional range handling (configurable), `Connection: close` semantics.
|
//! optional range handling (configurable), simulated failures for range
|
||||||
|
//! GETs (flaky/exhausted/non-retryable modes), and a broken exit whose
|
||||||
|
//! client fails at transport level, for retry tests.
|
||||||
|
|
||||||
#![forbid(unsafe_code)]
|
#![forbid(unsafe_code)]
|
||||||
// Tests are allowed to unwrap/expect; production code denies them via
|
// Tests are allowed to unwrap/expect; production code denies them via
|
||||||
@@ -8,8 +10,36 @@
|
|||||||
|
|
||||||
use std::io::{BufRead, BufReader, Write};
|
use std::io::{BufRead, BufReader, Write};
|
||||||
use std::net::{SocketAddr, TcpListener, TcpStream};
|
use std::net::{SocketAddr, TcpListener, TcpStream};
|
||||||
|
use std::sync::atomic::{AtomicU32, Ordering};
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
use std::thread::JoinHandle;
|
use std::thread::JoinHandle;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use sn_fetch::exit::HttpExit;
|
||||||
|
use sn_fetch::types::ExitId;
|
||||||
|
|
||||||
|
/// When the test origin should fail `Range` GETs (retry simulation).
|
||||||
|
/// Only used by the retry test binary; allow dead_code for the others.
|
||||||
|
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub enum FailMode {
|
||||||
|
#[default]
|
||||||
|
Never,
|
||||||
|
/// Fail the first `n` range GETs with HTTP 500, then serve normally.
|
||||||
|
FirstN(u32),
|
||||||
|
/// Fail every range GET with HTTP 500.
|
||||||
|
Always,
|
||||||
|
/// Answer every range GET with the given status (e.g. 416 to
|
||||||
|
/// simulate a non-retryable range rejection).
|
||||||
|
AlwaysStatus(u16),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Knobs for the test origin.
|
||||||
|
#[derive(Debug, Clone, Copy, Default)]
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub struct ServerConfig {
|
||||||
|
pub fail_mode: FailMode,
|
||||||
|
}
|
||||||
|
|
||||||
pub struct TestServer {
|
pub struct TestServer {
|
||||||
pub addr: SocketAddr,
|
pub addr: SocketAddr,
|
||||||
@@ -21,14 +51,25 @@ impl TestServer {
|
|||||||
/// server behaves like a range-hostile origin: no `Accept-Ranges`
|
/// server behaves like a range-hostile origin: no `Accept-Ranges`
|
||||||
/// header, and `Range` requests are answered with the full body.
|
/// header, and `Range` requests are answered with the full body.
|
||||||
pub fn start(data: Vec<u8>, ranges: bool) -> Self {
|
pub fn start(data: Vec<u8>, ranges: bool) -> Self {
|
||||||
|
Self::start_with(data, ranges, ServerConfig::default())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Start with a custom [`ServerConfig`] (failure simulation).
|
||||||
|
pub fn start_with(data: Vec<u8>, ranges: bool, config: ServerConfig) -> Self {
|
||||||
let data = Arc::new(data);
|
let data = Arc::new(data);
|
||||||
|
let config = Arc::new(config);
|
||||||
|
let range_get_count = Arc::new(AtomicU32::new(0));
|
||||||
let listener = TcpListener::bind("127.0.0.1:0").expect("bind in test");
|
let listener = TcpListener::bind("127.0.0.1:0").expect("bind in test");
|
||||||
let addr = listener.local_addr().expect("addr in test");
|
let addr = listener.local_addr().expect("local addr in test");
|
||||||
let handle = std::thread::spawn(move || {
|
let handle = std::thread::spawn(move || {
|
||||||
for stream in listener.incoming().flatten() {
|
for stream in listener.incoming().flatten() {
|
||||||
let data = Arc::clone(&data);
|
let (data, config, range_get_count) = (
|
||||||
|
Arc::clone(&data),
|
||||||
|
Arc::clone(&config),
|
||||||
|
Arc::clone(&range_get_count),
|
||||||
|
);
|
||||||
std::thread::spawn(move || {
|
std::thread::spawn(move || {
|
||||||
let _ = handle_conn(stream, &data, ranges);
|
let _ = handle_conn(stream, &data, ranges, &config, &range_get_count);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -43,7 +84,13 @@ impl TestServer {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn handle_conn(stream: TcpStream, data: &[u8], ranges: bool) -> std::io::Result<()> {
|
fn handle_conn(
|
||||||
|
stream: TcpStream,
|
||||||
|
data: &[u8],
|
||||||
|
ranges: bool,
|
||||||
|
config: &ServerConfig,
|
||||||
|
range_get_count: &AtomicU32,
|
||||||
|
) -> std::io::Result<()> {
|
||||||
let mut reader = BufReader::new(stream.try_clone()?);
|
let mut reader = BufReader::new(stream.try_clone()?);
|
||||||
let mut request_line = String::new();
|
let mut request_line = String::new();
|
||||||
reader.read_line(&mut request_line)?;
|
reader.read_line(&mut request_line)?;
|
||||||
@@ -64,7 +111,6 @@ fn handle_conn(stream: TcpStream, data: &[u8], ranges: bool) -> std::io::Result<
|
|||||||
|
|
||||||
let mut stream = stream;
|
let mut stream = stream;
|
||||||
let len = data.len() as u64;
|
let len = data.len() as u64;
|
||||||
|
|
||||||
let method = request_line
|
let method = request_line
|
||||||
.split_whitespace()
|
.split_whitespace()
|
||||||
.next()
|
.next()
|
||||||
@@ -83,6 +129,10 @@ fn handle_conn(stream: TcpStream, data: &[u8], ranges: bool) -> std::io::Result<
|
|||||||
let range = range_header.as_deref().and_then(parse_range).flatten();
|
let range = range_header.as_deref().and_then(parse_range).flatten();
|
||||||
match (ranges, range) {
|
match (ranges, range) {
|
||||||
(true, Some((start, end))) => {
|
(true, Some((start, end))) => {
|
||||||
|
if should_fail_range_get(config, range_get_count) {
|
||||||
|
write_fail_response(&mut stream, config.fail_mode)?;
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
let slice_len = end - start + 1;
|
let slice_len = end - start + 1;
|
||||||
write!(
|
write!(
|
||||||
stream,
|
stream,
|
||||||
@@ -111,6 +161,28 @@ fn handle_conn(stream: TcpStream, data: &[u8], ranges: bool) -> std::io::Result<
|
|||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Decide (and count) whether this range GET should fail. HEAD and
|
||||||
|
/// non-range GETs are never counted nor failed.
|
||||||
|
fn should_fail_range_get(config: &ServerConfig, range_get_count: &AtomicU32) -> bool {
|
||||||
|
match config.fail_mode {
|
||||||
|
FailMode::Never => false,
|
||||||
|
FailMode::Always | FailMode::AlwaysStatus(_) => true,
|
||||||
|
FailMode::FirstN(limit) => range_get_count.fetch_add(1, Ordering::Relaxed) < limit,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn write_fail_response(stream: &mut TcpStream, mode: FailMode) -> std::io::Result<()> {
|
||||||
|
let status = match mode {
|
||||||
|
FailMode::AlwaysStatus(status) => status,
|
||||||
|
_ => 500,
|
||||||
|
};
|
||||||
|
write!(
|
||||||
|
stream,
|
||||||
|
"HTTP/1.1 {status} Error\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||||
|
)?;
|
||||||
|
stream.flush()
|
||||||
|
}
|
||||||
|
|
||||||
/// Parse `bytes=a-b` into inclusive `(start, end)`. Anything else
|
/// Parse `bytes=a-b` into inclusive `(start, end)`. Anything else
|
||||||
/// (open-ended, malformed) is treated as absent.
|
/// (open-ended, malformed) is treated as absent.
|
||||||
fn parse_range(value: &str) -> Option<Option<(u64, u64)>> {
|
fn parse_range(value: &str) -> Option<Option<(u64, u64)>> {
|
||||||
@@ -124,3 +196,35 @@ fn parse_range(value: &str) -> Option<Option<(u64, u64)>> {
|
|||||||
Some(None)
|
Some(None)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// An exit whose client sends every request through a proxy that
|
||||||
|
/// refuses connections — every request through this exit fails at
|
||||||
|
/// transport level (retryable), as needed for exit-reassignment tests.
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub struct BrokenProxyExit {
|
||||||
|
id: ExitId,
|
||||||
|
client: reqwest::Client,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BrokenProxyExit {
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub fn new(id: ExitId) -> Self {
|
||||||
|
let proxy = reqwest::Proxy::http("http://127.0.0.1:1").expect("static proxy url parses");
|
||||||
|
let client = reqwest::Client::builder()
|
||||||
|
.proxy(proxy)
|
||||||
|
.connect_timeout(Duration::from_secs(10))
|
||||||
|
.build()
|
||||||
|
.expect("client builds in test");
|
||||||
|
Self { id, client }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HttpExit for BrokenProxyExit {
|
||||||
|
fn id(&self) -> ExitId {
|
||||||
|
self.id
|
||||||
|
}
|
||||||
|
|
||||||
|
fn client(&self) -> &reqwest::Client {
|
||||||
|
&self.client
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,151 @@
|
|||||||
|
//! End-to-end test: a `MeshExit` fetches through a real HTTP proxy.
|
||||||
|
//!
|
||||||
|
//! The fake proxy is a minimal HTTP forward proxy: it accepts absolute-URI
|
||||||
|
//! requests (`GET http://origin/file.bin HTTP/1.1`), extracts the `Range`
|
||||||
|
//! header and serves the requested slice from an in-memory buffer —
|
||||||
|
//! exactly the shape of an `sn-exit` service bound to a neighbor's
|
||||||
|
//! Yggdrasil address (minus the overlay transport, which is transparent
|
||||||
|
//! to HTTP).
|
||||||
|
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
#![allow(clippy::unwrap_used, clippy::expect_used)]
|
||||||
|
|
||||||
|
use std::num::{NonZeroU32, NonZeroU64, NonZeroU8, NonZeroUsize};
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use sha2::{Digest, Sha256};
|
||||||
|
use tempfile::TempDir;
|
||||||
|
|
||||||
|
use sn_fetch::exit::{ExitRegistry, MeshExit};
|
||||||
|
use sn_fetch::plan::{ExitSpecs, FetchPlan};
|
||||||
|
use sn_fetch::types::{
|
||||||
|
Egress, ExitId, ExitWeight, FileSize, MinSegmentBytes, SegmentsPerExit, TargetUrl,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Minimal forward proxy serving one fixed object from memory.
|
||||||
|
async fn spawn_proxy(data: Arc<Vec<u8>>) -> std::net::SocketAddr {
|
||||||
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
|
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
let Ok((mut socket, _)) = listener.accept().await else {
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
let data = Arc::clone(&data);
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Read the request head.
|
||||||
|
let mut head = Vec::new();
|
||||||
|
let mut byte = [0u8; 1];
|
||||||
|
while !head.windows(4).any(|w| w == b"\r\n\r\n") {
|
||||||
|
if socket.read(&mut byte).await.unwrap_or(0) == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
head.push(byte[0]);
|
||||||
|
}
|
||||||
|
let head_text = String::from_utf8_lossy(&head);
|
||||||
|
|
||||||
|
// First line: `GET http://origin/path HTTP/1.1`.
|
||||||
|
let first_line = head_text.lines().next().unwrap_or_default();
|
||||||
|
if !first_line.starts_with("GET http://") {
|
||||||
|
socket
|
||||||
|
.write_all(b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\n\r\n")
|
||||||
|
.await
|
||||||
|
.ok();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract the requested byte range (if any). Header
|
||||||
|
// names arrive lowercased from some clients.
|
||||||
|
let range = head_text.lines().find_map(|line| {
|
||||||
|
let (name, value) = line.split_once(':')?;
|
||||||
|
if !name.eq_ignore_ascii_case("range") {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let value = value.trim().strip_prefix("bytes=")?;
|
||||||
|
let (start, end) = value.trim().split_once('-')?;
|
||||||
|
Some((start.parse::<u64>().ok()?, end.parse::<u64>().ok()?))
|
||||||
|
});
|
||||||
|
|
||||||
|
let (start, end_inclusive, status) = match range {
|
||||||
|
Some((start, end)) => (start, end, "206 Partial Content"),
|
||||||
|
None => (0, data.len() as u64 - 1, "200 OK"),
|
||||||
|
};
|
||||||
|
let slice = &data[start as usize..=(end_inclusive as usize).min(data.len() - 1)];
|
||||||
|
let response = format!(
|
||||||
|
"HTTP/1.1 {status}\r\nContent-Length: {}\r\n\r\n",
|
||||||
|
slice.len()
|
||||||
|
);
|
||||||
|
socket.write_all(response.as_bytes()).await.ok();
|
||||||
|
socket.write_all(slice).await.ok();
|
||||||
|
socket.shutdown().await.ok();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addr
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn mesh_exit_downloads_through_the_proxy() {
|
||||||
|
let data: Vec<u8> = (0..64_000u32).map(|i| (i % 251) as u8).collect();
|
||||||
|
let data = Arc::new(data.clone());
|
||||||
|
let proxy_addr = spawn_proxy(Arc::clone(&data)).await;
|
||||||
|
|
||||||
|
let proxy_url = format!("http://{proxy_addr}");
|
||||||
|
let specs: ExitSpecs = vec![(
|
||||||
|
ExitId::new(NonZeroU8::new(1).expect("1 is non-zero")),
|
||||||
|
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
||||||
|
Egress::proxy(&proxy_url).expect("proxy url is valid"),
|
||||||
|
)];
|
||||||
|
|
||||||
|
// The origin URL is fictional — the proxy serves the content; this
|
||||||
|
// proves the request really went through the proxy, not straight to
|
||||||
|
// an origin server.
|
||||||
|
let target = TargetUrl::parse("http://origin.invalid/file.bin").expect("valid url");
|
||||||
|
let file_size = FileSize::try_new(data.len() as u64).expect("non-zero in test");
|
||||||
|
|
||||||
|
let plan = FetchPlan::build(
|
||||||
|
target.clone(),
|
||||||
|
file_size,
|
||||||
|
&specs,
|
||||||
|
SegmentsPerExit::new(NonZeroUsize::new(2).expect("2 is non-zero")),
|
||||||
|
MinSegmentBytes::new(NonZeroU64::new(1024).expect("1024 is non-zero")),
|
||||||
|
)
|
||||||
|
.expect("plan builds");
|
||||||
|
plan.validate().expect("plan validates");
|
||||||
|
|
||||||
|
let mut registry = ExitRegistry::empty();
|
||||||
|
registry.register(Arc::new(
|
||||||
|
MeshExit::new(
|
||||||
|
ExitId::new(NonZeroU8::new(1).expect("1 is non-zero")),
|
||||||
|
reqwest::Url::parse(&proxy_url).expect("proxy url is valid"),
|
||||||
|
)
|
||||||
|
.expect("mesh exit builds"),
|
||||||
|
));
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let stats = sn_fetch::exec::run_segments(&plan, ®istry, parts.path())
|
||||||
|
.await
|
||||||
|
.expect("segments succeed through the proxy");
|
||||||
|
let total: u64 = stats.iter().map(|s| s.bytes).sum();
|
||||||
|
assert_eq!(total, data.len() as u64);
|
||||||
|
|
||||||
|
let output = parts.path().join("mesh.bin");
|
||||||
|
let expected_digest =
|
||||||
|
sn_fetch::types::Sha256Digest::from_bytes(Sha256::digest(data.as_ref().as_slice()).into());
|
||||||
|
let digest = sn_fetch::exec::assemble(parts.path(), &output, file_size, Some(&expected_digest))
|
||||||
|
.await
|
||||||
|
.expect("assembly verifies");
|
||||||
|
assert_eq!(digest, expected_digest);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The proxy's address (IPv4, arbitrary port) is not a valid proxy URL
|
||||||
|
/// unless it has an http scheme — Egress::proxy must reject junk.
|
||||||
|
#[test]
|
||||||
|
fn egress_rejects_non_http_proxy_schemes() {
|
||||||
|
assert!(Egress::proxy("ftp://proxy:2121").is_err());
|
||||||
|
assert!(Egress::proxy("not a url").is_err());
|
||||||
|
assert!(Egress::proxy("http://127.0.0.1:8080").is_ok());
|
||||||
|
assert!(Egress::proxy("https://proxy.example:3128").is_ok());
|
||||||
|
}
|
||||||
@@ -0,0 +1,428 @@
|
|||||||
|
//! Retry behaviour against a local HTTP server: a flaky first attempt
|
||||||
|
//! (retried, reassigned to another exit), a consistently failing exit
|
||||||
|
//! (reassigned), retry exhaustion (whole fetch fails, no output file)
|
||||||
|
//! and a non-retryable status (immediate abort).
|
||||||
|
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
// Tests are allowed to unwrap/expect; production code denies them via
|
||||||
|
// workspace lints (server/Cargo.toml).
|
||||||
|
#![allow(clippy::unwrap_used, clippy::expect_used)]
|
||||||
|
|
||||||
|
mod common;
|
||||||
|
|
||||||
|
use std::num::{NonZeroU32, NonZeroU64, NonZeroU8, NonZeroUsize};
|
||||||
|
use std::path::Path;
|
||||||
|
use std::sync::{Arc, Mutex};
|
||||||
|
|
||||||
|
use sha2::{Digest, Sha256};
|
||||||
|
use tempfile::TempDir;
|
||||||
|
|
||||||
|
use sn_fetch::error::FetchError;
|
||||||
|
use sn_fetch::exec::{self, RetryPolicy};
|
||||||
|
use sn_fetch::exit::{ExitRegistry, LocalExit};
|
||||||
|
use sn_fetch::plan::{ExitSpecs, FetchPlan};
|
||||||
|
use sn_fetch::progress::{BytesTracker, ProgressEvent, ProgressSink};
|
||||||
|
use sn_fetch::types::{
|
||||||
|
Egress, ExitId, ExitWeight, FileSize, MinSegmentBytes, SegmentsPerExit, TargetUrl,
|
||||||
|
};
|
||||||
|
|
||||||
|
use common::{BrokenProxyExit, FailMode, ServerConfig, TestServer};
|
||||||
|
|
||||||
|
fn spec(id: u8) -> (ExitId, ExitWeight, Egress) {
|
||||||
|
(
|
||||||
|
ExitId::new(NonZeroU8::new(id).expect("test ids are 1..=255")),
|
||||||
|
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
||||||
|
Egress::Direct,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn exit_id(id: u8) -> ExitId {
|
||||||
|
ExitId::new(NonZeroU8::new(id).expect("test ids are 1..=255"))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn local_registry(specs: &ExitSpecs) -> ExitRegistry {
|
||||||
|
let mut registry = ExitRegistry::empty();
|
||||||
|
for (id, _, _) in specs {
|
||||||
|
registry.register(Arc::new(LocalExit::new(*id).expect("local exit")));
|
||||||
|
}
|
||||||
|
registry
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Single-exit plan + registry, for exhaustion tests.
|
||||||
|
fn single_exit_setup(
|
||||||
|
url: &TargetUrl,
|
||||||
|
file_size: u64,
|
||||||
|
min_segment: u64,
|
||||||
|
) -> (FetchPlan, ExitRegistry) {
|
||||||
|
let specs: ExitSpecs = vec![spec(1)];
|
||||||
|
let plan = FetchPlan::build(
|
||||||
|
url.clone(),
|
||||||
|
FileSize::try_new(file_size).expect("non-zero in test"),
|
||||||
|
&specs,
|
||||||
|
SegmentsPerExit::new(NonZeroUsize::MIN),
|
||||||
|
MinSegmentBytes::new(NonZeroU64::new(min_segment).expect("non-zero in test")),
|
||||||
|
)
|
||||||
|
.expect("plan builds");
|
||||||
|
let registry = local_registry(&specs);
|
||||||
|
(plan, registry)
|
||||||
|
}
|
||||||
|
|
||||||
|
type RecordedEvents = Arc<Mutex<Vec<ProgressEvent>>>;
|
||||||
|
|
||||||
|
fn recording_sink() -> (ProgressSink, RecordedEvents) {
|
||||||
|
let events: RecordedEvents = Arc::new(Mutex::new(Vec::new()));
|
||||||
|
let sink = ProgressSink::new({
|
||||||
|
let events = Arc::clone(&events);
|
||||||
|
move |event| {
|
||||||
|
events
|
||||||
|
.lock()
|
||||||
|
.expect("mutex poisoned in test")
|
||||||
|
.push(event.clone())
|
||||||
|
}
|
||||||
|
});
|
||||||
|
(sink, events)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn failed_events(events: &RecordedEvents) -> Vec<(u32, u32, String)> {
|
||||||
|
events
|
||||||
|
.lock()
|
||||||
|
.expect("mutex poisoned in test")
|
||||||
|
.iter()
|
||||||
|
.filter_map(|event| match event {
|
||||||
|
ProgressEvent::SegmentFailed {
|
||||||
|
index,
|
||||||
|
exit: _,
|
||||||
|
attempt,
|
||||||
|
error,
|
||||||
|
} => Some((index.value(), *attempt, error.clone())),
|
||||||
|
_ => None,
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Names of leftover part files in `parts_dir`.
|
||||||
|
fn part_files(dir: &Path) -> Vec<String> {
|
||||||
|
std::fs::read_dir(dir)
|
||||||
|
.expect("parts dir readable in test")
|
||||||
|
.flatten()
|
||||||
|
.filter(|entry| entry.file_name().to_string_lossy().ends_with(".part"))
|
||||||
|
.map(|entry| entry.file_name().to_string_lossy().into_owned())
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn flaky_segment_retried_on_other_exit() {
|
||||||
|
let data: Vec<u8> = (0..(2 * 1024 * 1024)).map(|i| (i % 253) as u8).collect();
|
||||||
|
let server = TestServer::start_with(
|
||||||
|
data.clone(),
|
||||||
|
true,
|
||||||
|
ServerConfig {
|
||||||
|
fail_mode: FailMode::FirstN(1),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
let url = TargetUrl::parse(&server.url("/file.bin")).expect("valid url");
|
||||||
|
|
||||||
|
let specs: ExitSpecs = vec![spec(1), spec(2)];
|
||||||
|
let plan = FetchPlan::build(
|
||||||
|
url,
|
||||||
|
FileSize::try_new(data.len() as u64).expect("non-zero in test"),
|
||||||
|
&specs,
|
||||||
|
SegmentsPerExit::new(NonZeroUsize::new(2).expect("2 is non-zero")),
|
||||||
|
MinSegmentBytes::new(NonZeroU64::new(1024).expect("1024 is non-zero")),
|
||||||
|
)
|
||||||
|
.expect("plan builds");
|
||||||
|
let registry = local_registry(&specs);
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let (progress, events) = recording_sink();
|
||||||
|
let tracker = BytesTracker::new();
|
||||||
|
let combined = ProgressSink::fanout(&progress, &tracker.sink());
|
||||||
|
|
||||||
|
let stats = exec::run_segments_with_policy(
|
||||||
|
&plan,
|
||||||
|
®istry,
|
||||||
|
parts.path(),
|
||||||
|
&RetryPolicy::default(),
|
||||||
|
&combined,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("fetch succeeds despite one flaky attempt");
|
||||||
|
|
||||||
|
assert_eq!(stats.len(), plan.segments.len());
|
||||||
|
let total: u64 = stats.iter().map(|s| s.bytes).sum();
|
||||||
|
assert_eq!(total, data.len() as u64);
|
||||||
|
assert_eq!(
|
||||||
|
tracker.total(),
|
||||||
|
data.len() as u64,
|
||||||
|
"tracker must see all finished bytes"
|
||||||
|
);
|
||||||
|
|
||||||
|
let retried: Vec<_> = stats.iter().filter(|s| s.attempts > 1).collect();
|
||||||
|
assert_eq!(retried.len(), 1, "exactly one segment needed a retry");
|
||||||
|
let retried = retried[0];
|
||||||
|
let planned_exit = plan
|
||||||
|
.segments
|
||||||
|
.iter()
|
||||||
|
.find(|seg| seg.index == retried.index)
|
||||||
|
.expect("planned segment")
|
||||||
|
.exit;
|
||||||
|
assert_ne!(retried.exit, planned_exit, "retry must use another exit");
|
||||||
|
assert_eq!(retried.attempts, 2);
|
||||||
|
|
||||||
|
let started: Vec<(u8, u32)> = events
|
||||||
|
.lock()
|
||||||
|
.expect("mutex poisoned in test")
|
||||||
|
.iter()
|
||||||
|
.filter_map(|event| match event {
|
||||||
|
ProgressEvent::SegmentStarted {
|
||||||
|
index,
|
||||||
|
exit,
|
||||||
|
attempt,
|
||||||
|
} if index == &retried.index => Some((exit.value(), *attempt)),
|
||||||
|
_ => None,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
started,
|
||||||
|
vec![(planned_exit.value(), 1), (retried.exit.value(), 2)],
|
||||||
|
"the retried segment must start once per attempt, on different exits"
|
||||||
|
);
|
||||||
|
let failed = failed_events(&events);
|
||||||
|
assert_eq!(
|
||||||
|
failed,
|
||||||
|
vec![(retried.index.value(), 1, failed[0].2.clone())],
|
||||||
|
"exactly one failed attempt, recorded on attempt 1"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
failed[0].2.contains("500"),
|
||||||
|
"failure reason should mention the 500 status: {}",
|
||||||
|
failed[0].2
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn exhausted_retries_fail_without_output() {
|
||||||
|
let server = TestServer::start_with(
|
||||||
|
vec![7u8; 32 * 1024],
|
||||||
|
true,
|
||||||
|
ServerConfig {
|
||||||
|
fail_mode: FailMode::Always,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
let url = TargetUrl::parse(&server.url("/file.bin")).expect("valid url");
|
||||||
|
let (plan, registry) = single_exit_setup(&url, 32 * 1024, 32 * 1024);
|
||||||
|
assert_eq!(plan.segments.len(), 1);
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let (progress, events) = recording_sink();
|
||||||
|
let tracker = BytesTracker::new();
|
||||||
|
let combined = ProgressSink::fanout(&progress, &tracker.sink());
|
||||||
|
|
||||||
|
let result = exec::run_segments_with_policy(
|
||||||
|
&plan,
|
||||||
|
®istry,
|
||||||
|
parts.path(),
|
||||||
|
&RetryPolicy::default(),
|
||||||
|
&combined,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let err = result.expect_err("fetch must fail when every attempt fails");
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
err,
|
||||||
|
FetchError::SegmentAttemptsExhausted { attempts: 3, .. }
|
||||||
|
),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
tracker.total(),
|
||||||
|
0,
|
||||||
|
"nothing finished, tracker stays at zero"
|
||||||
|
);
|
||||||
|
|
||||||
|
let failed: Vec<u32> = failed_events(&events)
|
||||||
|
.into_iter()
|
||||||
|
.map(|(_, attempt, error)| {
|
||||||
|
assert!(error.contains("500"), "failure reason: {error}");
|
||||||
|
attempt
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(failed, vec![1, 2, 3], "default policy means three attempts");
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
part_files(parts.path()).is_empty(),
|
||||||
|
"failed part files must be cleaned up between attempts"
|
||||||
|
);
|
||||||
|
let output = parts.path().join("output.bin");
|
||||||
|
assert!(
|
||||||
|
!output.exists(),
|
||||||
|
"no output file may be created for a failed fetch"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn single_attempt_policy_fails_fast() {
|
||||||
|
let server = TestServer::start_with(
|
||||||
|
vec![5u8; 16 * 1024],
|
||||||
|
true,
|
||||||
|
ServerConfig {
|
||||||
|
fail_mode: FailMode::Always,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
let url = TargetUrl::parse(&server.url("/file.bin")).expect("valid url");
|
||||||
|
let (plan, registry) = single_exit_setup(&url, 16 * 1024, 16 * 1024);
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let (progress, events) = recording_sink();
|
||||||
|
let policy = RetryPolicy::new(NonZeroU32::new(1).expect("1 is non-zero"));
|
||||||
|
|
||||||
|
let result =
|
||||||
|
exec::run_segments_with_policy(&plan, ®istry, parts.path(), &policy, &progress).await;
|
||||||
|
|
||||||
|
let err = result.expect_err("fetch must fail after the single attempt");
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
err,
|
||||||
|
FetchError::SegmentAttemptsExhausted { attempts: 1, .. }
|
||||||
|
),
|
||||||
|
"unexpected error: {err}"
|
||||||
|
);
|
||||||
|
assert_eq!(failed_events(&events).len(), 1);
|
||||||
|
assert!(part_files(parts.path()).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn failing_exit_reassigned_to_working_exit() {
|
||||||
|
let data: Vec<u8> = (0..(1024 * 1024)).map(|i| (i % 241) as u8).collect();
|
||||||
|
let expected_digest = sn_fetch::types::Sha256Digest::from_bytes(Sha256::digest(&data).into());
|
||||||
|
let server = TestServer::start(data.clone(), true);
|
||||||
|
let url = TargetUrl::parse(&server.url("/file.bin")).expect("valid url");
|
||||||
|
|
||||||
|
let specs: ExitSpecs = vec![spec(1), spec(2)];
|
||||||
|
let plan = FetchPlan::build(
|
||||||
|
url,
|
||||||
|
FileSize::try_new(data.len() as u64).expect("non-zero in test"),
|
||||||
|
&specs,
|
||||||
|
SegmentsPerExit::new(NonZeroUsize::MIN),
|
||||||
|
MinSegmentBytes::new(NonZeroU64::new(1024).expect("1024 is non-zero")),
|
||||||
|
)
|
||||||
|
.expect("plan builds");
|
||||||
|
|
||||||
|
let mut registry = ExitRegistry::empty();
|
||||||
|
registry.register(Arc::new(BrokenProxyExit::new(exit_id(1))));
|
||||||
|
registry.register(Arc::new(LocalExit::new(exit_id(2)).expect("local exit")));
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let (progress, events) = recording_sink();
|
||||||
|
|
||||||
|
let stats = exec::run_segments_with_policy(
|
||||||
|
&plan,
|
||||||
|
®istry,
|
||||||
|
parts.path(),
|
||||||
|
&RetryPolicy::default(),
|
||||||
|
&progress,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("fetch succeeds by reassigning off the broken exit");
|
||||||
|
|
||||||
|
for stat in &stats {
|
||||||
|
let planned = plan
|
||||||
|
.segments
|
||||||
|
.iter()
|
||||||
|
.find(|seg| seg.index == stat.index)
|
||||||
|
.expect("planned segment");
|
||||||
|
if planned.exit.value() == 1 {
|
||||||
|
assert_eq!(
|
||||||
|
stat.attempts, 2,
|
||||||
|
"segment planned on the broken exit needs exactly one retry"
|
||||||
|
);
|
||||||
|
assert_eq!(stat.exit.value(), 2, "retry must land on the working exit");
|
||||||
|
} else {
|
||||||
|
assert_eq!(stat.attempts, 1);
|
||||||
|
assert_eq!(stat.exit.value(), 2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let started_exits: Vec<u8> = events
|
||||||
|
.lock()
|
||||||
|
.expect("mutex poisoned in test")
|
||||||
|
.iter()
|
||||||
|
.filter_map(|event| match event {
|
||||||
|
ProgressEvent::SegmentStarted { index, exit, .. } if index.value() == 0 => {
|
||||||
|
Some(exit.value())
|
||||||
|
}
|
||||||
|
_ => None,
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
started_exits,
|
||||||
|
vec![1, 2],
|
||||||
|
"segment 0 must be tried on the broken exit first, then reassigned"
|
||||||
|
);
|
||||||
|
|
||||||
|
let output = parts.path().join("out.bin");
|
||||||
|
let digest = exec::assemble(
|
||||||
|
parts.path(),
|
||||||
|
&output,
|
||||||
|
FileSize::try_new(data.len() as u64).expect("non-zero in test"),
|
||||||
|
Some(&expected_digest),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.expect("assembly verifies");
|
||||||
|
assert_eq!(digest, expected_digest);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn non_retryable_status_aborts_immediately() {
|
||||||
|
let server = TestServer::start_with(
|
||||||
|
vec![3u8; 32 * 1024],
|
||||||
|
true,
|
||||||
|
ServerConfig {
|
||||||
|
fail_mode: FailMode::AlwaysStatus(416),
|
||||||
|
},
|
||||||
|
);
|
||||||
|
let url = TargetUrl::parse(&server.url("/file.bin")).expect("valid url");
|
||||||
|
let (plan, registry) = single_exit_setup(&url, 32 * 1024, 32 * 1024);
|
||||||
|
|
||||||
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
let (progress, events) = recording_sink();
|
||||||
|
|
||||||
|
let result = exec::run_segments_with_policy(
|
||||||
|
&plan,
|
||||||
|
®istry,
|
||||||
|
parts.path(),
|
||||||
|
&RetryPolicy::default(),
|
||||||
|
&progress,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let err = result.expect_err("416 must abort the fetch");
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
err,
|
||||||
|
FetchError::Http(ref http_err)
|
||||||
|
if http_err.status() == Some(reqwest::StatusCode::RANGE_NOT_SATISFIABLE)
|
||||||
|
),
|
||||||
|
"the raw 416 must surface unchanged, got: {err}"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Non-retryable: one started attempt, one failure event, no retry.
|
||||||
|
let guard = events.lock().expect("mutex poisoned in test");
|
||||||
|
let started = guard
|
||||||
|
.iter()
|
||||||
|
.filter(|event| matches!(event, ProgressEvent::SegmentStarted { .. }))
|
||||||
|
.count();
|
||||||
|
assert_eq!(started, 1, "non-retryable failure must not be retried");
|
||||||
|
let failed = guard
|
||||||
|
.iter()
|
||||||
|
.filter(|event| matches!(event, ProgressEvent::SegmentFailed { .. }))
|
||||||
|
.count();
|
||||||
|
assert_eq!(failed, 1);
|
||||||
|
drop(guard);
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
part_files(parts.path()).is_empty(),
|
||||||
|
"failed part files must be cleaned up"
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -21,8 +21,8 @@ use sn_fetch::exit::LocalExit;
|
|||||||
use sn_fetch::plan::{ExitSpecs, FetchPlan};
|
use sn_fetch::plan::{ExitSpecs, FetchPlan};
|
||||||
use sn_fetch::probe;
|
use sn_fetch::probe;
|
||||||
use sn_fetch::types::{
|
use sn_fetch::types::{
|
||||||
ByteRange, ExitId, ExitWeight, FileSize, MinSegmentBytes, RangeSupport, SegmentsPerExit,
|
ByteRange, Egress, ExitId, ExitWeight, FileSize, MinSegmentBytes, RangeSupport,
|
||||||
Sha256Digest,
|
SegmentsPerExit, Sha256Digest,
|
||||||
};
|
};
|
||||||
|
|
||||||
fn exit_specs(n: u8) -> ExitSpecs {
|
fn exit_specs(n: u8) -> ExitSpecs {
|
||||||
@@ -31,6 +31,7 @@ fn exit_specs(n: u8) -> ExitSpecs {
|
|||||||
(
|
(
|
||||||
ExitId::new(NonZeroU8::new(i).expect("test ids are 1..=255")),
|
ExitId::new(NonZeroU8::new(i).expect("test ids are 1..=255")),
|
||||||
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
||||||
|
Egress::Direct,
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
.collect()
|
.collect()
|
||||||
@@ -68,7 +69,7 @@ async fn segmented_download_reassembles_and_hashes() {
|
|||||||
plan.validate().expect("plan validates");
|
plan.validate().expect("plan validates");
|
||||||
|
|
||||||
let mut registry = sn_fetch::exit::ExitRegistry::empty();
|
let mut registry = sn_fetch::exit::ExitRegistry::empty();
|
||||||
for id in specs.iter().map(|(id, _)| *id) {
|
for id in specs.iter().map(|(id, _, _)| *id) {
|
||||||
registry.register(Arc::new(LocalExit::new(id).expect("local exit")));
|
registry.register(Arc::new(LocalExit::new(id).expect("local exit")));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -109,8 +110,9 @@ async fn range_hostile_server_falls_back_to_single_stream() {
|
|||||||
segments_per_exit: SegmentsPerExit::new(NonZeroUsize::new(1).expect("1 is non-zero")),
|
segments_per_exit: SegmentsPerExit::new(NonZeroUsize::new(1).expect("1 is non-zero")),
|
||||||
min_segment: MinSegmentBytes::new(NonZeroU64::new(1024).expect("1024 is non-zero")),
|
min_segment: MinSegmentBytes::new(NonZeroU64::new(1024).expect("1024 is non-zero")),
|
||||||
expected_sha256: None,
|
expected_sha256: None,
|
||||||
|
retries: sn_fetch::exec::RetryPolicy::default(),
|
||||||
};
|
};
|
||||||
let registry = sn_fetch::runner::local_registry(&specs).expect("registry");
|
let registry = sn_fetch::runner::registry_from_specs(&specs).expect("registry");
|
||||||
let parts = TempDir::new().expect("tempdir in test");
|
let parts = TempDir::new().expect("tempdir in test");
|
||||||
|
|
||||||
let report = sn_fetch::runner::run(&target, &options, ®istry, parts.path())
|
let report = sn_fetch::runner::run(&target, &options, ®istry, parts.path())
|
||||||
@@ -131,10 +133,12 @@ fn plan_is_contiguous_and_weighted() {
|
|||||||
(
|
(
|
||||||
ExitId::new(NonZeroU8::new(1).expect("1 is non-zero")),
|
ExitId::new(NonZeroU8::new(1).expect("1 is non-zero")),
|
||||||
ExitWeight::new(NonZeroU32::new(3).expect("3 is non-zero")),
|
ExitWeight::new(NonZeroU32::new(3).expect("3 is non-zero")),
|
||||||
|
Egress::Direct,
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
ExitId::new(NonZeroU8::new(2).expect("2 is non-zero")),
|
ExitId::new(NonZeroU8::new(2).expect("2 is non-zero")),
|
||||||
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
ExitWeight::new(NonZeroU32::new(1).expect("1 is non-zero")),
|
||||||
|
Egress::Direct,
|
||||||
),
|
),
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,15 @@
|
|||||||
|
[package]
|
||||||
|
name = "sn-node"
|
||||||
|
description = "SpiderNet node helper: Yggdrasil overlay config generation and status queries"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
tokio = { workspace = true }
|
||||||
|
serde_json = { workspace = true }
|
||||||
|
thiserror = { workspace = true }
|
||||||
|
clap = { workspace = true }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
@@ -0,0 +1,343 @@
|
|||||||
|
//! Yggdrasil configuration generation for a `SpiderNet` node.
|
||||||
|
//!
|
||||||
|
//! Yggdrasil's own `-genconf` output is used as the base (so we never
|
||||||
|
//! hard-code version-specific defaults); the `SpiderNet` settings are then
|
||||||
|
//! patched into that JSON base as validated, typed values. The result is
|
||||||
|
//! written as JSON — Yggdrasil accepts JSON config via `-useconffile`.
|
||||||
|
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
use crate::error::NodeError;
|
||||||
|
use crate::yggdrasil::AdminEndpoint;
|
||||||
|
|
||||||
|
/// A validated Yggdrasil peering URI (`tcp://`, `tls://`, `quic://` or
|
||||||
|
/// `socks://`).
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct PeerUri(String);
|
||||||
|
|
||||||
|
impl PeerUri {
|
||||||
|
pub fn parse(input: &str) -> Result<Self, NodeError> {
|
||||||
|
Ok(Self(validate_scheme(input, "peering")?))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
&self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A validated listener URI a node binds for incoming peerings
|
||||||
|
/// (`tcp://[::]:1337` style).
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct ListenUri(String);
|
||||||
|
|
||||||
|
impl ListenUri {
|
||||||
|
pub fn parse(input: &str) -> Result<Self, NodeError> {
|
||||||
|
let trimmed = input.trim();
|
||||||
|
if trimmed.starts_with("socks://") {
|
||||||
|
return Err(NodeError::InvalidUri {
|
||||||
|
uri: input.to_string(),
|
||||||
|
hint: "listeners bind an address: use tcp://, tls:// or quic://",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(Self(validate_scheme(trimmed, "listen")?))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
&self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const PEER_SCHEMES: [&str; 4] = ["tcp://", "tls://", "quic://", "socks://"];
|
||||||
|
|
||||||
|
fn validate_scheme(input: &str, kind: &'static str) -> Result<String, NodeError> {
|
||||||
|
let trimmed = input.trim();
|
||||||
|
let scheme = PEER_SCHEMES
|
||||||
|
.iter()
|
||||||
|
.find(|scheme| trimmed.starts_with(**scheme));
|
||||||
|
match scheme {
|
||||||
|
Some(scheme) if trimmed.len() > scheme.len() => Ok(trimmed.to_string()),
|
||||||
|
_ => Err(NodeError::InvalidUri {
|
||||||
|
uri: input.to_string(),
|
||||||
|
hint: match kind {
|
||||||
|
"peering" => "peer URIs must start with tcp://, tls://, quic:// or socks://",
|
||||||
|
_ => "listen URIs must start with tcp://, tls:// or quic://",
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A validated Yggdrasil public key: 64 hex characters.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct PublicKeyHex(String);
|
||||||
|
|
||||||
|
impl PublicKeyHex {
|
||||||
|
pub fn parse(input: &str) -> Result<Self, NodeError> {
|
||||||
|
let trimmed = input.trim().to_lowercase();
|
||||||
|
let valid = trimmed.len() == 64 && trimmed.chars().all(|c| c.is_ascii_hexdigit());
|
||||||
|
if valid {
|
||||||
|
Ok(Self(trimmed))
|
||||||
|
} else {
|
||||||
|
Err(NodeError::InvalidPublicKey(input.to_string()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
&self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Which TUN interface Yggdrasil should use.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum IfName {
|
||||||
|
/// `auto` — let Yggdrasil pick (default).
|
||||||
|
Auto,
|
||||||
|
/// `none` — headless router-only mode, no TUN device.
|
||||||
|
Headless,
|
||||||
|
Named(String),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Multicast (same-LAN) auto-peering mode.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum MulticastMode {
|
||||||
|
/// Leave Yggdrasil's generated default untouched.
|
||||||
|
PlatformDefault,
|
||||||
|
/// Disable multicast peering entirely.
|
||||||
|
Disabled,
|
||||||
|
/// Explicit interface entries (regex-matched).
|
||||||
|
Interfaces(Vec<MulticastInterface>),
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One multicast peering entry, mirroring Yggdrasil's JSON shape.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct MulticastInterface {
|
||||||
|
/// Interface name regex, e.g. `eth0` or `.*`.
|
||||||
|
pub regex: String,
|
||||||
|
pub beacon: bool,
|
||||||
|
pub listen: bool,
|
||||||
|
pub port: u16,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Everything `SpiderNet` wants to control in the Yggdrasil config.
|
||||||
|
/// Unlisted fields keep the values Yggdrasil's `-genconf` produced.
|
||||||
|
#[derive(Debug, Clone, Default)]
|
||||||
|
pub struct NodeSettings {
|
||||||
|
pub peers: Vec<PeerUri>,
|
||||||
|
pub listen: Vec<ListenUri>,
|
||||||
|
pub multicast: Option<MulticastMode>,
|
||||||
|
pub admin_listen: Option<AdminEndpoint>,
|
||||||
|
pub if_name: Option<IfName>,
|
||||||
|
/// Neighbor keys allowed to peer with us; empty means "allow all".
|
||||||
|
pub allowed_public_keys: Vec<PublicKeyHex>,
|
||||||
|
pub private_key_path: Option<PathBuf>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run `yggdrasil -genconf -json` and patch the `SpiderNet` settings into
|
||||||
|
/// the generated base config. Returns the pretty-printed JSON config.
|
||||||
|
pub fn generate_config(binary: &str, settings: &NodeSettings) -> Result<String, NodeError> {
|
||||||
|
let output = std::process::Command::new(binary)
|
||||||
|
.arg("-genconf")
|
||||||
|
.arg("-json")
|
||||||
|
.output()?;
|
||||||
|
if !output.status.success() {
|
||||||
|
let stderr = String::from_utf8_lossy(&output.stderr).trim().to_string();
|
||||||
|
return Err(NodeError::YggdrasilBinary {
|
||||||
|
binary: binary.to_string(),
|
||||||
|
stderr,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
let mut base: serde_json::Value = serde_json::from_slice(&output.stdout)?;
|
||||||
|
apply_to_base_config(settings, &mut base)?;
|
||||||
|
Ok(serde_json::to_string_pretty(&base)?)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Patch the `SpiderNet` settings into an existing (parsed) Yggdrasil base
|
||||||
|
/// config. Pure function so it can be tested without the binary.
|
||||||
|
pub fn apply_to_base_config(
|
||||||
|
settings: &NodeSettings,
|
||||||
|
base: &mut serde_json::Value,
|
||||||
|
) -> Result<(), NodeError> {
|
||||||
|
let obj = base
|
||||||
|
.as_object_mut()
|
||||||
|
.ok_or(NodeError::UnexpectedConfigShape)?;
|
||||||
|
|
||||||
|
if !settings.peers.is_empty() {
|
||||||
|
let peers: Vec<&str> = settings.peers.iter().map(PeerUri::as_str).collect();
|
||||||
|
obj.insert("Peers".to_string(), serde_json::json!(peers));
|
||||||
|
}
|
||||||
|
if !settings.listen.is_empty() {
|
||||||
|
let listen: Vec<&str> = settings.listen.iter().map(ListenUri::as_str).collect();
|
||||||
|
obj.insert("Listen".to_string(), serde_json::json!(listen));
|
||||||
|
}
|
||||||
|
match settings.multicast.as_ref() {
|
||||||
|
None | Some(MulticastMode::PlatformDefault) => {}
|
||||||
|
Some(MulticastMode::Disabled) => {
|
||||||
|
obj.insert("MulticastInterfaces".to_string(), serde_json::json!([]));
|
||||||
|
}
|
||||||
|
Some(MulticastMode::Interfaces(interfaces)) => {
|
||||||
|
let entries: Vec<serde_json::Value> = interfaces
|
||||||
|
.iter()
|
||||||
|
.map(|iface| {
|
||||||
|
serde_json::json!({
|
||||||
|
"Beacon": iface.beacon,
|
||||||
|
"Listen": iface.listen,
|
||||||
|
"Port": iface.port,
|
||||||
|
"Regex": iface.regex,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
obj.insert(
|
||||||
|
"MulticastInterfaces".to_string(),
|
||||||
|
serde_json::json!(entries),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(admin_listen) = settings.admin_listen.as_ref() {
|
||||||
|
obj.insert(
|
||||||
|
"AdminListen".to_string(),
|
||||||
|
serde_json::json!(admin_listen.config_value()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if let Some(if_name) = settings.if_name.as_ref() {
|
||||||
|
let value = match if_name {
|
||||||
|
IfName::Auto => "auto".to_string(),
|
||||||
|
IfName::Headless => "none".to_string(),
|
||||||
|
IfName::Named(name) => name.clone(),
|
||||||
|
};
|
||||||
|
obj.insert("IfName".to_string(), serde_json::json!(value));
|
||||||
|
}
|
||||||
|
if !settings.allowed_public_keys.is_empty() {
|
||||||
|
let keys: Vec<&str> = settings
|
||||||
|
.allowed_public_keys
|
||||||
|
.iter()
|
||||||
|
.map(PublicKeyHex::as_str)
|
||||||
|
.collect();
|
||||||
|
obj.insert("AllowedPublicKeys".to_string(), serde_json::json!(keys));
|
||||||
|
}
|
||||||
|
if let Some(private_key_path) = settings.private_key_path.as_ref() {
|
||||||
|
obj.insert(
|
||||||
|
"PrivateKeyPath".to_string(),
|
||||||
|
serde_json::json!(private_key_path.display().to_string()),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Write the config to `path` and restrict permissions to the owner
|
||||||
|
/// (the file contains the node's private key — it must never be readable
|
||||||
|
/// by other users, and it must never be committed to any repository).
|
||||||
|
pub fn write_config(path: &Path, json: &str) -> Result<(), NodeError> {
|
||||||
|
use std::io::Write as _;
|
||||||
|
|
||||||
|
let mut file = std::fs::File::create(path)?;
|
||||||
|
file.write_all(json.as_bytes())?;
|
||||||
|
file.flush()?;
|
||||||
|
#[cfg(unix)]
|
||||||
|
{
|
||||||
|
use std::os::unix::fs::PermissionsExt;
|
||||||
|
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
#![allow(clippy::unwrap_used, clippy::expect_used)]
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn peer_uri_accepts_supported_schemes() {
|
||||||
|
for uri in [
|
||||||
|
"tcp://host:1337",
|
||||||
|
"tls://host:1337",
|
||||||
|
"quic://host:1337",
|
||||||
|
"socks://p:1/h:2",
|
||||||
|
] {
|
||||||
|
assert!(PeerUri::parse(uri).is_ok(), "{uri} should be valid");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn peer_uri_rejects_bare_or_unknown_schemes() {
|
||||||
|
for uri in ["host:1337", "http://host:80", "tcp://", ""] {
|
||||||
|
assert!(PeerUri::parse(uri).is_err(), "{uri} should be invalid");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn listen_uri_rejects_socks() {
|
||||||
|
assert!(ListenUri::parse("socks://[::]:1337").is_err());
|
||||||
|
assert!(ListenUri::parse("tcp://[::]:1337").is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_key_requires_64_hex_chars() {
|
||||||
|
assert!(PublicKeyHex::parse(&"a".repeat(64)).is_ok());
|
||||||
|
assert!(PublicKeyHex::parse(&"a".repeat(63)).is_err());
|
||||||
|
assert!(PublicKeyHex::parse(&"z".repeat(64)).is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn admin_endpoint_parse_and_roundtrip() {
|
||||||
|
let tcp = AdminEndpoint::parse("tcp://localhost:9001").unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
tcp,
|
||||||
|
AdminEndpoint::Tcp {
|
||||||
|
host: "localhost".into(),
|
||||||
|
port: 9001
|
||||||
|
}
|
||||||
|
);
|
||||||
|
assert_eq!(tcp.config_value(), "tcp://localhost:9001");
|
||||||
|
|
||||||
|
let unix = AdminEndpoint::parse("unix:///var/run/yggdrasil.sock").unwrap();
|
||||||
|
assert_eq!(unix.config_value(), "unix:///var/run/yggdrasil.sock");
|
||||||
|
|
||||||
|
assert!(AdminEndpoint::parse("http://localhost:9001").is_err());
|
||||||
|
assert!(AdminEndpoint::parse("tcp://localhost").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn patches_land_in_base_config() {
|
||||||
|
let mut base: serde_json::Value =
|
||||||
|
serde_json::from_str(r#"{"PrivateKey": "secret", "Peers": [], "Listen": []}"#).unwrap();
|
||||||
|
let settings = NodeSettings {
|
||||||
|
peers: vec![PeerUri::parse("tcp://[200:1234::1]:1337").unwrap()],
|
||||||
|
listen: vec![ListenUri::parse("tls://[::]:1337").unwrap()],
|
||||||
|
multicast: Some(MulticastMode::Disabled),
|
||||||
|
admin_listen: Some(AdminEndpoint::parse("tcp://localhost:9001").unwrap()),
|
||||||
|
if_name: Some(IfName::Headless),
|
||||||
|
allowed_public_keys: vec![PublicKeyHex::parse(&"ab".repeat(32)).unwrap()],
|
||||||
|
private_key_path: Some(PathBuf::from("/etc/spidernet/ygg.key")),
|
||||||
|
};
|
||||||
|
apply_to_base_config(&settings, &mut base).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(base["Peers"].as_array().unwrap().len(), 1);
|
||||||
|
assert_eq!(base["Listen"][0], "tls://[::]:1337");
|
||||||
|
assert_eq!(base["MulticastInterfaces"].as_array().unwrap().len(), 0);
|
||||||
|
assert_eq!(base["AdminListen"], "tcp://localhost:9001");
|
||||||
|
assert_eq!(base["IfName"], "none");
|
||||||
|
assert_eq!(base["AllowedPublicKeys"][0], "ab".repeat(32));
|
||||||
|
assert_eq!(base["PrivateKeyPath"], "/etc/spidernet/ygg.key");
|
||||||
|
// The base's own keys survive the patch.
|
||||||
|
assert_eq!(base["PrivateKey"], "secret");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn multicast_interfaces_are_serialized_in_yggdrasil_shape() {
|
||||||
|
let mut base: serde_json::Value = serde_json::from_str("{}").unwrap();
|
||||||
|
let settings = NodeSettings {
|
||||||
|
multicast: Some(MulticastMode::Interfaces(vec![MulticastInterface {
|
||||||
|
regex: "eth0".to_string(),
|
||||||
|
beacon: true,
|
||||||
|
listen: true,
|
||||||
|
port: 1337,
|
||||||
|
}])),
|
||||||
|
..NodeSettings::default()
|
||||||
|
};
|
||||||
|
apply_to_base_config(&settings, &mut base).unwrap();
|
||||||
|
let entry = &base["MulticastInterfaces"][0];
|
||||||
|
assert_eq!(entry["Regex"], "eth0");
|
||||||
|
assert_eq!(entry["Beacon"], true);
|
||||||
|
assert_eq!(entry["Listen"], true);
|
||||||
|
assert_eq!(entry["Port"], 1337);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
//! Error type for all sn-node operations.
|
||||||
|
|
||||||
|
#[derive(Debug, thiserror::Error)]
|
||||||
|
pub enum NodeError {
|
||||||
|
#[error("io error: {0}")]
|
||||||
|
Io(#[from] std::io::Error),
|
||||||
|
|
||||||
|
#[error("json error: {0}")]
|
||||||
|
Json(#[from] serde_json::Error),
|
||||||
|
|
||||||
|
#[error("yggdrasil binary `{binary}` failed: {stderr}")]
|
||||||
|
YggdrasilBinary { binary: String, stderr: String },
|
||||||
|
|
||||||
|
#[error("invalid uri `{uri}`: {hint}")]
|
||||||
|
InvalidUri { uri: String, hint: &'static str },
|
||||||
|
|
||||||
|
#[error("invalid admin endpoint `{0}`: expected `tcp://host:port` or `unix:///path`")]
|
||||||
|
InvalidAdminEndpoint(String),
|
||||||
|
|
||||||
|
#[error("invalid public key `{0}`: expected 64 hex characters")]
|
||||||
|
InvalidPublicKey(String),
|
||||||
|
|
||||||
|
#[error("generated config base has unexpected shape (expected a JSON object)")]
|
||||||
|
UnexpectedConfigShape,
|
||||||
|
|
||||||
|
#[error("admin protocol violation: {0}")]
|
||||||
|
AdminProtocol(String),
|
||||||
|
|
||||||
|
#[error("admin socket answered with an error: {0}")]
|
||||||
|
AdminFailed(String),
|
||||||
|
|
||||||
|
#[error("admin socket operation timed out")]
|
||||||
|
Timeout,
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
//! sn-node — `SpiderNet` node helper for the Yggdrasil overlay.
|
||||||
|
//!
|
||||||
|
//! `SpiderNet` is software for neighborhood fiber meshes: households peer
|
||||||
|
//! their internet uplinks and pool bandwidth. This crate is the node-side
|
||||||
|
//! glue for the overlay layer:
|
||||||
|
//!
|
||||||
|
//! - [`config`]: generate a Yggdrasil configuration file from validated,
|
||||||
|
//! typed `SpiderNet` settings (peers, listeners, admin socket, allowed
|
||||||
|
//! neighbor keys).
|
||||||
|
//! - [`yggdrasil`]: client for the Yggdrasil admin socket (JSON over
|
||||||
|
//! TCP/unix, default `localhost:9001`) to query node status (`getSelf`,
|
||||||
|
//! `getPeers`).
|
||||||
|
//!
|
||||||
|
//! Yggdrasil provides encrypted end-to-end routing between households.
|
||||||
|
//! It is **not** an anonymity network, and `SpiderNet` never claims to
|
||||||
|
//! provide anonymity.
|
||||||
|
//!
|
||||||
|
//! `SpiderNet` software carries no warranty and accepts no responsibility
|
||||||
|
//! for its usage or for transported content (AGPL-3.0, sections 15/16).
|
||||||
|
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
#![warn(clippy::pedantic)]
|
||||||
|
// PoC scope: doc-annotation noise, not correctness. Revisit at release.
|
||||||
|
#![allow(clippy::must_use_candidate, clippy::missing_errors_doc)]
|
||||||
|
|
||||||
|
pub mod config;
|
||||||
|
pub mod error;
|
||||||
|
pub mod yggdrasil;
|
||||||
@@ -0,0 +1,229 @@
|
|||||||
|
//! CLI entry point for sn-node.
|
||||||
|
//!
|
||||||
|
//! KEINE GARANTIE — this software is provided "as is" without warranty of
|
||||||
|
//! any kind (AGPL-3.0, sections 15/16). Users are solely responsible for
|
||||||
|
//! what they transmit or receive with it.
|
||||||
|
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use clap::{Parser, Subcommand};
|
||||||
|
use sn_node::config::{
|
||||||
|
self, IfName, ListenUri, MulticastMode, NodeSettings, PeerUri, PublicKeyHex,
|
||||||
|
};
|
||||||
|
use sn_node::yggdrasil::{self, AdminEndpoint};
|
||||||
|
|
||||||
|
#[derive(Debug, Parser)]
|
||||||
|
#[command(
|
||||||
|
name = "sn-node",
|
||||||
|
about = "SpiderNet node helper: Yggdrasil config generation + overlay status"
|
||||||
|
)]
|
||||||
|
struct Args {
|
||||||
|
#[command(subcommand)]
|
||||||
|
command: Command,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Subcommand)]
|
||||||
|
enum Command {
|
||||||
|
/// Generate a Yggdrasil config file for this SpiderNet node.
|
||||||
|
Genconf {
|
||||||
|
/// Peering URIs to connect to, comma separated (tcp://, tls://,
|
||||||
|
/// quic://, socks://).
|
||||||
|
#[arg(long = "peer", value_delimiter = ',')]
|
||||||
|
peers: Vec<String>,
|
||||||
|
|
||||||
|
/// Listener URIs for incoming peerings, comma separated
|
||||||
|
/// (e.g. `tcp://[::]:1337`).
|
||||||
|
#[arg(long = "listen", value_delimiter = ',')]
|
||||||
|
listen: Vec<String>,
|
||||||
|
|
||||||
|
/// Admin socket endpoint (tcp://host:port or unix:///path).
|
||||||
|
#[arg(long, default_value = "tcp://localhost:9001")]
|
||||||
|
admin: String,
|
||||||
|
|
||||||
|
/// Public keys of neighbors allowed to peer with us (64 hex
|
||||||
|
/// characters, comma separated). Empty: any node may peer.
|
||||||
|
#[arg(long = "allow-key", value_delimiter = ',')]
|
||||||
|
allow_key: Vec<String>,
|
||||||
|
|
||||||
|
/// Store the node's private key in an external file instead of
|
||||||
|
/// embedding it in the config.
|
||||||
|
#[arg(long)]
|
||||||
|
private_key_path: Option<PathBuf>,
|
||||||
|
|
||||||
|
/// TUN interface: `auto`, `none` (headless router-only), or a name.
|
||||||
|
#[arg(long, default_value = "auto")]
|
||||||
|
ifname: String,
|
||||||
|
|
||||||
|
/// Disable multicast (same-LAN) auto-peering.
|
||||||
|
#[arg(long)]
|
||||||
|
no_multicast: bool,
|
||||||
|
|
||||||
|
/// Path to the yggdrasil binary.
|
||||||
|
#[arg(long, default_value = "yggdrasil")]
|
||||||
|
yggdrasil_bin: String,
|
||||||
|
|
||||||
|
/// Output path for the generated config.
|
||||||
|
#[arg(short, long, default_value = "yggdrasil.conf")]
|
||||||
|
out: PathBuf,
|
||||||
|
},
|
||||||
|
/// Query the local Yggdrasil node via the admin socket.
|
||||||
|
Status {
|
||||||
|
/// Admin socket endpoint (tcp://host:port or unix:///path).
|
||||||
|
#[arg(long, default_value = "tcp://localhost:9001")]
|
||||||
|
admin: String,
|
||||||
|
|
||||||
|
/// Operation timeout in seconds.
|
||||||
|
#[arg(long, default_value_t = 3)]
|
||||||
|
timeout_secs: u64,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_ifname(input: &str) -> Result<IfName, String> {
|
||||||
|
match input.trim() {
|
||||||
|
"auto" => Ok(IfName::Auto),
|
||||||
|
"none" => Ok(IfName::Headless),
|
||||||
|
name if !name.is_empty() => Ok(IfName::Named(name.to_string())),
|
||||||
|
_ => Err("--ifname must be `auto`, `none` or an interface name".to_string()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_uris<T>(
|
||||||
|
inputs: &[String],
|
||||||
|
what: &str,
|
||||||
|
parse: fn(&str) -> Result<T, sn_node::error::NodeError>,
|
||||||
|
) -> Result<Vec<T>, String> {
|
||||||
|
inputs
|
||||||
|
.iter()
|
||||||
|
.map(|input| parse(input).map_err(|err| format!("{what}: {err}")))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::main]
|
||||||
|
async fn main() {
|
||||||
|
if let Err(report) = run().await {
|
||||||
|
eprintln!("NO WARRANTY — SpiderNet sn-node (AGPL-3.0, no liability)");
|
||||||
|
eprintln!("error: {report}");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run() -> Result<(), String> {
|
||||||
|
let args = Args::parse();
|
||||||
|
match args.command {
|
||||||
|
Command::Genconf {
|
||||||
|
peers,
|
||||||
|
listen,
|
||||||
|
admin,
|
||||||
|
allow_key,
|
||||||
|
private_key_path,
|
||||||
|
ifname,
|
||||||
|
no_multicast,
|
||||||
|
yggdrasil_bin,
|
||||||
|
out,
|
||||||
|
} => {
|
||||||
|
run_genconf(
|
||||||
|
peers,
|
||||||
|
listen,
|
||||||
|
admin,
|
||||||
|
allow_key,
|
||||||
|
private_key_path,
|
||||||
|
ifname,
|
||||||
|
no_multicast,
|
||||||
|
yggdrasil_bin,
|
||||||
|
out,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
}
|
||||||
|
Command::Status {
|
||||||
|
admin,
|
||||||
|
timeout_secs,
|
||||||
|
} => run_status(&admin, timeout_secs).await,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
async fn run_genconf(
|
||||||
|
peers: Vec<String>,
|
||||||
|
listen: Vec<String>,
|
||||||
|
admin: String,
|
||||||
|
allow_key: Vec<String>,
|
||||||
|
private_key_path: Option<PathBuf>,
|
||||||
|
ifname: String,
|
||||||
|
no_multicast: bool,
|
||||||
|
yggdrasil_bin: String,
|
||||||
|
out: PathBuf,
|
||||||
|
) -> Result<(), String> {
|
||||||
|
let peers = parse_uris(&peers, "peer", PeerUri::parse)?;
|
||||||
|
let listen = parse_uris(&listen, "listen", ListenUri::parse)?;
|
||||||
|
let admin_listen = AdminEndpoint::parse(&admin).map_err(|err| err.to_string())?;
|
||||||
|
let allowed_public_keys = allow_key
|
||||||
|
.iter()
|
||||||
|
.map(|key| PublicKeyHex::parse(key).map_err(|err| err.to_string()))
|
||||||
|
.collect::<Result<Vec<_>, String>>()?;
|
||||||
|
let if_name = parse_ifname(&ifname)?;
|
||||||
|
|
||||||
|
let settings = NodeSettings {
|
||||||
|
peers,
|
||||||
|
listen,
|
||||||
|
multicast: Some(if no_multicast {
|
||||||
|
MulticastMode::Disabled
|
||||||
|
} else {
|
||||||
|
MulticastMode::PlatformDefault
|
||||||
|
}),
|
||||||
|
admin_listen: Some(admin_listen),
|
||||||
|
if_name: Some(if_name),
|
||||||
|
allowed_public_keys,
|
||||||
|
private_key_path,
|
||||||
|
};
|
||||||
|
|
||||||
|
let json = config::generate_config(&yggdrasil_bin, &settings).map_err(|err| err.to_string())?;
|
||||||
|
config::write_config(&out, &json).map_err(|err| err.to_string())?;
|
||||||
|
|
||||||
|
println!("NO WARRANTY — SpiderNet sn-node (AGPL-3.0, no liability)");
|
||||||
|
println!();
|
||||||
|
println!("config written: {}", out.display());
|
||||||
|
println!(
|
||||||
|
"mode: start with `yggdrasil -useconffile {}`",
|
||||||
|
out.display()
|
||||||
|
);
|
||||||
|
println!();
|
||||||
|
println!(
|
||||||
|
"NOTE: the config contains this node's PRIVATE key. Keep it at \
|
||||||
|
permission 600, never commit it, never share it."
|
||||||
|
);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn run_status(admin: &str, timeout_secs: u64) -> Result<(), String> {
|
||||||
|
let endpoint = AdminEndpoint::parse(admin).map_err(|err| err.to_string())?;
|
||||||
|
let timeout = Duration::from_secs(timeout_secs);
|
||||||
|
|
||||||
|
let info = yggdrasil::AdminClient::get_self(&endpoint, timeout)
|
||||||
|
.await
|
||||||
|
.map_err(|err| err.to_string())?;
|
||||||
|
|
||||||
|
println!("NO WARRANTY — SpiderNet sn-node (AGPL-3.0, no liability)");
|
||||||
|
println!();
|
||||||
|
println!("address: {}", info.address);
|
||||||
|
if let Some(subnet) = info.subnet.as_ref() {
|
||||||
|
println!("subnet: {subnet}");
|
||||||
|
}
|
||||||
|
if let Some(version) = info.build_version.as_ref() {
|
||||||
|
let name = info.build_name.as_deref().unwrap_or("yggdrasil");
|
||||||
|
println!("version: {name} {version}");
|
||||||
|
}
|
||||||
|
println!("key: {}", info.key);
|
||||||
|
|
||||||
|
let peers = yggdrasil::AdminClient::get_peers(&endpoint, timeout)
|
||||||
|
.await
|
||||||
|
.map_err(|err| err.to_string())?;
|
||||||
|
println!("peers: {}", peers.len());
|
||||||
|
for peer in &peers {
|
||||||
|
match peer.remote.as_ref() {
|
||||||
|
Some(remote) => println!(" {} via {remote}", peer.address),
|
||||||
|
None => println!(" {}", peer.address),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -0,0 +1,268 @@
|
|||||||
|
//! Client for the Yggdrasil admin socket.
|
||||||
|
//!
|
||||||
|
//! Protocol (per the official docs): JSON stanza + `\n` over a TCP or unix
|
||||||
|
//! socket; the default endpoint is `localhost:9001`. A response always has
|
||||||
|
//! a `"status"` field (`"success"` or `"error"`), optionally a `"response"`
|
||||||
|
//! section and an `"error"` field. By default Yggdrasil closes the
|
||||||
|
//! connection after answering, so this client sends one request per
|
||||||
|
//! connection.
|
||||||
|
|
||||||
|
use std::net::Ipv6Addr;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crate::error::NodeError;
|
||||||
|
|
||||||
|
/// Where the admin socket lives, in Yggdrasil endpoint syntax
|
||||||
|
/// (`tcp://host:port` or `unix:///path/to/socket`).
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum AdminEndpoint {
|
||||||
|
Tcp { host: String, port: u16 },
|
||||||
|
Unix { path: PathBuf },
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AdminEndpoint {
|
||||||
|
/// Yggdrasil's default admin endpoint: `tcp://localhost:9001`.
|
||||||
|
pub fn default_local() -> Self {
|
||||||
|
Self::Tcp {
|
||||||
|
host: "localhost".to_string(),
|
||||||
|
port: 9001,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn parse(input: &str) -> Result<Self, NodeError> {
|
||||||
|
let trimmed = input.trim();
|
||||||
|
if let Some(rest) = trimmed.strip_prefix("tcp://") {
|
||||||
|
let (host, port) = rest
|
||||||
|
.rsplit_once(':')
|
||||||
|
.ok_or_else(|| NodeError::InvalidAdminEndpoint(trimmed.to_string()))?;
|
||||||
|
let port: u16 = port
|
||||||
|
.parse()
|
||||||
|
.map_err(|_| NodeError::InvalidAdminEndpoint(trimmed.to_string()))?;
|
||||||
|
if host.is_empty() {
|
||||||
|
return Err(NodeError::InvalidAdminEndpoint(trimmed.to_string()));
|
||||||
|
}
|
||||||
|
Ok(Self::Tcp {
|
||||||
|
host: host.to_string(),
|
||||||
|
port,
|
||||||
|
})
|
||||||
|
} else if let Some(rest) = trimmed.strip_prefix("unix://") {
|
||||||
|
if rest.is_empty() {
|
||||||
|
return Err(NodeError::InvalidAdminEndpoint(trimmed.to_string()));
|
||||||
|
}
|
||||||
|
Ok(Self::Unix {
|
||||||
|
path: PathBuf::from(rest),
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
Err(NodeError::InvalidAdminEndpoint(trimmed.to_string()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Value for the `AdminListen` field of a Yggdrasil config file.
|
||||||
|
pub fn config_value(&self) -> String {
|
||||||
|
match self {
|
||||||
|
Self::Tcp { host, port } => format!("tcp://{host}:{port}"),
|
||||||
|
Self::Unix { path } => format!("unix://{}", path.display()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The node's own Yggdrasil address — a validated IPv6 address inside the
|
||||||
|
/// Yggdrasil range.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct YggAddr(Ipv6Addr);
|
||||||
|
|
||||||
|
impl YggAddr {
|
||||||
|
pub const fn new(addr: Ipv6Addr) -> Self {
|
||||||
|
Self(addr)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub const fn inner(self) -> Ipv6Addr {
|
||||||
|
self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Display for YggAddr {
|
||||||
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
write!(f, "{}", self.0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One connection to the admin socket; one request per connection.
|
||||||
|
pub struct AdminClient {
|
||||||
|
stream: AdminStream,
|
||||||
|
}
|
||||||
|
|
||||||
|
enum AdminStream {
|
||||||
|
Tcp(tokio::net::TcpStream),
|
||||||
|
Unix(tokio::net::UnixStream),
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn connect_stream(endpoint: &AdminEndpoint) -> Result<AdminStream, NodeError> {
|
||||||
|
match endpoint {
|
||||||
|
AdminEndpoint::Tcp { host, port } => tokio::net::TcpStream::connect((host.as_str(), *port))
|
||||||
|
.await
|
||||||
|
.map(AdminStream::Tcp)
|
||||||
|
.map_err(NodeError::Io),
|
||||||
|
AdminEndpoint::Unix { path } => tokio::net::UnixStream::connect(path)
|
||||||
|
.await
|
||||||
|
.map(AdminStream::Unix)
|
||||||
|
.map_err(NodeError::Io),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AdminClient {
|
||||||
|
pub async fn connect(endpoint: &AdminEndpoint, timeout: Duration) -> Result<Self, NodeError> {
|
||||||
|
let stream = tokio::time::timeout(timeout, connect_stream(endpoint))
|
||||||
|
.await
|
||||||
|
.map_err(|_| NodeError::Timeout)??;
|
||||||
|
Ok(Self { stream })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Send one request verb (e.g. `getself`, `getpeers`) and return the
|
||||||
|
/// raw response section.
|
||||||
|
pub async fn request(&mut self, verb: &str) -> Result<serde_json::Value, NodeError> {
|
||||||
|
// `verb` is a compile-time-known constant at every call site, not
|
||||||
|
// user input — no injection risk.
|
||||||
|
let request = format!("{{\"request\":\"{verb}\"}}\n");
|
||||||
|
let bytes = request.as_bytes();
|
||||||
|
|
||||||
|
match &mut self.stream {
|
||||||
|
AdminStream::Tcp(stream) => {
|
||||||
|
tokio::io::AsyncWriteExt::write_all(stream, bytes).await?;
|
||||||
|
let mut buffer = Vec::new();
|
||||||
|
tokio::io::AsyncReadExt::read_to_end(stream, &mut buffer).await?;
|
||||||
|
Self::parse_response(&buffer, verb)
|
||||||
|
}
|
||||||
|
AdminStream::Unix(stream) => {
|
||||||
|
tokio::io::AsyncWriteExt::write_all(stream, bytes).await?;
|
||||||
|
let mut buffer = Vec::new();
|
||||||
|
tokio::io::AsyncReadExt::read_to_end(stream, &mut buffer).await?;
|
||||||
|
Self::parse_response(&buffer, verb)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query the node's own status (address, subnet, key, build info).
|
||||||
|
pub async fn get_self(
|
||||||
|
endpoint: &AdminEndpoint,
|
||||||
|
timeout: Duration,
|
||||||
|
) -> Result<SelfInfo, NodeError> {
|
||||||
|
let mut client = Self::connect(endpoint, timeout).await?;
|
||||||
|
let value = client.request("getself").await?;
|
||||||
|
SelfInfo::from_response(&value)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Query the node's active peer sessions.
|
||||||
|
pub async fn get_peers(
|
||||||
|
endpoint: &AdminEndpoint,
|
||||||
|
timeout: Duration,
|
||||||
|
) -> Result<Vec<PeerInfo>, NodeError> {
|
||||||
|
let mut client = Self::connect(endpoint, timeout).await?;
|
||||||
|
let value = client.request("getpeers").await?;
|
||||||
|
peers_from_response(&value)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_response(buffer: &[u8], verb: &str) -> Result<serde_json::Value, NodeError> {
|
||||||
|
let envelope: serde_json::Value = serde_json::from_slice(buffer)?;
|
||||||
|
let status = envelope
|
||||||
|
.get("status")
|
||||||
|
.and_then(serde_json::Value::as_str)
|
||||||
|
.ok_or_else(|| {
|
||||||
|
NodeError::AdminProtocol(format!("{verb}: response has no `status` field"))
|
||||||
|
})?;
|
||||||
|
if status != "success" {
|
||||||
|
let error = envelope
|
||||||
|
.get("error")
|
||||||
|
.and_then(serde_json::Value::as_str)
|
||||||
|
.unwrap_or("unknown error");
|
||||||
|
return Err(NodeError::AdminFailed(error.to_string()));
|
||||||
|
}
|
||||||
|
envelope
|
||||||
|
.get("response")
|
||||||
|
.cloned()
|
||||||
|
.ok_or_else(|| NodeError::AdminProtocol(format!("{verb}: response has no data")))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Information about the local Yggdrasil node (`getSelf`).
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct SelfInfo {
|
||||||
|
pub address: YggAddr,
|
||||||
|
pub key: String,
|
||||||
|
pub subnet: Option<String>,
|
||||||
|
pub coords: Option<serde_json::Value>,
|
||||||
|
pub build_name: Option<String>,
|
||||||
|
pub build_version: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SelfInfo {
|
||||||
|
/// Parse the `getSelf` response section: exactly one record, keyed by
|
||||||
|
/// the node's IPv6 address.
|
||||||
|
pub fn from_response(value: &serde_json::Value) -> Result<Self, NodeError> {
|
||||||
|
let obj = value.as_object().ok_or_else(|| {
|
||||||
|
NodeError::AdminProtocol("getSelf: response section is not an object".to_string())
|
||||||
|
})?;
|
||||||
|
let (address, record) = obj.iter().next().ok_or_else(|| {
|
||||||
|
NodeError::AdminProtocol("getSelf: response contains no record".to_string())
|
||||||
|
})?;
|
||||||
|
let address = YggAddr::new(address.parse::<Ipv6Addr>().map_err(|_| {
|
||||||
|
NodeError::AdminProtocol(format!("getSelf: `{address}` is not an IPv6 address"))
|
||||||
|
})?);
|
||||||
|
|
||||||
|
let str_field = |name: &str| {
|
||||||
|
record
|
||||||
|
.get(name)
|
||||||
|
.and_then(serde_json::Value::as_str)
|
||||||
|
.map(str::to_string)
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
address,
|
||||||
|
key: str_field("key").ok_or_else(|| {
|
||||||
|
NodeError::AdminProtocol("getSelf: record has no `key`".to_string())
|
||||||
|
})?,
|
||||||
|
subnet: str_field("subnet"),
|
||||||
|
coords: record.get("coords").cloned(),
|
||||||
|
build_name: str_field("build_name"),
|
||||||
|
build_version: str_field("build_version"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One active peer session (`getPeers`).
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct PeerInfo {
|
||||||
|
pub address: YggAddr,
|
||||||
|
pub key: Option<String>,
|
||||||
|
/// The peering URI of the remote peer, when reported.
|
||||||
|
pub remote: Option<String>,
|
||||||
|
pub coords: Option<serde_json::Value>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse the `getPeers` response section: records keyed by IPv6 address.
|
||||||
|
/// The first record typically refers to the current node itself.
|
||||||
|
pub fn peers_from_response(value: &serde_json::Value) -> Result<Vec<PeerInfo>, NodeError> {
|
||||||
|
let obj = value.as_object().ok_or_else(|| {
|
||||||
|
NodeError::AdminProtocol("getPeers: response section is not an object".to_string())
|
||||||
|
})?;
|
||||||
|
let mut peers = Vec::with_capacity(obj.len());
|
||||||
|
for (address, record) in obj {
|
||||||
|
let address = YggAddr::new(address.parse::<Ipv6Addr>().map_err(|_| {
|
||||||
|
NodeError::AdminProtocol(format!("getPeers: `{address}` is not an IPv6 address"))
|
||||||
|
})?);
|
||||||
|
peers.push(PeerInfo {
|
||||||
|
address,
|
||||||
|
key: record
|
||||||
|
.get("key")
|
||||||
|
.and_then(serde_json::Value::as_str)
|
||||||
|
.map(str::to_string),
|
||||||
|
remote: record
|
||||||
|
.get("remote")
|
||||||
|
.and_then(serde_json::Value::as_str)
|
||||||
|
.map(str::to_string),
|
||||||
|
coords: record.get("coords").cloned(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(peers)
|
||||||
|
}
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
//! Integration tests for the Yggdrasil admin socket client, against a
|
||||||
|
//! fake admin server speaking the documented protocol.
|
||||||
|
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
#![allow(clippy::unwrap_used, clippy::expect_used)]
|
||||||
|
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use sn_node::error::NodeError;
|
||||||
|
use sn_node::yggdrasil::{peers_from_response, AdminClient, AdminEndpoint, SelfInfo};
|
||||||
|
|
||||||
|
const SELF_RESPONSE: &str = concat!(
|
||||||
|
r#"{"request": {"request": "getself"}, "status": "success", "response": {"#,
|
||||||
|
r#""200:1111:2222:3333:4444:5555:6666:7777": {"#,
|
||||||
|
r#""address": "200:1111:2222:3333:4444:5555:6666:7777","#,
|
||||||
|
r#""key": "0000aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa1111","#,
|
||||||
|
r#""build_name": "yggdrasil", "build_version": "0.5.12","#,
|
||||||
|
r#""subnet": "300:1111:2222:3333::/64", "coords": [1, 2, 3]}}}"#
|
||||||
|
);
|
||||||
|
|
||||||
|
const PEERS_RESPONSE: &str = concat!(
|
||||||
|
r#"{"request": {"request": "getpeers"}, "status": "success", "response": {"#,
|
||||||
|
r#""200:1111:2222:3333:4444:5555:6666:7777": {"coords": [1]}, "#,
|
||||||
|
r#""200:9999:2222:3333:4444:5555:6666:7777": {"#,
|
||||||
|
r#""coords": [1, 2], "port": 1, "key": "ffff...", "#,
|
||||||
|
r#""remote": "tcp://[fe80::1%25eth0]:1337"}}}"#
|
||||||
|
);
|
||||||
|
|
||||||
|
const ERROR_RESPONSE: &str =
|
||||||
|
r#"{"request": {"request": "getself"}, "status": "error", "error": "boom"}"#;
|
||||||
|
|
||||||
|
/// Fake admin server: answers each connection's request line, then closes.
|
||||||
|
async fn fake_admin_server(response: &'static str) -> std::net::SocketAddr {
|
||||||
|
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||||
|
|
||||||
|
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let addr = listener.local_addr().unwrap();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
loop {
|
||||||
|
let Ok((mut socket, _)) = listener.accept().await else {
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
let mut line = Vec::new();
|
||||||
|
let mut byte = [0u8; 1];
|
||||||
|
while !line.ends_with(b"\n") {
|
||||||
|
if socket.read(&mut byte).await.unwrap_or(0) == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
line.push(byte[0]);
|
||||||
|
}
|
||||||
|
socket.write_all(response.as_bytes()).await.ok();
|
||||||
|
socket.shutdown().await.ok();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
addr
|
||||||
|
}
|
||||||
|
|
||||||
|
fn tcp_endpoint(addr: std::net::SocketAddr) -> AdminEndpoint {
|
||||||
|
AdminEndpoint::Tcp {
|
||||||
|
host: addr.ip().to_string(),
|
||||||
|
port: addr.port(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn get_self_parses_address_subnet_and_version() {
|
||||||
|
let endpoint = tcp_endpoint(fake_admin_server(SELF_RESPONSE).await);
|
||||||
|
let info = SelfInfo::from_response(
|
||||||
|
&AdminClient::connect(&endpoint, Duration::from_secs(2))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.request("getself")
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
info.address.inner().to_string(),
|
||||||
|
"200:1111:2222:3333:4444:5555:6666:7777"
|
||||||
|
);
|
||||||
|
assert_eq!(info.subnet.as_deref(), Some("300:1111:2222:3333::/64"));
|
||||||
|
assert_eq!(info.build_version.as_deref(), Some("0.5.12"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn get_peers_lists_remote_uris() {
|
||||||
|
let endpoint = tcp_endpoint(fake_admin_server(PEERS_RESPONSE).await);
|
||||||
|
let value = AdminClient::connect(&endpoint, Duration::from_secs(2))
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.request("getpeers")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let peers = peers_from_response(&value).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(peers.len(), 2);
|
||||||
|
assert_eq!(
|
||||||
|
peers[1].remote.as_deref(),
|
||||||
|
Some("tcp://[fe80::1%25eth0]:1337")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn admin_error_is_surfaced() {
|
||||||
|
let endpoint = tcp_endpoint(fake_admin_server(ERROR_RESPONSE).await);
|
||||||
|
let mut client = AdminClient::connect(&endpoint, Duration::from_secs(2))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let err = client.request("getself").await.unwrap_err();
|
||||||
|
assert!(matches!(err, NodeError::AdminFailed(ref text) if text == "boom"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn connection_refused_becomes_io_or_timeout_error() {
|
||||||
|
// Port 1 on localhost: virtually guaranteed to be closed.
|
||||||
|
let endpoint = AdminEndpoint::Tcp {
|
||||||
|
host: "127.0.0.1".into(),
|
||||||
|
port: 1,
|
||||||
|
};
|
||||||
|
let result = AdminClient::get_self(&endpoint, Duration::from_millis(500)).await;
|
||||||
|
assert!(matches!(
|
||||||
|
result,
|
||||||
|
Err(NodeError::Io(_)) | Err(NodeError::Timeout)
|
||||||
|
));
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user