Skip to main content

Module surb_telemetry

Module surb_telemetry 

Source
Expand description

Records SURB round-trips as network graph edge telemetry. Turns SURB round-trips into edge telemetry for the network graph.

A SURB rides a forward path to reach its destination and carries a return path for the reply to come back on. When the reply arrives, that is proof both legs passed end to end – evidence the graph already wants, produced by traffic a session was sending anyway. Unlike a probe it costs no extra packets and accrues at data rates, which is what lets a dead relayer be noticed in seconds rather than after a probe success rate has moved behind a path cache.

§Why one layer owns both directions

Minting and consuming happen at opposite ends of the pipeline, but only together do they mean anything: the mint says what was expected, the reply says what was observed. Wrapping both codec halves keeps the SURB-to-path association private to this module, so nothing below has to carry a path identity it has no other use for.

§Mitigations driven from here

Each names its trigger, its effect, its setting and what it must not work against.

  • Relayer health. Trigger: a relayer losing reply races (two SURBs of one packet, different return paths) by more than a round trip. Effect: its health in the graph scales every edge into it, so forward and return paths through it are drawn less. Settings: HoprProtocolConfig::latency_race_memory, HoprProtocolConfig::relayer_health_weight (0 off). Must not fight: the return-path temper, which flattens it along with path quality.
  • Dead verdict. Trigger: at least MIN_RACES replies trailing their partners by more than any race settled by a relayer on their leg, none home through the relayer since, and a reply home through some relayer after the first of them fell due. A reply is overdue only while others are delivering: a pause in the replier’s stream, or an outage of every return relayer, leaves nothing arriving and so nothing overdue, however long the replies stay out. A relayer that has settled too few races to have a margin of its own borrows the bar of the best-measured relayer, raised to whatever its own settled races measure, and none is judged while no relayer has one; its own races that expired open are booked as lost and are not replies that came home, so they do not reset the clock. Of the relayers the same overdue replies point at, the one that explains the most of them is taken, the one further up their legs if two explain the same, and two that the replies do not tell apart wait for a reply that does (explain_away). A relayer has two hops: from the counterparty, where it is a leg’s first relayer, and from another relayer, whichever that is. One the counterparty can no longer reach still delivers behind other relayers, which says nothing about the replies sent to it directly: a relayer that delivers over one hop is judged over the other on MIN_RACES_OVER_ONE_HOP replies with none home over that hop since, is only suspected at first, and judged when the next flush finds the hop still silent (RelayerHealth::has_stood). Effect: health 0, the walk prunes it. A verdict is lifted by a reply coming home through the relayer, over the hop it is about if it was reached while the other delivered, not by time. From the moment the verdict is acted on (VERDICT_CONFIRMATION), only a reply on a SURB minted after it counts: the late replies a slow relayer still delivers are what the verdict was about, not news that it ended (RelayerHealth::is_fresh_for). Once no reply through it is out any more (about PENDING_SURB_TTL) the condition lapses, and the relayer goes on probation: published at PROBATION_HEALTH, which weighted selection all but never draws while the planner’s exploration still does, and that draw is the probe. A probe is minted after the verdict, so a probe reply home through it revives it, to the health its lost races leave it, so gradually; a probe that goes missing leaves it where it is, and one that is a black hole again goes back to dead without a new SURB generation. Must not fight: common-mode outages, where no partner gets home and nothing is judged.
  • Invalid SURBs. Trigger: a dead verdict, or probation. Effect: each session is told how many SURBs in its counterparty’s store run through that relayer; the balancer subtracts them from its level and sends replacements. A lapsed verdict is only a timeout and the counterparty still holds the SURBs, probe SURBs included, so the write-off outlives it; each written-off SURB counts as held only until WRITE_OFF_HORIZON, the time the counterparty needs to spend it, and a reply through the relayer that revives it (one on a SURB minted after the verdict, see RelayerHealth::is_fresh_for) or the end of the session ends the write-off sooner. A session’s count is dropped when its SURB generation is advanced, though: the wipe takes the SURBs it counted, and from then on it counts only the probes minted since. Must not fight: flow control, which keeps the raw level so the replacements are not throttled.
  • Fresh SURB generation. Trigger: a dead verdict that held for a second flush (VERDICT_CONFIRMATION) reaching a session for the first time, at most once per (session, relayer) while the relayer is known dead, and per REBUMP_COOLDOWN after it is revived. Effect: the session’s SURB generation advances, so the counterparty drops its whole stock on its next push, the dead SURBs it cannot recognise along with the rest, and the balancer refills from empty, paced; the session’s write-off goes with the stock it counted. Probation is not a new verdict: a pair bumped in this death is not bumped again until a reply revives the relayer, so one that dies again on probation costs the session nothing more. Only stock advances a generation: a SURB minted after its relayer was already known dead is a probe, written off without one, also in a session the wipe never reached. Must not fight: the cooldown, since each advance costs the counterparty everything it holds, and a PIX cycle can absorb only so many such losses.
  • Lost neighbour. Trigger: our own connection drops. Effect: re-plan only; the graph’s connection status already rejects that edge, so its SURBs are not invalidated.

With HoprProtocolConfig::relayer_health_weight at 0 none of these acts, except the lost-neighbour re-plan.

Structs§

Judgement 🔒
A relayer’s standing in its races: median margin behind its partners, that median’s uncertainty, and the fastest round-trip among them.
LastReply 🔒
When a reply last came home, kept in atomics: it is recorded for every reply, on the packet hot path.
PendingSurb 🔒
What a minted SURB is remembered by until its reply comes home.
Race 🔒
One race as one of its relayers saw it.
RaceHistory 🔒
RelayerHealth
Per-relayer health from reply races: two SURBs of one packet share mint time, forward path and store slot, so the gap between their replies is down to the return paths alone.
StockThroughDead 🔒
An outstanding SURB whose return path runs through a relayer confirmed dead, as found by SurbRoundTripRegistry::dead_stock.
SurbRoundTripCounters
Expected/observed counts for one pair of legs, accumulated between flushes.
SurbRoundTripRegistry
Round-trip counts keyed by the legs they were observed over.
SurbTelemetryCodec
Wraps a codec so the SURBs it mints and consumes are counted per pair of legs.
Suspect 🔒
A relayer whose overdue replies would make it a black hole.

Enums§

Hop 🔒
Where a reply reaches a relayer from on its return leg: the replier, when the relayer is the leg’s first, or another relayer.
NoPathTelemetry
Counterpart to NoPeerTelemetry for the path half of record_edge.
NoPeerTelemetry
record_edge is generic over peer and path telemetry that a SURB observation does not carry.

Constants§

FIRST_SPENT_INDEX 🔒
The position, in a pair of SURBs minted together, of the one the replier spends first: the first, as a replier spends its SURBs oldest first (FIFO is the only pop order, see hopr_protocol_hopr::SurbPopOrder).
MIN_RACES 🔒
The fewest races whose median no single race decides.
MIN_RACES_OVER_ONE_HOP 🔒
The fewest overdue replies that make a relayer a black hole over one of its hops while it delivers over the other (see RaceHistory::hop_at_fault).
PATH_ID_SLOTS 🔒
Slots in a [PathId], and therefore the longest leg that can be reported.
PENDING_SURB_HORIZON 🔒
How long a minted SURB stays eligible to be matched with its reply.
PENDING_SURB_TTL 🔒
How long a race stays open: a reply still out this long after its partner came home is lost.
PROBATION_HEALTH 🔒
The health published for a relayer on probation: a thousandth.
REBUMP_COOLDOWN 🔒
How long after a relayer is revived, and no longer known dead, the session that was advanced for its verdict is not advanced again for it: twice PENDING_SURB_TTL, counted from the revival.
VERDICT_CONFIRMATION 🔒
How long a dead verdict must have held before it is acted on: the write-off and the SURB generation advance wait for the next flush to find the relayer still dead.
WRITE_OFF_HORIZON 🔒
How long a SURB written off through a relayer known dead keeps counting as held by the counterparty, and therefore as invalid.

Functions§

explain_away 🔒
Of the relayers whose overdue replies would make them black holes, those whose lateness no other one explains.
flush_into
Turns accumulated counts into graph observations.
hops_of 🔒
The relayers of a leg, each with the hop a reply reaches it over and its position on the leg.
invalid_updates
The invalid SURBs to hand each session this tick: every current mark, re-applied so a session opened since picks it up, and (0, 0.0) for sessions whose mark has gone.
median 🔒
Lower median; values must not be empty.
no_path_slots
A resolver that places no node, so nothing is ever attributed.
path_id 🔒
Builds the [PathId] of a leg from the nodes it visits.
path_slots_of
Reads path slots out of a network graph.
race_partners 🔒
The SURB each one races: its neighbour in mint order (first with second, third with fourth), when both return over relayers and not over the same ones.
refresh_relayer_health
Re-judges relayer health, records what changed in the graph, and re-plans when it or a neighbour changed, before writing off the SURBs of relayers judged dead. Recording precedes the re-plan, so the re-plan already avoids them; the write-off follows it, so the refill goes over new paths.
relayers_of 🔒
round_trip_paths 🔒
Derives both legs of a round-trip from the routing that produced the SURB.

Type Aliases§

PathSlotResolver
Resolves a node to the slot it occupies in a [PathId].
ReturnRelayers 🔒
The relayers of a reply leg, from the first after the replier to the last before us.