pub struct RelayerHealth {
relayers: DashMap<OffchainPublicKey, RaceHistory>,
lost_neighbours: DashSet<OffchainPublicKey>,
neighbours_changed: AtomicBool,
dead: DashSet<OffchainPublicKey>,
dead_since: DashMap<OffchainPublicKey, Instant>,
confirmed: DashSet<OffchainPublicKey>,
probation: DashSet<OffchainPublicKey>,
known_dead_since: DashMap<OffchainPublicKey, Instant>,
dead_over: DashMap<OffchainPublicKey, Hop>,
suspected: DashMap<OffchainPublicKey, (Hop, Instant)>,
published: DashMap<OffchainPublicKey, f64>,
last_reply: LastReply,
}Expand description
Per-relayer health from reply races: two SURBs of one packet share mint time, forward path and store slot, so the gap between their replies is down to the return paths alone.
Fields§
§relayers: DashMap<OffchainPublicKey, RaceHistory>§lost_neighbours: DashSet<OffchainPublicKey>Neighbours we lost our own connection to, to re-plan when one comes or goes.
neighbours_changed: AtomicBool§dead: DashSet<OffchainPublicKey>Relayers judged dead: black holes as of the last flush, published at 0.
dead_since: DashMap<OffchainPublicKey, Instant>When each relayer in dead was judged, for VERDICT_CONFIRMATION. Removed with the relayer.
confirmed: DashSet<OffchainPublicKey>Relayers in dead whose verdict held for VERDICT_CONFIRMATION, and so is acted on. A verdict
that lapses before it is confirmed is withdrawn rather than put on probation: it never held.
probation: DashSet<OffchainPublicKey>Relayers whose dead verdict has lapsed without a reply that is news about them coming home through them,
published at PROBATION_HEALTH. Disjoint from dead; a relayer leaves both only by such a reply (see
RelayerHealth::is_fresh_for).
known_dead_since: DashMap<OffchainPublicKey, Instant>When each relayer known dead was first judged dead in its current death, kept through probation
and any later verdict until a reply revives it. A SURB minted through it after that is a probe the
planner’s exploration sent, not stock the counterparty held when it died; see
SurbRoundTripRegistry::write_off_dead.
dead_over: DashMap<OffchainPublicKey, Hop>For a relayer judged dead while it still delivered over its other hop, the hop its verdict is
about: only a reply over that one lifts it (see Self::revive_by_reply). None for a relayer
that delivered nothing at all, which any reply through it revives. Kept from the verdict through
probation, removed with the death.
suspected: DashMap<OffchainPublicKey, (Hop, Instant)>Relayers that would be judged over one of their hops while delivering over the other, and since
when: suspected, with nothing published or written off for them yet (see Self::has_stood). The
hop is the one they are suspected over: a suspicion over the other hop is a new one.
published: DashMap<OffchainPublicKey, f64>Health last handed to the graph, below 1.0 only.
last_reply: LastReplyWhen a reply last came home, through any relayer, for any session.
Implementations§
Source§impl RelayerHealth
impl RelayerHealth
Sourcepub(crate) fn record_race(
&self,
later: &[Option<OffchainPublicKey>; 5],
later_surb: HoprSurbId,
earlier: &[Option<OffchainPublicKey>; 5],
winning_round_trip: Duration,
by: Duration,
)
pub(crate) fn record_race( &self, later: &[Option<OffchainPublicKey>; 5], later_surb: HoprSurbId, earlier: &[Option<OffchainPublicKey>; 5], winning_round_trip: Duration, by: Duration, )
Records that the reply over later came home by after the one over earlier, which took
winning_round_trip, both on SURBs of one packet. Relayers on both legs cancel out.
Sourcepub(crate) fn open_race(
&self,
waiting: &[Option<OffchainPublicKey>; 5],
surb: HoprSurbId,
home: &[Option<OffchainPublicKey>; 5],
winning_round_trip: Duration,
)
pub(crate) fn open_race( &self, waiting: &[Option<OffchainPublicKey>; 5], surb: HoprSurbId, home: &[Option<OffchainPublicKey>; 5], winning_round_trip: Duration, )
Records that the reply over home came home in winning_round_trip while surb, over
waiting, is still out.
Sourcefn discard_race(
&self,
waiting: &[Option<OffchainPublicKey>; 5],
surb: &HoprSurbId,
)
fn discard_race( &self, waiting: &[Option<OffchainPublicKey>; 5], surb: &HoprSurbId, )
Drops the race surb, over waiting, opened without settling or booking it: its record is gone
before its reply could close it, so the race says nothing about the relayers.
Sourcefn discard_race_if_fresh(
&self,
waiting: &[Option<OffchainPublicKey>; 5],
surb: &HoprSurbId,
minted_at: Instant,
)
fn discard_race_if_fresh( &self, waiting: &[Option<OffchainPublicKey>; 5], surb: &HoprSurbId, minted_at: Instant, )
discard_race for the relayers on waiting that a reply on a SURB minted at
minted_at is news about (see Self::is_fresh_for), the filter note_home applies.
fn record(&self, relayer: OffchainPublicKey, race: Race)
Sourcepub fn set_neighbour_connected(
&self,
neighbour: OffchainPublicKey,
connected: bool,
)
pub fn set_neighbour_connected( &self, neighbour: OffchainPublicKey, connected: bool, )
Our own connection to a neighbour came or went.
Sourcepub fn take_neighbour_change(&self) -> bool
pub fn take_neighbour_change(&self) -> bool
Whether a neighbour came or went since the last call.
Sourcepub(crate) fn record_reply(&self)
pub(crate) fn record_reply(&self)
Records that a reply came home now, through any relayer, for any session.
fn record_reply_at(&self, at: Instant)
Sourcefn last_reply(&self) -> Option<Instant>
fn last_reply(&self) -> Option<Instant>
When a reply last came home, to millisecond resolution, or None if none has.
Sourcepub fn any_known_dead(&self) -> bool
pub fn any_known_dead(&self) -> bool
Whether any relayer is judged dead or on probation: when none is, there is nothing to write off.
Sourcefn is_known_dead(&self, relayer: &OffchainPublicKey) -> bool
fn is_known_dead(&self, relayer: &OffchainPublicKey) -> bool
Whether the relayer is judged dead or on probation after such a verdict: its SURBs are written off either way.
Sourcefn is_confirmed_dead(&self, relayer: &OffchainPublicKey) -> bool
fn is_confirmed_dead(&self, relayer: &OffchainPublicKey) -> bool
Known dead and old enough to act on: on probation, or dead with a verdict that a flush at
least VERDICT_CONFIRMATION after it found still standing (refresh marks that, before any
re-plan of the same flush, so a slow re-plan cannot stand in for the second look).
Sourcefn note_home(&self, leg: &[Option<OffchainPublicKey>; 5], minted_at: Instant)
fn note_home(&self, leg: &[Option<OffchainPublicKey>; 5], minted_at: Instant)
A reply on a SURB minted at minted_at came home over leg: every relayer on it that the reply
is news about (see Self::is_fresh_for) delivered, over the hop it has on this leg. This is what
“home since” means for a black hole (see RaceHistory::unanswered), whether or not the reply
settled a race.
Sourcefn mark_dead(&self, relayer: OffchainPublicKey)
fn mark_dead(&self, relayer: OffchainPublicKey)
Records a relayer entering dead now, which starts its death.
Sourcefn has_stood(&self, relayer: &OffchainPublicKey, hop: Hop) -> bool
fn has_stood(&self, relayer: &OffchainPublicKey, hop: Hop) -> bool
Whether a relayer that is a black hole over hop only has been one for
VERDICT_CONFIRMATION, with nothing home over it in between. It is suspected from the first
flush that finds it so, and judged at the first one that finds this.
Over one hop a few late replies are all the evidence there is: only what travels that hop answers them, not every reply through the relayer as for one judged as a whole. And a path is late when it is first used, at the start of a session or after its stock was wiped, by up to a second on a cluster. Measured there with a verdict at the first flush: 136 verdicts in 84 runs on relayers that had nothing wrong with them, each steering the planner off the relayer, 129 of them gone by the next flush and 6 lifted before it by a reply over the hop; of the 90 on the relayer that was cut off, stalled or killed, 5 were gone. So the first flush only suspects, and the planner is not told. The stock is written off no later for it: that waited for the second flush anyway. A relayer suspected over the other hop is suspected anew over this one, and has a flush of its own to stand.
Sourcefn known_dead_since(&self, relayer: &OffchainPublicKey) -> Option<Instant>
fn known_dead_since(&self, relayer: &OffchainPublicKey) -> Option<Instant>
When the relayer’s current death began, while it is known dead.
Sourcefn is_fresh_for(&self, relayer: &OffchainPublicKey, minted_at: Instant) -> bool
fn is_fresh_for(&self, relayer: &OffchainPublicKey, minted_at: Instant) -> bool
Whether a reply on a SURB minted at minted_at is news about relayer. It is, unless the
relayer’s verdict was acted on (confirmed, or the relayer is on probation) and the SURB is older
than the verdict. A relayer that is slow rather than dead delivers such replies late, after the
verdict that judged them overdue: they are what the verdict was about, not proof that it ended,
and taking them as proof put a stalling relayer back into routing after every verdict. Before
the verdict is confirmed they still count, so that a relayer that is merely slow is revived by
its own late replies without costing a wipe, and a probe minted after the verdict always counts.
Sourcefn revive_by_reply(
&self,
leg: &[Option<OffchainPublicKey>; 5],
minted_at: Instant,
)
fn revive_by_reply( &self, leg: &[Option<OffchainPublicKey>; 5], minted_at: Instant, )
A reply on a SURB minted at minted_at came home over leg, so every relayer on it is
delivering: any that was judged dead or on probation is alive again, unless the reply is not news
about it (see Self::is_fresh_for), or it was judged while delivering over its other hop and
this is that one (dead_over). Its health is then what its races say, which still holds the ones
it lost, so it recovers gradually as they are forgotten.
Sourcepub fn refresh(&self, memory: u32) -> Vec<(OffchainPublicKey, f64, f64)>
pub fn refresh(&self, memory: u32) -> Vec<(OffchainPublicKey, f64, f64)>
Re-judges every relayer once per flush; returns (relayer, previous, current) health changes for
the graph, dead being 0.0 and probation PROBATION_HEALTH. A race open longer than a SURB is
remembered counts as lost by that long. memory: margins a race is remembered for, 0 for dead,
on probation or healthy.
A relayer is alive until it is a black hole, which makes it dead. Its verdict does not end with the
condition: once no overdue reply is out any more it goes on probation instead, and stays there
until a reply on a SURB minted after the verdict comes home through it (revive_by_reply,
is_fresh_for), a probe’s above all. One on probation that is a black hole again is dead again.
Trait Implementations§
Source§impl Debug for RelayerHealth
impl Debug for RelayerHealth
Source§impl Default for RelayerHealth
impl Default for RelayerHealth
Source§fn default() -> RelayerHealth
fn default() -> RelayerHealth
Auto Trait Implementations§
impl !Freeze for RelayerHealth
impl !RefUnwindSafe for RelayerHealth
impl Send for RelayerHealth
impl Sync for RelayerHealth
impl Unpin for RelayerHealth
impl UnsafeUnpin for RelayerHealth
impl UnwindSafe for RelayerHealth
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Conv for T
impl<T> Conv for T
§impl<T> FmtForward for T
impl<T> FmtForward for T
§fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
self to use its Binary implementation when Debug-formatted.§fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
self to use its Display implementation when
Debug-formatted.§fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
self to use its LowerExp implementation when
Debug-formatted.§fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
self to use its LowerHex implementation when
Debug-formatted.§fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
self to use its Octal implementation when Debug-formatted.§fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
self to use its Pointer implementation when
Debug-formatted.§fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
self to use its UpperExp implementation when
Debug-formatted.§fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
self to use its UpperHex implementation when
Debug-formatted.§fn fmt_list(self) -> FmtList<Self>where
&'a Self: for<'a> IntoIterator,
fn fmt_list(self) -> FmtList<Self>where
&'a Self: for<'a> IntoIterator,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> Pipe for Twhere
T: ?Sized,
impl<T> Pipe for Twhere
T: ?Sized,
§fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
§fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read more§fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read more§fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
§fn pipe_borrow_mut<'a, B, R>(
&'a mut self,
func: impl FnOnce(&'a mut B) -> R,
) -> R
fn pipe_borrow_mut<'a, B, R>( &'a mut self, func: impl FnOnce(&'a mut B) -> R, ) -> R
§fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
self, then passes self.as_ref() into the pipe function.§fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
self, then passes self.as_mut() into the pipe
function.§fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
self, then passes self.deref() into the pipe function.§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> Read<Exclusive, BecauseExclusive> for Twhere
T: ?Sized,
§impl<T> Tap for T
impl<T> Tap for T
§fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
Borrow<B> of a value. Read more§fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
BorrowMut<B> of a value. Read more§fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
AsRef<R> view of a value. Read more§fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
AsMut<R> view of a value. Read more§fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
Deref::Target of a value. Read more§fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
Deref::Target of a value. Read more§fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
.tap() only in debug builds, and is erased in release builds.§fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
.tap_mut() only in debug builds, and is erased in release
builds.§fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
.tap_borrow() only in debug builds, and is erased in release
builds.§fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
.tap_borrow_mut() only in debug builds, and is erased in release
builds.§fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
.tap_ref() only in debug builds, and is erased in release
builds.§fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
.tap_ref_mut() only in debug builds, and is erased in release
builds.§fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
.tap_deref() only in debug builds, and is erased in release
builds.