Network incident post-mortem: Validator sampling bug
I. Summary
On June 26, 2025 at 2:40 AM UTC, Flare Mainnet experienced a brief outage, causing a temporary halt in block production. The issue was identified as a known and since-patched bug in the validator sampling mechanism inherited from an upstream dependency. A pre-planned, audited network upgrade containing the fix was deployed ahead of schedule, successfully restoring network functionality by 11.15 AM UTC on June 26, 2025.
II. Root cause
The outage was triggered by a bug in the go-flare v1.10 client software, originating from an upstream Avalanche dependency (avalanchego). The bug was located in the validator sampling algorithm, which is critical for block production and peer selection. Under specific network conditions, the cumulative weight of all validators surpassed the maximum value for a 64-bit signed integer (MaxInt64). This resulted in an errOutOfRange error, causing validator nodes to shut down. The loss of these validators interrupted the network's ability to produce new blocks.
III. Resolution
The underlying bug had already been rectified in avalanchego v1.11.0, which updated the sampling logic to support the larger uint64 data type. Flare had proactively audited and prepared its corresponding v1.11 release ahead of its original schedule. In response to the incident, this planned upgrade was deployed as an emergency patch. The new software was successfully rolled out, resolving the error and restoring network stability and operations.
IV. Next steps
The successful deployment of this upgrade has enhanced the network's resilience and stability, positioning it to handle future growth. We have reviewed our incident response process to ensure rapid deployment of critical patches if similar situations arise.
We extend our gratitude to our infrastructure providers and the wider community for their swift response and support, which were instrumental in the speedy resolution of this incident.
4,96k
0
Innholdet på denne siden er levert av tredjeparter. Med mindre annet er oppgitt, er ikke OKX forfatteren av de siterte artikkelen(e) og krever ingen opphavsrett til materialet. Innholdet er kun gitt for informasjonsformål og representerer ikke synspunktene til OKX. Det er ikke ment å være en anbefaling av noe slag og bør ikke betraktes som investeringsråd eller en oppfordring om å kjøpe eller selge digitale aktiva. I den grad generativ AI brukes til å gi sammendrag eller annen informasjon, kan slikt AI-generert innhold være unøyaktig eller inkonsekvent. Vennligst les den koblede artikkelen for mer detaljer og informasjon. OKX er ikke ansvarlig for innhold som er vert på tredjeparts nettsteder. Beholdning av digitale aktiva, inkludert stablecoins og NFT-er, innebærer en høy grad av risiko og kan svinge mye. Du bør nøye vurdere om handel eller innehav av digitale aktiva passer for deg i lys av din økonomiske tilstand.