The core developers have prepared an emergency release 2021.05.29.0 to fix the Mainnet chain halt noticed around 1:00 PM PST, May 29th, 2021. This is a mandatory release for all miners on the network. Please read further for more details.

What Happened?

The team changed the election_info API in blockchain-core as part of support for Validators but missed updating the relevant API calls in miner. Surprisingly xref and dialyzer, two tools used in the Erlang ecosystem to do static analysis, didn’t catch the incorrect calls either. This regression led to a chain halt as the code change broke a critical HoneyBadgerBFT recovery code path.

Timeline

  • Sat May 29 01:04:00 PM PDT 2021: The core team got a pagerduty alert that there were no new blocks after block 864000
  • Sat May 29 01:26:00 PM PDT 2021: status updated to reflect the blockchain halt and debugging.
  • Sat May 29 01:45:00 PM PDT 2021: The bug was identified as an incorrect call in miner to a critical election related function in blockchain-core, more details available in miner#797.
  • Sat May 29 02:00:00 PM PDT 2021: 2021.05.29.0 release candidate prepared for beta testing and imminent GA.

The issue was identified addressed quickly and the Consensus Group will be able to continue block production after this release.

Future Work

Bugs like these can slip through normal reviews and testing but improved test suites with better code coverage can help catch these sorts of regressions. Future regular maintenance releases will include these improvments.

Plan

We will begin beta testing 2021.05.29.0 as soon as the release is built and GA immediately after checking beta status.