The core developers have prepared an emergency release
2021.05.29.0 to fix the Mainnet chain halt noticed around 1:00 PM PST, May 29th, 2021. This is a mandatory release for all miners on the network. Please read further for more details.
The team changed the election_info API in
blockchain-core as part of support for Validators but missed updating the relevant API calls in miner. Surprisingly
dialyzer, two tools used in the Erlang ecosystem to do static analysis, didn’t catch the incorrect calls either. This regression led to a chain halt as the code change broke a critical HoneyBadgerBFT recovery code path.
- Sat May 29 01:04:00 PM PDT 2021: The core team got a pagerduty alert that there were no new blocks after block
- Sat May 29 01:26:00 PM PDT 2021: status updated to reflect the blockchain halt and debugging.
- Sat May 29 01:45:00 PM PDT 2021: The bug was identified as an incorrect call in miner to a critical election related function in blockchain-core, more details available in miner#797.
- Sat May 29 02:00:00 PM PDT 2021:
2021.05.29.0release candidate prepared for beta testing and imminent GA.
The issue was identified addressed quickly and the Consensus Group will be able to continue block production after this release.
Bugs like these can slip through normal reviews and testing but improved test suites with better code coverage can help catch these sorts of regressions. Future regular maintenance releases will include these improvments.
We will begin beta testing
2021.05.29.0 as soon as the release is built and GA immediately after checking beta status.