Blockchain Fixes and Chain Halt Postmortem
The 2020.10.27.0
blockchain release contains some critical fixes specifically meant to address the blockchain halt observed on Monday, October 26th, 2020. Please read on for a detailed breakdown.
October 26th, 2020: Chain halt postmortem
- 11AM PST: Miner tag
2020.10.23.0
is released to GA - 4:30PM PST: The team notices chain stall and begins investigation
- 5:00PM PST: Root cause identified as an unhandled
EXIT
error message caused while trying to create a block containing a stale poc receipt transaction. Patch deployed to the consensus group to regain block production. - 5:30PM PST: After having internal discussions, the team decided to roll back to previous GA tag
2020.10.12.1
. It was not worth it to rush a release, especially knowing that the bug was largely harmless and the previous GA had been running solidly. - 6:00PM PST: All hotspots on the network downgraded back to
2020.10.12.1
. By this time, network gossip caught up and block production resumed normally.
Highlights and Related PRs
- Better transaction error handling and testing. blockchain-core#665
- Enhance client side state channel validation. blockchain-core#667
- Fix state channel tests. blockchain-core#668
- Add EU868 region for Ukraine. miner#550
- Fix argument order for LoRaWAN rx2 window. miner#551
- Update blockchain-core and snapshot block to
560881
. miner#552
Plan
We have begun beta-testing 2020.10.27.0
around 5:15 PM PST, October 27th, 2020 and plan to GA on Thursday, October 29th, 2020 around 11AM PST.