The 2020.09.16.0 emergency release addresses the chain halt we observed after activating PoC-v9. The team will continue to closely monitor the situation but we’re fairly confident that this update will get the blockchain moving again.

Contents

With every chain variable update there’s a little uncertainty that older code paths may get broken. The team tries their best to test any such changes thoroughly but sometimes bugs slip through the cracks.

What exactly happened?

  • We believe that the root cause of the chain halt was the fact that we were re-validating poc_receipts_txn by using a lagging ledger while trying to absorb the transaction. We have addressed that by getting the required channel information for witnesses by walking the chain backwards instead of relying on a potentially old ledger.

  • Furthermore, rewards_txn were being extremely expensive due to unnecessary logging, thereby causing IO issues. We have reduced the log levels from info to debug to address that.

Plan

Since this is an emergency release, we have already began betaing it on some hotspots and will be deploying it throughout the network around 5:30PM PST, Wednesday 16th Sept.