After monitoring the chain overnight and seeing few signs of recovery, further investigation revealed that we were still seeing out of memory errors on v1 Hotspots. A closer reading of the docs revealed that a part of the Erlang runtime reports memory usage in machine words (the width of the processor's registers), which means that the earlier measurements we based our tunings on were too low by a factor of 8. This release does nothing but retune the cache to use much less memory, which we hope will stabilize the chain, if not fully restore block times.
Current block time troubles are largely related to the slowness of the disk on the Hotspots combined with certain problematic behaviors, mostly excessive clustering, engaged in by some users. We have work in train to ameliorate the worst effects of these behaviors, but this will require a new chain var, and so can't be rushed. Our in-progress Proof of Coverage redesign will include disincentives for this sort of behavior as well.
Contents
- Out of Memory fixes: PRs: miner/433
Plan
A 2020.06.18.0 release was deployed to the beta group on the afternoon of Thursday June 18th at
around 1:00 PDT. After testing we expect to release this to GA around 4:30PM PDT.