Blockchain Beta Deploy
The last beta deploy did not go well. Parsing of LongFi packets was broken in concentrate so the selected hotspots were not able to engage in Proof of Coverage with each other as planned.
Rolling back the hotspot firmware update resulted in a hard reboot of
the beta hotspots. This hard reboot resulted in minor file system
corruption which triggered reformatting of the /var
partition
mounted on /var because of an incorrect return code check from fsck.
WiFi credentials were lost so hotspots that were connected to the
internet over WiFi fell offline. Older pre-production units also
lost their swarm_keys
which reside on the SD card rather than the
hardware security module in production units.
Since the /var
partition was lost on the affected beta hotspots, the
blockchain cache was also lost resulting in any production hotspots in
the beta group having to resync the blockchain over about 24 hours.
To remedy the inadvertent reformatting of hotspots persistent file system we have fast-tracked a PR to correct the behavior of the file system check and repair script that runs on start up.
To minimize risk all the recent LongFi related commits were reverted from the master branch of the hotspot firmware so that shut down behavior remains unchanged.
Content
The beta group will be updated with Hotspot firmware version
2019.10.25.0
which includes:
- Revert concentrate upgrade for LongFi (not in
2019.10.21.0
GA release) - Do not treat all non-zero fsck return codes as unrecoverable errors
- Revert miner upgrade for LongFi (not in
2019.10.21.0
GA release) - Revert CMake upgrade (not in
2019.10.21.0
GA release)
Deployment Plan
We plan to let this beta run for at least a couple of hours. If that proves
to be stable we will tag the branch as 2019.10.25.0
and confirm that the GA
release OTA updates successfully before making it available to all hotspots.