The last beta deploy did not go well. Parsing of LongFi packets was broken in concentrate so the selected hotspots were not able to engage in Proof of Coverage with each other as planned.

Rolling back the hotspot firmware update resulted in a hard reboot of the beta hotspots. This hard reboot resulted in minor file system corruption which triggered reformatting of the /var partition mounted on /var because of an incorrect return code check from fsck.

WiFi credentials were lost so hotspots that were connected to the internet over WiFi fell offline. Older pre-production units also lost their swarm_keys which reside on the SD card rather than the hardware security module in production units.

Since the /var partition was lost on the affected beta hotspots, the blockchain cache was also lost resulting in any production hotspots in the beta group having to resync the blockchain over about 24 hours.

To remedy the inadvertent reformatting of hotspots persistent file system we have fast-tracked a PR to correct the behavior of the file system check and repair script that runs on start up.

To minimize risk all the recent LongFi related commits were reverted from the master branch of the hotspot firmware so that shut down behavior remains unchanged.

Content

The beta group will be updated with Hotspot firmware version 2019.10.25.0 which includes:

  • Revert concentrate upgrade for LongFi (not in 2019.10.21.0 GA release)
  • Do not treat all non-zero fsck return codes as unrecoverable errors
  • Revert miner upgrade for LongFi (not in 2019.10.21.0 GA release)
  • Revert CMake upgrade (not in 2019.10.21.0 GA release)

Deployment Plan

We plan to let this beta run for at least a couple of hours. If that proves to be stable we will tag the branch as 2019.10.25.0 and confirm that the GA release OTA updates successfully before making it available to all hotspots.