Skip to main content

· 5 min read

An incident involving oracle rewards implementation on Solana was identified on December 12, 2023, revealing a discrepancy in MOBILE token rewards due to configuration errors in the delegator rewards percentage. The issue led to an over-emission to delegators and an underfunded rewards escrow, amounting to a total discrepancy of approximately 274 million MOBILE tokens. To address this, the protocol will temporarily increase emissions, effectively replenishing the escrow and slightly increasing the circulating supply of MOBILE by 274 million, a minor deviation that does not affect the max supply of 230B.

· 5 min read

At the end of September, the Helium Foundation changed our map tile server to Protomaps. As a globally distributed network of physical infrastructure, maps are a critical piece of the interpretation and visualization of the network. Helium runs many services leveraging maps, including Explorer, which serves hundreds of thousands of users monthly. Through this refactor, we are on track to save several thousands of dollars a month.

· One min read

Around 18:30 PM PST 10/03/23, Dedicated Console started experiencing severe data transfer disruption. The core team received the automated alerts immediately and started investigating the issue, which was resolved around 19:30 PM PST 10/03/23.

· 6 min read

In early August Nova Labs posted a change in how the denylist is operated. This change has resulted in several positive effects on the network in terms of the number of hotspots that were improved and cleared from the denylist, as well as clusters of misbehaving hotspots getting turned off, clearing innocent hotspots that were swept up in the analysis.

The ongoing challenge with the current approach is that innocent hotspots are still being swept up by the classifiers if they have especially good antenna setups since they are attractive targets for beacon replay.

Nova Labs will be deploying a refinement to the denylist that instead of denying hotspots for exhibiting improbable RF behavior denying witness events, also known as edges or connections, that are improbable. This would allow hotspots to continue to operate and earn PoC rewards for RF behavior that benefits the network overall even if they have improbable RF links.

We expect this new approach to be deployed before the end of September.

· 2 min read

Gateway-rs, the software backing all IoT Hotspots on the Helium Network, has been upgraded to include several key performance improvements related to data delivery and Proof of Coverage response times.

· 2 min read

The legacy Explorer (hosted at explorer-old.helium.com) and the legacy API are set for retirement on July 28. The Solana-backed Helium Explorer hosted at explorer.helium.com will continue to serve the community for coverage maps and high-level network stats.

· 3 min read

Summary

Previously, the Helium Foundation Console faced an exploit that was amplifying existing network usage. Following the discovery of this bug in the legacy Router/Console code backing this Network Server, several patches were deployed to prevent the exploit. While those patches were effective in fixing those specific bugs, two more bugs went overlooked and have now been patched.

· 7 min read

Summary

The Helium Foundation Console faced an exploit that was amplifying existing network usage and allowing bad actors to earn more than they were spending in DC. In essence, they were paying for 1 DC while the Console was burning up to 5 DC due to an accounting bug. This bug also affected all instances of Console that weren't managing their routes. This bug only affected these public instances and did not result in unpaid traffic on the Helium Network as a whole.

· One min read

Helium Wallet 1.6.0

This latest update adds more behind the scenes support for the Solana migration, including transactions, adding, updating, and viewing Hotspots on Solana L1, and swap support.

In addition, 1.6.0 fixes a bug where users could not sign into Crowdspot and fixes the display of HNT price on iOS widgets.

· One min read

The Nova developers have tagged a new 2022.12.13.0 Hotspot release for Original Helium Hotspots. This release prepares Hotspots for off-chain proof-of-coverage by updating the automatic beacon rate to every 6 hours, adds "jitter" to the beacon rate such that in the event all Hotspots come online at the same time, the PoC system is not overwhelmed with PoC.

· 2 min read

Helium Wallet 1.5.0

Prepare for the Helium migration to the Solana blockchain! This app update makes it easier for "blue app" users to successfully migrate to the Wallet app to manage their tokens. For those already on the Wallet app, migration will be easy. Just make sure your app is updated to the latest versions and your tokens will automatically be updated to use Solana.

· 5 min read

 

On Wednesday, 2022-11-16, at approximately 22:06 UTC, Validators stopped producing blocks at block 1,619,546, and a chain halt occurred. Normal chain operations resumed 18 hours on 2022-11-17 at approximately 15:57 UTC.

· 6 min read

 

On Wednesday, 2022-10-26, at approximately 1:57 AM UTC, Validators stopped producing blocks at block 1,587,313, and a chain halt occurred. Consequently, two days later, at 3:57 PM UTC, a second chain halt occurred at block 1,589,496. Normal chain operations resumed approximately 32 hours later at 2022-10-30 1:32 AM UTC.

· One min read

 

Burn HNT for DC from Console

The Wallet app now supports burn transactions from Helium Console. To burn Data Credits for packet transfer, simply tap the "Send" button and the QR code in the top left corner and scan the QR code.

· One min read

The core developers have tagged a new 2022.10.28.0 miner release. This release adds exponential back-off for Hotspots when requesting ledger information from Validators. This also ensures timely back-off when block times are too high and avoids too many Hotspots attempting to connect to a Validator. The pull request specifically addresses the blockchain outages over the past week from a Hotspot's perspective.

· 4 min read

 

On Wednesday 2022-10-19 at approximately 2:20 AM UTC Validators stopped producing blocks at block 1,575,935 and a chain halt occurred. Consequently, at 5:55 AM UTC a second chain halt occurred at block 1,575,955. Normal chain operations resumed approximately 5.5 hours later at 7:49 AM.

· One min read

 

Brand New UI

The UI has been cleaned to show a chart based on $HNT oracle price, made voting more intuitive, and improved payments on subtokens.

· One min read

The core developers have tagged a new 2022.08.17.1 miner release. This release disables peer-to-peer activity when the Hotspot is no longer syncing the chain. This improves overall performance for the peer-to-peer network for other participants and stability of the network as a whole. Other improvements include an update to HTTP2 process that the Hotspot uses to communicate to Validators (over GRPC), and a Kerlink-specific improvement that fixes the RSSI computation if the SNR is lower than zero to reduce the number of invalid Proof-of-Coverage challenges and receipts.

· 4 min read

 

On Monday 2022-08-08 at approximately 19:34 PM UTC Validators stopped producing blocks at block 1,469,951 and a chain halt occurred. Normal chain operations resumed at approximately 08:00 AM UTC just over 7 hours later.

· 2 min read

The team has released the latest version of Console 2.2.20 to production. This release includes a rollup of a number of updates:

· One min read

The team is prioritizing its focus on issues to improve network reliability. Any new features or enhancement requests, while appreciated, will be added to our backlog and prioritized accordingly.

· One min read

The core developers have tagged a new 2022.08.02.0 miner release. This release contains improvements to packet delivery such that when a Hotspot sees a packet, it sends the payload right away to the Router (and thus, recipient) immediately. Data transfer reliability is expected to improve with this release.

· 2 min read

The team has released the latest version of Console 2.2.17 to production. This release includes a rollup of out of band releases:

· One min read

The Nova developers have tagged a new 2022.07.15.0 firmware release for Original Helium Hotspots. This minor release enables mux and disables blockchain sync. Enabling mux allows the transferring of packets to use the Light Hotspot part of the codebase and is a further step forward in the Light Hotspot transition.

· 2 min read

The core developers have tagged a new 2022.07.14.0 miner release. This release contains fixes to packet delivery to make it more reliable when Hotspots fall offline for a brief period of time (it will queue up to 10 device packets it saw) and fixes to let Hotspots send unknown packets if it's not synced with the blockchain.

· 2 min read

 

On Monday 2022-07-11 at approximately 12:42 PM UTC the Consensus Group stopped producing blocks at block 1,435,692. Normal chain operations continued at 17:15 PM UTC just over 4 hours later.

· One min read

 

Ledger Payment Support

Users that have imported their Ledger Nano S (Android Only) or Ledger Nano X can now send payments from the app! Requires Ledger Helium App version 2.2.4. Update your Nano app using Ledger Live.

· One min read

The core developers have tagged a new 2022.06.01.0 miner release. This release is mandatory for all Hotspot Makers. Contained in this release are further improvements to the flatline issue (fixes HTTP2) and enables mux. Enabling mux allows all data packet traffic to be routed through gateway-rs and away from the erlang miner. This is a key step towards removing Erlang from Hotspots altogether.

· One min read

The core developers have tagged a new 2022.05.24.0 miner release. This release is mandatory for all Hotspot Makers. Contained in this release are chain variables to support Validator Performance (to be activated some time after May 25 9:00AM PT). Failure to update your fleet before the chain variables are activated will cause the Hotspot to stop functioning as Hotspots still follow the blockchain.

· One min read

 

This update adds Light Hotspot support to the app, specifically for Transferring Hotspots and removes pre-Light Hotspot terminology from Hotspot details and Diagnostic Reports.

· One min read

 

This update is to fix a few bugs found since the last release. The next release will address changes for Light Hotspots.

· 2 min read

Sprint: 75

The core team has released the latest version of Console 2.2.8 to production. Changes include merging the code for XOR filter fee reduction, however it has not been activated as the vote is still ongoing. For more information check the HIP.

· 5 min read

With HIP 55 the community approved increasing the scope of work and rewards for Validators. This document includes preliminary estimates to help provide further insight into the changes Light Hotspots have on Validator traffic.

· 2 min read

Sprint: 74

The team has released the latest version of Console 2.2.7 to production. Changes include MQTT connection overflow fixes, as well as usability and open source updates, for example payment processor Stripe is available to open source users, and new environment variables have been added to the .env template file.

· One min read

The core developers have tagged a new 2022.04.19.0 miner release. This release is mandatory for all Hotspot Makers. We request that Hotspot Manufacturers upgrade their fleets by 00:00 UTC on 2022-05-03. This update contains Light Hotspot software, and when activated via chain variable on May 3 2022, will change how Hotspots receive challenges, remove block sync from Hotspots, and stabilize Hotspot performance. This will allow the approved HIP 55 chain variable to be activated and prepare for the activation of Light Hotspots.

· One min read

 

Require Seed Words to Sign Out

Requires users to enter their 12 word seed phrase in order to sign out to prevent accidental sign outs. Thanks to our community Mentor for writing this request!

· 2 min read

 

OS Widgets (Perronef5 / Peroni_F5#3413)

iPhone users - our community member Perroni_F5 has added an incredible feature to the Hotspot app, adding 2 widgets to your app experience:

· 3 min read

The core developers are changing the PoC Interval from 475 to 950. Hotspots will PoC less often however total average rewards will remain unchanged. PoC rate has no effect on average individual Hotspot earnings, but may increase variability of rewards. This will help the chain process transactions more efficiently. Please read the details below for our reasoning.

· 2 min read

Sprint: 72

Console v2.2.5 has been released to production. During this sprint the team also launched Helium Roaming Services. LoRaWAN network providers can learn more about how to leverage Helium coverage here.

· 2 min read

Sprint: 71

The team has released v2.2.4 release to production. A number of usability updates were completed including minimizing user input for Azure IoT Hub, making it simpler to add a label to new devices, and adding abiliity to use mobile UI to scan QR codes for devices that have implemented the LoRaWAN Device Identification QR Codes specification.

· One min read

 

Identify Hotspots on Denylist

When browsing the network map on the app, Hotspots on the denylist will appear with a red badge next to their name. Denylist Hotspots are those identified to be gaming on the Helium Network.

· 2 min read

Sprint: 70

The latest v2.2.3 release has been releaseed to production by the team. Based on a number of conversations with users, a large number of Console API’s were added to this release and existing ones updated.

· 2 min read

The core developers have tagged a new 2022.02.22.0 miner and blockchain-core release. This release is mandatory for all Hotspot Makers (please update your fleet by March 3, 2021 00:00 UTC) and will be mandatory in the near future for Validators, ETLs, Routers and Nodes. It contains a network improvement that is gated behind some currently inactive chain variables. Hotspots without this update will not be able to sync or mine once the chain variable is activated later.

· 6 min read

 

On Monday 2022-02-07 5:46 AM UTC the team announced a chain halt caused by enlarged block 121528. Normal chain operations were announced at 3:04 PM UTC, less than 10 hours later.

· 2 min read

Sprint: 69

The team has pushed the latest v2.2.2 release to production. Highlights include updates to Receive Window Delay feature, added message queue (for Production Console only) to scale frontend database, and ux tweaks. In addition, for open source users the front end team added prebuilt Console images hosted on Quay.io so open source users do not have to build their own Console images on docker.

· One min read

The Helium team has tagged a new 2022.02.10.0 firmware release for Original Helium and RAK/MNTD Hotspots. All Original Helium/RAK MNTD Hotspots have a designated hardware_type to identify its hardware version at the manufacturing stage. By targeting Hotspots by its hardware_type, this release will enable RAK/MNTD Support teams to investigate and diagnose Hotspot-related issues in a secure manner without interaction from the Helium Support team.

· 2 min read

 

Transfer Hotspot v2

A new way to transfer Hotspots is now available on the app. Now transferring Hotspots is as easy as sending a payment transaction. Hotspot sellers can initiate the transfer from Hotspot Settings, the same as before, but instead of waiting for the buyer to accept the transaction, the transfer is immediately submitted to the blockchain. Once cleared, the buyer will see the new Hotspot in their app. When the transfer is successful, both buyers and sellers will receive a notification.

· One min read

The Helium team has tagged a new 2022.02.08.0 firmware release for Original Helium and RAK/MNTD Hotspots. This minor release changes the interval between automated update checks for a Hotspot's miner container. It has been updated to every 2 hours, from the current 5 minute interval. This update also changes how a Hotspot prunes old miner images, removing the possibility of re-downloading the latest image multiple times. These changes should reduce the amount of bandwidth an Original or RAK/MNTD Hotspot consumes.

· One min read

The core developers have tagged a new 2022.01.29.0 miner release. This release is recommended for all makers, but is only mandatory for Hotspots that are no longer able to load the latest snapshot. As the network gets larger and the snapshot size increases, Hotspots with less RAM will have trouble loading them. We will continue to support these Hotspots until all manufacturers have an upgrade path to Light Hotspot firmware.

· 2 min read

Sprint: 68

This release includes mostly backend under the hood fixes. The team plans to do an out of band release next week to address the latest issue related to receive window delay, specifically this issue.

· 2 min read

 

Updated Witnessed List

Witnessed Lists in both the App and Explorer reflects the list of Hotspots that your Hotspot has seen beacon. This is an improved change from displaying a list of Hotspots that has witnessed your Hotspot as the new list directly correlates with mining rewards.

· 2 min read

Sprint: 67

Based on almost 5000 responses to our Console UI survey, 70% respondents reported accessing Console at least multiple times a week, and almost 50% daily.

· 2 min read

The core developers have tagged a new 2022.01.12.1 Hotspot firmware release which includes recommended improvements to snapshot loading and reduces their memory footprint. Furthermore, this release also contains bug fixes for state channel handling specifically for the client side which should now clean the internally stored channels appropriately when they observe a state_channel_close transaction.

· One min read

The core developers have tagged a new 2022.01.04.0 Hotspot firmware release which includes recommended improvements in miner specifically related to ensure that Hotspots with unasserted location are still able to sync to the head of the blockchain. Secondly this release also includes ability to use additional packet metadata provided by certain packet forwarders.

· One min read

 

See how your Hotspot stacks up to the Network! We’ve added the average expected network earnings per Hotspot to see how yours perform against other Hotspots on the network. The average earnings is based on how many Hotspots are online that day, for creating challenges, beaconing, witnessing, and transferring data.

· 2 min read

Sprint: 66

The latest version of Console (sprint 66) has been deployed to production. Due to holidays we had an extended sprint 65, which shortened sprint 66. T The team also began sprint 67 which will be the last sprint of 2021 with a release planned for early January 2022.

· One min read

Sprint: 65

The latest version of Console (sprint 65) was released to production, including user enhancements, multi-factor authentication, and a host of backend updates.

· One min read

The core developers have tagged a new 2021.12.09.0 Hotspot firmware release which includes a recommended bug fix in erlang-libp2p. This change potentially mitigates the zero-witness beacon issue that has been afflicting, mostly, relayed Challenger Hotspots failing to receive witness receipts. Special thanks to @groot#3396 and others in the #poc-discussion channel on Discord for their efforts in tracking this issue down.

· 2 min read

 

Updating Hotspot Antenna and Gain

Several users have noted that submitting the antenna and elevation was not working as expected.

· 2 min read

 

This release includes improvements to API loading, adds rewards caching, and updates how earnings are calculated.

· One min read

The core developers have tagged a new 2021.11.11.0 Hotspot firmware release which includes critical bug fixes for gossip and ARP in libp2p. We hope that this release will help Hotspots with improvements on block gossip and keeping an up to date peerbook.

· 3 min read

The core developers have tagged a new 2021.11.10.0 Hotspot firmware release which includes critical bug fixes, supplementary code for finalizing activation of POC V11 related chain variables, packet transmission and blockchain-etl fixes.

· One min read

 

Improvements

  • Optimizes calls to the API and cacheing
  • Adds loading indicators when data calls to longer than usual
  • Updates React Native - Android screenshots are no longer enabled due to a security flag that the app is required to set.
  • Changes how the app retrieves chain variables from the blockchain.

· One min read

The core developers have tagged a new 2021.11.04.2 Hotspot firmware release which includes a few minor usability improvements, specifically allowing Makers to access a Hotspot's region parameters via JSON-RPC and miner's CLI.

· One min read

The team added another prebuilt integration with a partner akenza IoT platform, as well as updates to Configuration Profiles, and the Coverage feature.

· One min read

The core developers have tagged a new 2021.11.02.0 Hotspot firmware release which includes two critical blockchain-core fixes, firstly, blockchain core can now handle v1 and v2 block gossip seamlessly; secondly, the developers noticed that the transaction_validity_version chain variable definition was set incorrectly in the header file, this has been fixed. An unfortunate effect of this is that transfer_hotspot_v2 support has been slightly delayed.

· One min read

 

Improvements

  • updates account screen when API calls fail to avoid showing 0 balance and 0 Hotspots
  • improves how the app retrieves data from the API
  • fixes a bug where users couldn’t send payments or transfer Hotspot based on a security setting
  • added an option to clear map cache to troubleshoot when hexagons aren’t loading or app is malfunctioning due to tileserver

· 2 min read

The core developers have tagged a new 2021.10.27.0 Hotspot firmware release which includes new functionality for Hotspots to automatically ingest latest on-chain snapshots. Makers, please see the note below as there are some updates you'll need to incorporate to use this feature. This release also reverts the var cache which was causing unforeseen issues and upgrades blockchain-core to include many State Channel related optimizations.

· 2 min read

App Version 3.7.0

 

StakeJoy as API Provider

We've added StakeJoy API support to the Helium App. StakeJoy provides data to explorer.helium.com and now the Helium app to provide users with an improved experience. To use StakeJoy, go to Settings, tap Choose API Provider and select StakeJoy.

· 2 min read

Included in this sprint, the team delivered Configuration Profiles provides users with a streamlined way to configure, save, and apply device settings (ADR, CF List) and a new authentication system.

· One min read

The core developers have tagged a new 2021.10.18.0 Hotspot firmware release which includes some bug fixes and a new blessed snapshot at height 1059841. This release addresses a channel selection bug causing unnecessary incorrect witnesses and fixes a potential race condition when initializing var-cache. Furthermore, the updated blockchain-core has improved state channel support for router.

· 2 min read

State channel updates to improve network performance was released to resolve a choke point caused by the exponential growth of Hotspots providing coverage. In addition, Flows has become popular to quickly send device data over The People’s Network and this release includes a number of usability updates.

· 2 min read

App Version 3.6.0

 

Deployment Mode

Ever need to hand your phone to someone for installation but don’t want them seeing your seed phrase, use the wallet payment functionality, or transfer Hotspots?

· One min read

The core developers have tagged a new 2021.09.27.0 Hotspot firmware release which includes some performance fixes and a bug fix to block sync. It also includes a new core developer blessed snapshot at height 1029491. We've identified the cause of lack of agreements on on-chain snapshots and have issued a fix for it as well but this will require a validator release.

· 2 min read

This release includes an integration for Microsoft Azure IoT Hub. Microsoft had requested an integration with Console based on feedback from their customers. In addition, the team made usability updates based on community feedback, and enhanced device APIs.

· 2 min read

A major feature the team’s worked on for the last two sprints is Coverage. Unlike Hotspot monitoring tools focused on maximizing mining rewards, Coverage is built for users of the network who want to have more visibility about which Hotspots are transferring device packets as well as displaying the status of those Hotspots all from within Console.

· 2 min read

App Version 3.5.0

 

Hide Hotspot

Have a Hotspot that has met their untimely demise and the daily reminders just reminds you of the good old days and it just hurts too much? Or maybe you want to hide it for other reasons - we don't judge. Now you can hide it from app view (but it’ll stay associated with your account). To unhide Hotspots, go to Settings to view hidden Hotspots.

· One min read

The core developers have tagged a new 2021.09.03.0 Hotspot firmware release which contains a snapshot update. Please note that this tag is based off of 2021.08.31.1 and not on miner master as the team wants to continue testing PoCv11 changes that landed on master yesterday.

· 2 min read

While the team focused on the Gateway Monitoring feature, this sprint they also knocked off a number of frontend and backend issues reported by the community. In addition, an out-of-band update was released last week to address a number of issues.

· One min read

The core developers have tagged a new 2021.08.21.0 Hotspot firmware release which contains major blockchain-core updates specifically ensuring that snapshots are correctly getting loaded. Furthermore this release supplies a new snapshot to allow new and un-synced hotspots to continue syncing. This is mandatory release for all hotspot manufacturers.

· One min read

The core developers have tagged a new 2021.08.16.0 Hotspot firmware release which is a maintenance update to primarily update static seed nodes in miner. This is an optional but recommended release for Hotspot owners and manufacturers to improve network performance.

· One min read

New Features

Support for Data-Only Hotspots

  • Now you can search and follow data-only Hotspots on the app, and see information specific to data-only Hotspots.

· One min read

The core developers have tagged a new 2021.08.05.0 Hotspot firmware release which updates the currently blessed snapshot and includes some fixes for state channel handling for packet routing. The release also includes support for fetching more seed nodes from DNS and some fixes for other applications using blockchain-core (router and ETL).

· 2 min read

 

  • Update LongAP Descriptions
  • Add Maker type to Hotspot Details*
  • Add Linxdot, ClodPi, and Pisces to Hotspot Onboarding

· One min read

The core developers have tagged a new 2021.08.02.0 Hotspot firmware release which updates the currently blessed snapshot and handles IPv6 addresses in bogon check. This release also removes the default Helium production router from the static configuration to help meet the demands on utilization on the network

· One min read

The core developers have tagged a new 2021.07.30.0 Hotspot firmware release which updates the currently blessed snapshot, fixes a minor bug in snapshot downloads from the Amazon S3 source, and a community contribution from @PaulVMo that improves libp2p identification of private IP addresses. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

· One min read

The core developers have tagged a new 2021.07.29.0 Hotspot firmware release which updates the currently blessed snapshot, adds a new seed node to the statically configured list, adjusts address hashing for Hotspot identification, and contains minor bug fixes. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

· 4 min read

After many months of work and testing based on community feedback, the team is excited to announce the official release of Console 2.0! The team not only worked on new features and capabilities, but also focused on making sure migration to Console 2.0 would be seamless for existing users.

· One min read

The core developers have tagged a new 2021.07.22.0 Hotspot firmware release which updates the currently blessed snapshot and contains networking and snapshot improvements. This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

· One min read

Bug Fixes

  • Improves Shortcut loading performance
  • Fixes Discovery Mode modal not extending under status bar
  • Fixes the “Next” button logic for Web/QR Code onboarding

· One min read

The core developers have tagged a new 2021.07.19.0 Hotspot firmware release which updates the currently blessed snapshot, introduces state channel related improvements and switches all Hotspots to non-validation (follow mode). This is an optional but recommended release for Hotspot owners and manufacturers to continue syncing with newly blessed snapshots.

· One min read

 

Improvements

  • Users with 15+ Hotspots in their wallet account were experiencing performance degradation. This is now fixed
  • Add Web Onboarding type to support new Hotspot Makers

· One min read

The core developers have tagged a new 2021.07.11.0 Hotspot firmware release which contains performance improvements to snapshot ingestion. This is an optional but highly recommended release for Hotspot owners and manufacturers especially for those Hotspots which are based on older Raspberry Pi3 hardware.

· One min read

The core developers have tagged a new 2021.07.09.2 Hotspot firmware release which contains a new blessed snapshot and gossip improvements. This is an optional but recommended release for Hotspot owners.

· One min read

The core developers have tagged a new 2021.07.02.0 Hotspot firmware release which contains a new blessed snapshot and minor maintenance updates. This is an optional but recommended release for Hotspot owners.

· 2 min read

This week we’re excited to release Console 2.0 beta on our staging server. Console 2.0 beta represents months of work by the team and we’re excited to share with our community and get their feedback.

· One min read

The core developers have tagged a new 2021.06.30.0 Hotspot firmware release which contains updates to improve peerbook performance and a new blessed snapshot. This is an optional but recommended release for Hotspot owners.

· One min read

The core developers have tagged a new 2021.06.29.0 Hotspot firmware release which primarily contains maintenance updates to potentially reduce block times and enhance block sync. This is an optional but recommended release for Hotspot owners.

· 4 min read

The core developers prepared an emergency release 2021.06.25.3 to fix the Mainnet blockchain halt noticed around 4:00 PM PST, June 25th, 2021. This was a mandatory release for all blockchain participants on the network. Hotspot manufacturers who have not picked up this release should update their fleets. Please read further for more details.

· 2 min read

The core developers have tagged a new 2021.06.25.0 Hotspot firmware release which contains updates to support better transaction validation for Validators and improvements to State Channel scalability. This is a required release for Hotspot owners as it introduces new chain variables which will get activated at a later date.

· One min read

The core developers have tagged a new 2021.06.21.0 Hotspot firmware release which only has a new blessed snapshot update. This is an optional but recommended release for Hotspot owners; the updated blessed snapshot will help new and lagging Hotspots to sync to blockchain head faster.

· One min read

 

This release fixes 2 bugs:

  • Fixes a bug where RAK and Syncrob.it were not on the Hotspot onboarding list.
  • Fixes a bug when the app goes in background, the privacy splash screen disappears.

· 2 min read

The core developers have tagged a new 2021.06.18.0 Hotspot firmware release which contains network improvements related to NAT and bug fixes related to staking for Validators. This is a required release for Hotspot owners. As some of the community members have noticed that the blockchain has not generated a new snapshot since the last GA, we believe that the improvements to networking and an increment to snapshot memory generation limit will fix snapshot generation. The core developers plan to do another minor release with a snapshot update once it gets generated by the blockchain.

· One min read

The team focused on delivering an pre-built integration for Microshare in addition to continuing to build out our roaming capabilities for non-Helium traffic.

· 2 min read

The core developers have tagged a new 2021.06.14.0 Hotspot firmware release which contains transaction processing improvements, preparatory code for a new chain variable, and a newly generated snapshot. This is a required release for Hotspot owners. Further announcements will be made for chain variable activation schedule but we expect about a week or so of time to allow downstream clients of the chain (etl, node, and router users) to update.

· One min read

The core developers have tagged a new 2021.06.11.2 Hotspot firmware release which has minor test related fixes and a newly generated snapshot. This is an optional release for Hotspot owners but recommended to get newly onboarded and/or un-synced hotspots to sync to the head of the blockchain.

· One min read

The core developers have tagged a new 2021.06.09.0 Hotspot firmware release which contains a newly generated snapshot. This would allow newly onboarded and previously un-synced hotspots to sync to the head of the blockchain much faster.

· 4 min read

The core developers have tagged a new 2021.06.08.0 Hotspot firmware release which fixes Snapshot creation and ingestion. It also adds support for storing transaction with implicit burns. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we release to General Availability.

· One min read

 

Bug Fixes

  • Fixes a bug where the discovery mode timer does not continue when the app is put in background
  • Prevents users from submitting multiple assert location transactions before the first one clears
  • Fixes onboarding errors
  • Adds session length to discovery mode

· One min read

As some of the community has noticed, block times and elections have been struggling the last week or so as the network continues to grow. The core developers have prepared the 2020.05.29.1 release which will adjust validation times for two known slower transactions in our blockchain. This is an optional but recommended release for miners on the network.

· One min read

The core developers have tagged a new 2021.05.24.1 Hotspot firmware release which changes S3 snapshot download timeout to a more relaxed limit and updates the blessed snapshot to 858241. This is an optional, but recommended upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

· One min read

The core developers have tagged a new 2021.05.22.0 Hotspot firmware release which adds S3 download functionality for snapshots and updates the blessed snapshot to 855361. This is an optional, but recommended upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

· One min read

The core developers have tagged a new 2021.05.20.0 Hotspot firmware release which primarily fixes libp2p peerbook issues affecting a small number of Hotspots and also updates the blessed snapshot. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

· 2 min read

The team has a new release tagged 2021.05.18.0 which primarily fixes out-of-memory and sync issues observed on older, less capable hardware, and other bug fixes. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA.

· One min read

 

Use Discovery Mode to understand your Hotspot’s coverage without waiting for Proof-of-Coverage/Beacons, or setting a location!

· One min read

 

Added Relayed Hotspot pill

Hotspots that are relayed through other Hotspots on the network may experience syncing or reduced mining rewards. We now surface that up to the user in the app.

· 6 min read

Last night, during the rollout of the 2021.05.04.2 release including the code for HIP-29 and HIP-30, several members of the community observed a long election including multiple proposed Consensus Groups where only 15 of the 16 new members succeeded in their Distributed Key Generation (DKG). After several hours of monitoring the Consensus Group, the Decentralized Wireless Alliance (DeWi) in conjunction with several core developers ended the 450 block epoch with a Rescue Block, which later restored the network to a functioning state. The network has been operating correctly since. All funds remained safe through the entire process.

· One min read

 

Updated Hotspot UI

A modified Hotspot UI that allows for navigating to the next Hotspot in a list, without going back to the Hotspot tab.

· 3 min read

The team has a new release tagged 2021.04.26.1 which contains some new features and bug fixes. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA. Specifically, this release includes support for Payment Memos which requires a chain variable to activate and makes this a required release. We don't intend to activate this chain variable for about a week. Please read further for more details.

· 2 min read

 

Remote Assert

Remote Assert is finally here! Update your Hotspot's location without being physically next to the Hotspot, and without Bluetooth pairing! This improved user experience will make it much easier for Owners and Hosts to set up Hotspots.

· One min read

The team has a new minor release tagged 2021.04.08.0 which contains a minor bug fix that addresses an issue that prevented snapshots from being generated by the chain. We are deploying this change today so that we can do a subsequent bump of the snapshot tomorrow. A list of other minor changes are below.

· 2 min read

April 4-6 API Outage Retrospective

Starting on April 4, our daily active users (DAU) on Helium Explorer doubled from 15,000 to 30,000. As a result, the API services that Explorer relies on was unable to handle the additional load. The reason for this cannot be fully attributed to DAU, but rather a symptom of the root cause. This API is used by multiple services including the Explorer, Helium App, and Wallet CLI. Many third party services also depend on this API.

· One min read

The team has a new minor release tagged 2021.04.05.0 which contains a minor bug fix that caused the miner to restart every few hours, plus a snapshot bump.

· 2 min read

The team has a new release tagged 2021.03.24.0 which contains a significant overhaul of the Rewards transaction and maintenance improvements including a regular snapshot bump.

· One min read

 

New Features:

  1. Explore Hotspots around you if you don’t own hotspots
  2. Express HNT to your local currency with CoinGecko. Tap on your HNT balance to toggle between display options. Note that data uses current $HNT.
  3. Option to disable haptic feedback

· 2 min read

The team has a new release tagged 2021.03.08.0. This release includes a fix to Hotspot firmware where onboarding keys were incorrectly generated, a snapshot bump to 750241, and several reliability fixes that were identified in our Validators Testnet.

· 2 min read

The team has a new release tagged 2021.03.05.0. This release collects some small but important maintenance upgrades to the blockchain but primarily is focused on adding a new functionality to the Proof of Coverage system. We also included a few code changes that will support Light Hotspots in the future and snapshot bump.

· One min read

The team has a new release tagged 2021.03.02.2. This release collects several performance and maintenance upgrades to the blockchain that should improve stability and performance of members of the Consensus Group. We also included a few code changes that will support Light Hotspots in the future, a bugfix for oracle prices our testnet chain, and snapshot bump.

· 2 min read

 

New Features:

  1. See Hotspots around you
  • Tap on them to view their mining details and stats
  1. Show the time period when scrubbing charts on Hotspot Details

  2. Make addresses compatible on the Network selectable: either copy to clipboard or open in explorer

· 2 min read

 

After months of waiting, we’re so happy to announce the new version of the Helium Hotspot app is here! Version 3.0 introduces a whole new design and introduces a new way to interact with the app. Gone are the days of scrolling for hours toiling over challenges (we have Explorer for that!).

· 2 min read

This release the team added another pre-built integration with Ubidots and extended regional support for China. In addition, the team completed a number of usability enhancements, including making it easier to add Labels to devices, and identify which devices have sent data.

· 3 min read

Maker Update

The three newly approved Makers of the Helium network have made great progress in their audit and integration with the Hotspot app. We’d like to take this opportunity to inform the community on their progress and how long each Makers took to progress through the milestones so you, as a potential Hotspot owner, can understand when to expect your Hotspot once a Maker starts their audit process.

· 3 min read

We’re excited to launch a beta release of Flows, a new user experience designed to simplify performing Console core actions. Also, in this release notifications can be sent via webhooks, and APIs for notifications have been added.

· 2 min read

The team has a new release tagged 2021.02.01.1 which contains minor bug fixes, routing transaction updates and a snapshot bump. This is a required upgrade for all miners and will be automatic for all Hotspots using the Helium firmware when we GA. Please read further for more details.

· 4 min read

 

The Helium community places a lot of trust in HIP19 approved hardware vendors and grants them the ability to add full mining gateways to the Helium blockchain. These manufacturers, also known as “Makers”, are limited by chain variables and the Onboarding Server as described previously on the Helium blog. The community trusts that the Maker will produce and deliver a real physical Hotspot with approved security methods.

· 3 min read

In this release we’ve included beta support for LoRaWAN adaptive data rate (ADR), increased the number of email notifications, and added a new API for multiple packet purchase. ADR allows devices to use an optimal data rate which reduces power consumption and airtime on the network based on RF conditions. It's recommended to only use this setting for fixed or non-mobile devices to ensure reliable connectivity.

· 2 min read

During the last sprint of 2020, the team delivered email notifications, the ability to pay for additional packets, and a number of backend updates. Configuration for email notifications and the ability to pay for additional packets has been added to Labels. This allows for flexibility over which devices trigger the notifications or receive additional packets.

· 2 min read

The team has successfully deployed support for HIP 15 and HIP 17 and switched to a beacon-only PoC model. One of the things we decided not to change with all of the other chain variables this morning was an increase to the number of max witnesses for each PoC. Now that we are confident with the way the Consensus Group is able to handle the new PoC receipts and new rewards calculations, we feel like it's safe to increase the max number of witnesses in each PoC receipt.

· 5 min read

The 2020.12.15.2 release was a monumental release adding support for HIP17. Note that we have decided to scrap the 2020.12.09.0, 2020.12.11.0, and 2020.12.15.1 releases because of bugs we identified during ongoing testing. Details about those bug fixes can be found here and here.

· One min read

The Helium blockchain team will be submitting a chain variable transaction that will increase the target interval between Proof-of-Coverage challenges (PoCs), changing the poc_challenge_interval variable from 120 to 240 blocks. This update will effectively halve the rate of PoC transactions that Hotspots generate.

· 2 min read

 

  • Migrate legacy secure store data in preparation for new app version (v3). We recommend all users update to this new app version. Failure to update to this version could mean losing access to your Wallet/Account and associated Hotspots if a user directly upgrades from 2.6.1 to 3.x.x. as a restore from the deprecated secure store will no longer be supported. This is also a good reminder to write down the 12 words used to generate the wallet's private key. You can find those 12 words by launching the Helium App > More > Advanced > Reveal 12 Words.

· One min read

 

Send Max HNT

In this new update, we've made it easier for anyone to send the full amount of HNT in their wallet. You no longer have to worry about calculating the transaction fee.

· 3 min read

 

View Witness List

In this new update, we've brought in a feature called "View Witnesses". This new feature can be found on your Hotspot's page. Simply toggle the View Witness to see all the Hotspots that your Hotspot has seen and interacted with!

· 2 min read

This release includes enhancements to help manage large numbers of devices including ability to sort and customize the device table, and making group import more flexible. In addition, a number of backend fixes were made including improving join request reliability, making integrations more robust, and resolving an arbitrage issue.

· 3 min read

The primary objective of the 2020.10.08.0 release is to address the state channel overpayment exploit. The fix is guarded behind reward_version chain variable which would be incremented to 4 to activate the fix. We intend to activate this chain variable after verifying this release.

· 3 min read

The 2020.09.30.1 release is a regular blockchain maintenance update. Notably, the team has added support for providing witness validity information to Proof of Coverage receipts, fixed blockchain snapshot determinism and updated state channel client to make device join accepts more robust.

· 4 min read

On 9/16/2020 the team activated PoC v9, which didn't go as planned, causing a complete blockchain halt for ~5 hours and subsequent slow down of blocks for ~7 hours. Longer term effects included several hotspots going out of sync for several hours after the chain resumed. Please read on for the details on what happened, what we did in the short term to fix it, and the steps we'll be taking to address these kinds of issues longer term.

· One min read

Due to the recent blockchain outage with the activation of PoC v9 and extremely slow election block times, the team has agreed to revert back to PoC v8 while a more optimized mechanism to run PoC v9 is being worked upon. We will continue to post updates as we make further progress.

· 2 min read

The 2020.09.16.0 emergency release addresses the chain halt we observed after activating PoC-v9. The team will continue to closely monitor the situation but we're fairly confident that this update will get the blockchain moving again.

· One min read

The 2020.09.08.0 release is a minor bugfix release to fix an issue with the stabilization interval calculations on newer Hotspots that have been loaded from a snapshot. There was an error in the how the interval was calculated, which the PR below fixes:

· 4 min read

On 8/12 the system changed the token allocation providing rewards for Hotspots that transferred data and required devices to pay for network traffic with Data Credits.

· One min read

This 2020.09.01.0 release is a minor bugfix release to fix state channel issues on the hotspots so they can continue to transfer data. We have also added initial support for third-party gateways.

· One min read

 

Updated Payments UI

In this new update, we've revamped the payments UI. Use the QR code scanner to not only populate the recipient address, but HNT amounts embedded in the QR code will populate as well!

· 3 min read

Final Update:

The updated release had some issues with loading ledger snapshots on first-generation Hotspot hardware, and we had to cut the 2020.08.26.0 release in order to reduce snapshot import memory usage. Since the only new behavior was core PR 599, we did a very short beta to confirm the fix for the issue and released the image to GA around 12:30PM PDT.

· One min read

This 2020.08.21.0 provides an implementation of HIP-10. The implementation is gated behind a chain var, which we will activate once everyone has upgraded and we've finalized and announced an activation date and time.

· 2 min read

Recent increases in the size of the network have caused some scaling issues. When the network was smaller, we wrote some code without concern for scaling, pulling the entirety of various data structures off of disk and into memory for the duration of various operations. The code was simpler that way, but these in-memory structures have come to cause a lot of problems as the number of gateways has gotten larger.

· One min read

 

This release fixes the websockets used on the main Accounts page (it was using the old API, now uses the new one) and updates the polling interval to 30 seconds so it can fetch recently cleared pending transactions.

It also fixes a bug where NaN is displayed for pending transactions.

· One min read

This 2020.08.12.0 release addresses some of the concerns regarding the opening/closing of state channels responsible for packet transfer. Essentially it allows the router to maintain state channels better as we have introduced proper rollover of the channels on DC exhaustion.

· One min read

 

This release is to prep for the new Hotspots coming online, plus a few bug fixes.

  • Fix crash when tapping reward transactions with 0 amount
  • Add missing translation for data credit rewards
  • Fix account addresses being cut off in Android transaction details
  • Change hotspot activity reward transactions to just show HNT earned
  • Fix payment transactions not rendering correctly in block details
  • Fix assert location for accounts with 0 balance

· 3 min read

Summary

  • We're moving the Phase 3 Data Credits go live date to 8/11 for further testing and fixing issues.
  • A fix has been pushed to address issue with Hotspots not transferring packets.
  • We're going to improve communication and status of issues with this tool, subscribe here.

· One min read

 

Forget Wi-Fi Network

  • This feature allows users to forget a wi-fi network that the Hotspot has remembered, so that you can either update the password, or connect to a new network
  • To forget a network, tap the Hotspot settings, pair with the Hotspot, and tap the network name. The app will cycle through all the networks profiles saved on the Hotspot until they're all forgotten. Then you can connect to a new network.

· 2 min read

We’ve moved the date when devices need Data Credits from 7/27 to 8/3. On 8/3 all device owners must pay Data Credits to transfer packets and Hotspots start earning HNT for data transfer from devices.

· One min read

This 2020.07.24.0 release completes the "Forget Network" feature that will be added to an upcoming version of the Helium mobile app. We also fixed a couple bugs and added debugging to prepare for upcoming state channels V2 chain variable.

· One min read

This release lands a bunch of work (4000+ lines of code) to enable state channels v2. This release lays the ground work to start charging and accounting for data credits to move data across the Helium network.

· One min read

This 2020.07.16.0 release adds a chain variable for radio data gathering to aid with improvements to proof-of-coverage. When the chain variable is activated the witness data collected will include signal-to-noise ratio and frequency fields.

· One min read

In this release, we are focused on squashing bugs and updating our community link to reflect our move to Discord!

· 2 min read

Removal of GPS Status

In our last release 1.9.0, we introduced and experimental feature on a Hotspot's GPS status that was meant to be informational.

· One min read

This 2020.07.09.0 release fixes longstanding bugs in gateway-config that resulted in incorrect WiFi network states being reported to the mobile app and via the Hotspot's LED.

· One min read

This release is a routine update of the onboard blessed snapshot to block height 396721. This allows newly onboarded Helium Hotspots to sync the whole blockchain in hours not days.

· One min read

The chain variable discussed here has been activated. The limit has been set at 20 for the time, being, and we'll analyze Proof of Coverage path variety and transaction validation times to see if we need to make any further adjustments.

· 3 min read

 

PLEASE UPDATE TO THIS RELEASE. This update includes the following new features and is not backwards compatible. Failure to update to 1.9.0 will cause all transactions in the mobile app to fail once we activate transaction fee chain variables.

· 2 min read

Much of the trouble that we've been having lately with maintaining consistent block times has been a result of Hotspot clusters, some of them quite large. These clusters, often hosted in the same place, lead to an abnormally high number of witnesses and a large disk representation. Since they're colocated and witness each other, our Proof of Coverage (PoC) pathing code has some pathological behavior when they're challenged, since it must consider sometimes hundreds of Hotspots to generate the path. This can lead to single transactions that take the Hotspot hardware multiple seconds to validate, which can cause a lot of problems for protocol completion and steady block times as the validation and the Honeybadger runners compete for the disk.

· 2 min read

Recently it's come to our attention that a growing number of Hotspots are having issues staying in sync with the chain. Hotspots use RocksDB as their storage layer for the ledger and the blockchain. RocksDB offers a feature called snapshots which purports to provide an immutable, read-only version of your store, in this case, old ledgers to make validation cheaper. Unfortunately, due to an as-yet-unidentified issue, either to do with RocksDB itself, how the RocksDB Erlang NIF library is written, or how our code is creating and using these snapshots, we're seeing some writes which have been synchronously written to disk missing from the snapshots.

· One min read

While we are aware of some Hotspots falling out of sync (although they will eventually recover) we have not yet isolated the root cause. This release only makes 2 changes, it fixes a problem with the timing of LoRaWAN packets, specifically around meeting the downlink window and it blesses a new snapshot at height 379441. The new snapshot should help unstick some Hotspots that are struggling, and we will continue to work on a more permanent fix.

· 2 min read

After monitoring the chain overnight and seeing few signs of recovery, further investigation revealed that we were still seeing out of memory errors on v1 Hotspots. A closer reading of the docs revealed that a part of the Erlang runtime reports memory usage in machine words (the width of the processor's registers), which means that the earlier measurements we based our tunings on were too low by a factor of 8. This release does nothing but retune the cache to use much less memory, which we hope will stabilize the chain, if not fully restore block times.

· One min read

We've identified some more failures that have been leading to long block times and included fixes for them in this release. In addition, now we have the price oracle running, we've landed the code for real transaction fees and staking fees. These fees won't be active until we've made sure all the wallets have been updated and everyone has had appropriate warning. Finally we've bumped the blessed snapshot to 376561.

· One min read

In an attempt to improve block times, we have developed a new Hotspot cache which keeps the deserialized spots in memory for longer, so we're not constantly fetching and refetching the same information from the disk. There's also a change make validation of Proof of Coverage(PoC) receipts substantially cheaper.

· One min read

This minor release increases some network timeouts again to compensate for snapshot sizes. It also corrects an error in the "bad GPS fix" reporting code and makes snapshots more compatible with the upcoming changes for data credits and transaction fees. Finally there's a small fix that may help with some cases of ledger corruption. A bigger fix on this front is in the works.

· One min read

Over the weekend we've identified a few minor issues with ledger snapshots, and since they're impacting block times, we're issuing a minor release to clear them up. Our old code assumed that all Hotspots would have all of the blocks, and there are a few bits of code that rely on that, one of which affects consensus nodes, which has been negatively affecting block times.

· 2 min read

Another big issue that we've been dealing with recently is sync times on new Hotspots. We've seen cases recently when new Hotspots have taken in excess of 10 days to fully sync and start issuing challenges. This release is the first to support a feature we're calling "ledger snapshots", which allow Hotspots to safely sync without downloading the entire blockchain first. A few times a day, the running consensus group will take a snapshot of its ledger, package it up with a relatively small number of blocks and then store its hash on the chain. We can then attest the validity of a particular snapshot to allow new Hotspots to download that snapshot and start using it, much as we now attest to the validity of a particular block (and all the blocks leading up to it). This should allow syncing in hours, rather than in days as is currently the case.

· 4 min read

What Happened?

We activated witness refresh at block 339044 (see audit 23), which was intended to gradually clear witness tables over the course of about a week. Unfortunately we neglected to apply this refresh to the lagging ledger in the same way. Once we realized this, we issued a deactivation of the new feature at block 339291. Unfortunately it was too late and about 37 ledger entries were now divergent between the leading and lagging ledger. This led to disagreements on the block contents between the consensus members and caused a chain halt.

· One min read

Some common feedback we get is that Hotspots use too much bandwidth. We've taken some steps to address this in the past, and today is the latest step in scaling bandwidth back. This release applies compression to all connections between Hotspots and their other gossip peers, and makes it easier to shift to more effective compression in the future. For non-Hotspot peers, this code is backwards compatible, but you'll need to upgrade to see the benefits.

· 2 min read

Key feature updates

Functions

We’ve added the ability to execute functions on Console, the first can be used to decode payloads. With the Decoder Function, users can can transform and/or parse a raw payload before it is posted to an endpoint, such that, for example, individual sensor values are reported rather than the base64 string. This transformation also applies to debug output, allowing it to be more human-readable.

· One min read

This release adds support for amd64 and arm64 Docker images containing miner. These images can be fetched from Helium's image registry at quay.io. They are Alpine Linux-based so specify /bin/ash as the interactive terminal. The amd64 images will run in the cloud and the arm64 images will run on a Raspberry Pi 3 or 4 that has Ubuntu 20.04 LTS 64-bit installed.

· One min read

Summary

We identified a flaw where we did not check for 0 amount transactions in some cases. While largely harmless, it can lead to transaction spam. Additionally, after doing some load testing, we've lowered the maximum block size (measured in number of transactions) while we continue to work on improving block propogation times.

Audit

You can find the chain variable transaction details here.

· 2 min read

Key feature updates

  • We’re releasing Debug mode Monday April 20 as we described last week.
  • Users can access Debug mode on a specific device or label page. This allows users to view data from a single or multiple devices.
  • Debug mode will display 10 packets at a time.
  • Each packet is made up of two parts: the information sent from Console to an integration, and information sent to Console from the backend independent of the integration.

· One min read

We've been doing some internal work to increase transaction throughput from our API, and this has required some Hotspot side support, with respect to how the consensus group prefilters incoming transactions. The old code wasn't playing well with ordered transactions (multiple consecutive payments, for example). Expect more related fixes in the next few releases.

· One min read

Summary

This 2020.04.13.0 release upgrades gateway-config to enable the removal of saved Wi-Fi networks from hotspots via the mobile app. A bug fix for a recent regression that disabled GPS on the hotspots is also included.

· One min read

The new API has been deployed to api.helium.wtf (development) and api.helium.io (prod) for a couple of weeks now and as part of moving the mobile application to it we realized that two routes were not quite consistent.

· One min read

Summary

This 2020.04.07.0 release adds support for multiple default LoRa routers to miner. A bug fix for the libp2p NAT server crashing is also included. Additionally this release bumps the assumed valid block to 282180.

· One min read

Summary

Key user experience enhancements:

  • Labels mapped to integrations are easily identifiable by Integration icon.
  • Device details: includes packet transferred numbers.
  • Device index page includes: Labels, Integration, Frames up, Frames down, Date added, Last connected.

· One min read

Summary

This 2020.04.01.0 release fixes a bug that results in hotspots getting stuck syncing. A gateway OUI field was also added to the blockchain protocol. The new gateway OUI field is a breaking change that requires a software upgrade.

· One min read

Summary

We continue to receive excellent feedback from the community regarding some of the decisions we made with the introduction of PoC V8. This update clarifies the new chain variable transaction we issued on Thu 13 Feb 2020 09:29:39 PM UTC.

· 4 min read

Summary

We have been steadily gathering data on the performance of Proof-of-Coverage and are ready to release the PoC version 8 update. This update brings a few key bug fixes and enhancements detailed below.

· 2 min read

Executive summary

Block time disruptions have lowered the rate of token production from the chain, and we're implementing a mechanism to shorten block times when this is the case, so that we catch up to our targeted production rates of 5 million HNT per month if we experience long block times.

· One min read

Block Weather

Despite our prior efforts, we have still been tracking (and debugging) high memory usage in our storage engine, RocksDB. We have identified some additional fixes and this issue has become critical as of block 186592. This block was somewhat large and has triggered Out of Memory kills on much of the network as it propogates out. This release contains improvements that should allow the network to absorb past that block.

· One min read

Block Weather

While the recent releases have done a fairly good job at cleaning up block times, the last firmware image introduced a bug that can prevent Hotspots from challenging. This release fixes that bug and should restore the normal challenge rate.

· One min read

Block Weather

While the out of memory improvements from yesterday have helped, the erratic block times have returned. We believe this has to do with the gossip network topolology degrading over time (an OTA resets the gossip network connections as the nodes reboot). We have identified some more fixes that should help block propogation times between the consensus group which should limit the impact of bad gossip topologies on the block production rate.

· One min read

Block Weather

After the Proof of Coverage v7 (PoC) firmware release, block times quickly began to suffer again. This time, the cause seems to be the Linux Out of Memory subsystem killing the miner process as it begins to use too much memory. We've tracked this down to RocksDB, which we use for storage for a number of subsystems, using too much memory. We applied a large number of tunings and upgraded the version of RocksDB, picking up a number of fixes and enhancements along with some memory use improvements.

· One min read

A number of assert location transactions had become stalled and were replayed from the API. Unfortunately they all landed in the same few blocks, which became nearly impossible to process quickly because of some index book-keeping overhead that must be done when gateways are asserted. This release disabled some of that book-keeping, which is obsolete, and optimized other cases where the expensive work doesn't need to be done. This should allow all of the stalled Hotspots (around 5% of the fleet) to resume syncing and challenging.

· 2 min read

Plan

Shortly, we are going to issue a chain variable which will increase the Proof of Coverage (PoC) interval from 30 blocks to 60 blocks. This will have the effect of reducing the rate of PoC requests sharply, which should decrease long blocks going forward.

· 2 min read

We've been seeing issues of ledger corruption when nodes are syncing the chain with increased frequency and have finally identified the fix. In the way we use rocksdb we split data in the ledger and the blockchain across column families (eg token balances are distinct from hotspot information). The ledger corruption issues were when part of a ledger update would commit but the other parts would not. The fix was to enable the atomic flush option in the rocksdb database options which ensures that all updates in a batch update are committed atomically, even across column families.

· One min read

After adding an ARP mechanism to libp2p with the release of 2020.01.05.0 on January 6, 2020 we noticed that gossip quality began to degrade again over time. To remedy this we decreased the ARP's peer staleness time from 24 hours down to 6 minutes and began throttling ARP requests. We also bumped the assumed valid block to 165,353 for faster resync times.

· 2 min read

With this update we hope to alleviate some of the known concerns regarding POC path construction while also fixing some overlooked typos.

· One min read

Over time we have seen our gossip quality degrade as the network has become larger. Over the holidays, we experienced multiple blockchain stalls. Some of these stalls were the result of out-of-memory errors. This 2020.01.05.0 release adds an ARP mechanism to libp2p to pro-actively refresh stale peer book entries. In addition, we fixed a couple of unfortunate typos in PoC receipt validation. We also bumped the assumed valid block to 163,853 for faster resync times.

· One min read

Over time we've seen our gossip quality degrade as the network has become larger. Quite suddenly, we crossed some threshold and gossip has become untenable without changes. libp2p groups, the abstraction that our consensus and dkg are built upon, are taking longer and longer to converge, leading to elections that don't complete, and occasionally to chain stalls. This emergency release speeds up gossip dissemination by increasing the number of peers to gossip per gossip round, and changes the seed node connection strategy to only connect to two seed nodes, rather than all of them. At the same time, we're increasing the number of seed nodes to deal with overload.

· 2 min read

With the launch and activation of PoC v5, the team quickly realized that we needed to bring back some of the PoC RSSI related probability weights back to the path construction algorithm to allow building paths with potentially known valid witnesses as next hops.

· One min read

Content

With the launch of PoC V5, we unintentionally caused an API fork, where the blockchain couldn't sync past the new election block. This happened due to the following reasons:

· 4 min read

The launch and activation of PoC v4 introduced some interesting new paths and brought in much needed performance improvements. We have essentially quadrupled the number of challenges on the network as validation of PoC receipts is now significantly more efficient.

· 9 min read

As part of our efforts to continually improve the network, we recently implemented Proof-of-Coverage (PoC) version 4. This blog attempts to detail some of the planning, decisions and analysis that went into overhauling the existing PoC mechanism.

· One min read

While debugging performance issues over the weekend it was realized we could significantly improve performance of an expensive operation, calculating the ledger at a specific height, by using a ledger snapshot we were already making. A simple cache was added to track the last 30 ledger snapshots and if the ledger_at height existed in the cache, it will be used. This has resulted in a significant performance improvement both to validating PoC receipts as well as block syncing. While we still plan to activate PoCv4 tomorrow we feel that this interim performance fix, which remains relevant for PoCv4, is worth it to get the amount of accepted PoC receipts that can be validated and added to the chain up from the levels they have dropped to.

· 2 min read

The prior update, shipped this morning, has had some unintented consequences. Because of a change in an init script, a latent race condition in the configuration of the WiFi interface has caused some Hotspots to be unable to connect to WiFi. Whether a Hotspot is affected depends on what firmware version it was running when it was initially configured for WiFi. Hotspots on Ethernet should not be affected.

· 2 min read

On November 13 around 7am PST the blockchain halted due to an out of memory bug. All the consensus members, having agreed on the next block's contents were unable to verify them. This was because, for performance reasons, the transactions in the block were being verified in parallel. Unfortunately there was no limit on the amount of concurrency used and this led to CPU and memory exhaustion as all 65 transactions in block 110,710 were being verified simultaneously.

· 2 min read

The Helium hotspot uses an ECC608 security chip which generates and stores private keys for the hotspot. The private keys generated by the ECC are not accessible outside of the secure confines of the ECC hardware.

· One min read

We have released a firmware based on the previous beta to general availability. This firmware is focused on improving Proof of Coverage construction and validation times. The main change from the last beta has been to refactor our PoC path calculation routines to be more efficent and to improve our caching of expensive to obtain information necessary for pathing. This change can be seen at blockchain-core#289 .

· 3 min read

On November 1 around 3 PM PDT we rolled out a new beta release to better distribute PoC requests throughout the election cycle, and including some other longer-term engineering improvements.

· One min read

 

  • Our previous update for Android 1.4.6 had a bug for one of our require libraries and caused X86 architecture devices running Android 8.0 and 9.0 to crash and never recover. This update fixes the library crash.

· One min read

In preparation for release of the LongFi SDK we are testing the latest version of the LongFi protocol on a select few San Francisco hotspots that are within radio range of each other.

· One min read

On October 25 around 6 PM PDT we rolled out a GA release to address blockchain syncing problems. This was a network-wide change that intentionally broke compatibility with the previous 1.0 version of the blockchain sync protocol.

· 2 min read

As more and more hotspots ship out, we are encountering some blockchain syncing problems. A number of the shipped hotspots were assembled over a month ago but had older firmwware that did not include some important bugfixes.

· One min read

In preparation for release of the LongFi SDK we are testing the latest version of the LongFi protocol on a select few San Francisco hotspots that are within radio range of each other.

· One min read

We rolled out the Beta update over the weekend and it appears to have helped some, but a number of issues were discovered that caused unneeded network load, slowing down blocks and elections.

· 7 min read

Overview

The Helium blockchain team has been working for several weeks on some incremental upgrades to our Proof of Coverage (PoC) and election systems, to make them more scalable and fair. Changes here include: