Monday's hard fork and next validator challenge

Jehan Tremback -

On Tuesday, the Althea validators finished a hard fork from altheatest3 to altheatest4.

Here are the changes that were made to the chain's settings and state:

  • Reduce governance proposal deposit to 1 ALTG. This lets us do governance proposals easily (it was 512 ALTG before, and most people only have 100 or 200 ALTG).
  • Reduce unbonding period to 1 day (86400000000000ns). This makes it easier to take your tokens out of delegation to send them around and stuff. It was formerly 21 days.
  • Minted 100 new ALTG for anyone who signed up for our validator notification mailing list at https://airtable.com/shrpjZDQYQa3Xd5oe. The newly minted tokens are to give people who weren't in altheatest3 a chance to get some tokens to start validating on altheatest4, and to give the old-timers a little extra worthless testnet pocket change.

Hard forks on Comsos chains are currently a surprisingly manual process. Here are the steps:

  1. Everyone waits until the predetermined time.
  2. Everyone stops gaiad, and exports a new genesis file which will be used to start the new chain.
  3. Everyone makes any necessary changes to the new genesis file (at least updating the chain id), and replaces the old genesis file with the new one.
  4. Everyone does gaiad unsafe-reset-all, and gaiad start.
  5. Once validators controlling 2/3rds of the voting power on the chain have completed these steps, the chain starts again.

We were stuck between steps 4 and 5 for almost 24 hours due to several factors.

  • First of all, a lot of people had issues with the error ERROR: error exporting state: failed to load Store: wanted to load target 206400 but only found up to 200000  This was caused by the state pruning algorithm. In the future, we will set the forking block in advance, and as soon as it is decided, everyone will restart their nodes with gaiad start --halt_height [fork_height]. This will make sure that the pruning algorithm is ready for an export at that height.  
  • Because of these issues, some validators had to start using an altheatest4 genesis file from someone else instead of exporting and processing their own. This led to some errors, such as people mistakenly running the script on the already processed altheatest4 genesis and ending up with the wrong number of tokens in accounts.  
  • Also, about 1/4 of the validators did not come online until later Tuesday. With the MIA validators, and the validators with genesis file issues, we were slightly under 2/3rds of the validators for a while. It also took a little while for the chain to start up once we got enough validation power online because it had been stalled so long (I think it's an exponential backoff thing).

Next validator challenge

I think we should do a hard fork to gaia v2.0.0, which I believe is ready. Here is the Cosmos Hub proposal about it: https://ipfs.io/ipfs/QmbXnLfx9iSDH1rVSkW5zYC8ErRZHUK4qUPfaGs4ZdHdc7. Additionally, we should do it as a scripted hard fork, which should minimize downtime, and be easier for validators. Validator Blitmore has already created a script which the Althea team can audit and possibly improve.

I've created a governance proposal to this effect: https://althea.zone/#/governance/2

I mistakenly titled it "Scripted hard fork to gaia v0.36.0". This is somewhat incorrect, since it is gaia v2.0.0, built with Cosmos-sdk v0.36.0. Now that it's on the blockchain it's permanent. Forgive my stupidity.