How to overwrite endowed account balances using additional accounts and balances from a JSON file for genesis block inclusion?

Question

My goal is to allocate a balance in the genesis block to both:

• A list of endowed accounts already included in the chain specification chain_spec.rs
• A list of further accounts and balances included in a JSON file called genesis.json genesis block (overwrite the balance value of any duplicate accounts that are also in the list of endowed accounts).

However, I have encountered the following problems:

• None of the endowed accounts or the accounts in the JSON file are being credited
• polkadot-launch generates invalid values for "genesis_head" and "validation_code" in the parachain chain specification rococo-local.json that it generates (see below for details).

I've tried creating this Pull Request https://github.com/DataHighway-DHX/DataHighway-Parachain/pull/5 which is based on Substrate v0.9.16 and it compiles successfully with ./scripts/init.sh && cargo build --release.

I have forked the polkadot-launch repository in this Pull Request https://github.com/DataHighway-DHX/polkadot-launch/pull/2, which includes instructions and shell scripts that I follow to configure it on a macOS M1, but when I follow the default polkadot-launch steps, for example copying the DataHighway-Parachain binary cp ./target/release/datahighway-collator ~/code/DataHighway-DHX/polkadot-launch/bin/datahighway-collator to the relevant polkadot-launch folder and after cloning and building the v0.9.16 branch of the polkadot repository and also copying the polkadot binary to the same polkadot-launch folder, and then running polkadot-launch yarn start config.json using a relevant version of Node.js (i.e. v16.3.0), the polkadot-launch logs give an error message when it generates the rococo-local.json chain specification file Error: 0: Other: Error parsing spec file: 0x prefix is missing at line 208 column 1756, as shown below:

🧹 Resolving parachain id...
2022-02-10 14:05:34 Building chain spec    

🧹 Starting with a fresh authority set...
  👤 Added Genesis Authority alice
  👤 Added Genesis Authority bob
  👤 Added Genesis Authority charlie
  👤 Added Genesis Authority dave

⚙ Updating Relay Chain Genesis Configuration
  ✓ Updated Genesis Configuration [ validation_upgrade_frequency: 10 ]
  ✓ Updated Genesis Configuration [ validation_upgrade_delay: 10 ]

⛓ Adding Genesis Parachains
  ✓ Added Genesis Parachain 2000
  ✓ Added Genesis Parachain 3000
⛓ Adding Genesis HRMP Channels
  ✓ Added HRMP channel 2000 -> 3000
Added Boot Nodes: /ip4/127.0.0.1/tcp/30336/p2p/12D3KooWF66thRTDXG2ksKm3nMCHeoMneuohs71Q4anPLVnBSsSp,/ip4/127.0.0.1/tcp/30337/p2p/12D3KooWAZ1VKfix84y8rrrLekd6xVksDgjiYooTNTBi32GSXCWN,/ip4/127.0.0.1/tcp/30338/p2p/12D3KooWEpmRWnuEga2oFPZ7P8BfcJBZj928XT46hhj6zyieA72R,/ip4/127.0.0.1/tcp/30339/p2p/12D3KooWFUiU4LrYD1MVwbvkE4wqvCTy2feU9LijMKPoEXBJhArG

2022-02-10 14:05:36 Building chain spec    
Error: 
   0: Other: Error parsing spec file: 0x prefix is missing at line 208 column 1756
...

I found that the error is caused because the rococo-local.json file that it generates isn't being generated correctly, and its including snippets of the actual codebase and newline characters in front of the hex values of the "genesis_head" value and "validation_code" properties as shown below:

rococo-local.json

...
"paras": {
  "paras": [
    [
      2000,
      {
        "genesis_head": "account_public_key_json "a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21"\naccount_balance_json 60000000000000000000000\naccount_ss58_address_json a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21 (5FmxcuFw...)\nno match b/w endowed and json for "a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21" so adding it to the list\n0x0000000000000000000000000000000000000000000000000000000000000000005c76496ba72...",
        "validation_code": "account_public_key_json "a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21"\naccount_balance_json 60000000000000000000000\naccount_ss58_address_json a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21 (5FmxcuFw...)\nno match b/w endowed and json for "a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21" so adding it to the list\n0x52bc5376

Since I can't figure out how to debug (i.e. using println! or log::trace, etc), in the chain_spec.rs file of DataHighway-Parachain codebase, I instead replicated the relevant parts of the code in a separate Rust project https://github.com/ltfschoen/RustTest/tree/master/projects/endow to see what was going on, but it appeared to be combining and returning the accounts and balances correctly:

[
  (
    a42b7518d62a942344fec55d414f1654bf3fd325dbfa32a3c30534d5976acb21 (5FmxcuFw...),
    30000000000000000000000,
  ),
...
]

So in the DataHighway-Parachain codebase, I tried to isolate the issue as follows:

I found that if I only use the for loop https://github.com/DataHighway-DHX/DataHighway-Parachain/pull/5/files#diff-f2256017c4e606a30f53708578da46675512cc1300be6542ed5f77e58af0d6dcR961 and remove the line that calls get_allocation so it doesn't try to load and included the accounts and balances in the genesis.json file into the genesis, then it generates an error Error parsing spec file: 0x prefix is missing at line when i use polkadot-launch.

Whereas if comment out that for loop (so it doesn't credit the endowed accounts), and I uncomment the get_allocation function again (so it only credits the accounts loaded from the genesis.json file), then it compiles successfully, and after I copy the latest datahighway-collator binary file to polkadot-launch and run polkadot-launch, it launches the relay chain and collator nodes successfully, but it doesn't credit the accounts loaded from the genesis.json file as expected, because I checked the balance of the accounts included in genesis.json by adding them to the Polkadot.js Apps address book https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1:9988#/addresses but they didn't have a balance.

I also suspect it may be because I am not sure when you use AccountId (of type type AccountId = <<Signature as Verify>::Signer as IdentifyAccount>::AccountId; that is included in DataHighway-Parachain/runtime/src/types.rs) instead of AccountId32 (which I understand is a struct that creates an instance of inner type [u8; 32]).

Answer 1

So you could create a function in you chain_spec.rs like this:

pub fn get_endowed_accounts_with_balance() -> Vec<(AccountId, u128)> {
    let accounts: Vec<AccountId> =
        vec![
            get_account_id_from_seed::<sr25519::Public>("Alice"),
            get_account_id_from_seed::<sr25519::Public>("Bob"),
            get_account_id_from_seed::<sr25519::Public>("Charlie"),
            get_account_id_from_seed::<sr25519::Public>("Dave"),
            get_account_id_from_seed::<sr25519::Public>("Eve"),
            get_account_id_from_seed::<sr25519::Public>("Ferdie"),
            get_account_id_from_seed::<sr25519::Public>("Alice//stash"),
            get_account_id_from_seed::<sr25519::Public>("Bob//stash"),
            get_account_id_from_seed::<sr25519::Public>("Charlie//stash"),
            get_account_id_from_seed::<sr25519::Public>("Dave//stash"),
        ];
    
    let accounts_with_balance: Vec<(AccountId, u128)> = accounts.iter().cloned().map(|k| (k, 1 << 60)).collect();
    let json_data = &include_bytes!("../../seed/balances.json")[..];
    let additional_accounts_with_balance: Vec<(AccountId, u128)> = serde_json::from_slice(json_data).unwrap();
    
    let mut accounts = additional_accounts_with_balance.clone();
    
    accounts_with_balance.iter().for_each(|tup1| {
        for tup2 in additional_accounts_with_balance.iter() {
            if tup1.0 == tup2.0 {
                return;
            }
        }
        accounts.push(tup1.to_owned());
    });

    accounts
}

I would also modify your JSON slightly like this:

[
    [
        "5HGjWAeFDfFCWPsjFQdVV2Msvz2XtMktvgocEZcCj68kUMaw",
        50000000000000
    ],
    [
        "5CiPPseXPECbkjWCa6MnjNokrgYjMqmKndv2rSnekmSK2DjL",
        40000000000000
    ]
]

I created a simple working example on substate-node-template for you here:

• https://github.com/brunopgalvao/substrate-node-template/blob/main/node/src/chain_spec.rs

Hope that helps!

Answer 2

I modified my code in this commit here to incorporate the following parts from Bruno's answer here:

• Initially I adopted Bruno's approach and got it to work in the DataHighway-Parachain codebase using branch "luke/genesis-hardspoon-accounts-backup" here, and followed my Polkadot-Launch steps here. The only thing different in those steps to run DataHighway-Parachain instead of substrate-parachain-template with Polkadot-Launch is:
• copy with cp ./target/release/datahighway-collator ~/parachain/polkadot-launch/bin instead of cp ./target/release/parachain-collator ~/parachains/polkadot-launch/bin
• change name of each parachain binary from "bin": "./bin/parachain-collator", to "bin": "./bin/datahighway-collator", in the config.json file
• Whilst trying to get it to work in the DataHighway-Parachain codebase I was able to get it to work using the substrate-parachain-template with the changes in this branch "luke/polkadot-v0.9.16/endow" here
• I used Bruno's approach that used &include_bytes! to parse the genesis.json file since I could figure out how to do it using my original approach with serde_json::from_str, as although it compiled the balances would not appear in the UI.
• I removed use of decimal values for the balance in the genesis.json file, as I found that if you use a decimal value as the number value (i.e. 1000000000.1) in the genesis.json file, then you'd get an error trailing characters when running the collator node with Polkadot-Launch
• I changed the genesis.json data structure so the vector of accounts and balances wasn't stored under the "balances" key. However, I used a string value for the balance instead of a number, because I am using this script that i created and updated, which generates the genesis.json file based on accounts and balances obtained from a request to the Subscan API, and since some of the balances obtained from the API are rounded and do not have the correct number of decimal places and have different amounts of decimal places, they don't match the chain's tokens decimals, so in that script I included convert.js that adds the necessary number of 0's at the end and removes the decimal point and stores it as a string. For example if an account balance obtained from the Subscan API was 18.400991637728733 (18.4 UNIT, which only has 15 decimal places instead of 18 decimal places that our chain uses) then we would to convert it to "18400991637728733000" for the DataHighway-Parachain.
• If i used a number like 51112223330000 instead of a string like "51112223330000" in the JSON file, like Bruno suggested, then that value must include all the token decimal places in that integer value, but the trailing 0's may get accidently cropped from the end of a number like, so I preferred to use a string. In the DataHighway runtime we use a tokenDecimals value of 18, so 60000000000000000000000 in genesis.json appears as 60,000.0000 UNIT in the Polkadot.js Apps Addresses UI, but a value of 40000000000000 appears as 0.0000 UNIT (which could make you think it isn't working when it actually is). Anything less than 100000000000000 (i.e. 99999999999999 and below) does not appear in the UI for the DataHighway-Parachain. So if you have an address that only has the EXISTENTIAL_DEPOSIT of 100000000000 then that may appear as if you have no balance in the UI.
• In addition to endowing the embedded accounts (i.e. get_account_id_from_seed::<sr25519::Public>("Alice"),, etc), I am also endowing accounts using hex!["<INSERT_PUBLIC_KEY_HEX_WITHOUT_0X_PREFIX>"].into(), format.
• Since I didn't want to deviate too much from the substrate-node-template or substrate-parachain-template, I used my original approach that didn't require changing the template code from endowed_accounts: Vec<(AccountId, u128)>, into endowed_accounts: Vec<(AccountId)>,.
• I used AccountId32 rather than just AccountId, although still don't quite understand why sometimes they maybe used interchangeably in some places and it still works.
• I removed use of the print! function, since I discovered that if you add a line with the print! function https://docs.substrate.io/v3/runtime/debugging/ on the first line of the function get_endowed_accounts_with_balance() (or any other line in the function), i.e.

use sp_runtime::traits::Printable;
use sp_runtime::print;

...

pub fn get_endowed_accounts_with_balance() -> Result<Vec<(AccountId, Balance)>, Error> {
    print!("hi");
    ...

then when you run your collator node with Polkadot-Launch you get an error like Other: Error parsing spec file: invalid hex character: h, at 0 at line 213 column 256, which is because for some reason the print! function text gets added as the start of the value in the rococo-local.json file that gets generated by Polkadot-Launch:

...
    "paras": {
      "paras": [
        [
          2000,
          {
            "genesis_head": "hi0x0000000000000000000000000000000000000000000000000000000000000000002c965766f0bc338fda737ccd2a40934c16247deed5bf574bdd6f72e82dd0284603170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c11131400",
            "validation_code": "hi0x52...",