How to perform a large migration?

Question

If I want to migrate the Balance::Locks to a new structure. Imagine there are millions of accounts in the storage.

This is what I came out. Is this the right way? Split them into multiple blocks. If yes, how can I record the pos?

storage OldLocks
storage NewLocks
storage LastPos

fn on_new_block() {
    LastPos::put(migrate_from(LastPos::get()));
}

// Aka the locks' `getter` during the migration
fn lock_of(account) -> Amount {
    let mut locks = [];

    if let Some(lock) = NewLocks::get(account) {
        locks.append(lock);
    }
    if let Some(lock) = OldLocks::get(account) {
        locks.append(lock);
    }

    return locks.cal_total_lock_amount();
}

Furthermore, what if I want to keep using the old name? I don't want to break the API. move_prefix(b"Locks", b"OldLocks") first, does this take too much weights?

Answer 1

This is basically the correct approach.

There isn't actually any need to record the position: you should be removing items from OldLocks and inserting equivalents into NewLocks. You can stop the migration once OldLocks is empty. As long as you can guarantee that migration will remove at least one item from OldLocks and that your runtime never introduces any items into OldLocks then this will complete in a finite time.

See the pallet-xcm in Polkadot for an example on multi-block migrations.