How to get historical substrate (extrinsic, events, etc) data using polkaDotJS API

Question

PolkaDotJS API docs show to get extrinsic, events, etc data for a particular block. Is there a way to get all the extrinsic, events, etc data for the substrate network without having to get for each block?

Answer 1

There is no single endpoint where you can get "all the extrinsic, events, etc data for the substrate network". This doesn't even really make sense, since data is inherently organized by blocks.

The way to get "all" of the data, is to iterate over all the blocks, and for each block, get the data you are interested in.

There may be indexing services which you can run that may provide you with easier ways to query all of this at once, but these are tools built on top of the Substrate chain, and are not provided by Substrate nodes to begin with.

Answer 2

Extrinsics are stored as part of a block and events are "intrinsic" to the chain's state so they are stored separately in the chain's system storage. You can query events based on block hash using the system state. With RPC, you would have to iterate per block and query the events based on the block. Here is example code:

import { ApiPromise, WsProvider } from '@polkadot/api';

export const getAllData = async () => {

    const wsProvider = new WsProvider('wss://rpc.polkadot.io');
    const api = await ApiPromise.create({ provider: wsProvider });
    const lastHeader = await api.rpc.chain.getHeader();
    let blockNumber = lastHeader.number;

    while (blockNumber > 0) {
        blockNumber = blockNumber - 1
        const blockHash = await api.rpc.chain.getBlockHash(blockNumber);
        const signedBlock = await api.rpc.chain.getBlock(blockHash);
        signedBlock.block.extrinsics.forEach(async (ex, index) => {
            console.log(index, ex.toHuman());
            const { method: { args, method, section } } = ex;
            console.log("******EXTRINSIC--BEGIN***");
            console.log(`${section}.${method}(${args.map((a) => a.toString()).join(', ')})`);
            console.log("******EXTRINSIC--END*****");
            
            const allRecords = await api.query.system.events.at(blockHash);
            // filter the specific events based on the phase and then the
            // index of our extrinsic in the block
            const events = allRecords
                .filter(({ phase }) =>
                    phase.isApplyExtrinsic &&
                    phase.asApplyExtrinsic.eq(index)
                )
                .map(({ event }) => `${event.section}.${event.method}`);

            console.log("******EVENT--BEGIN*****");
            console.log(`${section}.${method}:: ${events.join(', ') || 'no events'}`);
            console.log("******EVENT--END*******");
        });
    }

}

Reference: https://polkadot.js.org/docs/api/cookbook/blocks#how-do-i-map-extrinsics-to-their-events

If you're not interested in iterating per block than you may want to consider an indexer.