Substrate and Polkadot Stack Exchange is a question and answer site for developers building blockchains with the Substrate SDK. It only takes a minute to sign up.
Sign up to join this community
If we want to ask lots of queries about chain state (for example users activity over time etc.) is there a recommended or de-facto way of doing it? Ethereum has 'the graph' for example as a high performance way to interrogate the state.
(Bonus points awarded for services that provide predictable performance under heavy load)
I personally run an Archive node on my computer which has been sufficient for the queries I do, but there are some tools available within the Substrate ecosystem built more specifically for deep queries like you suggest.
Inspired by The Graph, it gives a smooth way to provide powerful GraphQL queries to app developers over your Substrate blockchain state and history.
Run alongside a substrate-backed chain to index all Blocks, State, and Extrinsic data into PostgreSQL.
A compact indexer for @paritytech substrate based nodes providing a graphql interface.
SubQuery is an open source project that allows developers to index, transform, and query Substrate chain data to power their applications.
And probably more to come!
There is also Subsquid, which you can check out
I recommend using Subsquid Archive (indexer) to store chain history, you can then use GraphQL to query instantly anything you want.
The Archive can be connected with a Squid (processor) that will allow to transform data per block into your application-specific structure - similar to the subgraph.
Squids then are connected to the array of query nodes that cache and serve data to the end-users.
This architecture means excellent scalability and pretty good reindexing times for processors as the most difficult work is already done by indexer.
For example this query on Archive returns last 10 extrinsics that account signed:
query Recent {
substrate_extrinsic(
where: {signer: {_eq: "bXhxg5XurkmitqYGsPwGXQdXxQAKZYEaHvFjbVPY1kpLrW5ek"}},
limit: 10,
order_by: {id: desc}
) {
blockNumber
name
args
}
}
You can play with it here.
I think substrate-archive is the most reliable one when it comes to performance. It is connected directly to the RocksDB, therefore its processing time is way faster than other applications that rely on RPC calls.
Of course it has its own caveats, for example, substrate-archive doesn't store events or results from extrinsics, making it harder to determine if the extrinsics registered in the PostgresDB succeed or not.
Subsquid is also a good indexer but even though it says that it is built on top of substrate archive, it only takes the idea from it, and creates an indexer that is basically querying the node for each block that it is being processed. If you overload the substrate-node with transactions using, for example, sub-flood, the subsquid indexer tends to slow down and it can cause the indexing process to be way behind the latest state of the chain. The good thing about subsquid is that it stores only finalized data, so you don't have to deal with finalization or else.
I think the best solution could be a combination of substrate-archive and maybe another process running next to it to enrich the already indexed information, adding events and extrinsic execution results, for example.
Substrate-archive also seems to be a way to index:
