Pagination issues in py-substrate-interface

Question

It is not clear how to use pagination in py-substrate-interface. In the sample code below, there are only 32 records so all the query should be complete, but result.loading_complete shows false.

from substrateinterface import SubstrateInterface

url = 'wss://shiden.api.onfinality.io/public-ws'
substrate = SubstrateInterface(url)

result = substrate.query_map('Contracts', 'CodeStorage', block_hash = None, page_size = 100, ignore_decoding_errors = True)
start_key = result.last_key
print(f"loading complete {result.loading_complete} | size {len(result.records)} | next start key {start_key}")

When I do want to query the next page, I am running the same query_map call with the added start_key parameter. Is that correct, because there is also a retrieve_next_page call in the result object, but I can't figure out how to use it.

result = substrate.query_map('Contracts', 'CodeStorage', block_hash = None, page_size = 100, start_key = start_key, ignore_decoding_errors = True)

Here are the relevant components of the results object: ..., 'block_hash', 'current_index', 'ignore_decoding_errors', 'last_key', 'loading_complete', 'max_results', 'module', 'page_size', 'params', 'records', 'retrieve_next_page', 'storage_function', 'substrate']

So unless I am misunderstanding something, loading_complete is not correctly reporting when it is done and retrieve_next_page doesn't seem to do what it implies. Also, the documentation could be improved by adding some examples, especially one that includes pagination. Nonetheless, py-substrate-interface is a very handy tool.

Answer 1

The easiest way to handle query_map is use it as an iterator instead of manual last-key tracking.
For example all accounts of System::Account for Kusama:

from substrateinterface import SubstrateInterface

substrate = SubstrateInterface(
    url="wss://kusama-rpc.polkadot.io",
)

query = substrate.query_map('System', 'Account', page_size=1000)

for (i, (account, account_info)) in enumerate(query):
    print(f"#{i} '{account.value}': {account_info.value['data']['free']}")

You can play around with the page size and see that it prints way more than 1000 accounts.