fork-off-substrate dying at 37% with rpc max size disconnect

Question

I'm trying to get fork-off to work with a pair of custom types(https://github.com/hicommonwealth/edgeware-node-types) and I am getting fork-off-substrate to work with a WS socket instead of a HTTP RPC socket(because i couldnt get the HttpProvider to work with the custom specs):

const { ApiPromise, WsProvider } = require('@polkadot/api');
const { HttpProvider } = require('@polkadot/rpc-provider');
const { spec } = require('@edgeware/node-types');

//const provider = new HttpProvider(process.env.HTTP_RPC_ENDPOINT || 'http://localhost:9933')
// NEW
const provider = new WsProvider(process.env.HTTP_RPC_ENDPOINT);
//....

// Connect
api = await ApiPromise.create({ provider, ...spec });

but it breaks at 37%

HTTP_RPC_ENDPOINT=ws://mynode:9989 npm start
We are intentionally using the HTTP endpoint. If you see any warnings about that, please ignore them.
Custom Schema missing, using default schema.
2022-07-31 15:26:21        API/INIT: RPC methods not decorated: contracts_upload_code, dev_getBlockStats, eth_feeHistory, eth_maxPriorityFeePerGas
api promise ok
Fetching current state of the live chain. Please wait, it can take a while depending on the size of your chain.
 ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 37% | ETA: 117s | 95/256





2022-07-31 15:29:43          API-WS: disconnected from ws://5.9.83.210:9989: 1009:: Maximum message size exceeded.
2022-07-31 15:29:43        RPC-CORE: health(): Health:: disconnected from ws://host:9989: 1009:: Maximum message size exceeded.
/fork-off-substrate/node_modules/@polkadot/rpc-provider/ws/index.cjs:197
        const error = new Error(`disconnected from ${(0, _classPrivateFieldLooseBase2.default)(this, _endpoints)[_endpoints][(0, _classPrivateFieldLooseBase2;
                      ^

Error: disconnected from ws://5.9.83.210:9989: 1009:: Maximum message size exceeded.
    at W3CWebSocket.value (//chain_data/fork-off-substrate/node_modules/@polkadot/rpc-provider/ws/index.cjs:197:23)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/chain_data/fork-off-substrate/node_modules/yaeti/lib/EventTarget.js:107)
    at W3CWebSocket.onClose (/chain_data/fork-off-substrate/node_modules/websocket/lib/W3CWebSocket.js:228:10)
    at WebSocketConnection.<anonymous> (/chain_data/fork-off-substrate/node_modules/websocket/lib/W3CWebSocket.js:201:17)
    at WebSocketConnection.emit (node:events:527:28)
    at WebSocketConnection.drop (/chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:475:14)
    at WebSocketConnection.processFrame (/chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:578:2)
    at //chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:323:40
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

Node.js v17.8.0

Substrate node: https://github.com/edgeware-network/edgeware-node/tree/erup-5-latest

I tried to increase the rpc-max-limit flag, but it doesnt work :/ Does anyone have any ideas on how to solve this?

/tmp/edgeware-node # ./../bc/edgeware-node/target/re
lease/edgeware --chain=edgeware -d datal --rpc-port 9988 --unsafe-rpc-external --rpc-external --ws-external --rpc-cors=all --ws-port 9989 --rpc-methods=Unsafe --name flipchan_testing --max-runtime-instances 256  --ws-max-out-buffer-capacity=9999 --rpc-max-payload=9991024                                

Answer 1

It looks like that fork-off-substrate was not maintained actively.

You could try the subalfred.

It uses WS/Rust/Async. So, it's really fast. For me, it scrapes 200k pairs in 10 seconds (in localhost). If you are connecting to a remote node, the network would be the bottleneck.

./node build-spec --chain xxx-dev --raw > xxx.json

subalfred export-state ws://localhost:9944 --fork-off --simple-governance --skip-pallets Staking --output xxx.json --log subalfred::core::node,subalfred::core::substrate_client

--simple-governance will replace the council and tech.comm member with //Alice.

--skip-pallets could skip some large pallets if you want.

Use subalfred export-state --help to see other options detail.

It needs an xxx.json to generate a runnable spec file. If you just want the storage data. You could use --output <any name> or drop this option (it will export data to a export.json file).