2

I'm trying to get fork-off to work with a pair of custom types(https://github.com/hicommonwealth/edgeware-node-types) and I am getting fork-off-substrate to work with a WS socket instead of a HTTP RPC socket(because i couldnt get the HttpProvider to work with the custom specs):

const { ApiPromise, WsProvider } = require('@polkadot/api');
const { HttpProvider } = require('@polkadot/rpc-provider');
const { spec } = require('@edgeware/node-types');

//const provider = new HttpProvider(process.env.HTTP_RPC_ENDPOINT || 'http://localhost:9933')
// NEW
const provider = new WsProvider(process.env.HTTP_RPC_ENDPOINT);
//....

// Connect
api = await ApiPromise.create({ provider, ...spec });

but it breaks at 37%

HTTP_RPC_ENDPOINT=ws://mynode:9989 npm start
We are intentionally using the HTTP endpoint. If you see any warnings about that, please ignore them.
Custom Schema missing, using default schema.
2022-07-31 15:26:21        API/INIT: RPC methods not decorated: contracts_upload_code, dev_getBlockStats, eth_feeHistory, eth_maxPriorityFeePerGas
api promise ok
Fetching current state of the live chain. Please wait, it can take a while depending on the size of your chain.
 ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 37% | ETA: 117s | 95/256





2022-07-31 15:29:43          API-WS: disconnected from ws://5.9.83.210:9989: 1009:: Maximum message size exceeded.
2022-07-31 15:29:43        RPC-CORE: health(): Health:: disconnected from ws://host:9989: 1009:: Maximum message size exceeded.
/fork-off-substrate/node_modules/@polkadot/rpc-provider/ws/index.cjs:197
        const error = new Error(`disconnected from ${(0, _classPrivateFieldLooseBase2.default)(this, _endpoints)[_endpoints][(0, _classPrivateFieldLooseBase2;
                      ^

Error: disconnected from ws://5.9.83.210:9989: 1009:: Maximum message size exceeded.
    at W3CWebSocket.value (//chain_data/fork-off-substrate/node_modules/@polkadot/rpc-provider/ws/index.cjs:197:23)
    at W3CWebSocket._dispatchEvent [as dispatchEvent] (/chain_data/fork-off-substrate/node_modules/yaeti/lib/EventTarget.js:107)
    at W3CWebSocket.onClose (/chain_data/fork-off-substrate/node_modules/websocket/lib/W3CWebSocket.js:228:10)
    at WebSocketConnection.<anonymous> (/chain_data/fork-off-substrate/node_modules/websocket/lib/W3CWebSocket.js:201:17)
    at WebSocketConnection.emit (node:events:527:28)
    at WebSocketConnection.drop (/chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:475:14)
    at WebSocketConnection.processFrame (/chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:578:2)
    at //chain_data/fork-off-substrate/node_modules/websocket/lib/WebSocketConnection.js:323:40
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

Node.js v17.8.0

Substrate node: https://github.com/edgeware-network/edgeware-node/tree/erup-5-latest

I tried to increase the rpc-max-limit flag, but it doesnt work :/ Does anyone have any ideas on how to solve this?

/tmp/edgeware-node # ./../bc/edgeware-node/target/re
lease/edgeware --chain=edgeware -d datal --rpc-port 9988 --unsafe-rpc-external --rpc-external --ws-external --rpc-cors=all --ws-port 9989 --rpc-methods=Unsafe --name flipchan_testing --max-runtime-instances 256  --ws-max-out-buffer-capacity=9999 --rpc-max-payload=9991024                                
1
  • 1
    This looks more like an issue for github to me.
    – pepyakin
    Sep 8, 2022 at 12:45

1 Answer 1

6

It looks like that fork-off-substrate was not maintained actively.

You could try the subalfred.

It uses WS/Rust/Async. So, it's really fast. For me, it scrapes 200k pairs in 10 seconds (in localhost). If you are connecting to a remote node, the network would be the bottleneck.

./node build-spec --chain xxx-dev --raw > xxx.json

subalfred export-state ws://localhost:9944 --fork-off --simple-governance --skip-pallets Staking --output xxx.json --log subalfred::core::node,subalfred::core::substrate_client

--simple-governance will replace the council and tech.comm member with //Alice.

--skip-pallets could skip some large pallets if you want.

Use subalfred export-state --help to see other options detail.

It needs an xxx.json to generate a runnable spec file. If you just want the storage data. You could use --output <any name> or drop this option (it will export data to a export.json file).

1
  • Great one, will check it out
    – flipchan
    Aug 15, 2022 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.