3

I currently have a node running on the cloud and I've set its websocket port to a custom domain following Polkadot's Nginx config (here). I would like to be able to use this node as a bootnode for other nodes in the future. Ideally, I would like to use my custom domain instead of the node's IP. While following yet another Polkadot tutorial (here) I modified my nginx configuration to the following:

server {

        server_name {CUSTOM_DOMAIN};

        root /var/www/html;
        index index.html;

        location / {
          proxy_buffering off;
          proxy_pass http://localhost:30333;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        location /ws {
          proxy_buffering off;
          proxy_pass http://localhost:9944;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
        }

        listen [::]:443 ssl ipv6only=on;
        listen 443 ssl;
        ssl_certificate /etc/ssl/certs/cloudflare.crt;
        ssl_certificate_key /etc/ssl/certs/cloudflare.pem;

        ssl_session_cache shared:cache_nginx_SSL:1m;
        ssl_session_timeout 1440m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;

        ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";

}

I can successfully use my custom domain to access the WebSocket stuff, but when trying to use it for the p2p stuff it does not work.

I've tried setting the --public-addr parameter to my custom domain, running the bootnode like:

./path/to/binary --chain ./chainSpecRaw.json --base-path /tmp/node \
--ws-external --rpc-external --rpc-cors all --name "Bootnode 1" --pruning archive \
--public-addr="/dns/{CUSTOM_DOMAIN}/tcp/443/p2p/{PEER_ID}"

But when I try to run a new node as:

./target/release/contractuo-chain-node --chain ./chainSpecRaw.json --base-path /tmp/node \
    --name "Test Node" --ws-external --rpc-external --rpc-cors all \
    --bootnodes "/dns/{CUSTOM_DOMAIN}/tcp/443/p2p/{PEER_ID}"

It does not manage to find the bootnode. But if I swap the param to --bootnodes "/ip4/{CLOUD_IP}/tcp/30333/ws/p2p/{PEER_ID}" then it connects successfully.

Anyway, I don't know if I'm missing something on the nginx side, or on the node configuration, but I would like to be able to connect the nodes using my custom domain rather than the ip address.

3
  • What is your chainspec file? Did you edit this to include your bootnodes?
    – Nuke
    Commented May 2, 2022 at 21:46
  • Sounds dumb: are you able to dig {CUSTOM_DOMAIN} from the node within this nginx instance? Perhaps DNS isn't working properly.
    – Nuke
    Commented May 3, 2022 at 4:22
  • I am able to dig it and get a response. I do think that the problem is with the nginx config, but I have no idea what it is. Commented May 3, 2022 at 8:10

4 Answers 4

1

You need to modify your chain specification, best place being in the source itself, secondarily in the plain chain spec generated from that source, you can modify it and then convert it to raw. This is outlined in the Custom Chain Specifications How-to Guide.

Polkadot for example sets bootnodes in the node/service/res/polkadot.json file:

{
  "name": "Polkadot",
  "id": "polkadot",
  "chainType": "Live",
  "bootNodes": [
    "/dns/polkadot-connect-0.parity.io/tcp/443/wss/p2p/12D3KooWEPmjoRpDSUuiTjvyNDd8fejZ9eNWH5bE965nyBMDrB4o",
    "/dns/polkadot-connect-1.parity.io/tcp/443/wss/p2p/12D3KooWLvcA24g6sT9YTaQyinwowMbLF5z7iMLoxZpEiV9pSmNf",
    "/dns/polkadot-connect-2.parity.io/tcp/443/wss/p2p/12D3KooWDhp18HYzJuVX2jLhtjQgAhT1XWGqah42StoUJpkLvh2o",
    "/dns/polkadot-connect-3.parity.io/tcp/443/wss/p2p/12D3KooWEsPEadSjLAPyxckqVJkp54aVdPuX3DD6a1FTL2y5cB9x",
    "/dns/polkadot-connect-4.parity.io/tcp/443/wss/p2p/12D3KooWFfG1SQvcPoUK2N41cx7r52KYXKpRtZxfLZk8xtVzpp4d",
    "/dns/polkadot-connect-5.parity.io/tcp/443/wss/p2p/12D3KooWDmQPkBvQGg9wjBdFThtWj3QCDVQyHJ1apfWrHvjwbYS8",
    "/dns/polkadot-connect-6.parity.io/tcp/443/wss/p2p/12D3KooWBKtPpCnVTTzD7fPpCdFsrsYZ5K8fwmsLabb1JBuCycYs",
    "/dns/polkadot-connect-7.parity.io/tcp/443/wss/p2p/12D3KooWP3BsFY6UaiLjEJ3YbDp6q6SMQgAHB15qKj41DUZQLMqD",
    "/dns/p2p.0.polkadot.network/tcp/30333/p2p/12D3KooWHsvEicXjWWraktbZ4MQBizuyADQtuEGr3NbDvtm5rFA5",
    "/dns/p2p.1.polkadot.network/tcp/30333/p2p/12D3KooWQz2q2UWVCiy9cFX1hHYEmhSKQB2hjEZCccScHLGUPjcc",
    "/dns/p2p.2.polkadot.network/tcp/30333/p2p/12D3KooWNHxjYbDLLbDNZ2tq1kXgif5MSiLTUWJKcDdedKu4KaG8",
    "/dns/p2p.3.polkadot.network/tcp/30333/p2p/12D3KooWGJQysxrQcSvUWWNw88RkqYvJhH3ZcDpWJ8zrXKhLP5Vr",
    "/dns/p2p.4.polkadot.network/tcp/30333/p2p/12D3KooWKer8bYqpYjwurVABu13mkELpX2X7mSpEicpjShLeg7D6",
    "/dns/p2p.5.polkadot.network/tcp/30333/p2p/12D3KooWSRjL9LcEQd5u2fQTbyLxTEHq1tUFgQ6amXSp8Eu7TfKP",
    "/dns/cc1-0.parity.tech/tcp/30333/p2p/12D3KooWSz8r2WyCdsfWHgPyvD8GKQdJ1UAiRmrcrs8sQB3fe2KU",
    "/dns/cc1-1.parity.tech/tcp/30333/p2p/12D3KooWFN2mhgpkJsDBuNuE5427AcDrsib8EoqGMZmkxWwx3Md4"
  ],

For reference,The source from How-to Guides

1. Create a plain chain specification

Starting in the working directory of your node's working directory, and assuming the bin is node-template:

./target/release/node-template build-spec > chain-spec-plain.json

We have just generated a plain chain spec file for the default network set in your chain_spec.rs file. This file can be passed to other nodes

2. Modify the plain chain specification (optional)

This optional step we can leverage an existing plain chain specification for a network that otherwise would require modification of the source of the node to run on a new network. For example, this can be quite useful in the Cumulus Tutorial where we want to create a custom relay chain without customizing Polkadot's source.

Here we use the same chain spec, but pass a flag to disable bootnodes, as we want a new network where these nodes will be different.

./target/release/node-template build-spec --chain chain-spec-plain.json --raw --disable-default-bootnode > no-bootnodes-chain-spec-plain.json

This no-bootnodes-chain-spec-plain.json can be used to generate a SCALE storage encoded, distributable raw chain spec.

3. Generate the raw chain specification

With a plain spec available, you generate a final raw chain spec by:

./target/release/node-template build-spec --chain chain-spec-plain.json --raw > chain-spec.json

4. Publish the chain specification

Note: Non-Determinism in Wasm & chain specs: Because Rust -> Wasm optimized builds aren't reproducible, each person will get a slightly different Wasm blob which will break consensus if each participant generates the file themselves. For the curious, learn more about this issue in this blog post.

It is conventional to include the chain specification files for your node within the source code itself so that anyone can build your node in the same way, whereby it becomes easy to check for non-determinism by comparing a genesis blob with another. Polkadot, Kusama, Rococo, and more network chain spec files are found in the source here along with a .gitignore file to ensure that you don't accidentally change these !/*.json files as you build further on your node's software and do runtime upgrades.

5. Start a new node

If you publish a node binary, or have users build their own and then they want to join your network, all then need is the same chain spec file and to run your binary with:

# binary named `node-template`
# `chain-spec.json` obtained from canonical common source
node-template --chain chain-spec.json

This can also simply be configured to be the default network. For reference, you can see how Polkadot implements a default command that uses the chain specs for various networks in the source here

3
  • how is adding it to the chain spec different than passing it as a param when running the node? Commented May 2, 2022 at 23:13
  • 1
    The cli flag appends to the list, it doesn't replace it. github.com/paritytech/substrate/blob/master/client/cli/src/…
    – Nuke
    Commented May 3, 2022 at 4:18
  • But that's exactly my point, if it still adds it to the list then there's no difference. I tried it anyway, but did not work. Commented May 3, 2022 at 8:02
1

The issue here might be that you are just missing /wss

So it should be --bootnodes "/dns/{CUSTOM_DOMAIN}/tcp/443/wss/p2p/{PEER_ID}"

Let us know if this solved your problem!

3
  • tried it both with and without, did not work. I managed to set up the reverse proxy with the IP but in a different port by using Nginx streams. but I could not find a way to set up the stream to use DNS. Commented Jun 3, 2022 at 22:11
  • Are there new errors or information that you can provide so we can get you a proper answer here? Definitely the examples you posted in your main question are missing the /wss, so maybe you need to post updated error messages with the proper commands
    – Shawn Tabrizi
    Commented Jun 4, 2022 at 12:00
  • There aren't exactly error messages. The node just don't find each other. I believe the problem is that, using the DNS, the node can be found, but it replies through its own IP, and the connection can't be established because it should come from the same "origin". Not sure if this makes sense. Commented Jun 4, 2022 at 17:33
0

I don't think your problem comes from DNS but from archive mode. We just faced the same problem with other nodes not syncing in archive mode, using --validator and removing --pruning archive solved the problem. Not sure this is the expected behavior.

The difference is that in archive mode, the node will listen on ws/p2p URI while in collator mode it listens only on p2p. But simply adding ws or wss into bootnode URI doesn't help reaching peers.

As per DNS config, you have a simpler alternative than proxy with nginx:

  • Create a TXT record in DNS _dnsaddr.{SUB_DOMAIN} pointing to "dnsaddr=/ip4/{BOOTNODE_IP}/tcp/30333/p2p/{PEER_ID}".
  • Add "/dnsaddr/{SUB_DOMAIN}/p2p/{PEER_ID}" in chain specs.
0

Maybe it is related to following flag, which was in our case:

      --listen-addr <LISTEN_ADDR>...
          Listen on this multiaddress.
          
          By default: If `--validator` is passed: `/ip4/0.0.0.0/tcp/<port>` and `/ip6/[::]/tcp/<port>`. Otherwise: `/ip4/0.0.0.0/tcp/<port>/ws` and `/ip6/[::]/tcp/<port>/ws`.

You may try with --listen-addr /ip4/0.0.0.0/tcp/30333 --listen-addr /ip4/0.0.0.0/tcp/30334/ws enabled, then for example try to run them in docker-compose and expose this ports:

  node-docker-compose:
    image: node-image
    container_name: node-docker-compose
    restart: always
    ports:
      - "30333:30333"
      - "30334:30334"

Here is sample nginx configuration that forwards wss traffic to ws port:

upstream node_wss {
    server node-docker-compose:30334;
}

server {
    listen 12345 ssl;

    server_name node.example.com;

    root /var/www/html;
    index index.html;

    ssl_certificate     /etc/letsencrypt/live/node.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/node.example.com/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
      resolver 127.0.0.11;

      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $host;
      proxy_pass http://node_wss;
    }
}

TXT record _dnsaddr.node: dnsaddr=/dns/node.example.com/tcp/12345/p2p/12D3KooWSz8r2WyCdsfWHgPyvD8GKQdJ1UAiRmrcrs8sQB3fe2KU

With bootnode specificaton in chainspec.json:

  "bootNodes": [
    "/dnsaddr/node.example.com/p2p/12D3KooWSz8r2WyCdsfWHgPyvD8GKQdJ1UAiRmrcrs8sQB3fe2KU"
  ],

options-ssl-nginx.conf can be found here

I highly recomend to use libp2p dnsaddr instead of dns resolution, due you do not have to modify chain-specs of the nodes if you want to reloacte some of them, all chanes are in DNS setting of the domains.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.