3

I'm getting the following error while running rococo alice validator node in kubernetes cluster:

cannot bind unix socket: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" } debug_id=prepare program_path="/usr/bin/polkadot" extra_args=["prepare-worker", "--node-impl-version", "0.9.41-e203bfb396e"]
failed to spawn a prepare worker: Bind

Basically, I'm running my own rococo instance with 3 validators in Kubernetes. Everything works fine until the parachain is connected and start collating - then this error occurs in the alice validator node and parachain is stuck on importing new blocks. The validator is still running okay.

P.S. Everything works fine locally so possibly there is some problem with the deployment configuration. I'm using parity helm charts for deployment.

Have someone faced this problem? Any help is appreciated.

Thanks in advance!

UPD: My configuration:

AWS EKS Managed cluster. Kubernetes version: 1.24.

Polkadot relay chain image: polkadot/polkadot:latest (or v0.9.37 either)

Running statefulset node pod with securityContext parameter set to root user(0). No specific permission limitations applied.

2
  • The error states ReadOnlyFilesystem. Looks like your filesystem does not have write permissions. Is your disk full?
    – Bruno
    Apr 13, 2023 at 14:49
  • @Bruno no. As I mentioned above - the validator node itself is working fine and the chain is producing and finalizing blocks correctly. Apr 13, 2023 at 14:58

2 Answers 2

1

The problem is in the helm charts configuration, especially when node.customNodeKey value is defined in custom values provided.

The thing is that it mounts custom node key into the /tmp/ folder with readOnly option set to true. This way the /tmp/ folder in the filesystem becomes read-only and when relay chain wants to spawn a PVF worker for parachain validation - it is using /tmp/ folder for that which is read-only by the default configuration above.

So ideally - the custom node key should be mounted under custom path that shouldn't interfere with such generic paths like /tmp, especially when it is getting restricted to 'read-only`.

2
  • Hi @Oleg Kalenik, how did you solve this
    – lara
    Jul 7, 2023 at 6:14
  • 1
    @lara, the issue was fixed with this PR in helm-charts repo: github.com/paritytech/helm-charts/pull/236. So now it should work out of the box if you pull the latest code from the repo. Jul 8, 2023 at 9:08
0

It's hard to say without knowing more details about what your configuration is, where you're running it, what version of kubernetes, how your cluster is configured, etc.

That being said, it could be that it's trying to spawn a worker on the root filesystem which is likely read-only as managed kubernetes clusters tend to make read only pods by default. The node will work fine because it's using a mounted volume for the database, but that's not where the bind is attempting to be established.

2
  • How can I find out where the worker is going to be spawned? I mean like what’s the default path. I’ve been trying to run validator node with the root account using custom security context but it didn’t help either. Apr 15, 2023 at 8:05
  • I’ve also updated the question adding the eks configuration we’re using. However I’m not sure if it helps since there are no specific restrictions applied - cluster out of the box. Apr 15, 2023 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.