Collator node starts failing to import blocks with "database" and "block has an unknown parent" errors after its been running for a while

Question

I'm running collators for both kusama and polkadot network, and both started failing to import blocks after running for a while, these are the logs that I'm getting:

2022-12-13 22:49:39 [Relaychain] 💔 Error importing block 0x6843471352677f914d51277815a5ebc17ba6af7564b6fea1bfdf2b544bd9477f: consensus error: Import failed: Import failed: Database    
2022-12-13 22:49:39 [Relaychain] 💔 Error importing block 0x53c82a1ca03c9ddd3c3cfb6c3715a1a1c1ce2320c5cf8da118f95d5b14e47ea0: block has an unknown parent    
2022-12-13 22:49:39 [Relaychain] Block import error: Database    
2022-12-13 22:49:39 [Relaychain] 💔 Error importing block 0x6843471352677f914d51277815a5ebc17ba6af7564b6fea1bfdf2b544bd9477f: consensus error: Import failed: Import failed: Database    
2022-12-13 22:49:39 [Relaychain] 💔 Error importing block 0x53c82a1ca03c9ddd3c3cfb6c3715a1a1c1ce2320c5cf8da118f95d5b14e47ea0: block has an unknown parent    
2022-12-13 22:49:39 [Relaychain] ⚙️  Syncing  0.0 bps, target=#13344031 (24 peers), best: #9352018 (0x973d…5cf4), finalized #9351680 (0x50f4…b6c1), ⬇ 55.2MiB/s ⬆ 49.2kiB/s    
2022-12-13 22:49:40 [Relaychain] Block import error: Database    
2022-12-13 22:49:40 [Relaychain] 💔 Error importing block 0x6843471352677f914d51277815a5ebc17ba6af7564b6fea1bfdf2b544bd9477f: consensus error: Import failed: Import failed: Database    
^[[A2022-12-13 22:49:40 [Relaychain] Block import error: Database    
2022-12-13 22:49:40 [Relaychain] 💔 Error importing block 0x6843471352677f914d51277815a5ebc17ba6af7564b6fea1bfdf2b544bd9477f: consensus error: Import failed: Import failed: Database    
2022-12-13 22:49:41 failed to associate send_message response to the sender    
2022-12-13 22:49:41 [Parachain] 💤 Idle (0 peers), best: #0 (0xba71…7873), finalized #0 (0xba71…7873), ⬇ 0 ⬆ 0    
2022-12-13 22:49:41 failed to associate send_message response to the sender    
2022-12-13 22:49:41 failed to associate send_message response to the sender    
2022-12-13 22:49:41 failed to associate send_message response to the sender    
2022-12-13 22:49:42 [Relaychain] Block import error: Database    
2022-12-13 22:49:42 [Relaychain] 💔 Error importing block 0x6843471352677f914d51277815a5ebc17ba6af7564b6fea1bfdf2b544bd9477f: consensus error: Import failed: Import failed: Database    
2022-12-13 22:49:42 [Relaychain] 💔 Error importing block 0xbe59ff69c5698389d999ddadf8aa2624994a28656198d8c8e4ccd53619dba4ce: block has an unknown parent    

Does this mean that the database got corrupted? Is there a way to fix or prevent this?

Edited:

The polkadot version being used is: polkadot-v0.9.27. I didn't make any changes to the collators, before the issue started. The parachain the collator is for does not have a slot yet.

It seems that the issue was that the node ran out of disk, how much storage does a collator need, it filled a 50GB disk in a couple of days, is this usual?

Thanks.

Answer 1

It seems that the issue was that the node ran out of disk, how much storage does a collator need, it filled a 50GB disk in a couple of days, is this usual?

It should fill that in a day at most. Even a pruned relay node for Kusama or Polkadot requires more than that.
You should plan to have AT LEAST 250 GB, but 500 would be more sane IMO. The reference hardware recommends a terrabyte to be future proof.
In newer Substrate versions you can opt for a remote relay node, if you have access to a trusted one. This would only require local space for the parachain data, which will minimize the required space.