The primary thing that nodes look for is a common
ProtocolID, this is what peers at the LibP2P layer your node will try and communicate with at all.
This is best outlined in this issue to replace
protocolId with the genesis hash, where failure to set a unique ID can lead to nodes of different networks (even IPFS nodes) start to communicate but because they are incompatible protocols above this, just case noise and bloat your node's bandwidth.
One mitigation here is traditional: use firewalls to ensure that you are screening out connections you do not want. Substrate in itself only protects what happens once it receives messages, and should not be tasked with more resource intensive tasks like DOS protection. Also a good firewall and opsec will make it much harder for your node to be compromised in general, where you likely store your keys for nodes' network ID and possibly validator associated signing keys.