This was partially only a logging issue, that logged this in the collator protocol in _some_ occassions where it should not have logged the message.

The issue of fluctuating block times is a known issue and is planned to be fixed with the v0.9.19 release (at the time of writing), also see https://github.com/paritytech/polkadot/issues/4911