Ignored block announcement because all validation slots for this peer are occupied

[dmitry-markin]

Sometimes during syncing block announcements from a specific peer start to be getting discarded with a warning

WARN tokio-runtime-worker sync: 💔 Ignored block (<block>) announcement from <peer_id> because all validation slots for this peer are occupied.

This error first appeared during testing of #2944 and is likely introduced by that PR.
It's also a reason for failing sc-consensus-grandpa tests::voter_persists_its_votes test as tracked in #4100.

[dmitry-markin]

As for sc-consensus-grandpa tests::voter_persists_its_votes, the test was flaky at least back to Dec 18, 2023, so it was broken long before #2944. Adding I10-unconfirmed label.

If you come across this issue, add logs and details on the node configuration and frequency of this error happening across runs to this issue, please.