Flush command and PathProcessor periodic flush

[agouin]

Consolidates rly tx relay-packets and rly tx relay-acknowledgements into a single rly tx flush command.

Main reason for this: simplify the operator experience of flushing old messages. The tx relay-packets and tx relay-acknowledgements commands require both the specific pathName and the channelID as arguments. This means if you don't have them memorized, you have to reference the config and do channel queries to determine the args.

The new flush command still allows passing pathName and channelID arguments to isolate the scope of the flush, but it also allows passing the following number of args:

• one arg - only pathName. It will flush all channels allowed by the filter rule for the specific path.
• no args. It will flush all paths in the config on the channels allowed by the filter rule for each path.

This means the most simple operator experience is now rly tx flush without any arguments, and all paths in the config will be flushed. This was the main use case I wanted to cover.

This new command utilizes the PathProcessor by queueing up pending messages that may have occurred before the start query height of the ChainProcessors, then allowing the PathProcessor correlation framework to handle anything that needs to be relayed and then shut down.

This means that if a packet not yet been relayed, both the MsgRecvPacket to the destination chain and MsgAcknowledgement back to the source chain will be relayed in the flush. For packets that have been relayed to the destination chain with MsgRecvPacket but not yet acked back to the source chain, the MsgAcknowledgement will be relayed to the source chain during the flush.

The PathProcessor will terminate once all IBC message caches are clear. Messages are cleared from the caches when they are either successful, or given up on (after 5 retries). Additionally, the command is bounded by a 10 minute timeout.

Logs are printed for each channel that is flushed.

Since optimized flush capabilities were added to the PathProcessor for this, I added a periodic flush. A flush will happen when the relayer first starts up after both chains are in sync, and again on the flush interval (default of 5m, configurable with the --flush-interval/-i flag. This reduces the need for the --block-history flag for the chain processors, and may be worth removing in a follow up PR.

[jtieri]

This looks good to me! 🚀