[bug]: payment attempts inflight for old payments

[bitromortac]

Background

In some goroutine dumps investigated recently (see for example #8125 (comment)) it was found that payment results are being collected for old payments that have been timed out long ago (I have some of those on a node as well), without them being failed properly. Somehow the switch does not give back any result for cltv-timed out payments when collectResult is called, which leads to creation of goroutines for each of those unhandled attempts, see

goroutine 31252721 [select, 2742 minutes]:
github.com/lightningnetwork/lnd/routing.(*shardHandler).waitForShard(...)
	github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:374
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).resumePayment(0xc022c07c98)
	github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:216 +0x805
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).sendPayment(0xc00082ed20, 0x0, {0x97, 0x48, 0xf5, 0x3f, 0x26, 0x77, 0x11, 0x66, ...}, ...)
	github.com/lightningnetwork/lnd/routing/router.go:2375 +0x152
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).Start.func2(0xc035dbc180)
	github.com/lightningnetwork/lnd/routing/router.go:673 +0x2c8
created by github.com/lightningnetwork/lnd/routing.(*ChannelRouter).Start
	github.com/lightningnetwork/lnd/routing/router.go:631 +0x62e

goroutine 31222704 [select, 2742 minutes]:
github.com/lightningnetwork/lnd/routing.(*shardHandler).collectResult(0xc0002a1030, 0xc00ecf9760)
	github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:605 +0x41c
github.com/lightningnetwork/lnd/routing.(*shardHandler).collectResultAsync.func2()
	github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:507 +0x57
created by github.com/lightningnetwork/lnd/routing.(*shardHandler).collectResultAsync
	github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:503 +0x11f

Related issues are

• [bug]: trackpayment shows payment IN_FLIGHT but logs shows failed: no_route #8178
• [bug]: LND consumes a lot of memory during start up, subsequently releasing when start up is successful. #7829
• Payment status: IN_FLIGHT (stuck) - (updated) -> v0.13.1-beta.rc2 #5396
• Payment status: IN_FLIGHT (stuck) #6249
• LND has some very old inflight payments without associated HTLC  #6834
• [bug]: In flight status for payment without HTLCs #7697
  where log lines [INF] CRTR: Resuming payment shard appear (but probably not the main cause of the issues).

Your environment

• version of lnd v0.17.0

Steps to reproduce

It is unclear how to reproduce, it may happen if there are many payments in flight and the node is restarted.

Expected behaviour

Old attempts are canceled back properly, payments marked failed such that no new payment loops are created.

Actual behaviour

Payment loops are started for old payments that are still of status in flight but not failed, leading to long-lived goroutines.