Akka.Tests.Delivery.ReliableDeliveryRandomSpecs.ReliableDelivery_with_random_failures_must_work_with_flaky_network_without_resending failure

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5,14
Which Akka.NET Modules? Akka

Describe the bug

After two minutes Akka.Tests.Delivery.ReliableDeliveryRandomSpecs.ReliableDelivery_with_random_failures_must_work_with_flaky_network_without_resending was unable to complete successfully. Full log is attached.

This looks like a legitimate race condition with the feature itself, rather than the test testing it.

To Reproduce

Unknown, but the test can reproduce it periodically. Logs are attached:

Akka.Tests.Delivery.ReliableDelivery_with_random_failures_must_work_with_flaky_network_without_resending.log

[Aaronontheweb]

Found this issue after I relaxed the log levels:

[DEBUG][01/03/2024 20:31:42.040Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [11], requested [21], current [31]
[ERROR][01/03/2024 20:31:42.041Z][Thread 0031][akka://test/system/log-test] Received a malformed formatted message. Log level: [DebugLevel], Template: [Expanded requestedSeqNr from [{0}] to [{1}], because current [{3}] and all were probably lost.], args: [Akka.Event.LogValues`3[System.Int64,System.Int64,System.Int64]]
Cause: System.FormatException: Received a malformed formatted message. Log level: [DebugLevel], Template: [Expanded requestedSeqNr from [{0}] to [{1}], because current [{3}] and all were probably lost.], args: [Akka.Event.LogValues`3[System.Int64,System.Int64,System.Int64]]
 ---> System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
   at System.Text.ValueStringBuilder.AppendFormatHelper(IFormatProvider provider, String format, ReadOnlySpan`1 args)
   at System.String.FormatHelper(IFormatProvider provider, String format, ReadOnlySpan`1 args)
   at System.String.Format(String format, Object[] args)
   at Akka.Event.DefaultLogMessageFormatter.Format(String format, IEnumerable`1 args) in C:\Repositories\olympus\akka.net\src\core\Akka\Event\DefaultLogMessageFormatter.cs:line 28
   at Akka.Event.LogMessage`1.ToString() in C:\Repositories\olympus\akka.net\src\core\Akka\Event\LogMessage.cs:line 79
   at System.Runtime.CompilerServices.DefaultInterpolatedStringHandler.AppendFormatted[T](T value)
   at Akka.Event.LogEvent.ToString() in C:\Repositories\olympus\akka.net\src\core\Akka\Event\LogEvent.cs:line 96
   at Akka.TestKit.Xunit2.Internals.TestOutputLogger.HandleLogEvent(LogEvent e) in C:\Repositories\olympus\akka.net\src\contrib\testkits\Akka.TestKit.Xunit2\Internals\Loggers.cs:line 46
   --- End of inner exception stack trace ---
   at Akka.TestKit.Xunit2.Internals.TestOutputLogger.HandleLogEvent(LogEvent e) in C:\Repositories\olympus\akka.net\src\contrib\testkits\Akka.TestKit.Xunit2\Internals\Loggers.cs:line 56
   at lambda_method20(Closure, Object, Action`1, Action`1, Action`1, Action`1, Action`1)
   at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture`6.Handle(T value) in C:\Repositories\olympus\akka.net\src\core\Akka\Util\MatchHandler\PartialHandlerArgumentsCapture.cs:line 216
   at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 78
   at Akka.Actor.ReceiveActor.OnReceive(Object message) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\ReceiveActor.cs:line 73
   at Akka.Actor.UntypedActor.Receive(Object message) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\UntypedActor.cs:line 26
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\ActorBase.cs:line 214
   at Akka.Actor.ActorCell.ReceiveMessage(Object message) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 185
   at Akka.Actor.ActorCell.Invoke(Envelope envelope) in C:\Repositories\olympus\akka.net\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 87

This shouldn't be affecting the participating actors from doing their processing, but it's worth resolving anyway.