Started using Distributed data - get ddatareplicator dead letter messages

[giovanni-ferrari]

I started using Distributed Data to store temporary information that should be resilient across the cluster (i'm using distributed data in conjunction with sharding).

Since then I started seeing some [INFO] logs from akka.net, that repeats continuosly.

Sep 27 09:36:07 ubuntu service.service[3831814]: [INFO][09/27/2024 07:36:07.329Z][Thread 0017][akka://Service/user/ddataReplicator] Message [Status] from [akka.tcp://Service@xxx.xxx.xxx.xx0:9997/user/ddataReplicator#43346742] to [akka://Service/user/ddataReplicator] was not delivered. [9] dead letters encountered. If this is not an expected behavior then [akka://Service/user/ddataReplicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: Status(chunk=0, totalChunks=1, digest={Data1->Google.Protobuf.ByteString,Data2->Google.Protobuf.ByteString,Data3->Google.Protobuf.ByteString,})

Any idea why this is happening, and if it can be related to any configuration issue?

[Aaronontheweb]

Is the replicator not running on your current node? Because that's why you're getting the dead letter - the actor at your local path akka://Service/user/ddataReplicator doesn't exist. If the replicators are syncing based on role types, the issue might be that the replicator on xxx.xxx.xxx.xx0:9997 expects there to be one running locally on your other node as well, but there isn't one. If you didn't specify a role setting for the replicator, then that would also cause this issue - no role means everyone gets all the the replication messages.