Delaying PING due to remote client data or ping 48s ago

[RitikaLaddha]

Observed behavior

Hi
I am facing below intermittent issue because of that Nats is not being stable, as even if its's connected sometimes, it doesn't respond back
Although we get in log of application, that message has been sent. But still it does not returns message.
We have kept 30s timeout, so we do not receive response in 30s of time.

then we get timeout exception

Nats logs

[1] 2024/08/26 06:32:46.781883 [DBG] 10.0.6.146:45496 - cid:14 - Client Ping Timer
[1] 2024/08/26 06:32:46.781920 [DBG] 10.0.6.146:45496 - cid:14 - Delaying PING due to remote client data or ping 48s ago

Application logs

{"timestamp":"2024-08-28T09:47:48.534Z","sequence":6272,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"com.rockwell.datamosaix.common.beans.N
atsClientAdapter","level":"ERROR","message":"[NATS] Error sending message through NATS server","threadName":"vert.x-eventloop-thread-1","threadId":44,"host
Name":"datamosaix-access-management-795c499bf7-xx7gk","processName":"application.jar","processId":1,"stackTrace":"java.util.concurrent.CancellationExceptio
n: Future cancelled, response not registered in time, check connection status.\n\tat
io.nats.client.support.NatsRequestCompletableFuture.cancelTimedOut(NatsRequestCompletableFuture.java:42)\n\tat
io.nats.client.impl.NatsConnection.lambda$cleanResponses$6(NatsConnection.java:1023)\n\tat
java.base/java.util.concurrent.ConcurrentHashMap.forEach(Unknown Source)\n\tat
io.nats.client.impl.NatsConnection.cleanResponses(NatsConnection.java:1019)\n\tat io.nats.client.impl.NatsConnection$2.run(NatsConnection.java:500)\n\tat
java.base/java.util.TimerThread.mainLoop(Unknown Source)\n\tat java.base/java.util.TimerThread.run(Unknown
Source)\n","errorType":"java.util.concurrent.CancellationException","errorMessage":"Future cancelled, response not registered in time, check connection
status."}
        at io.nats.client.support.NatsRequestCompletableFuture.cancelTimedOut(NatsRequestCompletableFuture.java:42)
        at io.nats.client.impl.NatsConnection.lambda$cleanResponses$6(NatsConnection.java:1023)
        at io.nats.client.impl.NatsConnection.cleanResponses(NatsConnection.java:1019)
        at io.nats.client.impl.NatsConnection$2.run(NatsConnection.java:500)
        at io.nats.client.support.NatsRequestCompletableFuture.cancelTimedOut(NatsRequestCompletableFuture.java:42)
        at io.nats.client.impl.NatsConnection.lambda$cleanResponses$6(NatsConnection.java:1023)
        at io.nats.client.impl.NatsConnection.cleanResponses(NatsConnection.java:1019)
        at io.nats.client.impl.NatsConnection$2.run(NatsConnection.java:500)
        at io.nats.client.support.NatsRequestCompletableFuture.cancelTimedOut(NatsRequestCompletableFuture.java:42)

Nats configuration

server_name: $NATS_SERVER_NAME
listen: 0.0.0.0:4222
http: 0.0.0.0:8222

# Authorization for client connections
authorization {
  user: "nats_client"
  password: "paasss"
  timeout: 6000
}

# Logging options
debug: true
trace: true
logtime: true
# Pid file
pid_file: "/opt/bitnami/nats/tmp/nats-server.pid"

# Some system overrides

# Clustering definition
cluster {
  name: "nats"
  listen: 0.0.0.0:6222
  # Authorization for cluster connections
  authorization {
    user: "nats_cluster"
    password: "paass"
    timeout:  10
  }
  # Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above
  routes = [
    nats://nats_cluster:paass@nats:6222
  ]
}
# JetStream configuration
jetstream: enabled
jetstream {
  store_dir: /data/jetstream
  max_memory_store: 1G
  max_file_store: 8Gi
}

Expected behavior

Application should get response in time

Server and client version

Helm chart used: https://bitnami.com/stack/nats/helm
Kubernetes version: 1.28.12
Application Java version: 21
Nats version: 2.10.18

Host environment

Azure Kubernetes service

Steps to reproduce

No response

[wallyqs]

Thanks for the report @RitikaLaddha, could you share the nats-server version as well (the one used by the nats container pod)? You could also confirm by checking /varz, for example: https://demo.nats.io:8222/varz

[wallyqs]

Also the following configuration will not work ok on k8s, you need to add all the A records from the statefulset to the config or use the nats-io/k8s helm chart:

  routes = [
    nats://nats_cluster:paass@nats:6222
  ]

[RitikaLaddha]

Thanks for the report @RitikaLaddha, could you share the nats-server version as well (the one used by the nats container pod)? You could also confirm by checking /varz, for example: https://demo.nats.io:8222/varz

Hey @wallyqs
Sorry I forgot to add that, I have added just now.
Nats version: 2:10:18

[RitikaLaddha]

Also the following configuration will not work ok on k8s, you need to add all the A records from the statefulset to the config or use the nats-io/k8s helm chart:

  routes = [
    nats://nats_cluster:paass@nats:6222
  ]

can you please specify what are A records we would need to add in config?

[wallyqs]

The routes need the complete dns entries from the stateful set, the nats-io/k8s helm chart does this by default:

  routes = [
    nats://nats-0.nats_cluster:paass@nats:6222
    nats://nats-1.nats_cluster:paass@nats:6222
    nats://nats-2.nats_cluster:paass@nats:6222
  ]
``