Client stops responding after a day or so

Describe the bug
I have client that I need to keep open connection for months. After a day or so I see that the client stops responding. Spinify does not output ANY logs when this situation happens. On server, client is no longer visible in presence. It's not getting unsubscribed, nor disconnected, because I would be aware of that. It's simply not responding any more. The client program is still running though, because I have a timer inside which sends HTTP requests periodically as a health check, so I know its up and running, just Spinify not connecting to the server.

To Reproduce
Run client for two days

Expected behavior
Keep sending pings

Screenshots

Desktop (please complete the following information):

OS: Windows 11, Dart CLI

Additional context

That's strange because, in the tests, I emulate that time range.
Maybe something with the server?
Do you try so on a local computer with a local centrifuge?

Also, you can try to catch any errors and StackTrace with the runZonedGuarded

I have not tried to run in locally for this long, only few hours and they were not in a stand-by mode as production clients mostly are.

Centrifugo config I use is the same for dev and prod environment.

I have over 30 clients connected like this and they behave similarly, some last longer, some stop logging stuff and dropping put of presence earlier, it's really weird.

I did notice though that one client in particular has issues and I am losing him faster. The client runs on a Windows machine with 3G like speed network while also probably really unstable. Manual restart of thst client brings it back to working state for some time.

In which situation would Spinify stop completely logging? The one "zombie" client been running now for 9 hours without connection, still reporting to me with green health status and doing other jobs it needs to do, just no Spinify logs or presence

In which situation would Spinify stop completely logging?

This shouldn't happen because at least we should at least have ping/pong messages.

And without ping/pong, we should make a reconnection.

I'll probably make a simple headless flutter application inside a docker container to track that behavior.

In which situation would Spinify stop completely logging?

This shouldn't happen because at least we should at least have ping/pong messages.

I thought the same, but its happening. Pings just stop and theres no disconnect event/log, whatever, in client's Spinify log.

I have not used runZonedGuarded(), well, it didn't occur to me to wrap it, since client is still running and does not seem to be crashing.

By the way, I am not using Flutter, it's pure dart cli tool for windows, does that change anything?

@FluffyDiscord

@FluffyDiscord for now try to wrap everything at main with runZonedGuarded and collect some logs.

@FluffyDiscord

@FluffyDiscord for now try to wrap everything at main with runZonedGuarded and collect some logs.

Will try. Which logs would you want? I can send you spinify logs tomorrow morning.

This is the implementation. Maybe I have something horribly wrong? (Please excuse any syntax errors, its pain to work with code on a phone)

var client = Spinify(
      config: SpinifyConfig(
        client: (name: "client_name", version: version),
        getPayload: () async => utf8.encode(jsonEncode(Env.apiToken)),
        connectionRetryInterval: (min: Duration(milliseconds: 500), max: Duration(seconds: 5)),
        timeout: Duration(minutes: 1),
        codec: SpinifyProtobufCodec(),
        logger: (level, event, message, _) {
          switch(level) {
            case 0:
              Logger.debug("[WS]: [$event]: $message");
              break;
            case 1:
              Logger.info("[WS]: TRANSPORT [$event]: $message");
              break;
            case 2:
              Logger.info("[WS]: CONFIG [$event]: $message");
              break;
            case 3:
              Logger.info("[WS]: [$event]: $message");
              break;
            case 4:
              Logger.warning("[WS]: [$event]: $message");
              break;
            case 5:
              Logger.error("[WS]: [$event]: $message");
              break;
            case 6:
              Logger.critical("[WS]: [$event]: $message");
              break;
            default:
              Logger.warning("[WS]: Unknown LEVEL $level [$event]: $message");
          }
        },
      ),
    );

    String socketUrl = Env.apiBaseUrl;
    await client!.connect("$socketUrl/connection/websocket");

    Logger.info("connection open");

    String? clientId = client!.subscriptions.server.keys.firstOrNull;
    if(clientId == null) {
      Logger.emergency("backend did not response with subscription, exiting");

       exit(1)
    }
   client!.subscriptions.server[clientId]!.stream.publication().map((event) => utf8.decode(event.data)).listen((payload) async {
// Do stuff
});

Just get the idea, what happened before logs stop occurring.
It should be the last few entries.

Try it with clean centrifugo entry, not a production one.

I applied a workaround. If Spinify did not log anything within a minute, I simply reinitialize whole client. It's dirty, but works as needed. I had to enable history on Centrifugo though, because there was a slim chance that client could miss a message.

@FluffyDiscord

@FluffyDiscord Now I have a time to investigate that issue.
I'll create a Docker Compose with Dart Web + Dart Server + Centrifugo

And try to get that bug.

At test it should work perfectly even for years

Maybe the test is naive, as it's testing in a containerized network? I don't know. I assume it's either something network related or Windows related.

I am first person to happen to run into this weird behaviour. I think you might want to close this issue and maybe wait if another person happens to have similar troubles

@FluffyDiscord

@FluffyDiscord just wait a few days.
I creating a example application with frontend and backend dart.
Backend should push current time to Centrifugo and Client should render clock based on that time.

Current progress would be in that branch: https://github.com/PlugFox/spinify/tree/feature/clock-example

@FluffyDiscord

@FluffyDiscord you can try it by yourself.

https://github.com/PlugFox/spinify/tree/master/example/clock

docker compose up --build

Let's check should it work or not.

Yeap, I think at least I have a problem with reconnects.
I'll take a look when I have time to spare.

Navigation Menu

Repository navigation

Client stops responding after a day or so #18

Screenshots

Additional context

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Client stops responding after a day or so #18

Description

Screenshots

Additional context

Activity

PlugFox commented on Jan 13, 2025

FluffyDiscord commented on Jan 13, 2025

PlugFox commented on Jan 13, 2025

PlugFox commented on Jan 13, 2025

FluffyDiscord commented on Jan 13, 2025

PlugFox commented on Jan 13, 2025

FluffyDiscord commented on Jan 13, 2025

PlugFox commented on Jan 13, 2025

FluffyDiscord commented on Jan 28, 2025

PlugFox commented on Jan 28, 2025

PlugFox commented on Jan 28, 2025

FluffyDiscord commented on Jan 28, 2025

PlugFox commented on Jan 28, 2025

PlugFox commented on Jan 29, 2025

PlugFox commented on Jan 29, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions