Advanced Trade connection is being dropped for unknown reason

Hi all, I’m writing a python script using websocket.WebSocketApp and I’m getting a successful connection that is being maintained while subscribed to the “market_trades” and “heartbeats” channel under the “wss://advanced-trade-ws.coinbase.com” endpoint. However, after a few hours the connection seems to drop for no reason. The only information I get is “Connection to remote host was lost.” Is there any known reason or fix for this issue? I’m printing every single message I receive from the server and I see no warning or graceful shutdown of the connection. It just drops. I’ll post some of my code below.

def on_open(ws):
    try:
        print("Websocket opened.")
        channel = 'market_trades'
        product_ids = ['ETH-USD']
        # Get the current UNIX timestamp as a string
        timestamp = str(int(time.time()))

        # Concatenate the timestamp, channel, and product IDs
        message = timestamp + channel + ','.join(product_ids)

        # Create a signature
        signature = hmac.new(api_secret.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()

        subscribe_message = {
            'type': 'subscribe',
            'product_ids': product_ids,
            'channel': channel,
            'signature': signature,
            'api_key': api_key,
            'timestamp': timestamp,
        }

        ws.send(json.dumps(subscribe_message))

        channel = "heartbeats"
        timestamp = str(int(time.time()))

        # Concatenate the timestamp, channel, and product IDs
        message = timestamp + channel + ','.join(product_ids)

        # Create a signature
        signature = hmac.new(api_secret.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()

        subscribe_message = {
            "type": "subscribe",
            "product_ids": product_ids,
            "channel": channel,
            "signature": signature,
            "api_key": api_key,
            "timestamp": timestamp
        }
        ws.send(json.dumps(subscribe_message))
    except Exception as e:
        print("Exception caught: ", e)
        traceback.print_exc()
        raise Exception

trade_aggregator = TradeAggregator(BASE_SECONDS, TIMEFRAMES)

def on_message(ws, message):
    trade_aggregator.on_message(message)

def on_error(ws, error):
    print(f"We got an error: {error}")
    trade_aggregator.keepalive = False
    sys.exit(1)

def on_close(ws):
    print("### connection is closed ###")
    trade_aggregator.keepalive = False
    sys.exit(1)


if __name__ == "__main__":
    ws = websocket.WebSocketApp("wss://advanced-trade-ws.coinbase.com",
                                on_open=on_open,
                                on_message=on_message,
                                on_error=on_error,
                                on_close=on_close)
    ws.run_forever()

In the meantime I’ll attempt to modify the on_close() and on_error() functions to immediately reopen the connection and just hope that I don’t lose very much data.

Hi @idufresne! Thank you for using Advanced Trading API. We understand that you are facing a disconnection issue with our websockets. We are looking into it and will assist you further. Keep in touch.

I updated my code to automatically reconnect whenever the connection drops and log any errors. Here’s the resulting log file:

2023-07-26 10:01:30,453 - INFO - Websocket connected
2023-07-26 11:15:06,015 - INFO - 2023-07-26 11:15:06 - Error: Connection to remote host was lost.
2023-07-26 11:15:06,253 - INFO - Websocket connected
2023-07-26 13:44:35,799 - INFO - 2023-07-26 13:44:35 - Error: Connection to remote host was lost.
2023-07-26 13:44:36,611 - INFO - Websocket connected
2023-07-26 21:12:39,764 - INFO - 2023-07-26 21:12:39 - Error: Connection to remote host was lost.
2023-07-26 21:12:39,930 - INFO - Websocket connected
2023-07-27 05:01:30,721 - INFO - 2023-07-27 05:01:30 - Error: Connection to remote host was lost.
2023-07-27 05:01:30,996 - INFO - Websocket connected
2023-07-27 05:58:01,220 - INFO - 2023-07-27 05:58:01 - Error: Connection to remote host was lost.
2023-07-27 05:58:01,386 - INFO - Websocket connected

You can see that the connection is able to reconnect within a second, so the data loss is pretty minimal, but still sub optimal. Especially if this persists with the level2 connection and I want to maintain a copy of the order book.

@idufresne , Have you looked at the messages, do you receive the heartbeats every second? I am implementing a connector as well and trying to learn form others experience
Thanks

I am printing every message I receive. At the time of disconnect there is no message from the server. The connection just ends. I am getting the heartbeat messages, but they are nowhere near every second as advertised. They appear to occur every 30 seconds. The only information I am given regarding the disconnect is what is logged above. “Connection to remote host was lost.”

The doc does mention that the websocket dies after 60s or so, so if the heartbeats does not actually work as advertised and miss on of the 30s ticks. This kind of calls for a periodic websocket ping instead of using heartbeats. Hopefully, they’ll come back to you with more info
Thanks

Hey @idufresne! To look further into this, our team asked for a few details. Would you please share your registered email id and your IP via an inbox message? This will help them to investigate things further.

Sure thing. I’m not seeing a way to message people on this site though, so you may have to message me so I can follow the notification.

Hi @idufresne Thank you for your patience. Sorry that it is taking a little longer. All the details have been shared. We will share an update as soon as possible. Keep in touch.

Any update here @Loop_11 ? I am seeing this behavior as well. When the connection drops, I receive a web socket close code of 1006 without an close/error message.

@Loop_11 - Could be related, I get:
ConnectionResetError("Cannot write to closing transport") thrown by aiohttp/http_websocket.py

I establish a connection, immediately subscribe to heartbeats then later subscribe to a product_id when all the downstream pipe tasks have started