Real time candle data: An impossibility

I have been trying to get candle data from coinbase for weeks now. The deeper I go, the more broken it seems to be.

Option 1: Advanced trade API Websocket
Here I was trying to get the candles in real time from the websocket feed, but it is extremely inconsistent with the timings. This was actually the only way I could get proper quality data (where the close price matched Coinbase’s real close price), but the delays are tremendous. I left a socket open for a while to see how long it takes for the last update to come for a single 5m candle. At that point I would assume the candle is finalized, and the close price is locked.
Below are histograms for three different coins. Like one can see, STX performed the best where most candles were finalized within the first 20 seconds. There were many cases where the websocket would inconsistently update the candles 30+ seconds after the close. Note that this is still a very pleasant situation compared to the others.

See APT and GMT. Especially with GMT, updates coming more than 5 minutes late for a 5 minute candle. Here is the data:
[candle_timestamp, price, receive_timestamp]
1704337500,0.3036,2024-01-04T03:07:15.55726717Z
1704337500,0.3045,2024-01-04T03:08:25.556554524Z
1704337500,0.3056,2024-01-04T03:13:05.557189749Z
1704337800,0.3055,2024-01-04T03:13:35.556313962Z
1704337800,0.3054,2024-01-04T03:14:55.558373227Z
1704337800,0.3036,2024-01-04T03:20:35.556853355Z <—

Coinbase marked the close price correctly, but how can one get real time data when the candle updates are coming a candle length late? Tradingview marked the close at 0.3054, which by conventional thinking would be correct since that’s the last update before the close.
In conclusion: the websocket does provide correct price data, but the timings are extremely inconsistent.
Oh, and the updates even on the more busy pairs are more or less every 10 seconds, not every second like the API docs state.

Option 2: Exchange / Pro REST:
Here I was trying to just fetch the candles in a more simple way. I ended up noticing that Coinbase does have them built, but they are sent in a VERY inconsistent way. I did a simple pull which checks the latest 1m candle’s timestamp which is received for BTC-USD.
REST_delay

The UTC 10:15 candle is not yet generated, but the even more weird and alarming side is that the API is inconsistently delivering old and even older data. It clearly has the 10:14 candle there, but it chooses to deliver it as the latest one just whenever it wants to. The latest candle is fluctuating between 10:11 and 10:14. Why is this happening? How, even?

Option 3: Build the candles from the trade data
I have the exact same websocket script running for Bybit, Binance and Coinbase. It subscribes to the trade feed and takes the last trade until the candle is closed. After that it listens to a few hundred (270) milliseconds just in case the exchange sends the trades a bit late due to congestion. Understandable. By all logic, the last trade received before the candle closes should be the close price. Below is a of the literal same algorithm’s error rates on LTCUSD (LTCUSDT in Bybit and Binance):
641 candles (28/12/2023 06:00UTC - 03/01/2024 22:00 UTC)
Agg_error_rates

During the weeks I have observed that the more low volume the pair is, the greater the error rate is. It is almost doable to build the candles on something like ETH, but completely impossible to do on very low volume pairs since the official candles are almost random.

Bonus: Sometimes the closing price of a candle is taken from the first trade of the next candle.
The first trade from the next candle’s time period marks the closing price of the previous candle even though the previous candle should have been closed based on the actual last trade. See the picture below.

DOGE-USD 04 Jan 2024
A trade at 09:12:46 UTC 0.08223 is the last received trade of the 1min candle (by Tradingview)
A trade at 09:13:40 UTC 0.08218 is the next received trade and belongs to the next 1min candle
Coinbase closes the 09:12 UTC candle at 0.08218
The next trade is at 09:13:58 UTC 0.08217
Coinbase starts the 09:13 UTC candle at 0.08217, which is the second trade of the minute.
Why? How? The only feasible explanation I would have is that Tradingview receives the trades late and doesn’t check (very unlikely) the timestamp of the trade but marks them to be of whatever candle that’s currently ongoing.

Tradingview’s whole livelihood is wrapped around candles. They literally build candlestick charts and offer them to paying customers. Even they cannot get the candles right when it comes to this data.

At this point I have no proper options to get the data. It’s almost if the API is made like to be inconsistent to cut down algorithmic trading. I wish the developers could give one consistent way to either fetch or build the candles. Anyway, thanks for reading the rant.

1 Like

Asking this the same time in case someone related to the issue reads these:
A trade came through at 15:57:04 UTC on GMT-USD.
It got marked as the closing price of the 15:56 UTC 1min candle. Why?
What is the logic behind this?

Hi @Avada! Apologies for the delay. We get it, this seems weird! But we will try our best to resolve this. Please allow us some time to look into it. Keep in touch!

1 Like

@Avada I appreciate the thorough investigation and attention to detail you presented. I also am scratching my head wondering why it seems impossible to accurately retrieve candle close data in real time.

I am guessing the issues you are seeing in option 2 is data inconsistency between the api servers that are processing your request, i.e server1 might have data for 10:14 but server2 has data only up to 10:11. For some reason there is a big delay in relaying all this information to the necessary api servers.

I settled on an implementation similar to your option 3, and just accepted I have to live with the results for now. After dealing with the coinbase api, I have a greater appreciation for how binance trade feed marks trades with a ‘close’ flag, so you know if this info represents a candle close for the feed you are subscribed to.

1 Like

For @zuzusangry and maybe others who have seen / will see the thread:
The problem was taken under review by the support, and I think I managed to get it further. An engineering team has been looking into it since the 8th of Jan 2024. No ETA at the moment.

1 Like