Handling WebSocket Disconnects
The exchange keeps processing trades while a WebSocket client is disconnected. After reconnecting and resubscribing, the client starts receiving new messages. Unless the exchange also replays the missed messages, the outage leaves a gap that has to be filled from a history endpoint.
A price display can recover as soon as a fresh quote arrives. A volume calculation built from individual trades needs the missing trades as well. The same distinction applies to candles: a historical bar can restore a chart, but it can't replace the individual trades in a trade recorder.
This means the connection can be working before recovery is finished. The application can update a price label while keeping calculations that need the missing history paused. A readiness check keeps that decision separate from the socket's connection status.
When an SDK already reconnects and resubscribes, its lifecycle events are the place to start recovery. A second reconnect loop can open duplicate sockets and subscriptions. The package's examples document the event names and subscription responses.
Detecting Missing Messages
A disconnect is an obvious reason to check for missing data. A full message queue or failed database write can lose data locally even while the socket stays open. Sequence gaps and failed integrity checks can expose other problems, where the channel provides them. An integer ID alone isn't enough: some IDs aren't consecutive, and some are only meaningful within one symbol or connection.
Silence is harder to interpret. An idle market may have no trades to report. Heartbeats help detect a dead connection, but they don't confirm that a particular subscription is receiving messages.
That requires checking the subscription response. A rejected subscription fails the recovery attempt immediately. A missing response does the same after a timeout. Some channels send an initial snapshot instead of an acknowledgement. The message handler needs to be attached before subscribing so it can capture that first response and the updates that follow.
Backfilling Missed Trades
Once the subscription is active, new trades continue arriving while REST requests fetch older ones. A queue holds those live messages until backfill catches up. Processing them immediately could put later trades into a calculation before the missing earlier trades.
The starting point is the cursor saved with the last stored record. The last received message is less useful: it may still be waiting for a database write when the connection drops or the process crashes.
The backfill also needs an end. A trade already in the live queue can provide one, if the REST endpoint returns the same records and can retrieve everything through that trade. Fixing this endpoint before pagination starts prevents the request from continually chasing a moving live feed. For a quiet channel, the API needs another documented way to establish that the requested range is complete. A local receive timestamp isn't automatically a valid history cursor.
Paginating Trade History
Binance Spot aggregate-trade history returns at most 1,000 records per response. Its fromId and time bounds are inclusive, so requests can overlap. The matching WebSocket feed is the aggregate-trade stream; individual trades use different IDs.
Pagination has to follow the endpoint's cursor and sort order. Adding one millisecond to the last timestamp looks convenient, but can skip trades that share that timestamp. A short or empty page can also be misleading if REST history is behind the WebSocket feed. It only proves completion when the endpoint documents that behaviour.
Recovery can fail because the exchange no longer retains the trades or the account cannot access them. That leaves a gap to record, rather than a request to retry indefinitely. Delayed history can be retried, but within a time limit and request budget that respects the exchange's rate limits. Repeated pages, a cursor that stops advancing, or a live queue that exceeds its limit are reasons to stop the attempt with an error. Calculations that need the missing trades remain paused.
Saving Backfill Progress
A crash between writing records and saving the cursor creates another source of duplicates. The records are in the database, but the old cursor causes the next run to fetch them again. A unique key per trade makes that replay safe.
Writing the cursor first creates a different failure: a crash can leave the cursor pointing past records that were never saved. A transaction that commits records and cursor together avoids both partial-write cases. Separate stores need a restart procedure that tolerates replay without skipping or double-applying records.
Replaying history also shouldn't repeat its side effects. A backfilled trade can update a volume calculation without causing an old signal to submit a new order or send another notification.
Merging History and Live Trades
Consider a recorder with trade A already stored. After reconnecting, it buffers live trades D, E, and F. Its history endpoint returns every trade through E in order, and the live queue contains the complete ordered stream from D onwards. The letters label records in this example; they aren't exchange sequence numbers.
| Input | Records | Stored after processing |
|---|---|---|
| Previous run | A | A |
| First history page | A, B, C | A, B, C |
| Second history page | C, D, E | A, B, C, D, E |
| Live queue | D, E, F | A, B, C, D, E, F |
A and C repeat at history boundaries. D and E arrive through both REST and the WebSocket. They still appear only once in storage. The important assumption is that the history endpoint returned every trade through E. Removing duplicates wouldn't reveal a trade that neither source returned.
The deduplication key comes from the exchange's record IDs and their scope. A fill ID may also need an account and product. Price and timestamp aren't sufficient because different fills can share both. An order ID identifies an order, so using it to discard every later order update would lose valid state changes. String IDs also avoid rounding values outside JavaScript's safe integer range. The deduplication guide covers storage choices.
The same queue handles messages arriving during database writes. If G arrives while F is being saved, the processing loop picks it up next. Keeping the final queue check and switch to normal processing in that loop avoids a gap between the two operations. Limits on queued message count, bytes, and age depend on available memory and how much delay the consumer can tolerate. Exceeding those limits stops the attempt instead of silently discarding trades.
Handling Repeated Disconnects
A second disconnect can happen before the first backfill finishes. That leaves an old REST request running alongside a new recovery attempt. If the old response updates current state or marks the feed ready, it can undo the newer attempt's work.
Each attempt therefore gets an ID. A response is applied only if its ID still matches the active attempt for that feed or account. AbortController can cancel supported requests, but the ID check remains necessary when a response completes before cancellation takes effect.
Rebuilding Order Books
An order book needs a current snapshot and the updates after it. The timing matters: fetching a snapshot and then subscribing can miss changes between the two operations. Buffering updates before fetching the snapshot closes that gap, provided the protocol identifies where the snapshot ends and subsequent updates begin.
Binance Spot's depth-stream procedure uses this approach. The snapshot's lastUpdateId and each event's U/u range determine which buffered updates to discard, where to start applying updates, and how to detect later gaps. The snapshot also has a depth limit: price levels outside it remain unknown until they change.
Bybit's order-book stream supplies a snapshot after subscription, so its startup procedure is different. Every new snapshot replaces the local book. The docs identify u = 1 as a service-restart snapshot, and the level-one feed sends snapshots only.
These differences prevent a single “fetch REST, then replay everything” implementation from working across all order-book feeds. The snapshot and ordering rules belong to the exchange and channel.
Recovering Account State
During normal operation, accepted private events update the corresponding entries in Account State, and consumers read those cached values directly. A REST read supplies a required value when it is missing at startup or needs recovery. Healthy updates don't need another REST request to confirm each change.
The stream's fields determine what that cache can answer. Bybit's wallet stream, for example, sends no initial snapshot and doesn't emit an event just because unrealised PnL changes. The application may need an initial wallet read, but a recent wallet event still doesn't establish current marked equity. Wallet balance, equity, and funds available for another order have different meanings.
Tracking Order Progress
For derivatives with exchange-provided position rows, accepted position updates provide the current position. The matching order's status and cumulative filled quantity describe its progress. Bybit exposes those as orderStatus and cumExecQty on its order stream. Individual fills supply execution prices, fees, and settlement details for accounting; their arrival needn't block an action that already has the position and order facts it requires.
Consider a single two-unit order opening a position from flat. It reports Filled, but the position cache still shows a one-unit partial update. A consumer that needs the resulting position waits for the matching quantity. If the order was instead cancelled after filling one unit, and the cache already shows that one unit, requiring another position event after cancellation could leave the consumer waiting unnecessarily. Checking the quantity and order status resolves the distinction.
This depends on the product's data. Products without native position rows may need executions, fees, and account events to derive their position. An accepted position message containing the current size can replace that cached field. An order-book delta usually needs the preceding updates to produce the right book.
Resolving Unknown Orders
A disconnect adds another possibility: an order submission reached the exchange, but its response never reached the application. The exchange can accept and fill the order while the application is still waiting. Sending it again can create a duplicate.
Saving the Order Intent and a supported client order ID before submission gives recovery something to look up. Order-status or order-history requests can establish the order's state; execution history supplies missing fill details when those are needed. Open orders alone aren't enough because an order that filled during the outage may no longer appear there. An inconclusive lookup leaves the order marked as unknown and prevents an automatic retry.
Balances, positions, orders, and executions may also come from separate REST endpoints with no shared sequence number. Those responses can describe different moments. Applying every buffered private event over them without checking the endpoint's update rules can count an update twice or overwrite newer state. The Exchange State Management workflow uses scoped recovery to repair the affected state, then returns to incoming events and direct cache reads.
Checking Data Freshness
An old position-update timestamp is expected if the position hasn't changed. Giving every cached value a 30-second expiry would make an unchanged position disappear from ordinary reads and trigger requests even while its stream remained healthy. Connection health, subscription status, known gaps, and conflicting updates determine whether private state needs REST Hydration.
A calculation can have its own requirement for price age. Exchange timestamps show when events occurred; receive timestamps show when they reached the application. A calculation can reject a quote that is too old for its purpose without deleting the cached value or making the price getter depend on unrelated account state. Missing trade history is a separate reason to pause calculations that need it. Display updates can be batched separately from processing the underlying messages.
Recovery Diagnostics
A stalled recovery is easier to diagnose when the logs include the affected symbol or account, attempt ID, last saved cursor, current history page, queued-message count, and elapsed time. Those fields show whether history retrieval or queue processing has stopped. Rejected subscriptions, repeated pages, conflicting copies of an immutable record, and discarded responses from older attempts should be visible as well. Credentials and sensitive account data stay out of the logs.
During shutdown, the application stops new calculations and order submissions, prevents late callbacks from restarting them, finishes pending writes, and saves recovery progress before closing clients. The next process resumes from that saved cursor and checks any orders whose outcome is still unknown.
The SDK examples provide a starting point for each channel. Queues, timeouts, and attempt tracking can be shared across integrations. Pagination, record IDs, ordering, and snapshot handling depend on the channel's protocol. Pin the SDK version used by the application and review release notes before upgrading.
A useful final check is a reconnect through the application's normal entrypoint, using demo or testnet for order flows where supported. A standalone socket example can verify message handling while missing a problem in the application's cache, sizing, or order-management code.
Related articles
Continue from here