---
title: "Deduplicating WebSocket Messages in Node.js"
description: "Deduplicating exchange WebSocket messages in Node.js, with examples of trade IDs, repeated order updates, bounded caches, and database writes."
canonical: "https://siebly.io/blog/websocket-message-deduplication-in-nodejs-a-practical-guide"
---

# Deduplicating WebSocket Messages in Node.js

Deduplicating exchange WebSocket messages in Node.js, with examples of trade IDs, repeated order updates, bounded caches, and database writes.

## Technical Disclaimer

These articles are software engineering references for exchange API integrations. They are not financial, investment, legal, tax, compliance, or trading advice. Use public data, demo, testnet, or paper workflows first. Keep API credentials out of frontend code and disable withdrawal permissions for automation keys.

## Duplicate Trades {#overview}

A trade recorder receives a trade over WebSocket and saves it. After a disconnect, a history request returns the same trade again. Inserting both copies doubles its contribution to volume, even though the exchange only executed it once.

<span id="why-websocket-message-deduplication-in-nodejs-needs-explicit-event-identity"></span><span id="can-websocket-deliver-the-same-message-more-than-once"></span>The duplicate can come from overlapping history pages, two active subscriptions, or a retry after a database timeout. A reconnect doesn't necessarily replay old messages. Often the overlap comes from the application's own recovery work, described in [WebSocket Reconnects and Backfill](/blog/crypto-websocket-reliability-account-state#history-live-handoff).

<span id="key-takeaways"></span>For an individual trade, a stable trade ID gives the recorder something to compare. Order updates need different treatment: the same order ID appears when an order opens, partially fills, and finishes. Discarding everything after the first message would leave the order permanently open in the cache.

## Choosing Event Keys {#build-a-reliable-deduplication-key-from-websocket-event-semantics}

<span id="what-counts-as-a-duplicate-websocket-message"></span>Consider these simplified trade records. The IDs are illustrative, and all three deliveries belong to the same exchange, market, and symbol.

| Delivery | Trade ID | Price | Quantity | Result |
| --- | --- | --- | --- | --- |
| Live message | `trade-A` | `65000` | `0.001` | Store |
| Next live message | `trade-B` | `65000` | `0.001` | Store |
| History response | `trade-A` | `65000` | `0.001` | Already stored |

`trade-A` and `trade-B` happen to have identical prices and quantities. They are still separate trades. A price-and-quantity key would lose `trade-B`; adding a timestamp doesn't fix this when multiple trades share that timestamp.

<span id="what-is-the-best-deduplication-key-for-a-websocket-message"></span>The ID's meaning depends on the stream. [Bybit public trades](https://bybit-exchange.github.io/docs/v5/websocket/public/trade) carry the trade ID in `i`. Private [execution messages](https://bybit-exchange.github.io/docs/v5/websocket/private/execution) carry an `execId` for each execution, and one message can contain several executions for the same order. The loop needs to process each record in `data`.

<span id="spot-trades-on-binance"></span>[Binance Spot](https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md#aggregate-trade-streams) uses `t` for an individual trade and `a` for an aggregate trade. Those are different record types. Comparing their numeric IDs or matching their prices won't merge the streams correctly.

The stored key includes the exchange, environment, product, record type, symbol, and record ID. Private executions also need an account scope. REST and WebSocket copies of the same record must produce the same key, so the transport name and a newly created socket ID don't belong in it. Adding the connection ID would make an old trade look new after every reconnect.

<span id="normalize-events-before-computing-identity"></span>Keep textual IDs as strings. If an API sends numeric IDs beyond JavaScript's safe integer range, the JSON parser must preserve them before they become a `Number`. Calling `String()` on an already rounded number cannot restore the missing digits. A record with no usable ID needs an explicit error or separate handling; a payload hash can't establish an identity the schema doesn't provide.

### Updating an Existing Order {#choose-identifiers-without-suppressing-valid-updates}

An order cache uses the order ID to find the row to update. That doesn't make the order ID a suitable key for discarding subsequent events. For a two-unit order, these messages all describe useful progress:

| Order ID | Status | Cumulative filled quantity |
| --- | --- | --- |
| `order-A` | `New` | `0` |
| `order-A` | `PartiallyFilled` | `1` |
| `order-A` | `PartiallyFilled` | `1.5` |
| `order-A` | `Filled` | `2` |

A key of `orderId + orderStatus` still loses the second partial fill. The status hasn't changed, but the filled quantity has. The cache accepts each valid update and stores the reported cumulative quantity. Adding that cumulative quantity to the previous total would count earlier fills again.

[Bybit documents a cancel race](https://bybit-exchange.github.io/docs/v5/websocket/private/order) that can produce two `Filled` updates for one order. One reports the fill; the other reports that cancellation failed because the order had filled. The cached filled quantity remains `2`, while the cancellation result can still be useful. An individual execution record has its own ID for fee and fill accounting. Deduplication also doesn't establish ordering: a delayed `New` update mustn't reopen a filled order. Accepting or rejecting an update still follows the channel's ordering rules and the order's current state.

<span id="awaitable-websocket-requests"></span>A command response is separate again. A successful order-placement request doesn't mean the order has filled, and its request ID doesn't identify a later execution. [Recovering Account State](/blog/crypto-websocket-reliability-account-state#idempotency-and-unknown-outcomes) covers order progress and requests whose outcome is unknown.

The same issue appears in candle feeds. [Bybit kline updates](https://bybit-exchange.github.io/docs/v5/websocket/public/kline) share a candle's `start` while its prices and volume change; `confirm` becomes true when it closes. Keeping the first update and discarding the rest freezes the candle before it is complete.

## Deduplication in Memory {#when-an-in-memory-set-or-map-is-sufficient}

<span id="where-deduplication-belongs-in-a-node-js-pipeline"></span><span id="how-do-you-deduplicate-websocket-messages-in-node-js"></span>A single-process counter can keep recently accepted trade IDs in a `Set`. The example below uses the `i` and `s` fields from Bybit public trades. Its scope is fixed to Bybit mainnet spot trades, and its only effect is incrementing an in-memory counter.

Save it as `deduplicate.mjs` and run `node deduplicate.mjs`. The sample records are synthetic and require no exchange connection.

```js title="deduplicate.mjs"
export function createTradeCounter(maxKeys = 10_000) {
  if (!Number.isSafeInteger(maxKeys) || maxKeys < 1) {
    throw new Error('maxKeys must be a positive safe integer');
  }

  const seen = new Set();
  let count = 0;

  return {
    accept(trade) {
      if (typeof trade?.i !== 'string' || !trade.i ||
          typeof trade.s !== 'string' || !trade.s) {
        throw new Error('Trade requires a string ID and symbol');
      }

      const key = JSON.stringify([
        'bybit', 'mainnet', 'spot', 'trade', trade.s, trade.i,
      ]);
      if (seen.has(key)) return 'duplicate';

      seen.add(key);
      count += 1;
      if (seen.size > maxKeys) {
        seen.delete(seen.values().next().value);
      }
      return 'accepted';
    },
    get count() {
      return count;
    },
  };
}

const counter = createTradeCounter();
for (const i of ['trade-A', 'trade-B', 'trade-A']) {
  console.log(i, counter.accept({ i, s: 'BTCUSDT' }));
}
console.log('Count:', counter.count);
```

The output is:

```text title="Output"
trade-A accepted
trade-B accepted
trade-A duplicate
Count: 2
```

The key is a serialized array of explicit fields. JSON encoding keeps the field boundaries intact without relying on a separator that might appear inside an ID. It isn't a hash of the payload.

<span id="implement-and-test-websocket-deduplication-without-hiding-failure-modes"></span>The lookup, insertion, and counter update are synchronous. Another callback in this process can't enter halfway through them. Adding `await saveTrade(trade)` between the lookup and insertion changes that: a second handler can find the key absent while the first write is still pending.

Moving `seen.add()` before the `await` closes that race but creates another problem. If the write fails, the next attempt finds the key and skips a trade that was never saved. The database example below handles this at the write itself.

### Cache Retention {#choose-in-memory-persistent-or-distributed-deduplication-for-nodejs}

<span id="how-long-should-a-websocket-deduplication-key-be-stored"></span>The counter remembers at most 10,000 keys, evicting the oldest accepted key first. That number is an example memory limit, not a guarantee about how far an exchange can replay. With a limit of two, receiving `A`, `B`, `C`, then `A` accepts all four: `C` has already evicted the first `A`.

Restarting the process forgets every key. A time-based expiry has the same limitation for events older than its retention window. A chart that rebuilds from a snapshot may tolerate that; a permanent trade archive cannot rely on this cache to prevent duplicate rows.

## Connecting the Trade Handler {#use-sieblyio-sdks-for-exchange-connectivity-and-keep-deduplication-explicit}

<span id="public-trades-on-bybit"></span><span id="listen-on-the-event-the-package-actually-emits"></span>The [Bybit JavaScript SDK](/sdk/bybit/javascript) delivers public trade batches on `update`. To use the counter with live data, replace everything after the function in `deduplicate.mjs` with this code, and install the package with `npm install bybit-api`:

```js title="Live Bybit spot trades"
const { WebsocketClient } = await import('bybit-api');
const counter = createTradeCounter();
const ws = new WebsocketClient();

ws.on('update', (message) => {
  if (message.topic !== 'publicTrade.BTCUSDT') return;

  for (const trade of message.data) {
    const result = counter.accept(trade);
    if (result === 'accepted') {
      console.log(trade.i, trade.p, trade.v);
    }
  }
});

ws.subscribeV5('publicTrade.BTCUSDT', 'spot');
```

<span id="does-a-websocket-sdk-automatically-prevent-duplicate-messages"></span>The handler is attached before subscription. Its counter survives reconnects for as long as the Node.js process stays alive. The SDK manages the connection and subscription; the counter decides which trade IDs it has seen. This example validates the fields needed for the key, not the complete exchange payload.

Only public spot data is subscribed here. A derivatives feed, testnet connection, or private execution stream needs its own scope and field mapping. The counter is still just an in-memory example; console output doesn't make it a trade archive.

<span id="rate-limit-numbers-you-can-read"></span>If recovery also fetches REST history, request pacing is separate from duplicate detection. The [backfill example](/blog/crypto-websocket-reliability-account-state#backfill-pagination) covers pagination and request limits.

## Saving Trades Once {#when-shared-or-durable-state-is-necessary}

<span id="should-node-js-websocket-deduplication-use-redis-or-an-in-memory-cache"></span>Two Node.js workers each have their own `Set`, so both can accept the same trade. A shared cache can coordinate them, but it still leaves a gap between claiming a key and writing the record. When the intended effect is a database insert, a unique constraint can handle the duplicate at that point.

This PostgreSQL table stores individual public trades. Its primary key uses the same fields across live delivery and history imports. Price and quantity use `numeric` so they can be passed as decimal strings without a JavaScript floating-point conversion.

```sql title="Trade storage"
CREATE TABLE recorded_trades (
  exchange text NOT NULL,
  environment text NOT NULL,
  category text NOT NULL,
  record_type text NOT NULL,
  symbol text NOT NULL,
  trade_id text NOT NULL,
  price numeric NOT NULL,
  quantity numeric NOT NULL,
  PRIMARY KEY (
    exchange, environment, category, record_type, symbol, trade_id
  )
);
```

The parameterized insert uses those columns as its conflict target:

```sql title="Insert or skip a duplicate"
INSERT INTO recorded_trades (
  exchange, environment, category, record_type,
  symbol, trade_id, price, quantity
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (
  exchange, environment, category, record_type, symbol, trade_id
) DO NOTHING
RETURNING trade_id;
```

[PostgreSQL returns a row only when this statement inserts one](https://www.postgresql.org/docs/current/sql-insert.html). A duplicate returns no rows. Two workers can attempt the same insert without first racing through separate `SELECT` and `INSERT` calls.

If the connection drops after the insert commits but before the response arrives, retrying the insert finds the stored trade. If it never committed, the retry can insert it. This assumes a trade record is immutable under that ID. A feed that publishes corrections needs an update policy; blindly ignoring a changed payload would hide the correction.

### Committing Related Changes {#structure-an-idempotent-event-handler}

Suppose the handler also adds each new trade's quantity to a stored volume total. Writing the trade and total separately leaves room for a crash between them. On restart, the duplicate insert is skipped, but the total is still missing that trade.

A transaction can insert the trade and increment the total together. The increment runs only if the insert returned a row, and both changes commit or roll back together. The same approach applies to a saved backfill cursor. A process-local `Set` can reduce redundant database calls, but it cannot replace that transaction.

An HTTP request or notification cannot join a normal database transaction. Recording pending work in the same transaction prevents the work from disappearing between the commit and the external call. A worker can send it later, but a crash after sending and before recording success still causes a retry. The receiver needs an idempotency key or another way to recognise the repeated operation. Deduplicating the incoming trade alone doesn't prevent that second side effect.

<span id="build-deduplication-into-your-event-processing-design"></span>A high duplicate count is also worth investigating. Repeated history boundaries are expected; two unintended live subscriptions are a connection-management bug. Recording the feed scope, duplicate count, and cache evictions helps distinguish them without logging every payload. Missing IDs and failed writes should remain errors rather than being counted as duplicates.

<span id="frequently-asked-questions"></span><span id="test-replay-and-concurrency-cases"></span><span id="how-can-i-test-websocket-deduplication-without-a-live-exchange-connection"></span>A small replay check is enough to start: send the same trade twice, then a different trade with the same price, and confirm that two trades are stored. For a database-backed handler, also check a retry after a failed write and a process restart; no live orders are needed.

## Related articles

- [Event-Driven Trading Design Patterns for Node.js](/blog/event-driven-trading-design-patterns-for-nodejs)
- [Mastering Crypto Order State Management in Node.js and TypeScript](/blog/mastering-crypto-order-state-management-in-nodejs-and-typescript)
- [Pre-Trade Checks and Kill Switches for Trading Bots](/blog/safety-boundaries-for-nodejs-trading-systems)


## Related Siebly Resources

- [WebSocket Reconnects and Backfill](/blog/crypto-websocket-reliability-account-state)
- [Batching WebSocket UI Updates](/blog/websocket-ui-batching-javascript)
- [Bybit JavaScript SDK](/sdk/bybit/javascript)
- [Binance JavaScript SDK](/sdk/binance/javascript)