Deduplicating WebSocket Messages in Node.js

A trade recorder receives a trade over WebSocket and saves it. After a disconnect, a history request returns the same trade again. Inserting both copies doubles its contribution to volume, even though the exchange only executed it once.

The duplicate can come from overlapping history pages, two active subscriptions, or a retry after a database timeout. A reconnect doesn't necessarily replay old messages. Often the overlap comes from the application's own recovery work, described in WebSocket Reconnects and Backfill.

For an individual trade, a stable trade ID gives the recorder something to compare. Order updates need different treatment: the same order ID appears when an order opens, partially fills, and finishes. Discarding everything after the first message would leave the order permanently open in the cache.

Consider these simplified trade records. The IDs are illustrative, and all three deliveries belong to the same exchange, market, and symbol.

trade-A and trade-B happen to have identical prices and quantities. They are still separate trades. A price-and-quantity key would lose trade-B; adding a timestamp doesn't fix this when multiple trades share that timestamp.

The ID's meaning depends on the stream. Bybit public trades carry the trade ID in i. Private execution messages carry an execId for each execution, and one message can contain several executions for the same order. The loop needs to process each record in data.

Binance Spot uses t for an individual trade and a for an aggregate trade. Those are different record types. Comparing their numeric IDs or matching their prices won't merge the streams correctly.

The stored key includes the exchange, environment, product, record type, symbol, and record ID. Private executions also need an account scope. REST and WebSocket copies of the same record must produce the same key, so the transport name and a newly created socket ID don't belong in it. Adding the connection ID would make an old trade look new after every reconnect.

Keep textual IDs as strings. If an API sends numeric IDs beyond JavaScript's safe integer range, the JSON parser must preserve them before they become a Number. Calling String() on an already rounded number cannot restore the missing digits. A record with no usable ID needs an explicit error or separate handling; a payload hash can't establish an identity the schema doesn't provide.

A single-process counter can keep recently accepted trade IDs in a Set. The example below uses the i and s fields from Bybit public trades. Its scope is fixed to Bybit mainnet spot trades, and its only effect is incrementing an in-memory counter.

Save it as deduplicate.mjs and run node deduplicate.mjs. The sample records are synthetic and require no exchange connection.

export function createTradeCounter(maxKeys = 10_000) {
  if (!Number.isSafeInteger(maxKeys) || maxKeys < 1) {
    throw new Error('maxKeys must be a positive safe integer');
  }

  const seen = new Set();
  let count = 0;

  return {
    accept(trade) {
      if (typeof trade?.i !== 'string' || !trade.i ||
          typeof trade.s !== 'string' || !trade.s) {
        throw new Error('Trade requires a string ID and symbol');
      }

      const key = JSON.stringify([
        'bybit', 'mainnet', 'spot', 'trade', trade.s, trade.i,
      ]);
      if (seen.has(key)) return 'duplicate';

      seen.add(key);
      count += 1;
      if (seen.size > maxKeys) {
        seen.delete(seen.values().next().value);
      }
      return 'accepted';
    },
    get count() {
      return count;
    },
  };
}

const counter = createTradeCounter();
for (const i of ['trade-A', 'trade-B', 'trade-A']) {
  console.log(i, counter.accept({ i, s: 'BTCUSDT' }));
}
console.log('Count:', counter.count);

The output is:

trade-A accepted
trade-B accepted
trade-A duplicate
Count: 2

The key is a serialized array of explicit fields. JSON encoding keeps the field boundaries intact without relying on a separator that might appear inside an ID. It isn't a hash of the payload.

The lookup, insertion, and counter update are synchronous. Another callback in this process can't enter halfway through them. Adding await saveTrade(trade) between the lookup and insertion changes that: a second handler can find the key absent while the first write is still pending.

Moving seen.add() before the await closes that race but creates another problem. If the write fails, the next attempt finds the key and skips a trade that was never saved. The database example below handles this at the write itself.

The Bybit JavaScript SDK delivers public trade batches on update. To use the counter with live data, replace everything after the function in deduplicate.mjs with this code, and install the package with npm install bybit-api:

const { WebsocketClient } = await import('bybit-api');
const counter = createTradeCounter();
const ws = new WebsocketClient();

ws.on('update', (message) => {
  if (message.topic !== 'publicTrade.BTCUSDT') return;

  for (const trade of message.data) {
    const result = counter.accept(trade);
    if (result === 'accepted') {
      console.log(trade.i, trade.p, trade.v);
    }
  }
});

ws.subscribeV5('publicTrade.BTCUSDT', 'spot');

The handler is attached before subscription. Its counter survives reconnects for as long as the Node.js process stays alive. The SDK manages the connection and subscription; the counter decides which trade IDs it has seen. This example validates the fields needed for the key, not the complete exchange payload.

Only public spot data is subscribed here. A derivatives feed, testnet connection, or private execution stream needs its own scope and field mapping. The counter is still just an in-memory example; console output doesn't make it a trade archive.

If recovery also fetches REST history, request pacing is separate from duplicate detection. The backfill example covers pagination and request limits.

Two Node.js workers each have their own Set, so both can accept the same trade. A shared cache can coordinate them, but it still leaves a gap between claiming a key and writing the record. When the intended effect is a database insert, a unique constraint can handle the duplicate at that point.

This PostgreSQL table stores individual public trades. Its primary key uses the same fields across live delivery and history imports. Price and quantity use numeric so they can be passed as decimal strings without a JavaScript floating-point conversion.

CREATE TABLE recorded_trades (
  exchange text NOT NULL,
  environment text NOT NULL,
  category text NOT NULL,
  record_type text NOT NULL,
  symbol text NOT NULL,
  trade_id text NOT NULL,
  price numeric NOT NULL,
  quantity numeric NOT NULL,
  PRIMARY KEY (
    exchange, environment, category, record_type, symbol, trade_id
  )
);

The parameterized insert uses those columns as its conflict target:

INSERT INTO recorded_trades (
  exchange, environment, category, record_type,
  symbol, trade_id, price, quantity
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (
  exchange, environment, category, record_type, symbol, trade_id
) DO NOTHING
RETURNING trade_id;

PostgreSQL returns a row only when this statement inserts one. A duplicate returns no rows. Two workers can attempt the same insert without first racing through separate SELECT and INSERT calls.

If the connection drops after the insert commits but before the response arrives, retrying the insert finds the stored trade. If it never committed, the retry can insert it. This assumes a trade record is immutable under that ID. A feed that publishes corrections needs an update policy; blindly ignoring a changed payload would hide the correction.