Skip to main content
Interviewers want to know whether you can keep a trading or data system healthy and respond calmly when it is not. This page has nine operations and incident questions with follow-ups and answer points. The answers are frameworks, and specific compliance and reporting duties follow each firm’s and regulator’s rules.
Status of this page. Last checked on 15 November 2026. These are general practices, not any firm’s incident process.

A framework for incident questions

  1. Stop the bleeding: Limit further damage first (pause, kill switch, fall back to a backup source).
  2. Assess impact: Who and what is affected, and by how much? Notify risk, compliance and managers as procedures require.
  3. Diagnose: Find the root cause with evidence from logs, metrics and recorded data.
  4. Fix and verify: Apply the fix, verify it, then resume carefully.
  5. Review: Write a blameless postmortem with a timeline, root cause, detection gap and follow-up actions.

Data and batch jobs

A common cause is running out of memory (an OOM kill).
  • Check the container’s exit status and reason, and the host’s system logs for OOM records.
  • Check the container’s memory and CPU limits.
  • Compare data volume with earlier runs, or look for recent code changes.
  • Switch to chunked processing or adjust the resource settings.
Prevention: Add memory monitoring and failure alerts to the job so there are no silent failures. See Data engineering Q9 and Q10.
  • Freshness monitoring: Alert when expected data for a trading day has not arrived by a set time, using the trading calendar to know what is expected.
  • Row count and value checks: Compare each day’s volume of data with recent history and flag large deviations.
  • Job-level alerts: Alert on failure and also on jobs that never started.
  • Safe re-runs: Make the pipeline idempotent so a backfill for the missing day is safe. See Data engineering Q8.
  • Ownership: Each dataset has a named owner and a documented recovery runbook.
The principle is that silence must never be treated as success.

Live connectivity and data

  • Detect: Track the expected next sequence number per channel and flag any jump.
  • Contain: Mark affected books as uncertain so strategies do not trade on a possibly wrong book. Widen quotes or pause for those instruments.
  • Recover: Use the redundant feed if you have one, request retransmission, or fetch a snapshot and rebuild the book from the incremental updates that follow it.
  • Verify and resume: Check the rebuilt book for sanity (for example it is not crossed) before trading again.
  • Record: Log the gap for later analysis, because frequent gaps point to a network or capacity problem.
  • Know your state: The system keeps its own sent and received sequence numbers and a durable log of all messages.
  • Reconnect: Use the backup connection if one is ready, otherwise reconnect and log on again.
  • Gap fill: Compare sequence numbers with the counterparty and request resends for messages you missed. Handle messages the counterparty did not receive.
  • Reconcile orders and positions: After reconnecting, query open orders and positions from the venue and compare them with your records. Treat the venue as the authority.
  • Do not blindly resend: Orders in an unknown state must be checked, because resending can create duplicates.
  • Safety: While disconnected, stop generating new orders, and consider whether open orders should be cancelled on disconnect.
Order: stop the bleeding, assess, fix, review.
  • Stop the bleeding: Use the kill switch or fall back to a backup source immediately to stop publishing the bad price.
  • Assess: Determine which clients, trades and positions were affected and by how much, and notify risk, compliance and the relevant managers.
  • Fix: Find the root cause (data, configuration or a code change), and verify the fix before resuming.
  • Review: Write a blameless postmortem with the timeline, root cause, why it was not detected sooner and follow-up actions (automated checks, limits, tests).
Specific compliance and reporting obligations follow the firm’s and regulators’ rules. In the interview, say that you would escalate according to company procedure.
  • Stop it first: trigger the strategy’s kill switch or cancel-on-disconnect, and block new orders for that strategy. Do not wait to understand the cause.
  • Check the damage: open orders, fills and positions created during the spike, and whether any limit was breached. Cancel what is still open.
  • Contain: check whether other strategies share the same code, configuration or data feed, and pause them if they might be affected.
  • Escalate: inform the trading desk, risk and compliance according to procedure, since venues may also contact you about message rates.
  • Diagnose afterwards: typical causes are a feedback loop (reacting to its own orders), a bad price or a stale book, a configuration error, or a retry loop after rejects.
Follow-up: What control would have stopped it automatically?→ An order and message rate limit in the pre-trade risk layer that is independent of the strategy, with automatic disable on breach. See System design Q8 and Q9.
Why it matters: event ordering across machines, latency measurement, as-of joins in research, and regulatory record keeping all assume clocks agree. Some regulators, such as the EU under MiFID II, set maximum allowed divergence from UTC for trading timestamps.Fixes:
  • Synchronize with a proper time source. NTP is usually accurate to milliseconds, while PTP with hardware timestamping can reach microseconds or better.
  • Monitor the offset of every host from the reference and alert when it drifts beyond a threshold.
  • Record which clock produced each timestamp, and keep the exchange’s timestamp next to your receive time.
  • For ordering within one process, use a monotonic clock, not the wall clock, which can jump when it is corrected.

Change management

  • Pre-release: Tests including replay of recorded sessions, and review of the change and its blast radius.
  • Shadow mode: Run the new version on live data without sending orders, and compare its decisions with the current version.
  • Staged rollout: Enable for a small set of instruments or a small size first, with tight limits and close monitoring.
  • Feature flags: Turn the new behavior on and off without redeploying.
  • Consistency checks: Verify that every instance runs the same version and configuration, because partial deployments are a classic cause of incidents.
  • Rollback: A rehearsed, fast path back to the previous version, with clear criteria for when to use it.
  • Timing: Avoid risky changes at market open, close and around major events.
Public interview guides frequently cite Knight Capital’s 2012 loss when motivating deployment controls and kill switches.
  • Blameless: Focus on how the system and process allowed the failure, not on who made the mistake.
  • Timeline: What happened, when it was detected, when it was mitigated and when it was resolved.
  • Impact: Clients, trades, positions and money affected, stated factually.
  • Root cause and contributing factors: Distinguish the trigger from the underlying weaknesses (missing checks, unclear ownership, weak monitoring).
  • Detection gap: Why it was not caught earlier, and which alert would have caught it.
  • Actions: Specific, owned, time-bound follow-ups such as new automated checks, tighter limits, extra tests or runbook updates, tracked to completion.

Sources