A framework for incident questions
- Stop the bleeding: Limit further damage first (pause, kill switch, fall back to a backup source).
- Assess impact: Who and what is affected, and by how much? Notify risk, compliance and managers as procedures require.
- Diagnose: Find the root cause with evidence from logs, metrics and recorded data.
- Fix and verify: Apply the fix, verify it, then resume carefully.
- Review: Write a blameless postmortem with a timeline, root cause, detection gap and follow-up actions.
Data and batch jobs
Q1. A container running a nightly job is killed with no error message. How do you investigate?
Q1. A container running a nightly job is killed with no error message. How do you investigate?
- Check the container’s exit status and reason, and the host’s system logs for OOM records.
- Check the container’s memory and CPU limits.
- Compare data volume with earlier runs, or look for recent code changes.
- Switch to chunked processing or adjust the resource settings.
Q2. A daily data pipeline silently skipped yesterday and nobody noticed until a researcher complained. How do you prevent a repeat?
Q2. A daily data pipeline silently skipped yesterday and nobody noticed until a researcher complained. How do you prevent a repeat?
- Freshness monitoring: Alert when expected data for a trading day has not arrived by a set time, using the trading calendar to know what is expected.
- Row count and value checks: Compare each day’s volume of data with recent history and flag large deviations.
- Job-level alerts: Alert on failure and also on jobs that never started.
- Safe re-runs: Make the pipeline idempotent so a backfill for the missing day is safe. See Data engineering Q8.
- Ownership: Each dataset has a named owner and a documented recovery runbook.
Live connectivity and data
Q3. You detect a gap in the sequence numbers of a market data feed. What do you do?
Q3. You detect a gap in the sequence numbers of a market data feed. What do you do?
- Detect: Track the expected next sequence number per channel and flag any jump.
- Contain: Mark affected books as uncertain so strategies do not trade on a possibly wrong book. Widen quotes or pause for those instruments.
- Recover: Use the redundant feed if you have one, request retransmission, or fetch a snapshot and rebuild the book from the incremental updates that follow it.
- Verify and resume: Check the rebuilt book for sanity (for example it is not crossed) before trading again.
- Record: Log the gap for later analysis, because frequent gaps point to a network or capacity problem.
Q4. Your connection to an exchange or broker drops in the middle of the day. Walk me through the recovery.
Q4. Your connection to an exchange or broker drops in the middle of the day. Walk me through the recovery.
- Know your state: The system keeps its own sent and received sequence numbers and a durable log of all messages.
- Reconnect: Use the backup connection if one is ready, otherwise reconnect and log on again.
- Gap fill: Compare sequence numbers with the counterparty and request resends for messages you missed. Handle messages the counterparty did not receive.
- Reconcile orders and positions: After reconnecting, query open orders and positions from the venue and compare them with your records. Treat the venue as the authority.
- Do not blindly resend: Orders in an unknown state must be checked, because resending can create duplicates.
- Safety: While disconnected, stop generating new orders, and consider whether open orders should be cancelled on disconnect.
Q5. A wrong price was published to clients for 10 minutes. Walk me through your response.
Q5. A wrong price was published to clients for 10 minutes. Walk me through your response.
- Stop the bleeding: Use the kill switch or fall back to a backup source immediately to stop publishing the bad price.
- Assess: Determine which clients, trades and positions were affected and by how much, and notify risk, compliance and the relevant managers.
- Fix: Find the root cause (data, configuration or a code change), and verify the fix before resuming.
- Review: Write a blameless postmortem with the timeline, root cause, why it was not detected sooner and follow-up actions (automated checks, limits, tests).
Q6. A strategy suddenly sends orders at ten times its normal rate. What do you do in the first five minutes?
Q6. A strategy suddenly sends orders at ten times its normal rate. What do you do in the first five minutes?
- Stop it first: trigger the strategy’s kill switch or cancel-on-disconnect, and block new orders for that strategy. Do not wait to understand the cause.
- Check the damage: open orders, fills and positions created during the spike, and whether any limit was breached. Cancel what is still open.
- Contain: check whether other strategies share the same code, configuration or data feed, and pause them if they might be affected.
- Escalate: inform the trading desk, risk and compliance according to procedure, since venues may also contact you about message rates.
- Diagnose afterwards: typical causes are a feedback loop (reacting to its own orders), a bad price or a stale book, a configuration error, or a retry loop after rejects.
Q7. Timestamps from two servers disagree by several milliseconds. Why does it matter and how do you fix it?
Q7. Timestamps from two servers disagree by several milliseconds. Why does it matter and how do you fix it?
- Synchronize with a proper time source. NTP is usually accurate to milliseconds, while PTP with hardware timestamping can reach microseconds or better.
- Monitor the offset of every host from the reference and alert when it drifts beyond a threshold.
- Record which clock produced each timestamp, and keep the exchange’s timestamp next to your receive time.
- For ordering within one process, use a monotonic clock, not the wall clock, which can jump when it is corrected.
Change management
Q8. How do you deploy a change to a live pricing or trading system safely?
Q8. How do you deploy a change to a live pricing or trading system safely?
- Pre-release: Tests including replay of recorded sessions, and review of the change and its blast radius.
- Shadow mode: Run the new version on live data without sending orders, and compare its decisions with the current version.
- Staged rollout: Enable for a small set of instruments or a small size first, with tight limits and close monitoring.
- Feature flags: Turn the new behavior on and off without redeploying.
- Consistency checks: Verify that every instance runs the same version and configuration, because partial deployments are a classic cause of incidents.
- Rollback: A rehearsed, fast path back to the previous version, with clear criteria for when to use it.
- Timing: Avoid risky changes at market open, close and around major events.
Q9. How do you write a good postmortem?
Q9. How do you write a good postmortem?
- Blameless: Focus on how the system and process allowed the failure, not on who made the mistake.
- Timeline: What happened, when it was detected, when it was mitigated and when it was resolved.
- Impact: Clients, trades, positions and money affected, stated factually.
- Root cause and contributing factors: Distinguish the trigger from the underlying weaknesses (missing checks, unclear ownership, weak monitoring).
- Detection gap: Why it was not caught earlier, and which alert would have caught it.
- Actions: Specific, owned, time-bound follow-ups such as new automated checks, tighter limits, extra tests or runbook updates, tracked to completion.

