> ## Documentation Index
> Fetch the complete documentation index at: https://raymondchang.me/llms.txt
> Use this file to discover all available pages before exploring further.

# Quant dev mock interview: operations and incidents

> Operations mock interview for quant developers: silent job failures, feed gaps, exchange disconnects, wrong prices, safe deployments and postmortems.

Interviewers want to know whether you can keep a trading or data system healthy and respond calmly when it is not. This page has nine operations and incident questions with follow-ups and answer points. The answers are frameworks, and specific compliance and reporting duties follow each firm's and regulator's rules.

> [!NOTE]
>
> **Status of this page.** Last checked on 15 November 2026. These are general practices, not any firm's incident process.

## A framework for incident questions

1. **Stop the bleeding:** Limit further damage first (pause, kill switch, fall back to a backup source).
2. **Assess impact:** Who and what is affected, and by how much? Notify risk, compliance and managers as procedures require.
3. **Diagnose:** Find the root cause with evidence from logs, metrics and recorded data.
4. **Fix and verify:** Apply the fix, verify it, then resume carefully.
5. **Review:** Write a blameless postmortem with a timeline, root cause, detection gap and follow-up actions.

## Data and batch jobs

> [accordion] Q1. A container running a nightly job is killed with no error message. How do you investigate?
>
> A common cause is running out of memory (an OOM kill).
>
> * Check the container's exit status and reason, and the host's system logs for OOM records.
> * Check the container's memory and CPU limits.
> * Compare data volume with earlier runs, or look for recent code changes.
> * Switch to chunked processing or adjust the resource settings.
>
> **Prevention:** Add memory monitoring and failure alerts to the job so there are no silent failures. See [Data engineering](https://raymondchang.me/quantitative-research/interview/data-engineering.md) Q9 and Q10.

> [accordion] Q2. A daily data pipeline silently skipped yesterday and nobody noticed until a researcher complained. How do you prevent a repeat?
>
> * **Freshness monitoring:** Alert when expected data for a trading day has not arrived by a set time, using the trading calendar to know what is expected.
> * **Row count and value checks:** Compare each day's volume of data with recent history and flag large deviations.
> * **Job-level alerts:** Alert on failure and also on jobs that never started.
> * **Safe re-runs:** Make the pipeline idempotent so a backfill for the missing day is safe. See [Data engineering](https://raymondchang.me/quantitative-research/interview/data-engineering.md) Q8.
> * **Ownership:** Each dataset has a named owner and a documented recovery runbook.
>
> The principle is that silence must never be treated as success.

## Live connectivity and data

> [accordion] Q3. You detect a gap in the sequence numbers of a market data feed. What do you do?
>
> * **Detect:** Track the expected next sequence number per channel and flag any jump.
> * **Contain:** Mark affected books as uncertain so strategies do not trade on a possibly wrong book. Widen quotes or pause for those instruments.
> * **Recover:** Use the redundant feed if you have one, request retransmission, or fetch a snapshot and rebuild the book from the incremental updates that follow it.
> * **Verify and resume:** Check the rebuilt book for sanity (for example it is not crossed) before trading again.
> * **Record:** Log the gap for later analysis, because frequent gaps point to a network or capacity problem.

> [accordion] Q4. Your connection to an exchange or broker drops in the middle of the day. Walk me through the recovery.
>
> * **Know your state:** The system keeps its own sent and received sequence numbers and a durable log of all messages.
> * **Reconnect:** Use the backup connection if one is ready, otherwise reconnect and log on again.
> * **Gap fill:** Compare sequence numbers with the counterparty and request resends for messages you missed. Handle messages the counterparty did not receive.
> * **Reconcile orders and positions:** After reconnecting, query open orders and positions from the venue and compare them with your records. Treat the venue as the authority.
> * **Do not blindly resend:** Orders in an unknown state must be checked, because resending can create duplicates.
> * **Safety:** While disconnected, stop generating new orders, and consider whether open orders should be cancelled on disconnect.

> [accordion] Q5. A wrong price was published to clients for 10 minutes. Walk me through your response.
>
> **Order: stop the bleeding, assess, fix, review.**
>
> * **Stop the bleeding:** Use the kill switch or fall back to a backup source immediately to stop publishing the bad price.
> * **Assess:** Determine which clients, trades and positions were affected and by how much, and notify risk, compliance and the relevant managers.
> * **Fix:** Find the root cause (data, configuration or a code change), and verify the fix before resuming.
> * **Review:** Write a blameless postmortem with the timeline, root cause, why it was not detected sooner and follow-up actions (automated checks, limits, tests).
>
> Specific compliance and reporting obligations follow the firm's and regulators' rules. In the interview, say that you would escalate according to company procedure.

> [accordion] Q6. A strategy suddenly sends orders at ten times its normal rate. What do you do in the first five minutes?
>
> * **Stop it first:** trigger the strategy's kill switch or cancel-on-disconnect, and block new orders for that strategy. Do not wait to understand the cause.
> * **Check the damage:** open orders, fills and positions created during the spike, and whether any limit was breached. Cancel what is still open.
> * **Contain:** check whether other strategies share the same code, configuration or data feed, and pause them if they might be affected.
> * **Escalate:** inform the trading desk, risk and compliance according to procedure, since venues may also contact you about message rates.
> * **Diagnose afterwards:** typical causes are a feedback loop (reacting to its own orders), a bad price or a stale book, a configuration error, or a retry loop after rejects.
>
> **Follow-up:** What control would have stopped it automatically?
>
> → An order and message rate limit in the pre-trade risk layer that is independent of the strategy, with automatic disable on breach. See [System design](https://raymondchang.me/quantitative-dev/mock/system-design.md) Q8 and Q9.

> [accordion] Q7. Timestamps from two servers disagree by several milliseconds. Why does it matter and how do you fix it?
>
> **Why it matters:** event ordering across machines, latency measurement, as-of joins in research, and regulatory record keeping all assume clocks agree. Some regulators, such as the EU under MiFID II, set maximum allowed divergence from UTC for trading timestamps.
>
> **Fixes:**
>
> * Synchronize with a proper time source. NTP is usually accurate to milliseconds, while PTP with hardware timestamping can reach microseconds or better.
> * Monitor the offset of every host from the reference and alert when it drifts beyond a threshold.
> * Record which clock produced each timestamp, and keep the exchange's timestamp next to your receive time.
> * For ordering within one process, use a monotonic clock, not the wall clock, which can jump when it is corrected.

## Change management

> [accordion] Q8. How do you deploy a change to a live pricing or trading system safely?
>
> * **Pre-release:** Tests including replay of recorded sessions, and review of the change and its blast radius.
> * **Shadow mode:** Run the new version on live data without sending orders, and compare its decisions with the current version.
> * **Staged rollout:** Enable for a small set of instruments or a small size first, with tight limits and close monitoring.
> * **Feature flags:** Turn the new behavior on and off without redeploying.
> * **Consistency checks:** Verify that every instance runs the same version and configuration, because partial deployments are a classic cause of incidents.
> * **Rollback:** A rehearsed, fast path back to the previous version, with clear criteria for when to use it.
> * **Timing:** Avoid risky changes at market open, close and around major events.
>
> Public interview guides frequently cite Knight Capital's 2012 loss when motivating deployment controls and kill switches.

> [accordion] Q9. How do you write a good postmortem?
>
> * **Blameless:** Focus on how the system and process allowed the failure, not on who made the mistake.
> * **Timeline:** What happened, when it was detected, when it was mitigated and when it was resolved.
> * **Impact:** Clients, trades, positions and money affected, stated factually.
> * **Root cause and contributing factors:** Distinguish the trigger from the underlying weaknesses (missing checks, unclear ownership, weak monitoring).
> * **Detection gap:** Why it was not caught earlier, and which alert would have caught it.
> * **Actions:** Specific, owned, time-bound follow-ups such as new automated checks, tighter limits, extra tests or runbook updates, tracked to completion.

## Related pages

* [System design](https://raymondchang.me/quantitative-dev/mock/system-design.md)
* [Python, SQL and concurrency](https://raymondchang.me/quantitative-dev/mock/python-sql-concurrency.md)
* [Behavioral questions](https://raymondchang.me/quantitative-dev/mock/behavioral.md)
* [Mock Interview overview](https://raymondchang.me/quantitative-dev/mock-interview.md)

## Sources

* [Algorithmic Trading System Architecture (techinterview.org)](https://www.techinterview.org/post/3233474598/algorithmic-trading-architecture/)
* [What Interview Questions Test Quant Development Skills (OfflinePixel)](https://offline-pixel.github.io/blogs/what-interview-questions-test-quant-development-skills/)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.