> ## Documentation Index
> Fetch the complete documentation index at: https://raymondchang.me/llms.txt
> Use this file to discover all available pages before exploring further.

# Quant interview: data engineering and systems

> Python, SQL, Linux, Docker and data pipeline interview questions for quantitative roles: market data quality, as-of joins, corporate actions and reproducibility.

This page collects data engineering and systems questions asked in quantitative roles, covering Python, SQL, Linux, Docker and market data handling. The answers are concepts and design principles and contain no code. Ask separately if you want code examples.

> [!NOTE]
>
> **Status of this page.** Last checked on 15 November 2026. These are general engineering practices, not any firm's specific technology stack.

## Market data

> [accordion] Q1. How do you check the quality of a market data feed or historical price table?
>
> Classify the problems first, then check each:
>
> * **Missing values and days:** Compare against the trading calendar to find missing dates.
> * **Duplicates and ordering:** Duplicate rows for the same timestamp and symbol, and out-of-order timestamps.
> * **Outliers:** Prices or returns outside a plausible range, zero or negative prices, abnormal volume.
> * **Stale prices:** A long run of unchanged prices may be a halt or a feed interruption.
> * **Consistency:** The open-high-low-close relationships (high not below open or close) and differences between sources.
> * **Corporate actions:** Price breaks caused by splits and dividends.
>
> Implementation: write the checks as automated rules and log the results, alert on anomalies instead of silently fixing them, and keep the raw data.

> [accordion] Q2. What are corporate actions and how do they affect backtests?
>
> Dividends, splits, reverse splits, rights issues, ticker or name changes, mergers and delistings.
>
> Without adjustment, a split looks like a crash and produces false signals. Separate **adjusted prices** (for return calculation and signals) from **unadjusted prices** (for actually tradable prices, lot sizes and notionals). A backtest should use the adjustment information that was known at the time of the event, to avoid look-ahead.

> [accordion] Q3. How do you align trades and quotes in time, and what is a point-in-time (as-of) join?
>
> An as-of join takes, for each trade, **the most recent quote at or before that timestamp**, rather than matching timestamps exactly, because quotes and trades rarely coincide.
>
> Watch for:
>
> * Time zone and exchange clock differences
> * Timestamp precision (milliseconds, microseconds) and source latency
> * Never taking a quote from after the trade, which is look-ahead bias
>
> Both SQL and pandas support this. In an interview, explain the logic rather than recite syntax.

> [accordion] Q4. How do you handle time zones and trading calendars across markets?
>
> Store timestamps in UTC internally and convert to the market's local time for display and business logic. Each market has its own trading calendar, half days and holidays, so use a maintained calendar rather than assuming Monday to Friday.
>
> For cross-market analysis (for example a Hong Kong ETF against US constituents), define explicitly what "the same moment" means, because opening and closing hours do not overlap. This also affects how fresh the iNAV is.

## Python and SQL

> [accordion] Q5. Why is vectorized code faster than a Python loop, and when can you not vectorize?
>
> NumPy and pandas run loops in compiled code (C), avoiding the per-iteration overhead of the Python interpreter and using contiguous memory and SIMD.
>
> Hard to vectorize: each step depends on the state of the previous step (path-dependent logic such as a step-by-step simulation with stops) and complex branching. Options then include Numba, restructuring to reduce loops, or implementing the critical part in C++. Measure before optimizing rather than going by feel.

> [accordion] Q6. Why does Python's GIL matter for research code? How do you use multiple cores?
>
> The GIL allows only one thread to execute Python bytecode at a time within a process, so multithreading does not speed up CPU-bound work.
>
> Solutions: **multiprocessing**, or libraries that release the GIL (many NumPy operations). For I/O-bound work (downloads, file reads), threads or asyncio are effective. Mind the cost of passing data between processes (serialization).

> [accordion] Q7. What SQL techniques do you use for time-series data? Explain window functions.
>
> A window function computes over a set of related rows without collapsing them. Common uses:
>
> * Latest row per symbol (ranking or row numbers within a partition)
> * Rolling averages, cumulative sums and differences (lag and lead)
> * Ranking and percentiles within a group
>
> Design and performance: index or partition by time and symbol, avoid unnecessary full table scans, and understand how join types affect row counts. For large tick data, columnar storage or a time-series database usually suits better than row-based OLTP.

> [accordion] Q8. How do you make a data pipeline idempotent and safe to backfill?
>
> Idempotent means re-running the same task gives the same result with no duplicate data.
>
> * Partition by date and overwrite the whole partition on write, or use an upsert keyed on the primary key
> * Make tasks depend only on explicit inputs (date parameter, versioned data), not on the current time
> * Keep raw and transformed data separately so transforms can be re-run
> * Limit backfill scope, write to staging and then swap, and record versions
>
> The same principles make retries after failure safe.

## Linux and Docker

> [accordion] Q9. A research job is running slowly on a Linux server. How do you investigate?
>
> Work from the outside in:
>
> * **CPU and memory:** Use top or htop to see whether CPU is saturated or the system is swapping.
> * **Disk I/O:** Use a tool such as iostat to see read and write waits.
> * **Network:** Check whether the job is waiting on a remote data source.
> * **The program itself:** Use a profiler to find hot spots, and strace to see whether it is blocked in system calls.
> * **Resource limits:** Check ulimit and cgroup limits (inside containers) for CPU or memory caps.
>
> Principle: first confirm whether the bottleneck is CPU, memory, I/O or waiting on an external system, then decide how to optimize.

> [accordion] Q10. What is the difference between a Docker image and a container? Why use Docker for research?
>
> An **image** is a read-only, layered template containing the OS layer, dependencies and code. A **container** is a running instance started from an image, with a writable layer.
>
> Benefits for research: reproducible environments (the same image behaves the same on a laptop and a server), pinned dependency versions, and easy delivery and scheduling. Keep data and results in volumes or external storage, not in the container's writable layer, which disappears when the container is removed. Do not bake passwords or keys into images.

> [accordion] Q11. Batch versus streaming for market data: how do you choose?
>
> * **Batch:** End-of-day backtests, research and reports. Simple to build, easy to re-run and validate, with latency measured in minutes to hours.
> * **Streaming:** Live quotes, risk, hedging and market making. Low latency, but you must handle out-of-order data, duplicates, disconnects and backpressure, and validation and replay are harder.
>
> A common pattern uses both: a real-time path for trading that also writes to storage, followed by a batch recompute and reconciliation so the two stay consistent.

## Related pages

* [Regression, time series and backtesting](https://raymondchang.me/quantitative-research/interview/statistics-backtesting.md)
* [ETF market making and arbitrage](https://raymondchang.me/quantitative-research/interview/etf-market-making.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.