Skip to main content
This page collects data engineering and systems questions asked in quantitative roles, covering Python, SQL, Linux, Docker and market data handling. The answers are concepts and design principles and contain no code. Ask separately if you want code examples.
Status of this page. Last checked on 15 November 2026. These are general engineering practices, not any firm’s specific technology stack.

Market data

Classify the problems first, then check each:
  • Missing values and days: Compare against the trading calendar to find missing dates.
  • Duplicates and ordering: Duplicate rows for the same timestamp and symbol, and out-of-order timestamps.
  • Outliers: Prices or returns outside a plausible range, zero or negative prices, abnormal volume.
  • Stale prices: A long run of unchanged prices may be a halt or a feed interruption.
  • Consistency: The open-high-low-close relationships (high not below open or close) and differences between sources.
  • Corporate actions: Price breaks caused by splits and dividends.
Implementation: write the checks as automated rules and log the results, alert on anomalies instead of silently fixing them, and keep the raw data.
Dividends, splits, reverse splits, rights issues, ticker or name changes, mergers and delistings.Without adjustment, a split looks like a crash and produces false signals. Separate adjusted prices (for return calculation and signals) from unadjusted prices (for actually tradable prices, lot sizes and notionals). A backtest should use the adjustment information that was known at the time of the event, to avoid look-ahead.
An as-of join takes, for each trade, the most recent quote at or before that timestamp, rather than matching timestamps exactly, because quotes and trades rarely coincide.Watch for:
  • Time zone and exchange clock differences
  • Timestamp precision (milliseconds, microseconds) and source latency
  • Never taking a quote from after the trade, which is look-ahead bias
Both SQL and pandas support this. In an interview, explain the logic rather than recite syntax.
Store timestamps in UTC internally and convert to the market’s local time for display and business logic. Each market has its own trading calendar, half days and holidays, so use a maintained calendar rather than assuming Monday to Friday.For cross-market analysis (for example a Hong Kong ETF against US constituents), define explicitly what “the same moment” means, because opening and closing hours do not overlap. This also affects how fresh the iNAV is.

Python and SQL

NumPy and pandas run loops in compiled code (C), avoiding the per-iteration overhead of the Python interpreter and using contiguous memory and SIMD.Hard to vectorize: each step depends on the state of the previous step (path-dependent logic such as a step-by-step simulation with stops) and complex branching. Options then include Numba, restructuring to reduce loops, or implementing the critical part in C++. Measure before optimizing rather than going by feel.
The GIL allows only one thread to execute Python bytecode at a time within a process, so multithreading does not speed up CPU-bound work.Solutions: multiprocessing, or libraries that release the GIL (many NumPy operations). For I/O-bound work (downloads, file reads), threads or asyncio are effective. Mind the cost of passing data between processes (serialization).
A window function computes over a set of related rows without collapsing them. Common uses:
  • Latest row per symbol (ranking or row numbers within a partition)
  • Rolling averages, cumulative sums and differences (lag and lead)
  • Ranking and percentiles within a group
Design and performance: index or partition by time and symbol, avoid unnecessary full table scans, and understand how join types affect row counts. For large tick data, columnar storage or a time-series database usually suits better than row-based OLTP.
Idempotent means re-running the same task gives the same result with no duplicate data.
  • Partition by date and overwrite the whole partition on write, or use an upsert keyed on the primary key
  • Make tasks depend only on explicit inputs (date parameter, versioned data), not on the current time
  • Keep raw and transformed data separately so transforms can be re-run
  • Limit backfill scope, write to staging and then swap, and record versions
The same principles make retries after failure safe.

Linux and Docker

Work from the outside in:
  • CPU and memory: Use top or htop to see whether CPU is saturated or the system is swapping.
  • Disk I/O: Use a tool such as iostat to see read and write waits.
  • Network: Check whether the job is waiting on a remote data source.
  • The program itself: Use a profiler to find hot spots, and strace to see whether it is blocked in system calls.
  • Resource limits: Check ulimit and cgroup limits (inside containers) for CPU or memory caps.
Principle: first confirm whether the bottleneck is CPU, memory, I/O or waiting on an external system, then decide how to optimize.
An image is a read-only, layered template containing the OS layer, dependencies and code. A container is a running instance started from an image, with a writable layer.Benefits for research: reproducible environments (the same image behaves the same on a laptop and a server), pinned dependency versions, and easy delivery and scheduling. Keep data and results in volumes or external storage, not in the container’s writable layer, which disappears when the container is removed. Do not bake passwords or keys into images.
  • Batch: End-of-day backtests, research and reports. Simple to build, easy to re-run and validate, with latency measured in minutes to hours.
  • Streaming: Live quotes, risk, hedging and market making. Low latency, but you must handle out-of-order data, duplicates, disconnects and backpressure, and validation and replay are harder.
A common pattern uses both: a real-time path for trading that also writes to storage, followed by a batch recompute and reconciliation so the two stay consistent.