5 exercises — choose the best-structured answer to common AI infrastructure interview questions. Focus on GPU cluster design, distributed training sharding, checkpointing, KV cache management, and LLM serving architecture.
Structure for AI infrastructure interview answers
Give bandwidth numbers: NVLink 900 GB/s, NDR IB 400 Gbps — concrete specs show hands-on experience
Separate intra-node from inter-node: NVLink/NVSwitch within a node, InfiniBand between nodes
Quantify memory: calculate actual GB for the model size — interviewers want to see you can size a system
Cover failure modes: atomic rename for checkpoints, PERMISSIVE left on for mesh — failure awareness signals seniority
0 / 15 completed
1 / 15
The interviewer asks: "Design the network topology for a GPU cluster training a 100B parameter model — what interconnect technologies do you use and why?" Which answer best covers GPU cluster networking?
Option B provides the complete topology picture: NVLink bandwidth numbers (H100 NVLink 4.0, 900 GB/s bidirectional), NVSwitch crossbar architecture, IB generation speeds (HDR 200 Gbps, NDR 400 Gbps), HCA-per-GPU to avoid CPU bottleneck, fat-tree and rail-optimised topologies at scale, GPUDirect RDMA for CPU-bypass, NCCL algorithm selection (tree vs ring based on message size), and RoCEv2 as the Ethernet alternative with its requirements. Options A, C, D name the correct components but provide no bandwidth numbers, topology design choices, or RDMA mechanics.
2 / 15
The interviewer asks: "Compare PyTorch FSDP and DeepSpeed ZeRO (Stage 1, 2, 3) for training a 70B parameter model — what do they shard and what are the trade-offs?" Which answer best covers distributed training sharding?
Option B provides quantified analysis: exact memory savings per ZeRO stage (4×, 8×, linear in N), concrete memory calculation for 70B model (140GB at FP16, 2.2GB/GPU with ZeRO-3 on 64 GPUs), FSDP all-gather/reduce-scatter mechanics with the discard pattern, FSDP vs ZeRO-3 comparison on torch.compile integration and CPU/NVMe offloading, ZeRO-Infinity for trillion-parameter models, and activation checkpointing as an orthogonal technique. Options A and C state the facts without the memory calculations, offload mechanics, or the "when to use which" guidance.
3 / 15
The interviewer asks: "What is your checkpointing strategy for training a 100B+ parameter model on a 1000-GPU cluster — what can go wrong and how do you mitigate it?" Which answer best covers checkpoint engineering?
Option B covers all seven dimensions: checkpoint size calculation (200GB), asynchronous checkpointing mechanics (CPU RAM copy + background thread), distributed checkpoint storage (DCP API + Lustre/GPFS/S3 with per-rank shards), rolling window policy with corruption rationale, failure mode mitigation (atomic rename, Lustre stripe count), elastic training for fault tolerance (TorchElastic + NVIDIA Resiliency Lib for in-GPU recovery), and checkpoint validation. Options C and D each mention 2-3 correct ideas but none cover the atomic rename pattern, Lustre stripe tuning, TorchElastic fault tolerance, or checkpoint validation.
4 / 15
The interviewer asks: "Explain KV cache management in LLM inference — what is the KV cache, why does it grow, and what eviction strategies exist?" Which answer best covers LLM inference cache engineering?
Option B provides the complete engineering picture: KV cache memory formula with a concrete Llama-3 70B calculation (2MB/token, 8GB for 4K context), growth factors, LRU eviction and its system-prompt problem, H2O attention-score-based eviction with the "heavy hitter" mechanism, PagedAttention with the OS virtual memory analogy and the fragmentation problem it solves, prefix caching with the system-prompt sharing use case, and disaggregated prefill/decode as a scaling strategy. Options A, C, D each mention 1-2 mechanisms without the memory calculation, fragmentation problem, or H2O algorithm.
5 / 15
The interviewer asks: "Explain disaggregated prefill and decode in LLM serving — why are they separated, and how does this affect your infrastructure design?" Which answer best covers serving infrastructure architecture?
Option B covers all six dimensions: prefill (compute-bound, single forward pass) vs decode (memory-bandwidth-bound, sequential) characteristics, the disaggregation architecture (router + prefill pool + decode pool + KV transfer), provisioning guidance (compute vs HBM bandwidth, H100 3.35 TB/s number), KV cache transfer latency calculation (8GB / 400 Gbps = 160ms, pipelining as mitigation), production systems implementing PD disaggregation (Mooncake, TetriInfer, vLLM), and continuous batching as the simpler alternative. Options A, C, D each identify the prefill/decode distinction but don't cover the transfer latency, provisioning guidance, or production system examples.
6 / 15
Code Review Comment: 'This Lambda function seems to be pulling data directly from S3 without any caching. Given the expected throughput of 10k requests/second, this could lead to significant S3 egress costs and potential throttling. What's your rationale for bypassing a cache layer here?'
This question tests your ability to respond constructively during a code review. The interviewer isn't just looking for an acknowledgement of the problem; they want you to demonstrate understanding of performance optimization strategies like caching and its impact on costs/throttling. Option 2 reflects this by acknowledging the need while hinting at a solution.
7 / 15
Slack Message from @DataEngineer: 'We're seeing high latency on our real-time inference endpoints for the new sentiment analysis model. The logs show a significant amount of time spent waiting for GPU resources. Any insights?'
This assesses your ability to respond to operational alerts. The interviewer wants you to move beyond a basic description of the problem by asking clarifying questions—specifically regarding the *cause* of the latency (GPU resource contention). Option 3 best captures this investigative approach.
8 / 15
PR Description: 'Implemented a new data pipeline using Kafka Connect and Flink. The pipeline ingests clickstream data from our website and transforms it into aggregated metrics for real-time dashboards. This significantly improves dashboard refresh rates.'
PR descriptions should be informative. This question tests your ability to provide context and detail for changes made. A good PR description includes technical choices (Kafka Connect, Flink), potential challenges (data quality, scaling), and the overall architecture. Option 1 highlights this omission.
9 / 15
Stand-up Update from @MLOpsEngineer: 'I've been working on automating the deployment of our new LLM model to the inference cluster. We're using Kubernetes and ArgoCD for continuous delivery. The biggest challenge is managing GPU resource allocation at scale.'
Stand-up updates are concise. The interviewer is probing your awareness of key operational considerations. Option 2 correctly identifies that simply listing tools doesn't address the core challenge of managing GPU resources at scale – a crucial aspect of AI infrastructure.
10 / 15
API Response (from a model monitoring service): `{"status": "error", "message": "KV cache exceeded maximum size. Evicted oldest entries.", "timestamp": "2024-10-27T14:30:00Z"}`
This scenario tests your ability to interpret an API response indicating a critical issue. The interviewer wants you to recognize that a KV cache overflow is a serious problem and trigger further investigation. Simply acknowledging the error is the appropriate initial response.
11 / 15
Code Review Comment: 'This Lambda function seems to be pulling data directly from S3 without any caching. Given the expected throughput of 10k requests/second, this could lead to significant S3 egress costs and potential throttling. What's your rationale for bypassing a cache layer here?'
This question tests your ability to respond constructively during a code review. The interviewer isn't just looking for an acknowledgement of the problem; they want you to demonstrate understanding of performance optimization strategies like caching and its impact on costs/throttling. Option 2 reflects this by acknowledging the need while hinting at a solution.
12 / 15
Slack Message from @DataEngineer: 'We're seeing high latency on our real-time inference endpoints for the new sentiment analysis model. The logs show a significant amount of time spent waiting for GPU resources. Any insights?'
This assesses your ability to respond to operational alerts. The interviewer wants you to move beyond a basic description of the problem by asking clarifying questions—specifically regarding the *cause* of the latency (GPU resource contention). Option 3 best captures this investigative approach.
13 / 15
PR Description: 'Implemented a new data pipeline using Kafka Connect and Flink. The pipeline ingests clickstream data from our website and transforms it into aggregated metrics for real-time dashboards. This significantly improves dashboard refresh rates.'
PR descriptions should be informative. This question tests your ability to provide context and detail for changes made. A good PR description includes technical choices (Kafka Connect, Flink), potential challenges (data quality, scaling), and the overall architecture. Option 1 highlights this omission.
14 / 15
Stand-up Update from @MLOpsEngineer: 'I've been working on automating the deployment of our new LLM model to the inference cluster. We're using Kubernetes and ArgoCD for continuous delivery. The biggest challenge is managing GPU resource allocation at scale.'
Stand-up updates are concise. The interviewer is probing your awareness of key operational considerations. Option 2 correctly identifies that simply listing tools doesn't address the core challenge of managing GPU resources at scale – a crucial aspect of AI infrastructure.
15 / 15
API Response (from a model monitoring service): `{"status": "error", "message": "KV cache exceeded maximum size. Evicted oldest entries.", "timestamp": "2024-10-27T14:30:00Z"}`
This scenario tests your ability to interpret an API response indicating a critical issue. The interviewer wants you to recognize that a KV cache overflow is a serious problem and trigger further investigation. Simply acknowledging the error is the appropriate initial response.
What does "AI Infrastructure Architect — Interview Questions — Best-Answer Practice" cover?
Practice answering AI Infrastructure Architect interview questions in professional English. 5 exercises on GPU cluster networking, FSDP vs DeepSpeed ZeRO, checkpoint strategy, KV cache management, and disaggregated LLM serving.
How many questions are in this interview set?
This set has 15 exercises, each with a full explanation.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do these exercises include model answers?
Yes. Each interview question gives you several possible responses and asks you to pick the one that communicates most clearly and completely — the explanation then breaks down exactly why that answer works, including the specific vocabulary a strong candidate would use.
What if I choose an answer that isn't the strongest one?
You'll see which option was correct and read a full explanation of why it's stronger than the alternatives, plus the key vocabulary and phrasing worth reusing in a real interview.
Can I retry the questions?
Yes — use the "Try again" button on the results screen to reset and go through the set again.
Is this the same as a real technical or behavioural interview?
No — it's focused practice for the language side of interviewing: recognising which phrasing sounds precise and confident versus vague, and knowing the vocabulary interviewers expect for this role. It won't replace mock interviews, but it builds the vocabulary you'll need in one.
Where can I find interview prep for other roles?
Browse the full Interview exercises hub for 170+ modules covering behavioural, technical, and system design rounds across dozens of IT roles, or check the "Next up" link below to continue.
Do I need an account, and is my progress saved?
No account is needed. Progress is tracked only for your current visit — reloading or leaving the page resets the counter.
Who writes these interview questions?
Every question is written by the CoderSlingo team based on real technical interview patterns for this role, then reviewed for accuracy and clarity.