Multimodal AI Engineer Interview Questions

Option B is strongest. The training mechanism section introduces the N×N similarity matrix (N matched + N²-N non-matching pairs) with the InfoNCE loss, which is the precise technical formulation. The zero-shot mechanism is explained step-by-step with the prompt template ("a photo of a {class_label}") making the procedure concrete. The "why it works" section provides the conceptual explanation — visual-semantic alignment — that ties the mechanism to the capability. The limitations section is comprehensive with four specific failure modes: fine-grained distinctions, spatial reasoning, prompt engineering sensitivity, and domain shift for medical/satellite images. Each limitation has a specific example (dog breeds, red ball spatial query) that shows genuine knowledge. CLIP vocabulary: InfoNCE loss — a contrastive loss that maximises similarity of positive pairs and minimises similarity of negative pairs in a batch. Contrastive pre-training — training by comparing matched vs. non-matched image-text pairs. Visual-semantic alignment — placing image and text representations of the same concept in the same region of embedding space. Zero-shot transfer — applying a model to a new task without any task-specific training. Prompt engineering — designing text prompts to maximise task performance. Options C and D are accurate but lack the N²-N negative pair formulation and the specific spatial reasoning limitation example.

Option B is strongest. The visual encoder section calculates the exact number of visual tokens (256 patch tokens + 1 CLS = 257 for 224×224 with 14×14 patches), which demonstrates real architectural knowledge rather than vague descriptions. The projection layer section explains the dimension mismatch (1024 → 4096) as the motivation for the projection, making the architecture choice non-arbitrary. The LLM section correctly explains that visual tokens are processed with standard causal self-attention — there is no special cross-attention mechanism in LLaVA, which is a common misconception. The two-stage training section is precise: stage 1 freezes everything except the projection (alignment), stage 2 also trains the LLM (instruction following). The context window impact section introduces the scaling problem that motivates LLaVA-HD and LLaVA-NeXT. LLaVA vocabulary: Vision Transformer (ViT) — transformer architecture applied to image patches as tokens. Patch embedding — the representation of a fixed-size image patch produced by ViT. Projection layer — a linear or MLP mapping from visual embedding space to language model embedding space. Feature alignment — stage 1 training that aligns the visual and language embedding spaces. Instruction tuning — fine-tuning on instruction-following pairs to enable task-specific behaviour. Options C and D are accurate but lack the patch count calculation and the causal attention clarification.

Option B is strongest. It presents two tokenisation approaches (patch-based and discrete VQ-VAE) with different use cases — patch-based is universal, discrete enables interleaved generation. The context window impact is quantified concretely (LLaMA-3 4096 context - 784 visual tokens = 3312 text positions). The three high-resolution strategies are named with specific implementations: dynamic tiling with the exact token count calculation (3456 tokens for a 1344×672 image in a 3×2 grid), the Q-Former's fixed 32-token compression regardless of resolution (the key advantage for high-res images), and the mixture of encoders approach. These three strategies represent the state of the art in 2024 VLM engineering. Visual tokenisation vocabulary: Patch tokenisation — dividing an image into fixed-size patches, each becoming one visual token. VQ-VAE — Vector Quantised Variational Autoencoder; maps image regions to discrete codebook tokens. Q-Former — BLIP-2's Query Transformer that compresses visual tokens via cross-attention from a fixed number of learned query vectors. Dynamic resolution — encoding high-resolution images by tiling into grid cells. Token compression — reducing the number of visual tokens after ViT encoding via pooling or cross-attention. Options C and D are accurate but lack the context window impact quantification and the Q-Former fixed-cost explanation.

Option B is strongest. It frames the comparison as two architectural families with named trade-offs, which is the correct interview structure. The self-attention section explains WHY it requires fine-tuning (the pre-trained LLM must adapt its attention patterns to include visual tokens) — a detail that distinguishes architectural understanding from surface knowledge. The cross-attention section precisely defines the Q/K/V assignment (Q from LLM hidden states, K and V from visual encoder), explains the parameter cost (new K/V projection matrices), and introduces the key architectural benefit (visual features separate from context window). The Flamingo tanh gate detail is a specific, elegant initialisation technique that experienced multimodal engineers recognise: by initialising cross-attention gates to zero, the model starts training as a standard text LLM and gradually incorporates visual information. Multimodal attention vocabulary: Cross-attention — attention where queries come from one modality and keys/values from another. Visual encoder output — the sequence of patch embeddings or compressed representations from the visual backbone. Tanh gate — a learned scalar gate that controls how much cross-attention output is added to the residual stream. Context consumption — visual tokens occupying context window positions, reducing available text space. Options C and D name the components correctly but lack the Q/K/V assignment explanation and the tanh gate initialisation rationale.

Option B is strongest. It names four challenges with specific solutions for each, which is the complete answer structure. The modality imbalance section provides three solution strategies (data rebalancing, loss weighting, staged training) with the key insight that text data outnumbers image-text pairs 100:1 — a concrete ratio that makes the problem real. The catastrophic forgetting section correctly names three mitigations with the specific data mix ratio (30-50% text-only). The alignment section explains the mechanism of failure (LLM misinterprets unaligned visual tokens as noise), which ties back to the LLaVA two-stage training rationale. The evaluation section introduces POPE for hallucination benchmarking — a specific current benchmark — and identifies the core evaluation problem (language prior shortcuts: VQA models scoring high by exploiting text statistics without understanding images). Multimodal training vocabulary: Modality imbalance — imbalanced data volume across modalities causing one modality to dominate training. Catastrophic forgetting — loss of previously learned capabilities when fine-tuning on new data. LoRA (Low-Rank Adaptation) — parameter-efficient fine-tuning that constrains weight updates to low-rank matrices. Language prior — the tendency of VQA models to answer questions using text statistics rather than image content. POPE — Polling-based Object Probing Evaluation, a benchmark for measuring VLM hallucination. Options C and D list the challenges correctly but lack the 100:1 imbalance ratio and the POPE benchmark explanation.