15 selected computer vision interview questions grouped by seniority level. Use them to review fundamentals, practical trade-offs, and senior-level production reasoning.
1What is a discrete 2D convolution on an image, and how does it relate to kernels, filters, and cross-correlation?
Discrete 2D convolution on an image is a linear, spatial filtering operation where a matrix of weights (a kernel) slides across the input image. At each spatial location, it computes an element-wise product between the kernel weights and the overlapping local image patch (receptive field) and sums the results (often adding a bias term) to produce a single pixel value in the output feature map. In classical mathematics and signal processing, true convolution involves flipping (rotating by 180 degrees) the kernel horizontally and vertically before computing the sliding inner product: $(I * K)(i, j) = \sum_m \sum_n I(i - m, j - n) K(m, n)$. In contrast, cross-correlation computes the sliding dot product directly without flipping the kernel: $(I \star K)(i, j) = \sum_m \sum_n I(i + m, j + n) K(m, n)$. In deep learning frameworks like PyTorch and TensorFlow, the operation implemented under the name "convolution" is actually cross-correlation. Because kernel weights are learnable parameters optimized directly via backpropagation, the network simply learns the appropriately oriented weights, making the mathematical kernel flip computationally redundant during both training and inference.
2When would you use RGB, HSV, Lab, YCbCr, or grayscale representations in a vision pipeline?
Use the representation that preserves the cues needed by the task and matches the model's training preprocessing. RGB is common for learned models; many OpenCV APIs use BGR, so verify channel order rather than assuming a framework enforces it. HSV/HSL can make color thresholding convenient, but hue is unstable near zero saturation and is not generally invariant to illumination or white-balance changes. CIE Lab separates lightness from opponent color coordinates and is approximately perceptually uniform under specified viewing/reference-white assumptions; Euclidean Delta E 1976 is only an approximate perceptual difference measure. YCbCr separates luma Y' from color-difference channels and supports chroma subsampling in many image/video formats; coefficients, range, and transfer functions depend on the standard. Grayscale is useful when color is unnecessary, reducing raw channel storage from three channels to one at equal precision, but it does not guarantee a two-thirds reduction in total model compute. Preserve color when it carries label information, and verify scaling and conversion conventions.
import cv2
import numpy as np
# Sample image: saturated red under normal and shaded lighting
img_bgr = np.zeros((100, 100, 3), dtype=np.uint8)
img_bgr[:, :50] = [0, 0, 255] # Bright Red in BGR
img_bgr[:, 50:] = [0, 0, 120] # Shaded/Dark Red in BGR
# Convert to HSV (OpenCV H: 0-179, S: 0-255, V: 0-255)
img_hsv = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2HSV)
# Red Hue wraps around 0/180; track red with single Hue range [0, 10] & high saturation
lower_red = np.array([0, 100, 50])
upper_red = np.array([10, 255, 255])
mask_hsv = cv2.inRange(img_hsv, lower_red, upper_red)
print("HSV segmented pixels:", np.count_nonzero(mask_hsv), "out of", mask_hsv.size)
3Why are channel-wise image normalization, per-image normalization, and consistent pixel scaling important in vision systems?
Use the pixel range, channel order, and normalization expected by the trained model consistently in training and inference. For 8-bit images, division by 255 maps values to [0,1], which rescales but does not center them at zero; mapping to [-1,1] centers the range, not necessarily the data mean. Images can also use other integer depths or floating-point physical units. Large unscaled values can make optimization poorly conditioned, but do not inevitably cause exploding gradients. Dataset channel standardization uses (x-mu_c)/sigma_c with training-set statistics or the pretrained model's prescribed constants; it gives zero mean and unit variance on the distribution used to estimate them, not every new image or dataset. Per-image z-score uses that image's mean and standard deviation with protection against near-zero variance; min-max scaling is a different operation. InstanceNorm is a neural-network layer that typically normalizes each sample and channel over spatial dimensions, not a synonym for all per-image preprocessing. Per-image normalization may reduce brightness/contrast variation but can remove useful absolute-intensity information, so validate it for the task.
import torch
import torchvision.transforms as T
from PIL import Image
import numpy as np
# Create a mock uint8 RGB image
raw_img = Image.fromarray(np.random.randint(0, 256, (224, 224, 3), dtype=np.uint8))
# Standard transform pipeline:
# 1. ToTensor scales uint8 [0, 255] -> float32 [0.0, 1.0] and permutes HWC -> CHW
# 2. Normalize standardizes per-channel using dataset mean and std
preprocess = T.Compose([
T.ToTensor(),
T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
tensor_img = preprocess(raw_img)
print("Tensor shape:", tensor_img.shape)
print("Tensor dtype:", tensor_img.dtype)
print("Channel 0 min/max:", round(tensor_img[0].min().item(), 2), round(tensor_img[0].max().item(), 2))
4How do geometric and photometric augmentations encourage invariance, and how do you decide whether an augmentation preserves labels?
Geometric augmentations (such as rotation, translation, scaling, and flipping) alter the spatial coordinate mapping of pixels, while photometric augmentations (such as brightness, contrast, hue jitter, and blur) alter pixel intensity and color channels without modifying spatial coordinates. In classification, both categories expose the model to realistic variations, encouraging it to learn invariant representations where predictions remain stable despite changes in pose, viewpoint, or lighting conditions. In spatial prediction tasks (detection and segmentation), geometric transforms require matching transformations to ground-truth boxes or masks to preserve equivariance. Deciding whether an augmentation preserves labels requires evaluating domain semantics and physical priors. A transform is label-preserving if the resulting image remains a plausible instance of the original target class without altering its semantic identity. For example, horizontal flipping preserves identity for general objects (like cars or animals), but alters identity or invalidates characters in optical character recognition (OCR) and digit classification. Similarly, full 360-degree rotation is valid in histology or satellite imagery where no canonical orientation exists, but unnatural in autonomous driving where upside-down objects violate real-world physical constraints.
5What are nearest-neighbor, bilinear, bicubic, and area interpolation, and how can resizing introduce aliasing or blur?
Interpolation methods estimate pixel intensities at continuous coordinates when resampling onto a new discrete grid: - **Nearest-Neighbor:** Assigns the intensity of the closest discrete pixel. It is computationally $O(1)$ and preserves exact original values, but creates blocky artifacts when upsampling and severe jagged edges when downsampling. - **Bilinear Interpolation:** Computes a distance-weighted average of the $2 \times 2$ (4 nearest) neighbor pixels using linear interpolation along both axes, yielding smooth transitions with moderate blurring. - **Bicubic Interpolation:** Samples a $4 \times 4$ (16 nearest) neighborhood using cubic polynomial kernels. It captures local intensity gradients, producing sharper edges and smoother curves than bilinear, but can produce overshoot/ringing artifacts around sharp transitions. - **Area (Box) Interpolation:** Computes the fractional area overlap of target pixels projected onto source pixels and averages the underlying source intensities. It is especially effective for downsampling without moiré. **Aliasing vs. Blur Mechanics:** - **Aliasing** occurs during downsampling when high-frequency content exceeds the Nyquist frequency ($f_s / 2$) of the new sampling rate without sufficient low-pass filtering. High frequencies fold back into lower frequency bands, producing false patterns, moiré, and jagged edges. - **Blur** occurs when interpolation kernels act as low-pass smoothing filters that attenuate legitimate high frequencies (e.g., bilinear smoothing) or when high-frequency details are averaged out during downsampling.
6What real-world visual failure modes should a vision system be evaluated against, and how do they shift the input distribution?
Real-world visual failure modes fall into several broad categories: environmental conditions (low light, direct glare, rain, snow, fog), sensor and optical effects (motion blur, defocus blur, high ISO sensor noise, rolling shutter skew), compression and transmission degradation (JPEG blockiness, video bitrate throttling, downsampling), and semantic/spatial shifts (partial occlusion, truncation, out-of-distribution object poses). These failure modes shift the input distribution away from nominal training domains at multiple levels. At the pixel level, they alter luminance and chrominance histograms, destroy local contrast, reduce the signal-to-noise ratio (SNR), and introduce artificial high-frequency artifacts or blur kernels. At the structural and semantic level, they attenuate edge gradients, obscure key visual features, distort silhouettes, and hide critical diagnostic regions. When models trained on clean data encounter these shifted inputs, low-level convolutional or attention feature extractors fail to activate correctly, producing dropped detections, hallucinated false positives on noise patterns, and overconfident errors.
7How do classical descriptors such as HOG represent image structure, and what limitations do they have compared with learned CNN features?
The Histogram of Oriented Gradients (HOG) is a classical handcrafted feature descriptor designed to capture local object shape and appearance through the distribution of intensity gradient orientations. The HOG computation pipeline involves: (1) calculating horizontal and vertical image gradients (e.g., with 1D derivative filters [-1, 0, 1]) to obtain gradient magnitude and orientation; (2) dividing the image into small spatial regions called 'cells' (such as 8x8 pixels) and accumulating 1D histograms of gradient orientations weighted by gradient magnitude; (3) grouping adjacent cells into larger overlapping 'blocks' (such as 2x2 cells) and normalizing block feature vectors (using L2-norm or L1-sqrt) to achieve robustness against local illumination, contrast, and shadowing variations; and (4) concatenating normalized block vectors into a final 1D feature representation, historically paired with linear SVMs for pedestrian and object detection. Compared to learned Convolutional Neural Network (CNN) features, HOG has major limitations: First, HOG features are fixed and handcrafted, capturing only low-level local edge orientation statistics without the ability to learn task-specific hierarchical representations (such as textures, object parts, and semantic concepts). Second, HOG provides very limited geometric invariance: while block normalization handles monotonic illumination changes and small spatial shifts within cells, HOG is fragile under 3D out-of-plane rotations, significant scale variations, non-rigid pose articulations, and heavy background clutter. In contrast, CNNs learn multi-layer non-linear abstractions optimized end-to-end for the target objective.
8How do stride, padding, dilation, and kernel size determine convolution output size and receptive field?
The spatial output dimensions of a convolutional layer and its cumulative receptive field are determined by kernel size $k$, stride $s$, padding $p$, and dilation $d$: 1. **Output Spatial Size**: Dilation introduces $(d - 1)$ gaps between kernel elements, giving an effective kernel size of $k' = d(k - 1) + 1$. The output dimension along an axis is given by: $$O = \left\lfloor \frac{I + 2p - k'}{s} \right\rfloor + 1 = \left\lfloor \frac{I + 2p - d(k - 1) - 1}{s} \right\rfloor + 1$$ Here, stride downsamples the resolution by advancing the kernel $s$ pixels per step, padding adds virtual border pixels to preserve or adjust spatial dimensions, and dilation expands the filter's footprint without adding extra parameters. 2. **Receptive Field (RF)**: The theoretical receptive field describes the spatial extent in the input image that influences a particular activation. Across stacked layers, the receptive field $RF_l$ and cumulative stride (jump $j_l$) update recursively: - Jump: $j_l = j_{l-1} \cdot s_l$, with $j_0 = 1$ - Receptive Field: $RF_l = RF_{l-1} + (k'_l - 1) \cdot j_{l-1}$, with $RF_0 = 1$ Strides and pooling expand the receptive field multiplicatively across depth because subsequent layer kernel steps correspond to larger pixel jumps in the original input coordinate space. Dilation expands the receptive field additively within a single layer by increasing $k'_l$ without spatial downsampling.
9How does a convolution kernel correspond to a classical linear image filter, and when can a 2D filter be made separable?
In classical linear signal and image processing, a convolution kernel represents the spatial impulse response (point spread function) of a Linear Shift-Invariant (LSI) system. Applying the kernel is a spatial-domain linear filtering operation that shapes the image's frequency response (its 2D Fourier transform). Low-pass kernels (e.g., Gaussian, box blur) attenuate high spatial frequencies to suppress noise and smooth textures, while high-pass or band-pass kernels (e.g., Sobel, Laplacian, Prewitt) amplify high frequencies to detect edges and gradients. A 2D filter kernel $K \in \mathbb{R}^{M \times N}$ is spatially separable if it can be decomposed into the outer product of two 1D filters: $K = u \cdot v^T$, where $u \in \mathbb{R}^{M \times 1}$ and $v \in \mathbb{R}^{N \times 1}$. In linear algebra terms, a 2D matrix is separable if and only if its matrix rank is 1. This can be verified via Singular Value Decomposition (SVD), where exactly one singular value is non-zero ($ sigma_1 > 0, \sigma_2 = \dots = 0$). Separability significantly reduces computational complexity. Applying a non-separable $K \times K$ kernel to an $H \times W$ image requires $O(H \cdot W \cdot K^2)$ multiplications and additions. When decomposed into two consecutive 1D passes (horizontal then vertical), the complexity drops to $O(H \cdot W \cdot 2K)$. For a $15 \times 15$ kernel, this provides roughly a $7.5\times$ speedup.
10How do resizing, center cropping, random-resized cropping, stretching, and letterboxing change the input distribution?
Different resizing and cropping strategies shift the input data distribution across geometry, scale, content coverage, and spatial boundaries: 1. **Direct Resizing (Stretching/Squishing):** Rescales the image non-isotropically to a target dimension $(H, W)$, distorting original aspect ratios. This forces the model to process deformed object shapes (e.g., circular objects become elongated ellipses). 2. **Center Cropping:** Extracts a central crop of fixed size or ratio. It preserves the native aspect ratio and local scale, but introduces a strong center bias (assuming targets are centrally framed) and discards peripheral context or cuts off edge-positioned objects. 3. **Random-Resized Cropping (RRC):** Extracts random sub-regions across varying area scales and aspect ratios, then resizes them to a fixed dimension. This broadens the training scale distribution and encourages part-based feature learning, but extreme crops may exclude the target object entirely. 4. **Letterboxing (Isotropic Resizing with Padding):** Scales the image uniformly until its longest dimension fits the target size, then pads the remaining borders with a constant value. It maintains true object proportions and aspect ratios, but introduces artificial high-contrast borders and uninformative padded pixels.
11What annotation noise modes occur in vision datasets, and how would you detect or mitigate them?
Computer vision datasets commonly exhibit four primary annotation noise modes: 1. **Categorical Label Noise:** An image or object is assigned the wrong class label (e.g., misclassifying a cat as a dog, or confusing subtle fine-grained classes). 2. **Missing Annotations (Omission Noise):** Valid foreground objects are left unannotated. In object detection, omitted targets are treated as background negatives, directly penalizing correct model detections during training. 3. **Bounding-Box / Keypoint Jitter (Localization Noise):** Inaccurate, loose, or shifted bounding box coordinates and landmark locations caused by human annotator inconsistency. 4. **Mask Boundary Ambiguity:** Inconsistent or coarse segmentation outlines along complex or blurry boundaries (e.g., hair, semi-transparent surfaces, motion blur). **Detection and Mitigation Strategies:** - **Loss Outlier Tracking & Confident Learning:** Tracking sample loss across training epochs identifies persistent high-loss outliers, which often indicate mislabeled or omitted instances. Algorithms like Confident Learning estimate noise distributions to prune or correct errors. - **Out-of-Fold (OOF) Prediction Discrepancies:** Training cross-validation models and comparing holdout predictions against ground truth highlights mislabeled images and missing bounding boxes. - **Inter-Annotator Agreement (IAA) & Adjudication:** Measuring metrics like Cohen's Kappa (classification) or mean Intersection-over-Union (localization) across multiple annotators flags low-agreement samples for consensus review. - **Robust Loss Formulations:** Using label smoothing, noise-tolerant classification losses, or robust IoU-based regression losses reduces sensitivity to jitter and mislabeling.
import torch
import torch.nn.functional as F
def find_label_noise_candidates(model, dataloader, device, top_k=50):
model.eval()
sample_losses = []
with torch.no_grad():
for batch_idx, (images, targets, sample_ids) in enumerate(dataloader):
images, targets = images.to(device), targets.to(device)
logits = model(images)
loss_per_sample = F.cross_entropy(logits, targets, reduction='none')
for sid, l, target, pred in zip(sample_ids, loss_per_sample.cpu(), targets.cpu(), logits.argmax(dim=-1).cpu()):
sample_losses.append({'id': sid, 'loss': l.item(), 'target': target.item(), 'pred': pred.item()})
sample_losses.sort(key=lambda x: x['loss'], reverse=True)
return sample_losses[:top_k]
12How does class imbalance affect image classification, detection, and dense prediction training?
Class imbalance affects optimization dynamics and loss landscapes across vision tasks: 1. **Image Classification:** When majority classes outnumber minority classes, empirical risk minimization causes parameter updates to be dominated by majority class gradients. The model learns the empirical class prior $P(Y)$ and biases its decision boundary against rare classes. This yields high overall top-1 accuracy while suffering from severe minority recall collapse. 2. **Object Detection:** - *Foreground-Background Imbalance:* In dense single-stage and anchor-based detectors, hundreds of thousands of candidate locations are evaluated per image, where >99% are background. Even if individual easy background anchors yield small losses, their vast aggregate gradient swamps the informative gradient signals from sparse foreground objects. - *Foreground Class Imbalance:* Common object categories appear orders of magnitude more frequently than rare categories, reducing the effective sample size and learning signal for rare classes. 3. **Dense Prediction (Segmentation):** Pixel-level imbalance is severe because large background and 'stuff' classes (e.g., road, sky) occupy millions of pixels, whereas small 'thing' classes (e.g., traffic signs, pedestrians) may occupy fewer than 0.1% of image pixels. Standard per-pixel cross-entropy loss is dominated by large-area classes, leading to the under-segmentation or complete disappearance of small, thin structures.
13How would you debug NaNs or unstable loss during large-scale distributed vision training?
Debugging NaNs or loss instability in large-scale Distributed Data Parallel (DDP) vision training requires isolating whether the root cause originates from bad data inputs, numerical overflow/underflow in mixed precision (AMP), or optimization instabilities across workers. First, capture reproducible inputs/seeds/configuration and add diagnostic hooks (anomaly detection itself does not make execution deterministic): enable anomaly detection (`torch.autograd.set_detect_anomaly(True)`), register gradient/activation hooks to catch the exact layer where NaNs appear, and add strict data validation in the DataLoader (asserting finite values, checking for 0-byte corrupt images, empty bounding boxes, or zero-variance normalization divisors). Minibatch-level logging should track per-rank loss, input URIs, gradient norm before clipping, and GradScaler scale factors. Second, inspect Automatic Mixed Precision (AMP) and dynamic loss scaling: in FP16, large gradients easily overflow (`> 65504`), causing GradScaler to skip invalid updates and reduce its scale by its configured backoff factor. Persistent scale collapse is a symptom to investigate, not normal convergence to zero; switching unstable operations (e.g., softmax, LayerNorm, focal loss exponents, or bounding box IoU denominators) to FP32 or adopting BF16 (which matches FP32's dynamic range) typically stabilizes training. Finally, check DDP-specific pitfalls like all-reduce operations propagating NaNs from a single worker to all ranks, learning rate warmup scaling (e.g., linear scaling rule with large global batch sizes), and gradient clipping after unscaling AMP gradients. Empty annotations can be valid negatives, so validate them against the task contract rather than rejecting every empty image.
import torch
from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
for step, (images, targets, uris) in enumerate(dataloader):
# 1. Input sanitization
if not torch.isfinite(images).all():
print(f"Corrupt input detected from URIs: {uris}")
continue
optimizer.zero_grad(set_to_none=True)
with autocast(dtype=torch.float16):
outputs = model(images)
loss = criterion(outputs, targets)
if not torch.isfinite(loss):
print(f"NaN/Inf loss at step {step} on rank {torch.distributed.get_rank()}; skipping step.")
continue
scaler.scale(loss).backward()
# Unscale before clipping to inspect true gradient norms
scaler.unscale_(optimizer)
grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
if torch.isnan(grad_norm) or torch.isinf(grad_norm):
print(f"Warning: Grad norm is {grad_norm}. Scaler will adjust.")
scaler.step(optimizer)
scaler.update()
14How would you design an online digital video stabilization component for a camera with frame jitter?
Estimate robust inter-frame motion from tracked corners or matched descriptors, rejecting foreground motion and selecting a suitable translation/similarity/affine/homography model. Define transform directions explicitly: with column vectors and H_t mapping frame t-1 coordinates to frame t, the reference-to-current path is P_t = H_t * P_{t-1}. Smooth that path to S_t using a causal filter or a bounded lookahead window, with proper rotation/transform handling rather than arbitrary averaging of matrix entries. The forward correction mapping current-frame coordinates into the stabilized frame is C_t = S_t * inverse(P_t); pass the transform or its inverse according to the warp API's sampling convention. Bound crop/zoom and stabilization strength to handle invalid borders; reset or recover on tracking loss and scene cuts. Strong parallax and rolling shutter may require richer motion models or inertial assistance. A strictly causal implementation uses no future frames; each lookahead frame adds capture latency, so choose buffering and crop policy from measured output quality and latency requirements.
15How would you optimize a real-time vision pipeline when the detector is slower than the target frame rate?
When an object detector cannot sustain the target frame rate (e.g., running at 15-20 FPS on a 60 FPS video feed), a robust production architecture decouples detection from the real-time presentation loop using an asynchronous multi-threaded pipeline combining heavy detection with lightweight tracking. In this hybrid architecture, an ingestion thread captures video frames continuously. The heavy detector runs asynchronously as a 'keyframe detector' on a background worker thread. Meanwhile, a lightweight tracker (such as Lucas-Kanade optical flow or a correlation filter, optionally combined with Kalman prediction) runs synchronously on every frame at full 60 FPS, maintaining predicted object state between detector updates. ByteTrack/BoT-SORT are tracking-by-detection systems: without new detections they need an explicit prediction/visual-tracking policy, and track age, uncertainty, and new-object recall must be monitored. To handle lag and backpressure without introducing unbounded latency or stale frames, bounded ring buffers and latest-frame drop policies are used. When the detector finishes a frame T_0 at time T_curr, its output is stale. The pipeline performs coordinate realignment/back-projection: it associates the delayed detection results with the historical tracklet state at T_0, updates identities and missed tracks, and propagates the corrections forward to T_curr via the tracking motion vectors or Kalman filter predict steps.
import queue
import threading
frame_queue = queue.Queue(maxsize=1) # Drop stale frames, keep latest
det_result_queue = queue.Queue()
def detector_worker():
while True:
frame, frame_id, timestamp = frame_queue.get()
boxes, scores, classes = heavy_detector.infer(frame)
det_result_queue.put({'frame_id': frame_id, 'boxes': boxes, 'timestamp': timestamp})
def realtime_pipeline(video_stream):
tracker = FastTracker() # e.g. Optical flow or Kalman tracker
for frame, frame_id, timestamp in video_stream:
if frame_queue.empty():
frame_queue.put((frame, frame_id, timestamp))
if not det_result_queue.empty():
det_result = det_result_queue.get()
tracker.reconcile_and_correct(det_result, current_frame_id=frame_id)
active_tracks = tracker.update(frame)
display_or_downstream(frame, active_tracks)