PyTorch interview questions

What a strong answer covers

• Explain that both are multi-dimensional array libraries, supporting similar mathematical operations.
• Highlight PyTorch Tensors' primary advantages: GPU acceleration and automatic differentiation (`autograd`).
• Mention the seamless interoperability between PyTorch Tensors and NumPy arrays (e.g., `.numpy()`, `torch.from_numpy()`).
• Discuss how Tensors are fundamental for building and training neural networks due to gradient tracking.

Where people lose the point

• ×Failing to mention `autograd` as a core differentiator.
• ×Incorrectly stating that NumPy arrays can directly utilize GPUs without external libraries.

What a strong answer covers

• Describe `autograd` as PyTorch's automatic differentiation engine that records operations on tensors to build a dynamic computation graph.
• Explain how `requires_grad=True` marks tensors for gradient tracking and how `grad_fn` attributes link operations.
• Detail the backward pass: `loss.backward()` traverses the graph backward, applying the chain rule to compute gradients for all leaf tensors with `requires_grad=True`.
• Emphasize its importance in deep learning for efficiently calculating gradients needed for optimization (e.g., gradient descent) without manual derivation.

Where people lose the point

• ×Confusing `autograd` with a static graph system (like TensorFlow 1.x).
• ×Misunderstanding that `loss.backward()` computes gradients for *all* tensors, not just those with `requires_grad=True`.

What a strong answer covers

• Define `requires_grad` as a boolean flag indicating whether PyTorch should track operations on a tensor to compute gradients.
• Explain that model parameters (weights, biases) typically have `requires_grad=True` by default, as their gradients are needed for optimization.
• Provide examples for `requires_grad=False`: input data, target labels, frozen layers in transfer learning, or during inference to save memory and computation.
• Mention how `torch.no_grad()` context manager can temporarily disable gradient tracking for blocks of code.

Where people lose the point

• ×Believing `requires_grad=False` prevents *any* computation, rather than just gradient tracking.
• ×Not understanding that `requires_grad` propagates through operations: if any input to an operation has `requires_grad=True`, the output will also have it.

What a strong answer covers

• Explain that `nn.Module` is a class that encapsulates layers with learnable parameters (weights, biases) and manages their state, providing methods like `to()` and `parameters()`.
• Describe `nn.functional` as a collection of stateless operations (e.g., `F.relu`, `F.conv2d`) that do not manage parameters or state.
• Illustrate usage: `nn.Module` for layers like `nn.Linear`, `nn.Conv2d`, or custom network architectures; `nn.functional` for activation functions, pooling operations, or within the `forward` method of an `nn.Module` for operations without learnable parameters.
• Highlight that `nn.Module` instances are typically added to `nn.Sequential` or composed within custom `nn.Module` classes, while `nn.functional` calls are direct function calls.

Where people lose the point

• ×Confusing `nn.functional` with `nn.Module` instances, e.g., trying to instantiate `F.relu()`.
• ×Not understanding that `nn.functional` operations can be used *inside* an `nn.Module`'s `forward` method.

What a strong answer covers

• Explain that custom layers/models inherit from `torch.nn.Module`.
• Detail the `__init__` method: where submodules (e.g., `nn.Linear`, `nn.Conv2d`) and learnable parameters (`nn.Parameter`) are defined and initialized.
• Describe the `forward` method: where the actual computation logic is implemented, defining how input tensors are transformed to output tensors.
• Mention other considerations like `super().__init__()` call, device management, and ensuring parameters are registered correctly.

Where people lose the point

• ×Forgetting to call `super().__init__()` in the custom module's `__init__`.
• ×Not registering custom tensors as `nn.Parameter` if they are meant to be learnable weights.

What a strong answer covers

• Data loading: Iterating through `DataLoader` to get batches of inputs and targets.
• Forward pass: Passing inputs through the model to get predictions (`outputs = model(inputs)`).
• Loss calculation: Comparing predictions with targets using a loss function (`loss = criterion(outputs, targets)`).
• Backward pass: Computing gradients of the loss with respect to model parameters (`loss.backward()`), preceded by `optimizer.zero_grad()` to clear old gradients.
• Parameter update: Adjusting model parameters based on computed gradients using an optimizer (`optimizer.step()`).

Where people lose the point

• ×Forgetting `optimizer.zero_grad()` or placing it incorrectly, leading to accumulated gradients.
• ×Confusing the order of `loss.backward()` and `optimizer.step()`.

What a strong answer covers

• Define an optimizer as an algorithm that adjusts model parameters (weights and biases) based on the gradients computed during the backward pass to minimize the loss function.
• Explain that optimizers take the model's parameters and a learning rate as input.
• Name common optimizers: SGD (Stochastic Gradient Descent), Adam, RMSprop.
• Briefly differentiate: SGD (basic, can be slow), Adam (adaptive learning rates for each parameter, generally good default), RMSprop (also adaptive, often good for recurrent networks).

Where people lose the point

• ×Confusing the optimizer's role with the loss function's role.
• ×Not understanding that optimizers operate on the *gradients* of the loss with respect to parameters.

What a strong answer covers

• Explain that the choice of loss function depends directly on the type of machine learning task (e.g., classification, regression) and the nature of the output.
• Provide examples for classification: `nn.CrossEntropyLoss` (for multi-class classification, combines `LogSoftmax` and `NLLLoss`), `nn.BCEWithLogitsLoss` (for binary classification, combines `Sigmoid` and `BCELoss`).
• Provide examples for regression: `nn.MSELoss` (Mean Squared Error) or `nn.L1Loss` (Mean Absolute Error).
• Mention that custom loss functions can be implemented by inheriting from `nn.Module` or as simple functions.

Where people lose the point

• ×Suggesting `nn.CrossEntropyLoss` for binary classification without understanding its internal `LogSoftmax`.
• ×Not considering the output activation function when choosing a loss (e.g., using `BCELoss` directly on logits without `Sigmoid`).

What a strong answer covers

• Define `Dataset` as an abstract class representing a collection of samples and their labels, requiring `__len__` and `__getitem__` methods.
• Define `DataLoader` as an iterator that wraps a `Dataset`, providing features like batching, shuffling, and multi-process data loading.
• Explain their synergy: `Dataset` handles how individual samples are fetched and preprocessed, while `DataLoader` handles how these samples are aggregated into batches and efficiently delivered to the model.
• Highlight benefits: efficient memory usage, parallel data loading, and simplified training loops.

Where people lose the point

• ×Confusing `Dataset` with `DataLoader` or thinking they are interchangeable.
• ×Not understanding that `DataLoader` is responsible for batching and shuffling, not `Dataset`.

What a strong answer covers

• Explain how to check for GPU availability (`torch.cuda.is_available()`) and define a `device` variable (e.g., `torch.device('cuda' if torch.cuda.is_available() else 'cpu')`).
• Describe moving tensors to a device using `.to(device)` (e.g., `tensor.to(device)`, `model.to(device)`).
• Emphasize the importance of moving *both* the model and *all* input data to the same device before computation.
• Best practices: define `device` once, pass it to functions/classes, and use `.to(device)` consistently for all tensors and modules.

Where people lose the point

• ×Moving only the model to GPU but not the input data, leading to runtime errors.
• ×Hardcoding device names ('cuda:0') instead of using `torch.device` for flexibility.

What a strong answer covers

• Recommend saving the model's `state_dict` using `torch.save(model.state_dict(), PATH)`.
• Explain that `state_dict` is a Python dictionary mapping each layer to its learnable parameters (weights and biases).
• Describe loading: instantiate the model architecture first, then load the `state_dict` using `model.load_state_dict(torch.load(PATH))`.
• Advantages of `state_dict`: smaller file size, more flexible (can load into different architectures if keys match), allows for transfer learning by loading partial weights.

Where people lose the point

• ×Attempting to load a `state_dict` without first instantiating the model architecture.
• ×Confusing saving the entire model object with saving just the `state_dict` and not understanding the implications.

What a strong answer covers

• Explain that `torch.no_grad()` is a context manager that disables gradient calculation within its scope.
• Primary use cases: during inference (evaluation, prediction) and validation loops.
• Reasons for use: saves memory by not storing intermediate activations for gradient computation, speeds up computation as `autograd` doesn't need to build the graph, and prevents accidental updates to model parameters.
• Mention that it's also useful when freezing parts of a model for transfer learning, though `requires_grad=False` on parameters is more explicit for that.

Where people lose the point

• ×Using `torch.no_grad()` during training, which would prevent parameter updates.
• ×Believing it completely stops all computation, rather than just gradient tracking.

What a strong answer covers

• Load a pre-trained model (e.g., from `torchvision.models`) and its pre-trained weights.
• Freeze the parameters of the pre-trained layers by setting `param.requires_grad = False` for all parameters in the base model.
• Modify the final classification/regression head of the pre-trained model to match the new task's output classes/dimensions.
• Train only the newly added/modified layers, allowing the model to adapt to the new task while leveraging the learned features from the pre-trained backbone.

Where people lose the point

• ×Forgetting to freeze the base model's parameters, leading to retraining the entire network.
• ×Not correctly adapting the final layer to the new task's output shape.

What a strong answer covers

• Define data augmentation as techniques used to artificially increase the diversity of a training dataset by applying random transformations to the existing data.
• Explain its benefits: reduces overfitting, improves model generalization, and makes models more robust to variations in input data.
• Describe implementation for image data using `torchvision.transforms`: composing multiple transformations (e.g., `RandomResizedCrop`, `RandomHorizontalFlip`, `ColorJitter`) into a `Compose` object.
• Mention that these transforms are typically applied within the `__getitem__` method of a `Dataset`.

Where people lose the point

• ×Applying data augmentation to the validation or test sets.
• ×Not understanding that augmentation should introduce *realistic* variations, not just random noise.

What a strong answer covers

• Define gradient clipping as a technique to prevent exploding gradients by scaling down gradients if their L2 norm exceeds a certain threshold.
• Explain the problem of exploding gradients: gradients become extremely large during backpropagation, leading to very large parameter updates and unstable training (NaNs or infs in loss).
• Describe its use in PyTorch: typically applied after `loss.backward()` and before `optimizer.step()` using `torch.nn.utils.clip_grad_norm_`.
• Highlight its importance for RNNs and LSTMs due to their sequential nature and repeated matrix multiplications, which can easily lead to exploding gradients over long sequences.

Where people lose the point

• ×Confusing gradient clipping with gradient vanishing (which is a different problem).
• ×Applying gradient clipping before `loss.backward()` or after `optimizer.step()`.

What a strong answer covers

• Explain that `model.train()` sets the model to training mode, enabling specific layers (like `nn.Dropout` and `nn.BatchNorm`) to behave as expected during training.
• Explain that `model.eval()` sets the model to evaluation mode, disabling `nn.Dropout` and freezing `nn.BatchNorm` layers to use their accumulated running statistics.
• Importance: `nn.Dropout` randomly zeroes out neurons during training to prevent overfitting; `nn.BatchNorm` uses batch statistics during training but global running statistics during evaluation for consistent predictions.
• Emphasize that failing to switch modes can lead to inconsistent or poor performance during evaluation/inference.

Where people lose the point

• ×Forgetting to call `model.eval()` before validation/inference, leading to non-deterministic results or poor performance.
• ×Believing these methods only affect gradient computation (which is handled by `requires_grad` and `torch.no_grad()`).