Large Language Models interview questions

What a strong answer covers

• Explain the fundamental limitation of RNNs/LSTMs: sequential processing, which hinders parallelization and long-range dependency capture.
• Describe how Transformers overcome this by processing sequences in parallel using self-attention.
• Mention the advantages of Transformers: faster training, better handling of long-range dependencies, and superior scalability for larger models.
• Briefly touch upon the role of self-attention in allowing direct interaction between any two tokens, regardless of their distance.

Where people lose the point

• ×Failing to mention parallelization as a key advantage of Transformers.
• ×Incorrectly attributing the ability to handle long-range dependencies solely to RNNs/LSTMs' memory cells.
• ×Not explaining *how* self-attention enables direct interaction between distant tokens.

What a strong answer covers

• Define self-attention as a mechanism that allows a model to weigh the importance of different words in an input sequence when processing each word.
• Explain that it helps the model understand the context and relationships between words, regardless of their position.
• Mention its role in capturing long-range dependencies within a sequence.
• Briefly describe how it computes a weighted sum of values based on query-key similarities.

Where people lose the point

• ×Confusing self-attention with simple word embeddings or convolutional filters.
• ×Not emphasizing the 'self' aspect – that it relates different parts of the *same* input sequence.
• ×Failing to connect it to understanding context or relationships between words.

What a strong answer covers

• Define zero-shot prompting as the ability of an LLM to perform a task without any explicit examples of that task provided in the prompt.
• Explain that the model relies solely on its pre-trained knowledge and general understanding of instructions.
• Provide a clear example, such as 'Translate 'Hello' to French.' or 'Summarize the following text: [text]'.
• Highlight that it demonstrates the model's generalization capabilities from its vast training data.

Where people lose the point

• ×Including examples in the prompt, which would make it few-shot.
• ×Confusing it with fine-tuning or training from scratch.
• ×Providing a vague example that doesn't clearly illustrate the concept.

What a strong answer covers

• Define hallucination as the phenomenon where an LLM generates information that is factually incorrect, nonsensical, or not supported by its training data or provided context.
• Explain that these outputs are often presented confidently, making them difficult to distinguish from accurate information.
• Discuss why it's a concern: it undermines trust, can lead to misinformation, and makes LLMs unreliable for critical applications.
• Mention potential causes like training data biases, model overconfidence, or lack of real-world grounding.

Where people lose the point

• ×Confusing hallucination with simply making a mistake or generating irrelevant text.
• ×Not emphasizing the 'factually incorrect' or 'unsupported' aspect.
• ×Failing to explain *why* it's a significant problem for practical applications.

What a strong answer covers

• **Pre-training:** Describe it as the initial, unsupervised phase on massive, diverse datasets (e.g., internet text, code) to learn general language understanding, grammar, and world knowledge. Mention objectives like next-token prediction.
• **Fine-tuning:** Explain it as the subsequent, supervised phase on smaller, task-specific, labeled datasets to adapt the pre-trained model for a particular downstream task (e.g., sentiment analysis, summarization).
• Highlight the difference in data size, task specificity, and computational cost between the two phases.
• Emphasize that fine-tuning leverages the broad knowledge acquired during pre-training (transfer learning).

Where people lose the point

• ×Suggesting that fine-tuning requires as much data as pre-training.
• ×Confusing the objectives of each phase (e.g., pre-training for a specific task).
• ×Failing to connect fine-tuning to the concept of transfer learning.

What a strong answer covers

• Explain that the self-attention mechanism processes all tokens in a sequence simultaneously, meaning it inherently lacks information about the relative or absolute order of tokens.
• State that positional encoding is added to the input embeddings to inject this crucial sequential information into the model.
• Describe how these encodings allow the model to understand word order, which is vital for grammar, syntax, and meaning.
• Mention that without positional encoding, 'dog bites man' and 'man bites dog' would be indistinguishable in terms of word order.

Where people lose the point

• ×Incorrectly stating that self-attention itself captures positional information.
• ×Failing to explain *why* the Transformer architecture loses positional information (parallel processing).
• ×Not clearly articulating that positional encoding *adds* this information back.

What a strong answer covers

• **Few-shot learning:** Explain it as providing a few examples directly within the prompt to guide the LLM's behavior for a specific task, without updating model weights. It's quick and requires no training data.
• **Fine-tuning:** Describe it as updating the model's weights by training on a labeled dataset for a specific task. It's more resource-intensive but can lead to higher performance and specialization.
• **Similarities:** Both aim to adapt an LLM for a specific task.
• **Differences:** Discuss resource requirements (compute, data), permanence of learning (temporary for few-shot, permanent for fine-tuning), and performance ceiling (fine-tuning generally achieves better task-specific performance).

Where people lose the point

• ×Confusing few-shot learning with updating model parameters.
• ×Understating the resource requirements for fine-tuning.
• ×Failing to acknowledge that few-shot learning is a form of in-context learning, not weight updates.

What a strong answer covers

• **Reduced Hallucinations:** Explain how RAG grounds the LLM's responses in factual, retrieved information, making outputs more reliable and less prone to generating incorrect facts.
• **Access to Up-to-Date/Proprietary Information:** Describe how RAG allows LLMs to incorporate external, real-time, or private knowledge bases that were not part of their original training data.
• **Improved Explainability/Attribution:** Mention that RAG can provide sources for the generated information, increasing transparency and allowing users to verify facts.
• **Cost-Effectiveness:** Highlight that RAG can achieve strong performance on specific tasks without requiring expensive and time-consuming fine-tuning or retraining of the base LLM.

Where people lose the point

• ×Suggesting RAG completely eliminates hallucinations.
• ×Failing to mention the ability to use proprietary or real-time data.
• ×Not connecting RAG to improved factual accuracy and trustworthiness.

What a strong answer covers

• **Data Curation:** Explain the importance of carefully selecting, filtering, and balancing training data to reduce over-representation or under-representation of certain groups.
• **Model Design/Training:** Discuss techniques like adversarial training, debiasing algorithms during fine-tuning, or using fairness-aware loss functions.
• **Prompt Engineering:** Describe how crafting prompts to explicitly request fair, neutral, or diverse responses can reduce biased outputs.
• **Post-deployment Monitoring & Feedback:** Emphasize continuous monitoring of LLM outputs for bias and establishing feedback loops for correction and improvement.
• **Human-in-the-Loop:** Mention the role of human review and oversight in critical applications to catch and correct biased outputs.

Where people lose the point

• ×Focusing solely on one stage (e.g., just data) and ignoring others.
• ×Suggesting that bias can be completely eliminated, rather than mitigated.
• ×Providing vague solutions without concrete technical or procedural examples.

What a strong answer covers

• Define CoT prompting as a technique where the LLM is prompted to generate a series of intermediate reasoning steps before providing the final answer.
• Explain that it mimics human thought processes, breaking down complex problems into smaller, manageable parts.
• Discuss its effectiveness: it allows the LLM to allocate more computation to the problem, reduces the likelihood of errors, and improves the transparency of the reasoning process.
• Provide an example, such as a multi-step arithmetic problem or a logical puzzle, where showing intermediate steps leads to a correct solution.

Where people lose the point

• ×Confusing CoT with simply asking for a detailed explanation *after* the answer.
• ×Failing to explain *why* it's effective (e.g., breaking down complexity, reducing errors).
• ×Not providing a clear example that demonstrates the step-by-step reasoning.

What a strong answer covers

• Describe the primary function of generative LLMs: to predict the next token in a sequence, effectively generating new text.
• Explain that the decoder-only architecture is inherently designed for this causal language modeling task, where each token's prediction can only depend on previous tokens.
• Contrast this with encoder-decoder architectures (like original Transformer or T5) which are better suited for sequence-to-sequence tasks (e.g., translation) where the entire input is available.
• Mention that the decoder's masked self-attention mechanism prevents it from 'seeing' future tokens, ensuring a realistic generation process.
• Highlight the simplicity and efficiency of scaling a single, unified decoder for diverse generative tasks.

Where people lose the point

• ×Incorrectly stating that decoder-only models can see future tokens during generation.
• ×Confusing the roles of encoders and decoders in the original Transformer.
• ×Failing to connect the architecture directly to the task of causal language modeling/text generation.

What a strong answer covers

• **Perplexity:** Explain it as a measure of how well a probability model predicts a sample, indicating how surprised the model is by new data. Lower perplexity is generally better, but it doesn't directly measure quality or factual accuracy.
• **BLEU/ROUGE:** Describe these as n-gram overlap metrics comparing generated text to reference text, useful for tasks like translation/summarization. Limitations include insensitivity to semantic meaning, fluency, and factual correctness.
• **Human Evaluation:** State it as the gold standard for assessing quality, coherence, factual accuracy, and safety. Limitations include high cost, subjectivity, and time consumption.
• **Task-Specific Metrics:** Mention metrics tailored for specific tasks (e.g., F1 for QA, BERTScore for semantic similarity). Limitations are their narrow scope and reliance on specific reference data.
• **Adversarial/Robustness Benchmarks:** Briefly mention the importance of evaluating against adversarial attacks or out-of-distribution data to assess robustness.

Where people lose the point

• ×Only listing metrics without explaining what they measure or their limitations.
• ×Overstating the effectiveness of automated metrics for capturing nuanced generative quality.
• ×Failing to mention human evaluation as the ultimate arbiter of quality.

What a strong answer covers

• **Tool Selection & Argument Generation:** Explain the challenge of the LLM accurately identifying the correct tool for a given user query and generating the precise, valid arguments required by that tool's API.
• **Error Handling & Recovery:** Discuss how to handle cases where tool calls fail (e.g., API errors, invalid arguments) and enable the LLM to recover gracefully or try alternative approaches.
• **Security & Permissions:** Highlight the critical need for robust security measures, access controls, and careful permission management to prevent misuse or unauthorized actions by the LLM.
• **Latency & Cost:** Mention that external tool calls introduce latency and can incur costs, requiring optimization and careful design to maintain responsiveness and manage expenses.
• **State Management & Context Window:** Explain the difficulty of maintaining conversational state and managing the LLM's context window when multiple tool calls and their outputs are involved.

Where people lose the point

• ×Underestimating the complexity of tool selection and argument generation.
• ×Ignoring critical security implications of giving LLMs access to external systems.
• ×Failing to consider the practical challenges of latency, cost, and state management.

What a strong answer covers

• **Bias and Discrimination:** Explain how LLMs can perpetuate and amplify biases present in their training data, leading to unfair or discriminatory outcomes in areas like hiring, lending, or content moderation.
• **Misinformation and Disinformation:** Discuss the risk of LLMs generating convincing but false information (hallucinations) or being intentionally misused to create propaganda, fake news, or deceptive content at scale.
• **Privacy and Data Security:** Highlight concerns around LLMs inadvertently revealing sensitive personal information from their training data or being used to extract private data from user inputs.
• **Safety and Harmful Content:** Address the potential for LLMs to generate harmful, toxic, or illegal content, or to be used for malicious purposes like phishing, cyberattacks, or social engineering.
• **Job Displacement and Economic Impact:** Briefly touch upon the societal implications of LLMs automating tasks, potentially leading to job losses and requiring workforce adaptation.

Where people lose the point

• ×Only listing risks without explaining *why* they are ethical concerns.
• ×Failing to connect risks back to the inherent properties of LLMs (e.g., training data, generative nature).
• ×Not considering the societal and human impact of these risks.

What a strong answer covers

• Define PEFT as a family of techniques that allow fine-tuning large pre-trained models by only updating a small subset of parameters, rather than all of them.
• Explain the core idea of LoRA (Low-Rank Adaptation): injecting small, trainable matrices (adapters) into the Transformer layers and only training these new matrices, keeping the original pre-trained weights frozen.
• Discuss the key advantages: significantly reduced computational cost and memory footprint during fine-tuning, faster training times, and the ability to store multiple task-specific adapters for a single base model.
• Mention that PEFT helps mitigate catastrophic forgetting and makes LLM adaptation more accessible and scalable.

Where people lose the point

• ×Confusing PEFT with full fine-tuning or pre-training.
• ×Failing to explain *how* LoRA works (low-rank matrices, frozen base weights).
• ×Not clearly articulating the benefits in terms of cost, speed, and storage.

What a strong answer covers

• **Advantage: Parallelization:** Explain how self-attention allows parallel computation across all tokens in a sequence, unlike RNNs, which significantly speeds up training on GPUs/TPUs.
• **Advantage: Long-Range Dependencies:** Describe how the direct connection between any two tokens via attention allows Transformers to effectively capture long-range dependencies, crucial for large contexts.
• **Challenge: Quadratic Complexity:** Discuss the quadratic computational and memory complexity of vanilla self-attention with respect to sequence length, which can become a bottleneck for very long inputs.
• **Challenge: Data & Compute Requirements:** Mention that scaling to billions of parameters requires immense amounts of high-quality training data and vast computational resources (GPUs/TPUs, energy).
• **Mitigation Strategies:** Briefly touch upon techniques like sparse attention, FlashAttention, or sequence truncation to address the quadratic complexity, and distributed training for compute.

Where people lose the point

• ×Only listing advantages without addressing challenges, or vice-versa.
• ×Incorrectly stating that RNNs scale better than Transformers.
• ×Failing to explain *why* self-attention has quadratic complexity or how it's mitigated.