NLP interview questions

What a strong answer covers

• Start with lowercasing, removing HTML tags, and handling contractions (e.g., 'don't' -> 'do not').
• Tokenize into words or subwords; consider using a tokenizer like BERT's WordPiece for consistency with pretrained models.
• Remove punctuation and stop words only if they don't carry sentiment (e.g., 'not' is important).
• Apply lemmatization to reduce inflectional forms while preserving meaning; avoid stemming if it changes word sense.
• Discuss trade-offs: aggressive cleaning may remove signal (e.g., emojis in reviews), and stop word removal can hurt negation detection.

Where people lose the point

• ×Removing all stop words without considering their role in negation (e.g., 'not good' becomes 'good').
• ×Using stemming when lemmatization is more appropriate for sentiment, leading to loss of meaning.
• ×Ignoring domain-specific preprocessing like handling product names or slang.

What a strong answer covers

• BoW/TF-IDF represents text as sparse vectors of word counts or weighted frequencies; it ignores word order and semantics.
• Word embeddings (e.g., Word2Vec) produce dense vectors that capture semantic similarity; they can be static or contextual.
• BoW is simple, interpretable, and works well for large vocabularies with limited data; embeddings require more data but generalize better.
• Choose BoW for small datasets, linear models, or when interpretability is key; choose embeddings for deep learning and capturing meaning.
• Contextual embeddings (BERT) outperform both for complex tasks but are computationally expensive.

Where people lose the point

• ×Claiming embeddings always outperform BoW without considering data size and task complexity.
• ×Ignoring that BoW can be competitive with linear models on some benchmarks.
• ×Assuming all embeddings are contextual; static embeddings like GloVe still have limitations.

What a strong answer covers

• RNNs suffer from vanishing/exploding gradients, making it hard to learn long-range dependencies.
• They process tokens sequentially, preventing parallelization and slowing training.
• LSTMs use gating mechanisms (input, forget, output gates) to control information flow, mitigating vanishing gradients but still sequential.
• Transformers replace recurrence with self-attention, allowing parallel processing and direct connections between any positions.
• Transformers use positional encodings to retain order information and multi-head attention to capture diverse relationships.

Where people lose the point

• ×Stating that LSTMs completely solve vanishing gradients; they mitigate but don't eliminate it.
• ×Forgetting to mention that transformers have quadratic complexity in sequence length.
• ×Confusing self-attention with cross-attention or ignoring positional encodings.

What a strong answer covers

• Attention computes a weighted sum of values: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V.
• Q, K, V are matrices derived from input embeddings; dot product measures similarity between query and key.
• Scaling by sqrt(d_k) prevents the dot products from growing large in magnitude, which would push softmax into regions with extremely small gradients.
• Without scaling, the softmax output becomes near one-hot, hindering learning.
• Multi-head attention runs multiple attention operations in parallel, each with different learned projections.

Where people lose the point

• ×Omitting the scaling factor or misstating it as 1/d_k.
• ×Thinking attention weights are always interpretable; they can be noisy.
• ×Confusing scaled dot-product with additive attention.

What a strong answer covers

• MLM (used by BERT) masks some tokens and predicts them from context; it is bidirectional and captures rich context.
• Autoregressive LM (used by GPT) predicts the next token given previous tokens; it is unidirectional but naturally generates text.
• MLM is better for understanding tasks (classification, NER) because it sees both left and right context.
• Autoregressive LM excels at generation tasks (text completion, translation) because it models sequential generation.
• MLM requires masking strategy and is less efficient due to only using a fraction of tokens per batch; autoregressive LM uses all tokens.

Where people lose the point

• ×Claiming MLM is always better; it depends on the downstream task.
• ×Ignoring that autoregressive models can be adapted for understanding via fine-tuning (e.g., GPT for classification).
• ×Forgetting that MLM introduces a mismatch between pretraining and fine-tuning (mask tokens not present at fine-tuning).

What a strong answer covers

• BLEU measures n-gram precision between generated and reference translations; it ignores recall and semantic similarity.
• It penalizes valid paraphrases that use different wording, leading to low scores for good translations.
• BLEU does not account for word order beyond n-gram overlap; it can be fooled by exact n-gram matches in wrong order.
• Complement with METEOR (considers synonyms and stemming) or human evaluation for fluency and adequacy.
• Use chrF (character n-gram F-score) for morphologically rich languages or BERTScore for semantic similarity.

Where people lose the point

• ×Thinking BLEU is a perfect metric; it correlates only moderately with human judgment.
• ×Ignoring that BLEU requires multiple references for reliability.
• ×Using BLEU for summarization without adaptation; ROUGE is more common there.

What a strong answer covers

• Full fine-tuning updates all parameters; it achieves highest accuracy but requires significant compute and risks overfitting on small datasets.
• Feature-based approach freezes the pretrained model and uses its embeddings as input to a classifier; it is faster but may underperform.
• Adapter layers insert small trainable modules between frozen layers; they are parameter-efficient and allow multi-task learning.
• Prompt-based fine-tuning reformulates the task as a cloze-style question; it works well with few examples but requires careful prompt engineering.
• Trade-offs include computational cost, data efficiency, and ability to generalize to new tasks.

Where people lose the point

• ×Assuming full fine-tuning is always best; it can overfit with limited data.
• ×Ignoring that adapters add inference latency despite being parameter-efficient.
• ×Confusing prompt-based fine-tuning with in-context learning (which doesn't update weights).

What a strong answer covers

• Subword tokenization (BPE, WordPiece, SentencePiece) splits rare words into frequent subword units, ensuring no OOV tokens.
• BPE iteratively merges the most frequent character pairs; WordPiece uses a likelihood-based merge criterion.
• Character-level models process each character, handling any word but producing longer sequences and losing word-level semantics.
• Subword tokenization balances vocabulary size and sequence length; it is the standard in transformers.
• Character models are useful for morphologically rich languages or noisy text but are less efficient.

Where people lose the point

• ×Claiming subword tokenization eliminates all OOV issues; it can still produce rare subword combinations.
• ×Thinking character-level models are always worse; they can be competitive with proper architecture.
• ×Ignoring that SentencePiece directly models whitespace, unlike BPE/WordPiece which require pretokenization.

What a strong answer covers

• Sparse attention patterns (e.g., Longformer, BigBird) restrict each token to attend only to local neighbors and a few global tokens.
• Linformer approximates self-attention with low-rank projections, reducing complexity to O(n).
• Reformer uses locality-sensitive hashing to group similar queries and keys, computing attention only within buckets.
• Performer uses kernel methods to approximate softmax attention with linear complexity.
• These methods trade off accuracy for efficiency; the choice depends on the task and sequence length.

Where people lose the point

• ×Assuming all efficient attention methods are equally accurate; some lose long-range dependency capture.
• ×Forgetting that many efficient variants still require careful implementation for speedups.
• ×Claiming that transformers cannot handle long sequences at all; they can with these modifications.

What a strong answer covers

• Use token-level precision, recall, and F1-score, but entity-level metrics are more meaningful (exact match of entity span and type).
• Entity-level F1 counts a prediction as correct only if both the boundary and type match the ground truth.
• Micro-averaging aggregates over all entities; macro-averaging computes per-entity type F1 and averages, useful for imbalanced types.
• Consider strict vs. relaxed matching: strict requires exact span, relaxed allows partial overlap.
• Also report confusion matrix for entity types to identify common misclassifications.

Where people lose the point

• ×Using only token-level accuracy, which can be high even if entities are missed.
• ×Ignoring entity type confusion; e.g., labeling a person as an organization.
• ×Not accounting for nested entities (e.g., 'New York City' as location and 'New York' as city).

What a strong answer covers

• Informal language: slang, abbreviations, misspellings (e.g., 'gr8' for 'great'). Use subword tokenization or a normalization dictionary.
• Sarcasm and irony: literal sentiment is opposite of intended. Use context-aware models (transformers) and consider emoji/emoticon features.
• Negation handling: 'not good' flips sentiment. Ensure tokenization preserves negation scope; use dependency parsing or n-gram features.
• Emoji and emoticons carry sentiment; map them to sentiment scores or treat as special tokens.
• Domain adaptation: social media language differs from formal text; fine-tune on in-domain data.

Where people lose the point

• ×Assuming standard preprocessing (stop word removal) works; it may remove important cues like 'not'.
• ×Ignoring emojis or treating them as noise; they are strong sentiment indicators.
• ×Using a model trained on formal text without adaptation; performance drops significantly.

What a strong answer covers

• Extractive summarization selects sentences or phrases from the original text; it is simpler, factual, and preserves original wording.
• Abstractive summarization generates new sentences that may paraphrase; it can produce more concise and coherent summaries.
• Extractive methods often use sentence scoring (e.g., TextRank) or sequence labeling; they guarantee factual correctness but may lack fluency.
• Abstractive methods use sequence-to-sequence models (e.g., BART, T5); they can produce novel phrases but risk hallucination.
• Hybrid approaches combine both: extract key sentences then abstractively rewrite them.

Where people lose the point

• ×Claiming abstractive is always better; it can introduce factual errors.
• ×Ignoring that extractive summaries can be redundant or incoherent.
• ×Assuming ROUGE is perfect for both; it favors extractive due to n-gram overlap.

What a strong answer covers

• POS taggers use context (surrounding words) to disambiguate; e.g., 'I bank at Chase' vs. 'river bank'.
• Statistical taggers (HMM, CRF) learn transition probabilities between tags and emission probabilities for words given tags.
• Neural taggers (BiLSTM-CRF, transformer) encode context via bidirectional representations, achieving high accuracy.
• Word embeddings help because similar contexts have similar representations; contextual embeddings (BERT) further improve disambiguation.
• Ambiguity is reduced with larger context windows and training on diverse corpora.

Where people lose the point

• ×Thinking that a word's most frequent tag is always correct; context is essential.
• ×Ignoring that rare or unseen words can still be tagged using morphological clues or subword information.
• ×Assuming neural taggers are perfect; they still make errors on highly ambiguous cases.

What a strong answer covers

• Static embeddings assign a single vector per word, averaging over all contexts; thus polysemous words have a blended representation.
• For example, 'bank' vector is a mix of financial and river meanings, which can hurt performance on tasks requiring disambiguation.
• Contextual embeddings (BERT, ELMo) produce different vectors for the same word depending on surrounding words.
• This allows the model to capture sense-specific semantics, improving tasks like word sense disambiguation and NER.
• Contextual embeddings are computed on-the-fly, making them more computationally expensive but more accurate.

Where people lose the point

• ×Claiming static embeddings cannot handle polysemy at all; they can to some extent if senses share context.
• ×Thinking contextual embeddings always outperform static; they require more data and compute.
• ×Ignoring that ELMo uses bidirectional LSTM, while BERT uses transformer; both are contextual but differ in architecture.

What a strong answer covers

• Synonym replacement: replace words with synonyms (using WordNet or embeddings) to create new training examples.
• Back-translation: translate text to another language and back; generates paraphrases that preserve meaning.
• Random insertion/deletion/swap: simple noise injection that can improve robustness to typos.
• Mixup: interpolate between two input sequences and their labels at the embedding level.
• These techniques increase data diversity, reduce overfitting, and improve generalization, especially for low-resource tasks.

Where people lose the point

• ×Applying synonym replacement without considering context; can change meaning (e.g., 'bank' synonyms differ).
• ×Assuming back-translation always produces valid paraphrases; it can introduce errors.
• ×Over-augmenting can distort label distribution or introduce unnatural examples.

What a strong answer covers

• Latency: transformer models can be slow; use quantization (e.g., ONNX, TensorRT), distillation, or smaller models (DistilBERT).
• Memory: large models may not fit on GPU; use model parallelism or offloading; consider CPU inference with optimized libraries.
• Monitoring: track input distribution drift (e.g., new vocabulary), prediction confidence, and performance metrics over time.
• Handle out-of-scope inputs: set confidence thresholds or use a fallback model.
• Versioning: maintain model registry and A/B test new versions before full rollout.

Where people lose the point

• ×Ignoring input drift; a model trained on news may fail on social media without retraining.
• ×Assuming GPU inference is always needed; CPU with quantization can be sufficient for low-latency requirements.
• ×Not monitoring for adversarial inputs or data poisoning.

What a strong answer covers

• Bias can come from training data (e.g., gender stereotypes in occupation words) or from model architecture.
• Examples: word embeddings exhibit gender bias ('doctor' closer to 'man' than 'woman'); sentiment models may be biased against dialects.
• Mitigation: use debiasing techniques (e.g., hard debiasing of embeddings), balanced datasets, and fairness metrics.
• During fine-tuning, evaluate performance across demographic groups and adjust loss weighting.
• Transparency: document model limitations and intended use; involve diverse teams in development.

Where people lose the point

• ×Thinking bias is only in data; models can amplify bias even with balanced data.
• ×Assuming debiasing completely removes bias; it often reduces but does not eliminate.
• ×Ignoring intersectional bias (e.g., race and gender combined).

What a strong answer covers

• Prompt engineering involves designing input text to guide LLM output without fine-tuning.
• Techniques: few-shot prompting (provide examples), chain-of-thought (step-by-step reasoning), and role prompting ('act as an expert').
• Use clear instructions, delimiters, and output format specifications (e.g., JSON).
• Temperature and top-p sampling control randomness; lower temperature for factual tasks.
• Iterative refinement: generate multiple outputs and select best via scoring or human feedback.

Where people lose the point

• ×Assuming LLMs always follow instructions perfectly; they can be sensitive to phrasing.
• ×Overlooking that few-shot examples must be representative and correctly formatted.
• ×Using high temperature for factual tasks, leading to hallucinations.