pandas interview questions

What a strong answer covers

• Define a Series as a one-dimensional labeled array capable of holding any data type, similar to a single column of data.
• Define a DataFrame as a two-dimensional labeled data structure with columns of potentially different types, resembling a spreadsheet or SQL table.
• Highlight that a DataFrame can be thought of as a collection of Series objects that share a common index.
• Explain that Series are suitable for single-column data or when working with time-series data, while DataFrames are ideal for tabular, multi-column datasets.

Where people lose the point

• ×Confusing the dimensionality, stating a Series is 2D or a DataFrame is 1D.
• ×Failing to mention the 'labeled' aspect of both structures, which is key to pandas.
• ×Not providing clear use cases for when to prefer one over the other.

What a strong answer covers

• Explain that `pd.DataFrame()` is the constructor used to create a DataFrame.
• Demonstrate creating a dictionary where each key represents a column name and its corresponding value is a list (or NumPy array) of data for that column.
• Show the syntax: `df = pd.DataFrame(data_dict)`.
• Provide a concrete Python code example with a dictionary and the resulting DataFrame.

Where people lose the point

• ×Incorrectly structuring the dictionary (e.g., values are not lists of equal length).
• ×Forgetting to import pandas as `pd`.
• ×Attempting to create a DataFrame from a dictionary of Series without explicitly calling `pd.DataFrame()`.

What a strong answer covers

• Explain the dictionary-like access method: `df['Age']`.
• Explain the attribute-like access method: `df.Age`.
• Mention that both methods return a pandas Series.
• Briefly discuss the pros and cons (e.g., attribute access doesn't work for column names with spaces or that conflict with DataFrame methods).

Where people lose the point

• ×Only providing one method of selection.
• ×Attempting to use attribute access for column names that are not valid Python identifiers (e.g., 'First Name').
• ×Not clarifying that the result of selecting a single column is a Series, not a DataFrame.

What a strong answer covers

• Explain that boolean indexing involves creating a boolean Series (True/False) based on a condition applied to a column.
• Demonstrate how to create this boolean Series, e.g., `df['Score'] > 80`.
• Show how to pass this boolean Series directly to the DataFrame to filter rows: `df[df['Score'] > 80]`.
• Provide a complete code example with a sample DataFrame and the filtering operation.

Where people lose the point

• ×Forgetting to wrap the boolean Series in square brackets when applying it to the DataFrame.
• ×Using `and` or `or` instead of `&` or `|` for combining multiple conditions.
• ×Attempting to filter using a string condition directly without creating a boolean Series.

What a strong answer covers

• Explain that new columns can be added by direct assignment, similar to adding a new key-value pair to a dictionary.
• Demonstrate the syntax: `df['new_column_name'] = value_or_series`.
• Show how to create the 'Full Name' column by concatenating existing 'First Name' and 'Last Name' columns, ensuring a space in between.
• Provide a complete code example with a sample DataFrame and the column addition.

Where people lose the point

• ×Forgetting to include a space or separator when concatenating strings for a new column.
• ×Attempting to use `df.add_column()` (which doesn't exist) instead of direct assignment.
• ×Not ensuring the length of the assigned Series/list matches the DataFrame's number of rows.

What a strong answer covers

• Explain that `.loc` is primarily label-based, used for selecting data by row and column labels (names).
• Explain that `.iloc` is primarily integer-position-based, used for selecting data by the integer position of rows and columns.
• Provide clear examples demonstrating selection of single elements, rows, columns, and slices for both methods (e.g., `df.loc['A':'C', 'col1']` vs `df.iloc[0:3, 0]`).
• Highlight that `.loc` slicing is inclusive of the end label, while `.iloc` slicing is exclusive of the end integer position.

Where people lose the point

• ×Confusing label-based (`.loc`) with integer-position-based (`.iloc`) indexing.
• ×Incorrectly assuming `.loc` slicing is exclusive of the end label.
• ×Attempting to use a non-integer index with `.iloc` or a non-label with `.loc`.

What a strong answer covers

• Explain the 'split-apply-combine' paradigm: splitting data into groups, applying a function to each group, and combining the results.
• Demonstrate using `df.groupby('category_column')` to split the DataFrame.
• Show how to select a numerical column and apply an aggregation function, e.g., `['numerical_column'].mean()`.
• Provide a complete code example with a sample DataFrame, grouping, and mean calculation.

Where people lose the point

• ×Forgetting to select a column after `groupby()` before applying an aggregation, leading to aggregation across all columns.
• ×Misunderstanding the output of `groupby()` before an aggregation (it returns a GroupBy object, not a DataFrame).
• ×Attempting to use `groupby()` on a non-categorical column without a clear purpose.

What a strong answer covers

• Define `pd.merge()` as a function for combining DataFrames based on common columns or indices, similar to SQL joins.
• Explain `inner` merge: returns only rows with matching keys in *both* DataFrames. Scenario: finding common customers in two sales lists.
• Explain `left` merge: returns all rows from the left DataFrame, and matching rows from the right. `NaN` for unmatched right keys. Scenario: adding customer details to all orders, even if some customers are missing.
• Explain `right` merge: returns all rows from the right DataFrame, and matching rows from the left. `NaN` for unmatched left keys. Scenario: adding order details to all customers, even if some customers have no orders.
• Explain `outer` merge: returns all rows when there is a match in *either* DataFrame, filling `NaN` for unmatched values. Scenario: combining all unique records from two datasets.

Where people lose the point

• ×Misunderstanding which rows are preserved or dropped for different merge types.
• ×Incorrectly specifying the `on` or `left_on`/`right_on` parameters, leading to unintended joins.
• ×Not considering potential duplicate keys and their impact on merge results.

What a strong answer covers

• Explain `df.dropna()`: removes rows or columns containing missing values. Discuss `axis` and `how` parameters.
• Explain `df.fillna()`: replaces missing values with a specified value (e.g., constant, mean, median, mode) or using imputation methods like `ffill`/`bfill`.
• Contrast their primary actions: `dropna` removes data, `fillna` modifies/imputes data.
• Discuss scenarios for `dropna` (e.g., small number of missing values, missing data is truly irrelevant) and `fillna` (e.g., preserving data, using statistical imputation, time-series data).

Where people lose the point

• ×Not mentioning the `axis` parameter for both functions, which controls row vs. column operation.
• ×Failing to discuss the `inplace` parameter and its implications.
• ×Suggesting `dropna()` as a universal solution, ignoring potential data loss.

What a strong answer covers

• Explain that the `apply()` method is used to apply a function along an axis of a DataFrame or Series.
• Demonstrate applying a function to a Series (e.g., `df['Age'].apply(my_function)`).
• Show how to define a custom function (e.g., `categorize_age`) that takes an age and returns a category string.
• Provide a complete code example with a sample DataFrame, the custom function, and its application to create a new 'Age Group' column.

Where people lose the point

• ×Confusing `apply()` with vectorized operations, using it when a direct pandas method exists (e.g., `df['col'].str.lower()` vs `df['col'].apply(lambda x: x.lower())`).
• ×Incorrectly defining the custom function to accept a Series when it should accept a single value (for Series.apply).
• ×Forgetting to specify `axis=1` when applying a function row-wise to a DataFrame that needs access to multiple columns.

What a strong answer covers

• Explain that `reset_index()` converts the DataFrame's index into a regular column and resets the index to the default integer index (0, 1, 2...).
• Describe a scenario where it's useful, such as after a `groupby()` operation where the grouping keys become the new index, and you want them back as regular columns.
• Explain the `drop=True` parameter: if `True`, the old index is discarded; if `False` (default), the old index is added as a new column.
• Provide a simple code example demonstrating `reset_index()` with and without `drop=True`.

Where people lose the point

• ×Not understanding that `groupby()` often changes the index, making `reset_index()` necessary.
• ×Confusing `reset_index()` with simply reassigning the index.
• ×Forgetting the `inplace=True` parameter if modifying the DataFrame directly is desired.

What a strong answer covers

• Explain `pivot_table`: transforms data from a 'long' format to a 'wide' format, aggregating values. Define `index`, `columns`, and `values` parameters.
• Explain `melt`: transforms data from a 'wide' format to a 'long' format, unpivoting columns into rows. Define `id_vars` and `value_vars` parameters.
• Provide a scenario for `pivot_table`: summarizing sales data by region and product type, with product types as new columns.
• Provide a scenario for `melt`: converting survey responses where each question is a column into a format suitable for statistical analysis (question and answer in separate columns).
• Emphasize that they are inverse operations, used for different data representation needs.

Where people lose the point

• ×Confusing the direction of transformation (wide to long vs. long to wide).
• ×Incorrectly identifying the `index`, `columns`, `values` for `pivot_table` or `id_vars`, `value_vars` for `melt`.
• ×Not understanding the aggregation aspect of `pivot_table` (it often requires an `aggfunc`).

What a strong answer covers

• Demonstrate grouping by multiple columns by passing a list of column names to `groupby()`: `df.groupby(['col1', 'col2'])`.
• Explain how to use the `.agg()` method with a dictionary to specify different aggregation functions for different columns.
• Show the dictionary structure: `{'column_A': ['sum', 'mean'], 'column_B': 'max'}`.
• Provide a complete code example with a sample DataFrame, multi-column grouping, and multiple aggregations using `.agg()`.

Where people lose the point

• ×Attempting to apply multiple aggregations without using the `.agg()` method.
• ×Incorrectly structuring the dictionary passed to `.agg()` (e.g., not using lists for multiple functions per column).
• ×Misinterpreting the resulting MultiIndex DataFrame after such an operation.

What a strong answer covers

• Explain the importance of vectorization: using built-in pandas/NumPy operations over explicit Python loops for significant speed gains.
• Discuss `apply()`: faster than pure Python loops but generally slower than vectorized operations. Use it when custom logic is unavoidable.
• Mention `astype()` for optimizing data types (e.g., using `category` for low-cardinality strings, smaller integer types) to reduce memory usage and improve performance.
• Suggest avoiding `inplace=True` where possible, as it can sometimes lead to less readable code and unexpected behavior, and often doesn't offer significant performance benefits.
• Briefly touch upon using `chunksize` when reading very large files to avoid memory issues.

Where people lose the point

• ×Relying heavily on explicit `for` loops for row-wise or element-wise operations instead of vectorized pandas functions.
• ×Misusing `apply()` when a direct vectorized method exists (e.g., `df['col'].str.lower()` instead of `df['col'].apply(lambda x: x.lower())`).
• ×Not considering the impact of data types on both memory and computational speed.

What a strong answer covers

• Explain method chaining as calling multiple pandas methods sequentially, where each method returns a DataFrame or Series, allowing the next method to be called directly.
• Highlight the benefits: improved readability, reduced need for intermediate variables, and often better performance due to optimized internal operations.
• Provide an example that combines several operations, such as filtering, grouping, aggregating, and sorting, into a single chain.
• Emphasize using parentheses and line breaks for clarity in long chains.

Where people lose the point

• ×Breaking the chain unnecessarily by assigning to intermediate variables.
• ×Not using parentheses to make the chained operations readable across multiple lines.
• ×Attempting to chain methods that do not return a DataFrame or Series (e.g., `inplace=True` operations).

What a strong answer covers

• Explain how to convert a column to datetime objects using `pd.to_datetime()`, mentioning the `errors='coerce'` parameter for handling invalid dates.
• Demonstrate extracting components like year, month, day, or hour using the `.dt` accessor (e.g., `df['date_col'].dt.year`).
• Show how to calculate the difference between two datetime columns, resulting in a `Timedelta` Series (e.g., `df['end_date'] - df['start_date']`).
• Explain how to extract specific units from a `Timedelta` (e.g., `.dt.days` for days difference).

Where people lose the point

• ×Forgetting to convert the column to datetime objects before attempting to use the `.dt` accessor.
• ×Not handling potential errors during conversion, leading to `ValueError` for malformed date strings.
• ×Attempting to perform arithmetic operations on datetime columns without ensuring they are of the correct `datetime64` type.