Apache Airflow interview questions

What a strong answer covers

• Define a DAG as a Directed Acyclic Graph, a collection of tasks with dependencies, representing a workflow.
• Explain that a DAG is defined in a Python file and acts as a blueprint, not the execution engine itself.
• List key components: tasks (instances of operators), dependencies (how tasks relate), and scheduling parameters (e.g., start_date, schedule_interval).

Where people lose the point

• ×Confusing a DAG with a running instance of a workflow (a DAG Run).
• ×Omitting the 'acyclic' nature and its importance for preventing infinite loops.
• ×Not mentioning that DAGs are defined in Python files.

What a strong answer covers

• Define an Operator as a predefined template or blueprint for a single unit of work (e.g., BashOperator, PythonOperator).
• Define a Task as an *instance* of an Operator within a specific DAG, configured with unique parameters and a task_id.
• Illustrate with an example: BashOperator is the class, while `bash_task = BashOperator(task_id='run_script', bash_command='echo hello')` creates a Task instance.

Where people lose the point

• ×Using 'Operator' and 'Task' interchangeably without distinguishing between the class and its instance.
• ×Failing to explain that a Task is what actually gets scheduled and executed.
• ×Not providing a clear example to illustrate the distinction.

What a strong answer covers

• Explain XComs (Cross-Communication) as a mechanism for tasks to exchange small amounts of data.
• Describe the push/pull mechanism: tasks push values (e.g., return value of PythonOperator), and other tasks pull them using `xcom_pull`.
• Provide use cases: passing file paths, small configuration parameters, or status flags between tasks.
• Discuss limitations: XComs are stored in the metadata database, making them unsuitable for large datasets; they should only be used for small, serializable data.

Where people lose the point

• ×Suggesting XComs for transferring large datasets (e.g., entire dataframes).
• ×Not mentioning that XComs are stored in the metadata database, which can become a performance bottleneck.
• ×Failing to explain the push/pull mechanism clearly.

What a strong answer covers

• Define Sensors as a special type of Operator designed to wait for a specific condition to be met before allowing downstream tasks to proceed.
• Explain that Sensors poll for a condition, consuming worker slots while waiting, but can be configured with `poke_interval` and `timeout`.
• Provide an example: `S3KeySensor` to wait for a file to appear in an S3 bucket, or `SqlSensor` to wait for a specific record in a database.
• Mention the `mode='reschedule'` option to free up worker slots while waiting.

Where people lose the point

• ×Confusing Sensors with regular Operators that perform work.
• ×Not mentioning the resource consumption of Sensors (worker slots) and how `mode='reschedule'` mitigates this.
• ×Failing to provide a concrete, real-world example.

What a strong answer covers

• Define Hooks as interfaces to external platforms and databases (e.g., S3Hook, PostgresHook, GoogleCloudStorageHook).
• Explain their utility: they abstract away connection details and provide a common API for interacting with external systems.
• Highlight benefits: promote code reusability, separate connection logic from business logic, and allow secure management of credentials via Airflow Connections.

Where people lose the point

• ×Confusing Hooks with Operators; Hooks are typically used *within* Operators or Python functions.
• ×Not emphasizing the role of Hooks in abstracting connection details and improving security.
• ×Failing to mention that Hooks leverage Airflow Connections.

What a strong answer covers

• Describe the Scheduler as the heart of Airflow, continuously monitoring all DAGs and tasks.
• List its primary responsibilities: parsing DAG files, creating new DAG runs based on schedules, evaluating task dependencies, and submitting eligible tasks to the executor.
• Mention its role in managing task states (e.g., marking tasks as failed, retrying tasks) and handling SLAs.

Where people lose the point

• ×Confusing the Scheduler's role with that of the Webserver or Workers.
• ×Omitting the parsing of DAG files or the submission of tasks to the executor.
• ×Not mentioning its role in managing task states and dependencies.

What a strong answer covers

• Local Executor: Runs tasks on the same machine as the Scheduler; simple for development/testing, not scalable or fault-tolerant for production.
• Celery Executor: Uses a Celery backend (e.g., RabbitMQ, Redis) to distribute tasks to a pool of Celery workers; offers horizontal scalability and fault tolerance, good for moderate to large workloads.
• Kubernetes Executor: Launches a new Kubernetes Pod for each task; provides excellent isolation, dynamic scaling, and resource management, ideal for highly dynamic or resource-intensive workloads in a Kubernetes environment.
• Discuss choice factors: scalability needs, fault tolerance requirements, existing infrastructure (e.g., Kubernetes cluster), and operational complexity.

Where people lose the point

• ×Incorrectly stating that the Local Executor is suitable for production.
• ×Not explaining the underlying mechanism for Celery (message queue) or Kubernetes (pods).
• ×Failing to connect the choice of executor to specific production requirements like isolation or dynamic scaling.

What a strong answer covers

• `start_date`: The absolute date and time from which the DAG is allowed to start scheduling. The scheduler will not create DAG runs for any `execution_date` before this `start_date`.
• `execution_date`: A logical timestamp that marks the start of the data interval for which the DAG run is processing. For a daily DAG scheduled at midnight, the `execution_date` for today's run would be yesterday's midnight.
• Clarify that the `execution_date` is typically passed to tasks as a parameter to define the data context, while `start_date` controls when the DAG *can* begin running.

Where people lose the point

• ×Confusing `execution_date` with the actual time a task runs.
• ×Incorrectly stating that `start_date` is when the first DAG run occurs (it's the earliest possible `execution_date`).
• ×Not explaining the concept of a 'data interval' for `execution_date`.

What a strong answer covers

• Define idempotency: a task or DAG is idempotent if running it multiple times with the same input produces the same output and state, without causing unintended side effects.
• Explain its importance in Airflow: crucial for handling retries, backfills, and unexpected failures gracefully, ensuring data consistency and preventing duplicate processing.
• Describe how to achieve it: use `execution_date` to define the processing window, write tasks that check for existing output before reprocessing, use transactional operations, or implement upsert logic in databases.
• Provide examples: deleting and recreating a partition, or using `INSERT ... ON CONFLICT UPDATE`.

Where people lose the point

• ×Failing to explain *why* idempotency is important in a distributed system like Airflow (retries, failures).
• ×Not providing concrete strategies or examples for implementing idempotency.
• ×Confusing idempotency with simply not failing on subsequent runs.

What a strong answer covers

• Configure `retries` and `retry_delay` parameters at the DAG or task level to automatically re-attempt failed tasks.
• Utilize `on_failure_callback` or `on_retry_callback` to execute custom Python functions for notifications (e.g., Slack, email) or logging.
• Implement `sla_miss_callback` to be notified if tasks don't complete within their Service Level Agreement.
• Consider using `ExternalTaskSensor` or `BranchPythonOperator` for conditional error handling or recovery workflows.

Where people lose the point

• ×Only mentioning `retries` without discussing `retry_delay` or callbacks.
• ×Not considering external notification mechanisms for failures.
• ×Failing to mention the importance of `sla_miss_callback` for monitoring.

What a strong answer covers

• Explain Airflow Connections: a centralized way to store connection parameters (host, port, user, password, schema) for external systems.
• Describe methods for storing connections: directly in the Airflow UI, environment variables (e.g., `AIRFLOW_CONN_MY_DB`), or external secret backends.
• Discuss external secret backends (e.g., HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) as the most secure and scalable option for production.
• Emphasize avoiding hardcoding credentials directly in DAG files.

Where people lose the point

• ×Suggesting hardcoding credentials in DAG files.
• ×Not mentioning environment variables as a common method for local/dev setups.
• ×Failing to highlight external secret backends as the best practice for production.

What a strong answer covers

• Define backfilling as the process of running a DAG for past `execution_date`s, typically to reprocess historical data or catch up on missed runs.
• Explain how to initiate backfills: using the `airflow dags backfill` CLI command or via the Airflow UI.
• Discuss considerations: ensuring tasks are idempotent to prevent data duplication or corruption, managing resource consumption during large backfills, and potential impact on current scheduled runs.
• Mention the importance of testing backfill logic in a staging environment.

Where people lose the point

• ×Not emphasizing the critical importance of idempotency for backfills.
• ×Failing to mention the resource implications of running many historical DAG runs concurrently.
• ×Confusing backfilling with regular scheduled runs.

What a strong answer covers

• Explain that task dependencies define the order in which tasks must execute within a DAG.
• Describe the common methods: using bitshift operators (`>>` for set_downstream, `<<` for set_upstream) or the `set_upstream`/`set_downstream` methods.
• Illustrate with examples: `task_a >> task_b` or `task_a.set_downstream(task_b)`.
• Mention more complex dependency patterns like `[task_a, task_b] >> task_c` (all tasks must complete) or `task_a >> [task_b, task_c]` (task_a must complete before either task_b or task_c can start).

Where people lose the point

• ×Incorrectly using dependency operators (e.g., `task_b >> task_a` when `task_a` should run first).
• ×Not explaining how to define multiple upstream or downstream dependencies.
• ×Failing to mention the `set_upstream`/`set_downstream` methods as alternatives to bitshift operators.

What a strong answer covers

• Scheduler bottleneck: Too many DAGs/tasks can overwhelm the scheduler. Address by optimizing DAG parsing, increasing scheduler resources, or using multiple schedulers (with HA setup).
• Metadata Database contention: High task volume leads to frequent database writes/reads. Address by optimizing database queries, scaling the database, or using a more performant database.
• Worker capacity: Not enough workers or workers with insufficient resources. Address by scaling workers horizontally (Celery/Kubernetes Executor), optimizing task resource usage, or using task-specific resource requests.
• Network latency and I/O: Data transfer between tasks or external systems can be slow. Address by optimizing data transfer mechanisms, co-locating resources, or using efficient storage solutions.

Where people lose the point

• ×Only focusing on one component (e.g., workers) without considering the entire architecture.
• ×Not mentioning the metadata database as a common bottleneck.
• ×Proposing solutions that don't align with Airflow's distributed nature (e.g., just adding more RAM to a single machine).

What a strong answer covers

• Define branching as the ability to dynamically choose which downstream tasks to execute based on the outcome of an upstream task.
• Explain the primary implementation: `BranchPythonOperator`, which takes a Python callable that returns the `task_id` (or list of `task_id`s) of the next task(s) to run.
• Describe how skipped tasks are handled: tasks not returned by the `BranchPythonOperator` are marked as 'skipped' and their downstream dependencies are also skipped.
• Provide a simple example: a `BranchPythonOperator` deciding between `process_data_a` or `process_data_b` based on a condition.

Where people lose the point

• ×Confusing branching with simple parallel execution.
• ×Not mentioning the `BranchPythonOperator` as the main mechanism.
• ×Failing to explain that skipped tasks propagate the 'skipped' state downstream.