Menu

Post image 1
Post image 2
1 / 2
0

How to monitor Apache Airflow DAGs so you know when they silently fail

DEV Community·Kriss·about 1 month ago
#ErD8cBxg
#option#airflow#python#devops#task#fullscreen
Reading 0:00
15s threshold

Your Airflow DAG ran last night. All tasks: green. All durations: normal. Export job completed at 02:14. Zero rows exported. Nobody knows. This is the silent failure Airflow's built-in alerting doesn't catch. on_failure_callback fires when a task crashes. It doesn't fire when a task exits 0 after connecting to a stale database replica and processing nothing. That's the failure mode that eats your Monday morning. This article shows you two ways to add external monitoring to Airflow DAGs — so you get paged for both kinds of failures. Why Airflow's built-in alerts aren't enough Airflow gives you several callback hooks: on_failure_callback — task or DAG run failed on_success_callback — task or DAG run succeeded on_retry_callback — task queued for retry on_execute_callback — task about to start on_skipped_callback — task raised AirflowSkipException These are useful. But none of them answer the question that actually matters for data pipelines: did the job do something?…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More