[AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists #4547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…G/task exists In ExternalTaskSensor, it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist. To provide an argument "check_existence". Set to True to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist. The default value is set to False (no check or ceasing will happen) so it will not affect any existing DAGs or current user expectation.
XD-DENG
commented
Jan 17, 2019
| @@ -26,7 +29,8 @@ | |||
|
|
|||
| class ExternalTaskSensor(BaseSensorOperator): | |||
| """ | |||
| Waits for a different DAG or a task in in a different DAG to complete | |||
| Waits for a different DAG or a task in a different DAG to complete for a | |||
XD-DENG
commented
Jan 17, 2019
|
|
||
| if self.external_task_id: | ||
| refreshed_dag_info = DagBag(dag_to_wait.fileloc).get_dag(self.external_dag_id) |
There was a problem hiding this comment.
-
Purposely leave
include_examplesforDagBagto be default valueTrue, since theexternal_dag_idmay be example DAG. -
The exact file path is given to
DagBag, so that it will only scan a single DAG file. This helps ensure no heavy overhead.
XD-DENG
commented
Jan 17, 2019
|
|
||
| if self.external_task_id: |
There was a problem hiding this comment.
If ExternalTaskSensor is waiting for a DAG, we will check database dag table + file system.
If it's waiting for a task, ADDITIONAL check will be done to check if that task exists in the latest DAG file.
Codecov Report
@@ Coverage Diff @@
## master #4547 +/- ##
==========================================
+ Coverage 74.07% 74.08% +<.01%
==========================================
Files 421 421
Lines 27649 27663 +14
==========================================
+ Hits 20481 20494 +13
- Misses 7168 7169 +1
Continue to review full report at Codecov.
|
|
LGTM overall for the logic, will take a closer look once in the office. |
|
LGTM, thanks @XD-DENG |
|
Thanks @feng-tao |
ashb
pushed a commit
that referenced
this pull request
Mar 21, 2019
…G/task exists (#4547) In ExternalTaskSensor, it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist. To provide an argument "check_existence". Set to True to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist. The default value is set to False (no check or ceasing will happen) so it will not affect any existing DAGs or current user expectation.
ashb
pushed a commit
that referenced
this pull request
Mar 22, 2019
…G/task exists (#4547) In ExternalTaskSensor, it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist. To provide an argument "check_existence". Set to True to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist. The default value is set to False (no check or ceasing will happen) so it will not affect any existing DAGs or current user expectation.
wmorris75
pushed a commit
to modmed/incubator-airflow
that referenced
this pull request
Jul 29, 2019
…G/task exists (apache#4547) In ExternalTaskSensor, it may be good to provide an option to cease waiting immediately if the external DAG/task specified doesn't exist. To provide an argument "check_existence". Set to True to check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist. The default value is set to False (no check or ceasing will happen) so it will not affect any existing DAGs or current user expectation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira
Description
Background
ExternalTaskSensorwill keep waiting even if the external DAG/task specified doesn't exist at all (this is making sense in some scenarios).But it may be good to provide an option to cease waiting immediately if the external task specified doesn't exist.
Proposal
Provide a flag
check_existence. Set toTrueto check if the external DAG/task exists, and immediately cease waiting if the external DAG/task does not exist.The default value is set to
False(no check or ceasing will happen), so it will not affect any existing DAGs or user expectation.This is a follow-up on #3688