Convert Step Functions Example DAG to System Test (AIP-47) #24643
Conversation
boring-cyborg
bot
added
area:providers
kind:documentation
provider:amazon
There was a problem hiding this comment.
since fetch_variable defaults default_value to None, we can just do this:
| if variable in self.variable_defaults: | |
| value = fetch_variable( | |
| variable, self.variable_defaults[variable], test_name=self.test_name | |
| ) | |
| else: | |
| value = fetch_variable(variable, test_name=self.test_name) | |
| default_value = self.variable_defaults.get(variable, None) | |
| value = fetch_variable(variable, default_value, test_name=self.test_name) |
There was a problem hiding this comment.
This code was written in such a way that any default could be provided, even None (since the user may want literally None to be the default value), but you're right that the fetch_variable code wasn't written with that philosophy so we can go with your suggestion for now until someone requires otherwise.
|
Looks nice! So the TaskFlow is used to manage system test configuration. What will happen if TaskFlow itself fails, will it be clear to users that it failed in setup phase rather than in system tests (of any other service)? |
I'm not sure which TaskFlow failures you're imagining here specifically, but one should get an exception like you would from any other catastrophic failure and they should be able to tell from that where the issue occurred. I'm happy to place some more error handling somewhere specific if you have any suggestions! |
This factory class builds a task that will generate the environment id and fetch any variables at runtime and store them in xcom
Moved example_step_functions example dag AWS system test dir. Convert to AIP-47 standard.
o-nikolas
force-pushed
the
onikolas/system-tests/step_functions
branch
from
611f17b to
6a838a6
Compare
* System test for EMR Serverless following the template in #24643 (AIP-47) * Remove example_emr_serverless.py from example_dags
3 participants
This PR moves another AWS example dag to a system test implementation. This example also includes a new test context builder class which produces a TaskFlow task that computes, fetches and stores test config at runtime instead of module load time, since doing novel work (including network requests) at module load time causes a couple specific issues for these tests and is generally bad practice for dags to be doing such work on dag parse/load time.
Related: #22438
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst, in newsfragments.