-
Notifications
You must be signed in to change notification settings - Fork 16.3k
implement api v1 for variables #9273
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update field name to confirm with RFC spec
|
@mik-laj the in openapi spec, the PATH method for variable takes the following body: I think the key in request body is redundant, should we update the api spec to remove it? |
airflow/www/app.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prevent CSRF check for all API write operations.
The structure of this request body results from the fact that it is based on the Variable schema. VariableCollectionItem:
# Divided into two schemas for sensitive data protection
type: object
properties:
key:
type: string
VariableCollection:
type: object
properties:
variables:
type: array
items:
$ref: '#/components/schemas/VariableCollectionItem'
Variable:
allOf:
- $ref: '#/components/schemas/VariableCollectionItem'
- type: object
properties:
value:
type: stringThe same schema is used to create as well as edit elements, so both fields must exist. We can create a new scheem that will not contain this one field, but it will complicate the API. This field should be immutable, which is well described in the AIP.
https://google.aip.dev/203 Unfortunately, OpenAPI has no support for immutable fields, so we have some confusion in the specification definition. I think we can add an additional description to explain this. What do you think about it? |
Cool, my implementation already matches this behavior.
Yeah, I think it would be helpful to document this in the spec. How would you want this to be documented? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mik-laj this field requires a uri for a human readable documentation, how do you envision this to be supported in airflow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not have a specific plan yet, but I will try to prepare a reference documentation that will be generated in Swagger UI or Redoc and add links where possible.
Here is issue about docs: #8143
3162023 to
92043b6
Compare
|
@OmairK the api tests should have already covered serialization and deserialization code path, no? Do you have an example in mind where a refactor will cause issue that won't be caught by the api tests but can be caught by testing schema module itself? |
I think, writing tests improve code coverage? |
Yes, I think you are right :) Initially we were dealing with only read endpoints so we had to explicitly test the de-serialization, which you are already covering in create/update endpoints. |
@ephraimbuddy No not necessarily, my understanding of code coverage is based on this. IMO it wouldn't matter much in this case, both serialization and de-serialization is already covered in the CRUD end points. |
As @OmairK already mentioned, duplicated tests that test the same code path won't increase code coverage. In fact it will make future refactoring harder because you will have to change more than one tests whenever you introduce a behavior change. If a code path can be covered by end to end api test, that should be preferred. It's better to test on expected behavior of the system instead of implementation details. |
For now I think it's enough to add a note in the specification. "This field is immutable". I will add a general description of what the invariant field means at the beginning of the documentation later. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's strange, assert always give me nested diff on errors:
assert response.status_code == 400
> assert response.json == {
"title": "Invalid post body",
"status": 401,
"type": "about:blank",
"detail": "key from request body doesn't match uri parameter",
}
E AssertionError: assert {'detail': "key from request body doesn't match uri parameter",\n 'status': 400,\n 'title': 'Invalid post body',\n 'type': 'about:blank'} == {'detail': "key from request body doesn't match uri parameter",\n 'status': 401,\n 'title': 'Invalid post body',\n 'type': 'about:blank'}
E Common items:
E {'detail': "key from request body doesn't match uri parameter",
E 'title': 'Invalid post body',
E 'type': 'about:blank'}
E Differing items:
E {'status': 400} != {'status': 401}
E Full diff:
E {
E 'detail': "key from request body doesn't match uri parameter",
E - 'status': 401,
E ? ^
E + 'status': 400,
E ? ^
E 'title': 'Invalid post body',
E 'type': 'about:blank',
E }
tests/api_connexion/endpoints/test_variable_endpoint.py:133: AssertionError
Do you mind sharing how you ran tests? I usually just invoke py.test tests/api_connexion/endpoints/test_variable_endpoint.py to run tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ha I see. I have not used IDEs myself, but definitely looks like something that can be easily fixed by tweaking the IDE config. Intelij IDEA is probably using some kind of custom test runner by default. Nested diffing with assert is supported natively by upstream py.test runner.
mik-laj
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What it looks looks very good, but I think it is worth adding to handle two cases.
- If an unknown field appears in the mask, the BadRequest exception should be raised
- If the variable does not exist and an attempt is made to delete it, exception 404 should be raised.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small nit: I suggest another way to exclude views from CSRF protection + order_by




close #9115 and #8133 .
Make sure to mark the boxes below before creating PR: [x]
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.