Skip to content

Conversation

@houqp
Copy link
Member

@houqp houqp commented Jun 13, 2020

close #9115 and #8133 .

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added the area:API Airflow's REST/HTTP API label Jun 13, 2020
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update field name to confirm with RFC spec

@houqp
Copy link
Member Author

houqp commented Jun 13, 2020

@mik-laj the in openapi spec, the PATH method for variable takes the following body:

PATCH /api/v1/variables/foo
{"key": "foo", "value": "bar"}

I think the key in request body is redundant, should we update the api spec to remove it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prevent CSRF check for all API write operations.

@mik-laj
Copy link
Member

mik-laj commented Jun 13, 2020

I think the key in request body is redundant, should we update the api spec to remove it?

The structure of this request body results from the fact that it is based on the Variable schema.

    VariableCollectionItem:
      # Divided into two schemas for sensitive data protection
      type: object
      properties:
        key:
          type: string

    VariableCollection:
      type: object
      properties:
        variables:
          type: array
          items:
            $ref: '#/components/schemas/VariableCollectionItem'

    Variable:
      allOf:
        - $ref: '#/components/schemas/VariableCollectionItem'
        - type: object
          properties:
            value:
              type: string

The same schema is used to create as well as edit elements, so both fields must exist. We can create a new scheem that will not contain this one field, but it will complicate the API.

This field should be immutable, which is well described in the AIP.

When a service receives an immutable field in an update request (or similar), even if included in the update mask, the service should ignore the field if the value matches, but should error with INVALID_ARGUMENT if a change is requested.

https://google.aip.dev/203

Unfortunately, OpenAPI has no support for immutable fields, so we have some confusion in the specification definition. I think we can add an additional description to explain this.

What do you think about it?

@houqp
Copy link
Member Author

houqp commented Jun 13, 2020

When a service receives an immutable field in an update request (or similar), even if included in the update mask, the service should ignore the field if the value matches, but should error with INVALID_ARGUMENT if a change is requested.

Cool, my implementation already matches this behavior.

Unfortunately, OpenAPI has no support for immutable fields, so we have some confusion in the specification definition. I think we can add an additional description to explain this.

Yeah, I think it would be helpful to document this in the spec. How would you want this to be documented?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mik-laj this field requires a uri for a human readable documentation, how do you envision this to be supported in airflow?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not have a specific plan yet, but I will try to prepare a reference documentation that will be generated in Swagger UI or Redoc and add links where possible.
Here is issue about docs: #8143

@houqp houqp requested a review from mik-laj June 13, 2020 09:03
@houqp houqp force-pushed the qp/variables branch 2 times, most recently from 3162023 to 92043b6 Compare June 13, 2020 19:12
@OmairK
Copy link
Contributor

OmairK commented Jun 13, 2020

Nitpicking here, but explicitly testing the out the schema's serialization and de-serialization could be helpful once we start refactoring the code. @houqp, what do you think ?

@houqp
Copy link
Member Author

houqp commented Jun 13, 2020

@OmairK the api tests should have already covered serialization and deserialization code path, no? Do you have an example in mind where a refactor will cause issue that won't be caught by the api tests but can be caught by testing schema module itself?

@ephraimbuddy
Copy link
Contributor

@OmairK the api tests should have already covered serialization and deserialization code path, no? Do you have an example in mind where a refactor will cause issue that won't be caught by the api tests but can be caught by testing schema module itself?

I think, writing tests improve code coverage?

@OmairK
Copy link
Contributor

OmairK commented Jun 14, 2020

@OmairK the api tests should have already covered serialization and deserialization code path, no? Do you have an example in mind where a refactor will cause issue that won't be caught by the api tests but can be caught by testing schema module itself?

Yes, I think you are right :) Initially we were dealing with only read endpoints so we had to explicitly test the de-serialization, which you are already covering in create/update endpoints.

@OmairK
Copy link
Contributor

OmairK commented Jun 14, 2020

I think, writing tests improve code coverage?

@ephraimbuddy No not necessarily, my understanding of code coverage is based on this. IMO it wouldn't matter much in this case, both serialization and de-serialization is already covered in the CRUD end points.

@mik-laj mik-laj linked an issue Jun 14, 2020 that may be closed by this pull request
@houqp
Copy link
Member Author

houqp commented Jun 14, 2020

I think, writing tests improve code coverage?

As @OmairK already mentioned, duplicated tests that test the same code path won't increase code coverage. In fact it will make future refactoring harder because you will have to change more than one tests whenever you introduce a behavior change.

If a code path can be covered by end to end api test, that should be preferred. It's better to test on expected behavior of the system instead of implementation details.

@mik-laj
Copy link
Member

mik-laj commented Jun 16, 2020

Yeah, I think it would be helpful to document this in the spec. How would you want this to be documented?

For now I think it's enough to add a note in the specification. "This field is immutable". I will add a general description of what the invariant field means at the beginning of the documentation later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using self.assertEquals is helpful because it provides clearer error messages, so it's easier to find an error, but I don't require it as per the mailing list voting. I only inform.
assert statement
image (1)
self.assertEquals
image (2)
image (3)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's strange, assert always give me nested diff on errors:

        assert response.status_code == 400
>       assert response.json == {
            "title": "Invalid post body",
            "status": 401,
            "type": "about:blank",
            "detail": "key from request body doesn't match uri parameter",
        }
E       AssertionError: assert {'detail': "key from request body doesn't match uri parameter",\n 'status': 400,\n 'title': 'Invalid post body',\n 'type': 'about:blank'} == {'detail': "key from request body doesn't match uri parameter",\n 'status': 401,\n 'title': 'Invalid post body',\n 'type': 'about:blank'}
E         Common items:
E         {'detail': "key from request body doesn't match uri parameter",
E          'title': 'Invalid post body',
E          'type': 'about:blank'}
E         Differing items:
E         {'status': 400} != {'status': 401}
E         Full diff:
E           {
E            'detail': "key from request body doesn't match uri parameter",
E         -  'status': 401,
E         ?              ^
E         +  'status': 400,
E         ?              ^
E            'title': 'Invalid post body',
E            'type': 'about:blank',
E           }

tests/api_connexion/endpoints/test_variable_endpoint.py:133: AssertionError

Do you mind sharing how you ran tests? I usually just invoke py.test tests/api_connexion/endpoints/test_variable_endpoint.py to run tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use green button in Intelij IDEA.
Screenshot 2020-06-18 at 03 16 39

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha I see. I have not used IDEs myself, but definitely looks like something that can be easily fixed by tweaking the IDE config. Intelij IDEA is probably using some kind of custom test runner by default. Nested diffing with assert is supported natively by upstream py.test runner.

Copy link
Member

@mik-laj mik-laj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What it looks looks very good, but I think it is worth adding to handle two cases.

  • If an unknown field appears in the mask, the BadRequest exception should be raised
  • If the variable does not exist and an attempt is made to delete it, exception 404 should be raised.

@houqp houqp requested a review from mik-laj June 17, 2020 23:59
Copy link
Member

@mik-laj mik-laj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: I suggest another way to exclude views from CSRF protection + order_by

@mik-laj mik-laj merged commit e2b2198 into apache:master Jun 18, 2020
@houqp houqp deleted the qp/variables branch June 18, 2020 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API Endpoint - CRUD - Variable API Endpoints - Read-only - Variable

5 participants