Skip to content

Conversation

@mik-laj
Copy link
Member

@mik-laj mik-laj commented Feb 27, 2019

Make sure you have checked all steps below.

Jira

Description

The current state is unexpected because:

  • When Javascript is generated by the template engine, we do metaprogramming, which raises the complexity of the problems.
  • It does not allow testing the code in isolation.
  • Jinja is a HTML Template Engine. Using it to the JS code may cause a security risk.

As a next step, I would like to:

  • move the JS code to separate file
  • introduce linting for all JS/HTML files.
  • extract inlined CSS to separate file

For a long-term goal, I would like to

  • introduce visual regression and snapshot testing. If the JS code will be in separate files then it will be relatively simple.
  • update a dependency - Bootstrap 4 including dropping glyphicons

@jmcarp I saw that you changed HTML / JS files recently. What do you think about the change and plans for the future? Does what I want to do in the future make sense for you?

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
    • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

  • Passes flake8

@mik-laj mik-laj force-pushed the extract-jinja-variable branch 2 times, most recently from 53394a9 to 4bdf387 Compare February 27, 2019 09:13
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice simplification of the UI variables. Enables more improvements in the UI code.

@mik-laj mik-laj force-pushed the extract-jinja-variable branch from 4bdf387 to 3e705b9 Compare February 27, 2019 13:40
@mik-laj mik-laj closed this Feb 27, 2019
@mik-laj mik-laj reopened this Feb 27, 2019
@mik-laj mik-laj changed the title [AIRFLOW-3967][WIP] Extract Jinja directive from Javascript [AIRFLOW-3967] Extract Jinja directive from Javascript Feb 27, 2019
@mik-laj
Copy link
Member Author

mik-laj commented Feb 27, 2019

I finished my work. It looks good to me. However, I encourage you to submit your suggestions.

@ashb Can I ask for review?

@mik-laj mik-laj force-pushed the extract-jinja-variable branch 2 times, most recently from aff94be to 223bc2d Compare February 27, 2019 18:21
@mik-laj mik-laj force-pushed the extract-jinja-variable branch 2 times, most recently from 5416927 to 553c368 Compare February 28, 2019 19:14
@codecov-io
Copy link

codecov-io commented Feb 28, 2019

Codecov Report

Merging #4787 into master will decrease coverage by 1.67%.
The diff coverage is 87.5%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4787      +/-   ##
==========================================
- Coverage   80.02%   78.35%   -1.68%     
==========================================
  Files         594      467     -127     
  Lines       34769    29771    -4998     
==========================================
- Hits        27824    23326    -4498     
+ Misses       6945     6445     -500
Impacted Files Coverage Δ
airflow/www/views.py 76.34% <87.5%> (+1.14%) ⬆️
...low/contrib/operators/datastore_import_operator.py 0% <0%> (-100%) ⬇️
...low/contrib/operators/datastore_export_operator.py 0% <0%> (-100%) ⬇️
airflow/contrib/operators/gcs_to_bq.py 0% <0%> (-100%) ⬇️
airflow/contrib/sensors/gcs_sensor.py 0% <0%> (-97.15%) ⬇️
airflow/contrib/hooks/gcp_mlengine_hook.py 19.49% <0%> (-80.51%) ⬇️
airflow/operators/pig_operator.py 0% <0%> (-76.93%) ⬇️
airflow/contrib/hooks/gcp_dataproc_hook.py 36.84% <0%> (-63.16%) ⬇️
airflow/executors/sequential_executor.py 50% <0%> (-50%) ⬇️
airflow/contrib/task_runner/cgroup_task_runner.py 0% <0%> (-34.38%) ⬇️
... and 462 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6fb8217...b1edcff. Read the comment docs.

@mik-laj mik-laj mentioned this pull request Mar 5, 2019
6 tasks
@verdan
Copy link
Contributor

verdan commented Mar 6, 2019

I'd use data- attributes for this simplification instead of adding meta tags.
It's pretty common to pass values from html tags to javascript using data- attributes. Something like this: https://github.com/verdan/incubator-airflow/pull/10/files#diff-144f95ea05d0c8420d739f38ce8caa04R33
And yes, by doing it through data attributes you can easily get the value by using something like this

$('#ELEMENT_ID').data('DATA_ATTRIBUTE_NAME');

@mik-laj
Copy link
Member Author

mik-laj commented Mar 6, 2019

Not all variables have an assigned element. in other cases, this will complicate the code unnecessarily ex. csrf token. In other cases, I will try to introduce your change.

@verdan
Copy link
Contributor

verdan commented Mar 6, 2019

You can add new empty html span tags, just the way you introduced the meta tags in the PR.
You can get the idea here.

@mik-laj
Copy link
Member Author

mik-laj commented Mar 6, 2019

I do not know if adding empty elements in a body is a good idea. HEAD better describes the purpose and use of the code. It is clearer for me.

The body element represents the content of the document.

https://www.w3.org/TR/html5/sections.html#the-body-element

The head element represents a collection of metadata for the Document.

https://www.w3.org/TR/html5/document-metadata.html#the-head-element

The CSRF token is not associated with a specific document element, but it's a metadata for current page.

The fact that there is a function in jQuery is not a reason for me to use it.

@verdan
Copy link
Contributor

verdan commented Mar 6, 2019

True, that the CSRF token here is a metadata and not associated with any element. That was just an idea to use the empty tags, you can of course use data attributes on the top container as this is more relevant there. Up to you how you want to handle this.

@mik-laj mik-laj force-pushed the extract-jinja-variable branch 2 times, most recently from 21600b7 to ac2586a Compare March 12, 2019 01:25
@mik-laj
Copy link
Member Author

mik-laj commented Mar 12, 2019

@ashb i updated a PR.Now, code bluring is done without JS. Hostname/Clock code has been simplified

@mik-laj mik-laj force-pushed the extract-jinja-variable branch from ac2586a to b7e9b84 Compare March 12, 2019 01:42
@mik-laj mik-laj force-pushed the extract-jinja-variable branch 2 times, most recently from ec8e226 to 77d40a2 Compare March 16, 2019 02:09
@astahlman
Copy link
Contributor

I'd love to see this get merged.

In #4893 (adding time-zone support on the UI), I've also been wrestling with how to invoke Javascript functions with Jinja-templated variables supplied by the server as arguments. The approach I settled on was to expose these functions in a webpack library and invoke them from inline scripts (example).

The approach in this PR seems cleaner though, since it gets rid of inline scripting altogether.

@mik-laj
Copy link
Member Author

mik-laj commented Mar 16, 2019

I chose the easiest way (global) to call the code, because it plans to extract all JS to separate files and then manage the dependencies in a modern/ES6 way. My solution will not require any changes in the code (except utils.js), but only the addition of correct imports that will load the functions into the current scope.

@mik-laj mik-laj force-pushed the extract-jinja-variable branch 3 times, most recently from 505b1a0 to b18bbc0 Compare March 25, 2019 22:07
@potiuk
Copy link
Member

potiuk commented Apr 20, 2019

@mik-laj -> are you still working on that? I will be happy to take a look once it is ready.

@mik-laj
Copy link
Member Author

mik-laj commented Apr 20, 2019

@potiuk Yes. I'm working on it. I think it's finished, but I did not have time to make sure it was merged. In addition, the previous commiter was busy preparing a new release of Airflow and I did not want to take his time.

@mik-laj
Copy link
Member Author

mik-laj commented Apr 20, 2019

I did rebase. I had to make one change because of #5059 but the rest of changes remained unchanged.

PTAL @ashb @potiuk

@ashb
Copy link
Member

ashb commented Apr 23, 2019

@mik-laj BTW I found when looking for something else that Flask-AppBuilder has inline script tags too, so to enforce our CSP policy we'd need to do a bit of work there in that project too (or to customize the base templates further)

@mik-laj
Copy link
Member Author

mik-laj commented Apr 23, 2019

@ashb, in the future, we can also deal with this problem. It's important to me that ESLInt will work right now. I want to do small steps to improve Javascript code

@ashb
Copy link
Member

ashb commented Apr 23, 2019

@mik-laj absolutely. I just mentioning it before I forget. Looking at this right now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't have | tojson on it - should it? Also how does this cope with a double-quote in the variable somewher?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quoting should be fine then, but I do think we need tojson on all of these (think, but an not certain)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now the quotation mark can not appear in the name of the dag.
If I tried to do it, I got an error message.

airflow.exceptions.AirflowException: The key (example_bash_operator">Awesome) has to be made of alphanumeric characters, dashes, dots and underscores exclusively

I have changed the value of the dag_id field in a function from views.py file to show this problem.

Screenshot 2019-04-23 at 13 47 51

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

| tojson on all of these too?

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions about escaping - I'm not sure they are needed or not but please verify.

And I think util.js is now include twice on many pages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dup include?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already discussed this in the past.
#4787 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already discussed this in the past. Double imports in rendered file mean that this views needs libraries several times in several views. This will be more readable when the code will be in separate files and we could use a "import" statement. Currently, we must use such syntax.
#4787 (comment)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already discussed this in the past. Double imports in rendered file mean that this views needs libraries several times in several views. This will be more readable when the code will be in separate files and we could use a "import" statement. Currently, we must use such syntax.
#4787 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this is inline it won't be compiled: what browsers do/don't support const? (I'm way out of date with what is ES "Next" and what is widely supported now)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we accept this patch, in the next change I will extract the Javascript code into separate files to be able to compile it into old browsers - babel. New JS features are too widely accepted in the community to write code in old standards.

Copy link
Member Author

@mik-laj mik-laj Apr 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, your suggestion is not compatible with the code rules adopted in the project. This project follow the Airbnb recommendation
https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#javascript-style-guide

2.1 Use const for all of your references; avoid using var. eslint: prefer-const, no-const-assign
2.2 If you must reassign references, use let instead of var. eslint: no-var

https://github.com/airbnb/javascript#references

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduction your suggestion will also add confusion in the code, because the "var" syntax is used in the same file. The code does not work on older versions of the Internet Explorer. However, there are a lot of reasons why this does not work and the best solution is to extract the code for separate files and transpile all JS using Babel. Transpilation will allow the developer to use new syntax and functionality, but browsers will execute code that conforms to the old standards.
Please note that some of the JS code is currently being transpiled.
https://github.com/apache/airflow/blob/master/airflow/www/package.json#L29-L34
This is not a problem that we want to solve in this PR. The JS code has many issues that have to be gradually solved.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: In the current state of the code, we can not worry about unsupported browsers. Support for other browsers can be introduced in a separate PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope that the arguments will be fine for you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dup include?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already discussed this in the past. Double imports in rendered file mean that this views needs libraries several times in several views. This will be more readable when the code will be in separate files and we could use a "import" statement. Currently, we must use such syntax.
#4787 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dup include?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have already discussed this in the past. Double imports in rendered file mean that this views needs libraries several times in several views. This will be more readable when the code will be in separate files and we could use a "import" statement. Currently, we must use such syntax.
#4787 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have two identical script tags - once here and again in the top level template. Even ignoring import statement we don't need a duplicate script tag do we?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

| tojson?

Copy link
Member Author

@mik-laj mik-laj Apr 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated a PR. Now this PR use | tojson | forceescape to make the code more explict.

@mik-laj mik-laj force-pushed the extract-jinja-variable branch from 7e8fb7f to b1edcff Compare April 24, 2019 10:28
@OmerJog
Copy link
Contributor

OmerJog commented Aug 16, 2019

What's the status of this PR?

@mik-laj
Copy link
Member Author

mik-laj commented Aug 16, 2019

Do you want to work on it? Unfortunately, I'm busy and I don't have time to finish it. I will be happy if someone take my changes and finishes them. I am going on vacation in 5 hours, so I will be AFK for a week.

@stale
Copy link

stale bot commented Nov 20, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 20, 2019
@stale stale bot closed this Nov 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants