Codecov Report

Merging #1321 into master will increase coverage by 0.47%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1321      +/-   ##
==========================================
+ Coverage   65.89%   66.36%   +0.47%     
==========================================
  Files          75       75              
  Lines        5785     5944     +159     
  Branches      885      934      +49     
==========================================
+ Hits         3812     3945     +133     
- Misses       1708     1730      +22     
- Partials      265      269       +4

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a129b6b...3e3a495. Read the comment docs.

Shall all/most code generation logic live in the Jinja template?

TL;DR: No.

Currently, we use code generators to produce parts of the CircleCI YAML config files. I will be continuing this course in the pytorch/builder repo for integration tests. I would like to establish a convention going forward regarding generation practices.

One important objective is to minimize complexity of the YAML generation logic. In fact, when possible, code generation should be avoided in favor of utilizing CircleCI's API in static YAML. However, generating a matrix of configs is one use case where we cannot currently avoid code generation.

Some initial demonstrations of using exclusively Jinja as a code generator were created. I like Jinja as a means for substituting variables and simple looping. However, personally the appeal of Jinja templating ends at macros; to me they look a lot like Python functions but with inferior ergonomics to Python (e.g. more syntactical overhead, no automatic linting in CI, absent syntax highlighting in editors/Github).

I've put up a side-by-side comparison of Jinja macros and equivalent native Python for a simple YAML generation example. I also made a Python alternative using a class for the sake of experiment.

There is a contention that storing the logic strictly in Jinja templates will help help establish boundaries for complexity, whereas for general purpose code (i.e. native Python), you would have to write some rules down about limiting complexity.

However, I contend that the upper bound on complexity imposed by Jinja is loose; the template language is still sufficiently rich to get into trouble. It doesn't seem to get around the need for writing down rules about acceptable usage.

Another rationale given for Jinja-only logic is to have all the logic in one file, making it easy to skim over and read in its totality. However, I'd contend that given equivalent code length, storing logic in one file vs. two files is merely a choice between two comparable navigation modes: scrolling up and down vs. flipping left/right through editor tabs (or even displaying them side-by-side).

A couple other advantages of using the PyYAML library are data composability (which allows correct indentation to be handled automatically) and automatic quoting of YAML values when required.

Finally, regarding the proliferation of languages in CI infra: we can reasonably expect PyTorch devs to be comfortable with Python code, but devs will be strictly less familiar with Jinja syntax and capabilities.

---

Conclusion: Internal consensus is that Jinja macros are not very readable, and that logic should be expressed in native Python, while exercising restraint toward dynamic config generation.