Is this superseded by #883 (comment) ?

@bryanwweber ... I don't think so, as this is (thus far) for Python.

Edit: I haven't looked into running (as opposed to building) of cxx yet.

@ischoegl Ray's work in #883 runs the C++ samples and I think #874 did Fortran, so no need to worry there. We usually refer to the Python code as examples rather than samples (hence my earlier confusion), do you mind renaming this?

We usually refer to the Python code as examples rather than samples (hence my earlier confusion), do you mind renaming this?

No problem.

Codecov Report

Merging #892 into main will increase coverage by 0.50%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #892      +/-   ##
==========================================
+ Coverage   70.95%   71.45%   +0.50%     
==========================================
  Files         376      376              
  Lines       45889    46690     +801     
==========================================
+ Hits        32560    33362     +802     
+ Misses      13329    13328       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 047e74b...8cfbc4e. Read the comment docs.

@bryanwweber ... will stop here. Ubuntu works, and there are some environmental variables left on windows that I need to work out. OSX runners currently won’t leave the queue, so I disabled the new runs. matplotlib causes some issues for sundials and Xenial runners, so those won’t work.

PS: matplotlib causes issues on OSX runners also

ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to 
function correctly if Python is not installed as a framework. See the Python documentation for 
more information on installing Python as a framework on Mac OS X. Please either reinstall 
Python as a framework, or try one of the other backends. If you are using (Ana)Conda please 
install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib 
on OSX' in the Matplotlib FAQ for more information.

Alright. At this point I believe I have come to a stopping point.

• check-examples works on Linux and Windows; I cannot test on OSX (Edit: the CI runner finally works, so it presumably is ok)
• matplotlib (which is used by many examples) causes issues on some CI environments (e.g. version requirements)
• CI runner works on Ubuntu, Windows, and macOS, as long as the Agg backend for matplotlib is used (this is essential for Windows and macOS)
• matplotlib isn't compatible with the OSX CI environment Edit: this is a Python 3.5 issue

Caveats: scons check-examples runs in the build folder, and some outputs will get copied during installation, which I'm not sure is an issue. Further, examples with known issues are skipped.

🎉 ... finally the CI runners work on all OS's - it was mainly a function of matplotlib backends, and avoiding Python 3.5.

Running the samples as part of the test process certainly makes sense. I was thinking about this myself as a follow on to #874 and #883. The approach that I had been considering was to run the Python examples as part of the Python test suite.

The advantage of that approach is that we would be able to use the unittest machinery to do some checks that the output of these samples is as expected. Perhaps not all of the detailed numerical value comparisons that we do for some other tests, but at least we could check that expected output files are generated, and that there are no deprecation warnings issued.

@speth ... thanks for the feedback.

I think the important thing is that examples are run, which is an infinite improvement over the status quo where examples can break without notice, despite being careful when closing out a PR. Going back to fix issues several months later is a painful exercise (thinking about #877 here; #875 was thankfully trivial, and the others I don't think were due to my work).

From a philosophical viewpoint, I view unit tests as sanity checks on small test problems, whereas examples are coding examples that are provided to end users. The handful of python examples currently take up about the same amount of run time as the entire test suite, so it's not particularly efficient to automatically add the examples. One other caveat is that writing examples that include regression tests will require a lot more developer input as it's not as straight forward as simply contributing a script.

Regarding my proposed implementation, I think that its simplicity is a feature: it's clear and simple to maintain (scons by itself is not for the faint of heart). I really think that just checking that examples don't break during modifications is a beyond 90% solution. Getting to a higher percentage will complicate things a lot more, and I am not sure that it is worth the effort. I honestly believe that beefing up on unit tests instead is preferable (Edit: probably there should be more that compare to a reference/'blessed' solution).

Ad warnings: I agree on DeprecationWarnings (or warnings in general). In this regard, however, not all are caught by Python, as warnings raised in C++ may 'escape'. One option would be to | tee log terminal output to a file, and simply search for deprecations. I also looked into the PYTHONWARNINGS=error environment variable, but it's failing due to an ImportWarning right from the get-go (a known issue caused by Cython); otherwise making sure that no warnings are raised would be an option. Edit: found a solution - see below.

PS: I added the option -W error::DeprecationWarning: which will throw errors. (Edit: already found a DeprecationWarning, but still need to circumnavigate the Windows issue with an imported library throwing a DeprecationWarning).

@speth and @bryanwweber ... with the latest force-push, the DeprecationWarnings are now raised as errors on all OS's (had to filter out an imp related warning that is specific to Windows). I could also include an additional test (and remove one redundant commit) once #876 gets merged.

Any feedback is appreciated at this point; if you'd rather pursue a different route, I'll simply close this PR.

Closing. While it was interesting to figure out scons for some simple checks, adding more comprehensive regression tests is not my cup of tea. I added fixes that deserve to survive to #876 and #899.

Reopening as an alternative to the CI runners proposed in #905 - while this implementation is still a stop-gap, it runs examples on all OS's, can be invoked via scons, and checks for DeprecationWarning's.

The PR needs to be rebased once #899 is resolved.

Closed again as there appears to be strong opposition to this suggestion.