Adding auto-reporting feature by mahendrark · Pull Request #232 · rl-institut/multi-vector-simulator

mahendrark · 2020-03-25T12:27:36Z

Fix #134

Changes proposed in this pull request:

Create a draft PR for this feature - "Basic report"
Develop "basic report" - using the plotly examples above recreate the word outline of the report in html (graphs inserted as png, no data from json file, but text and formatting) -> Add F2 module

The following steps were realized, as well (if applies):

Use in-line comments to explain your code
Write docstrings to your code
For new functionalities: Explain in readthedocs
Write test(s) for your new patch of code
Update the CHANGELOG.md
Apply black (black . --exclude docs/)
Check if tests pass

Please mark above checkboxes as following:

Open
Done

❌ Check not applicable to this PR

_{For more information on how to contribute check the CONTRIBUTING.md.}

smartie2076 · 2020-03-25T12:48:38Z

As discussed, you could use variables to define that input (strings) that are added to the html site that you create with plotly:

import dash_html_components as html

SCENARIO_NAME = "any string"

print("Bla, %s", INPUT_FOLDER)

html.Div([
    html.H1('%s', SCENARIO_NAME),
    html.Div([
        html.P('Dash converts Python classes into %s', INPUT_FOLDER),
        html.P('This conversion happens behind the scenes by Dash's JavaScript front-end')
    ])
])

(I do not konw if %s and so on is the right way to use variables with the dash library, but something like that).

Bachibouzouk · 2020-03-25T15:49:13Z

If you have questions with Dash, I can help :)

import dash_html_components as html

SCENARIO_NAME = "any string"

print("Bla, %s", INPUT_FOLDER)

html.Div([
    html.H1(SCENARIO_NAME),
    html.Div([
        html.P('Dash converts Python classes into {}'.format(INPUT_FOLDER)),
        html.P('This conversion happens behind the scenes by Dash's JavaScript front-end')
    ])
])

using '%s' is the older way, now the syntax uses '{}'.format(variable_name} or directly f'{variable_name}'

mahendrark · 2020-03-26T16:10:27Z

thank you @Bachibouzouk
I will get in touch should a doubt arise
I am just looking into Dash HTML docs on Plotly websit

smartie2076 · 2020-03-30T11:22:19Z

Hi @mahendrark, is anything you can push to this branch? I would be really happy to have a look at the HTML/dash HTML/code until our call on Wednesday! :)

src/F2_autoreport.py

smartie2076

Cool, thanks for the update!
Can you add the libraries I have to install to test the code in the requirements.txt?

src/F2_autoreport.py

smartie2076 · 2020-03-31T14:54:42Z

Can you add the libraries I have to install to test the code in the requirements.txt?

This will probably have to be added:

dash==1.4.1
dash-core-components==1.3.1
dash-html-components==1.0.1

mahendrark · 2020-04-01T09:39:56Z

@smartie2076 , I will add the packages used to requirements.txt once I am done with the HTML Layout script. Other packages might be needed as well.

In addition to the changes you suggested, I will also make branch-id as a separate variable so that we can use the technic @SabineHaas suggested in the issue thread #134

smartie2076 · 2020-04-01T10:09:15Z

@smartie2076 , I will add the packages used to requirements.txt once I am done with the HTML Layout script. Other packages might be needed as well.

I think it is better to include them step by step, so that when I or so. else pulls your branch to test it can also execute it effortlessly.

In addition to the changes you suggested, I will also make branch-id as a separate variable so that we can use the technic @SabineHaas suggested in the issue thread #134

👍

smartie2076 · 2020-04-01T14:19:08Z

Maybe this will help with the tables? Link to plotly dash tutorial
Search for "table" or "Reusable Components", it seems a really nice code excerpt.

smartie2076 · 2020-04-02T11:03:43Z

Hi @mahendrark, I suppose I did something not quite proper by directly committing "Add simDate=today()" to the branch - sorry. It defines simDate directly with todays date.

Bachibouzouk · 2020-04-03T06:50:41Z

src/F2_autoreport.py

+                                                  '/logo-eland-original.svg', width='370px'),
+                       html.H1('MULTI VECTOR SIMULATION - REPORT SHEET')],
+             style={
+                 'textAlign': 'center',


It is better to save all the style setting in a class def in a cssfile, then use the prop className to link to the class. For small project inline styling is ok though.

You can save the css file in a folde rnamed assets then dash will automatically load it (the folder should be in the same path as the file defining the app)

src/F2_autoreport.py

Bachibouzouk · 2020-04-03T06:59:06Z

src/F2_autoreport.py

+
+    html.Div(className='inpdatabox', children=[html.H2('INPUT DATA')],
+             style={
+                 'textAlign': 'center',


In the css file you can add this to

.inpdatabox h2{ ... }

Bachibouzouk · 2020-04-03T07:06:15Z

@mahendrark - I don't see graphs or tables here, could you push everytime you make progress on this branch? You can create functions which return an html.Div to avoid duplicate code, ie, if many graphs or dashtables have silimar layout you can generate them with these functions you defined (with content as argument of the function).

A good thing to do is to always make the rough html structure of the whole report, then fill the sections

smartie2076 · 2020-04-08T15:17:54Z

src/F2_autoreport.py

+import dash_html_components as html
+import time
+import pandas as pd
+import reverse_geocoder as rg


Please add this also to requirements.txt:

reverse_geocoder=1.5.1
scipy=1.4.1

Bachibouzouk · 2020-04-20T10:03:09Z

src/F2_autoreport.py

+# This script generates a report of the simulation automatically, with all the important data.
+
+# Importing necessary packages
 import dash


"Updates" is not a very helpful commit message, especially if all your commit messages are "Updates" :)

You can stlll change it by doing git commit --amendand then git push -f

mahendrark · 2020-04-21T15:13:07Z

@smartie2076
can you please run the updated f2 script and let me know if you see the demand and pv input time series images in the dash rendering on the browser?

For some reason, the browser doesnt render those when I run the script.

Bachibouzouk · 2020-04-21T16:20:55Z

@mahendrark - I know why, you can't directly provide png :) Let me find the process for you:

your_image = base64.b64encode(open('<path to your image>.png', 'rb').read())

html.Img(src='data:image/png;base64,{}'.format(your_image.decode()))

smartie2076 · 2020-04-22T11:13:36Z

You can find the plot functions in F1. Specifically, here

mahendrark · 2020-04-22T11:23:33Z

@smartie2076 & @Bachibouzouk,
I have some questions related to the time series plots
Since the aim is to automate everything, how should I go about reading in the images of the plots?

Which plots to choose if there are multiple PV components and demand time series?
Will the names of the plots' images always be in a standardized format?

smartie2076 · 2020-04-22T12:15:09Z

Which plots to choose if there are multiple PV components and demand time series?

In the future, there will be multiple PV components/demands - they should all be in the automatic report at some point. Basically, we only have to wrap a loop around the calls of html.Img() to add them all in later.

Will the names of the plots' images always be in a standardized format?

Yeah, there is a standardized format - we can create the loop from that format but also the json_processed file.

For now, please concencrate on the simple example given with the "inputs" on dev. The other stuff comes in the next iteration! :)

Bachibouzouk · 2020-04-22T16:42:26Z

@mahendrark - you can use the raw data to make plotly plots :)

mahendrark · 2020-04-27T15:29:47Z

@Bachibouzouk

I am struggling a bit with the pdf report generation. Can you please help me out?

Specific question.. how do we save a HTML file from the dash web app ?
If I have that, then I could xhtml2pdf library and create a PDF?

Bachibouzouk · 2020-04-27T16:52:39Z

src/F2_autoreport.py

+
+
+
 dfprojectData = pd.read_csv('../inputs/csv_elements/project_data.csv')


Did you run black . prior to committing? I am not sure why you commit 3 empty spaces?

Oops. This is a mistake. I wanted to read the data just below the three empty lines from JSONs as well. But then postponed it and forgot to remove these lines. Consider them placeholders xD

src/F2_autoreport.py

smartie2076 · 2020-04-27T17:19:44Z

@Bachibouzouk agreed to take over the task of integrating F2 into the mvs.

Add an argument (or a simulation_setting) to activate/deactivate creating the report with F2
Enable creating a pdf from the plotly dash html, store to output path (can also be finished together with @mahendrark )
Update requirements.txt
Reformat F2 to be a callable function and call it in F0 here, below l. 104, create a test in F0 that pdf is created
Update variable calls within F2 to constants.py and dict_values (the dict structure is as in json_with_results.json

Bachibouzouk · 2020-04-27T17:49:24Z

@mahendrark - I would say that this PR should be merged and subsequent PR should be made for

the integration of F2 with MVS
improvment of F2

mahendrark · 2020-04-27T18:28:17Z

@Bachibouzouk
please go ahead and merge it as the most important tasks have been ticked off.

I will take up all other tasks in the next PR, as you suggested.

Added sectors

…ormatting of sectors

Bachibouzouk

This will be merged as is and subsequent PR will be opened for improvements to the feature of auto-reporting and integration into mvs

smartie2076 assigned mahendrark Mar 25, 2020

smartie2076 self-requested a review March 25, 2020 12:49

smartie2076 added the enhancement New feature or request label Mar 25, 2020

smartie2076 mentioned this pull request Mar 25, 2020

New Feature: Automatic generation of simulation reports #134

Closed

smartie2076 reviewed Mar 31, 2020

View reviewed changes

src/F2_autoreport.py Outdated Show resolved Hide resolved

smartie2076 reviewed Mar 31, 2020

View reviewed changes

src/F2_autoreport.py Outdated Show resolved Hide resolved

src/F2_autoreport.py Outdated Show resolved Hide resolved

smartie2076 requested a review from Bachibouzouk April 1, 2020 14:39

Bachibouzouk reviewed Apr 3, 2020

View reviewed changes

src/F2_autoreport.py Outdated Show resolved Hide resolved

Bachibouzouk reviewed Apr 3, 2020

View reviewed changes

src/F2_autoreport.py Outdated Show resolved Hide resolved

Bachibouzouk reviewed Apr 3, 2020

View reviewed changes

smartie2076 reviewed Apr 8, 2020

View reviewed changes

Bachibouzouk reviewed Apr 20, 2020

View reviewed changes

smartie2076 mentioned this pull request Apr 22, 2020

Simulation results for storage not well readable #273

Closed

Bachibouzouk reviewed Apr 27, 2020

View reviewed changes

src/F2_autoreport.py Show resolved Hide resolved

mahendrark and others added 14 commits April 27, 2020 21:52

Create F2_autoreport.py

9399ab1

Part of the 1st page of the autoreport in Dash HTML

7b6e4ef

Update F2_autoreport.py

aea094b

Add simDate=today()

27d4d66

Update F2_autoreport.py

c487706

Add E-land logo

b1a9c12

Updates to the feature

0d11dc6

Added sectors

Add more tables, images and changed formatting

36b90dc

Add more data to the report and changed table formatting

1b85d6a

Add more information, changed some formatting.

13d3d4c

Fix map, add units and energy vector columns to data tables, change f…

fefe8b6

…ormatting of sectors

Add foundation CSS sytle sheets for formatting the display

ebe62da

Lint with black

fd23057

Update requirements.txt

44b6df9

Bachibouzouk force-pushed the feature/f2_autoreporting branch from 51c2e2d to 44b6df9 Compare April 27, 2020 19:57

Bachibouzouk marked this pull request as ready for review April 27, 2020 19:57

Bachibouzouk self-requested a review April 27, 2020 19:58

Bachibouzouk approved these changes Apr 27, 2020

View reviewed changes

Bachibouzouk added 2 commits April 27, 2020 22:00

Fix requirements.txt

df2ff9b

Update changelog

97725af

Bachibouzouk merged commit bc91f2c into dev Apr 27, 2020

Bachibouzouk deleted the feature/f2_autoreporting branch April 27, 2020 20:07

Bachibouzouk mentioned this pull request Apr 27, 2020

The module F2 is not integrated into the mvs #282

Closed

5 tasks

Bachibouzouk mentioned this pull request May 28, 2020

Release/v0.2.1 #341

Merged




		dfprojectData = pd.read_csv('../inputs/csv_elements/project_data.csv')

Conversation

mahendrark commented Mar 25, 2020 • edited by Bachibouzouk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smartie2076 commented Mar 25, 2020

Uh oh!

Bachibouzouk commented Mar 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahendrark commented Mar 26, 2020

Uh oh!

smartie2076 commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

smartie2076 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

smartie2076 commented Mar 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahendrark commented Apr 1, 2020

Uh oh!

smartie2076 commented Apr 1, 2020

Uh oh!

smartie2076 commented Apr 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smartie2076 commented Apr 2, 2020

Uh oh!

Bachibouzouk Apr 3, 2020

Choose a reason for hiding this comment

Uh oh!

Bachibouzouk Apr 3, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Bachibouzouk Apr 3, 2020

Choose a reason for hiding this comment

Uh oh!

Bachibouzouk commented Apr 3, 2020

Uh oh!

smartie2076 Apr 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bachibouzouk Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

Bachibouzouk Apr 20, 2020

Choose a reason for hiding this comment

Uh oh!

mahendrark commented Apr 21, 2020

Uh oh!

Bachibouzouk commented Apr 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smartie2076 commented Apr 22, 2020

Uh oh!

mahendrark commented Apr 22, 2020

Uh oh!

smartie2076 commented Apr 22, 2020

Uh oh!

Bachibouzouk commented Apr 22, 2020

Uh oh!

mahendrark commented Apr 27, 2020

Uh oh!

Bachibouzouk Apr 27, 2020

Choose a reason for hiding this comment

Uh oh!

mahendrark Apr 27, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smartie2076 commented Apr 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahendrark commented Mar 25, 2020 •

edited by Bachibouzouk

Loading

Bachibouzouk commented Mar 25, 2020 •

edited

Loading

smartie2076 commented Mar 30, 2020 •

edited

Loading

smartie2076 commented Mar 31, 2020 •

edited

Loading

smartie2076 commented Apr 1, 2020 •

edited

Loading

smartie2076 Apr 8, 2020 •

edited

Loading

Bachibouzouk commented Apr 21, 2020 •

edited

Loading

smartie2076 commented Apr 27, 2020 •

edited

Loading