I am taking over this PR now. I will document what I did here.

1. Download epa_benchmark.json
2. Create function to translate the EPA json to a MVS json (utils.helpers.translates_epa_strings_to_mvs_readable()), output:

>>> import multi_vector_simulator.utils.helpers as helpers
>>> helpers.translates_epa_strings_to_mvs_readable("./epa_benchmark", "epa_benchmark.json")
WARNING:root:The parameters group 'constraints' is not present in the EPA parameters to be parsed into MVS json format
WARNING:root:The parameters group 'fixcost' is not present in the EPA parameters to be parsed into MVS json format

[Trying to run the inputs with the MVS: python mvs_tool.py -i epa_benchmark -ext json -f -o epa_output]

3. Missing from the json file are fixcosts and constraints. Both are added through the parser, but the constraints are not added correctly. Instead of the valid:

    "constraints": {
        "minimal_renewable_factor": {
            "unit": "factor",
            "value": 0.3
        }
    },

This is added:

    "constraints": [
        {
            "minimal_renewable_factor": {
                "unit": "factor",
                "value": 0
            }
        }
    ],

Fixed with 12ffdf8

4. Manual change: simulation_settings/timestep is not entered as a key-value pair when it comes from the EPA. Therefore I had to change this:
   "timestep": 60,
   With resulted in following error:

  File "d:\pycharmprojects\mvs_eland\src\multi_vector_simulator\B0_data_input_json.py", line 234, in retrieve_date_time_info
    freq=str(simulation_settings[TIMESTEP][VALUE]) + UNIT_MINUTE,
TypeError: 'int' object is not subscriptable

To this:

        "timestep": {
            "unit": "minutes",
            "value": 60
        }

5. EPA allows label duplicates:

multi_vector_simulator.utils.exceptions.DuplicateLabels: Following asset label is not unique with 2 occurrences: Electricity_grid_DSO.
Please make sure that each label is only used once, as oemof otherwise can not build the model.

This should not be allowed. Is it possible to have a test for this?
I refactored the label of the energyConversion asset with label Electricity_grid_DSO to Electricity_grid_DSO_transformer.

6. It seems that the energy carrier name can be defined freely in the EPA. This is not possible for the MVS, it has to be one of the carriers in DEFAULT_WEIGHTS_ENERGY_CARRIERS, including capitalization. There should be a drop-down-list to choose from.

multi_vector_simulator.utils.exceptions.UnknownEnergyVectorError: The energy carrier electricity of asset group energyBusses, asset Electricity (DSO)_pdp is unknown, as it is not defined within the DEFAULT_WEIGHTS_ENERGY_CARRIERS.Please check the energy carrier, or update the DEFAULT_WEIGHTS_ENERGY_CARRIERS in contants.py (dev).

7. The peak_demand_pricing_period should be an int value, not a float

            "peak_demand_pricing_period": {
                "unit": "times per year (1,2,3,4,6,12)",
                "value": 1.0

Is this something that the EPA can ensure?

8. "energyProviders"/ "Electricity_grid_DSO" should have a field UNIT. It seems that this parameter is missing for energy provider assets in the EPA.

    "energyProviders": {
        "Electricity_grid_DSO": {
            "unit": "kWh",

9. "energyProviders"/ "Electricity_grid_DSO" should have a field input_bus_name. It seems that this parameter is missing for energy provider assets in the EPA.

    "energyProviders": {
        "Electricity_grid_DSO": {
            "input_bus_name": "a_name",

Note: input_bus_name and output_bus_name are deprecated, therefore translated with the parser: 15eba2f

10. Added timeseries/data list for PV plant and demand from tests/inputs, changed simulation_settings/evaluated_period to 1 day.

11. Now, timeseries is:

            "timeseries": {
                "unit": "kWh",
                "data": [
                    144.37,
                    ...
                    157.88
                ]
            }

However, as MVS input, "timeseries" should be defined by VALUE and DATA_TYPE_JSON_KEY: TYPE_SERIES and does not have a "unit" (eventhough that makes sense).
Fixed with a047410

12. energyProduction assets do not have a UNIT when provided by the EPA.

13. energyStorage assets have one parameter optimize_cap for ALL sub-assets [INPUT_POWER, OUTPUT_POWER, STORAGE_CAPACITY], but the EPA provides this option for each individually. In the end, this is better, but just not what happens currently. This should be changed in the EPA.
    I deleted from the individual assets and moved the option to the main asset manually.

14. Removed assets that are automatically generated by MVS, but were faultyly added manually when using the EPA interface for the blueprint inputs (excess sink, pdp bus, consumption period transformers)

15. The energyConsumption asset has too many unneeded parameters:
    "dispatchable", "age_installed", "development_costs", "dispatch_price", "lifetime", "specific_costs", "specific_costs_om", "installedCap", "optimizeCap"
    But does not have UNIT

16. energyProvider assets have value for optimizeCap==False, which is incorrect. It should always be optimizeCap==True

17. The MVS does not need to know the assets connected to each defined energyBusses - this is evaluated in pre-processing. It may be used as a cross-reference, but should be dropped before using the json file as MVS input:

    "energyBusses": {
        "Electricity (DSO)": {
            "label": "Electricity (DSO)",
            "assets": [
                "Electricity_grid_DSO_transformer",
                "Electricity_grid_DSO"
            ],
            "energyVector": "Electricity"

When reading from json, the REQUIRED_CSV_PARAMETERS do not work, so there are no error message for the additional parameters.

18. Parameter maximumCap is missing from the inputs for all assets that require it.

19. Parameter renewable_asset is not a key-value-pair:

            "renewableAsset": {
                "value": true
            },

20. For some reason, the energy system can not be solved if soc_initial/value == 0.0 for storages.

                "soc_initial": {
                    "unit": "None or factor",
                    "value": 0.0
                },

When setting "value" to None, the simulation runs through. Input data for later issue:
mvs_config-with-soc-initial-0.txt

With all of the above changes, a valid input file for the MVS can be created from the current EPA output (as per the data that @Bachibouzouk posted his EPA output first):

mvs_config.txt

The report can be generated:

simulation_report.pdf

As well as the results:

json_with_results.txt

It is a simple scenario with following energy system:

On EPA side it looks like this:

The results are equivalent to the results of the benchmark test.

Left to do:

• Decide which adaptations are to be automatized
• Automatize adaptations
• Replace string represenatations of EPA directly (not working)

@smartie2076 - the potential problem is that if EPA change and their outputed json change, then we need to redo a lot of manual steps. It also requires manual change to the assets fields and rebuilding the model everytime (this is why I ended up making a very simple benchmark) with drag and drop before we can generate the json, this is a lengthy process. I think it is better to make only the non-negotiable changes (such as unit field as a unitless value does not make physical sense) on EPA side and provide flexibility with the EPA-MVS and MVS-EPA parsers for the other formatting/parameters (like a parameter which should be an int and is provided as a float is relatively easy to cover on our side; that being said the user might want to know that 1.2 will be turned into 1. We could use https://github.com/rl-institut/multi-vector-simulator/blob/dev/docs/MVS_parameters_list.csv for enforcing the right type of parameter)

@smartie2076 - I pulled backup/adapt-epa-parser_martha locally, I still had the previous version of fix/adapt-epa-parser locally, so I added your commits with ´git cherry-pick ´ one by one (there is certainly a smarter way to do this), then I rebased onto latest dev and here we go. There were a few conflicts which I think I solved the right way. You'll be able to double check when reviewing.

One remark: With #184 and #726 there will be two parameters added to the input files. We need to remember to add them to the parser. I think that in connection with #750 we should think of using the default parameters also for the parser when we implement it...

This is the file which is provided by EPA benchmark model
json2MVS.json.txt

The following standalone script is then run on the file

from multi_vector_simulator.utils.data_parser import (
    MAP_EPA_MVS,
    MAP_MVS_EPA,
    convert_mvs_params_to_epa,
    convert_epa_params_to_mvs,
)

if __name__ == "__main__":
    import json

    with open("json2MVS.json") as json_file:
        epa_dict = json.load(json_file)

    dict_values = convert_epa_params_to_mvs(epa_dict)
    # to save the converted file to a json
    with open("epa_benchmark_converted_to_mvs.json", "w") as json_file:
       json.dump(dict_values, json_file, indent=4)

    from multi_vector_simulator import server

    server.run_simulation(dict_values)

With the parser in its current state it leads to the following error and warnings:

ERROR:root:The age of the asset `demand_01` (10 years) is lower or equal than the asset lifetime (10 years). This does not make sense, as a replacement is imminent or should already have happened. Please check this value.
WARNING:root:You might use an old input file! The efficiency of the storage capacity of 'storage' is 0.001, although it should represent the ability of the storage to hold charge over time; check PR #676.
ERROR:root:Energy system bus Electricity (DSO)_pdp has too few assets connected to it. The minimal number of assets that need to be connected so that the bus is not a dead end should be two, excluding the excess sink. These are the connected assets: Electricity_grid_DSO_consumption


WARNING:root:The calculation of the Total_excess per sector and their energy equivalent may currently be faulty. Please refer to issue #559
WARNING:root:Attention, on bus Electricity there is excessive excess generation, totalling up to 18% of the inflows. The total inflows are 105113 and outflows 86049  It seems to be cheaper to have this excess generation than to install more capacities that forward the energy carrier to other busses (if those assets can be optimized).

For more log message one can also copy and rename the generated epa_benchmark_converted_to_mvs.json file to mvs_config.json within a folder named epa_benchmark_converted_to_mvs.json and run
mvs_tool -i epa_benchmark_converted_to_mvs.json -o benchmarkcomparison -png

The energy system model schema is the same as the one in #675 (comment)

ERROR:root:The age of the asset demand_01 (10 years) is lower or equal than the asset lifetime (10 years). This does not make sense, as a replacement is imminent or should already have happened. Please check this value.

Energy consumption assets usually not use lifetime or an age. Both parameters shouldn't exist for this asset types (others that should not be there compage above, point 15. Their default value should be zero.

WARNING:root:You might use an old input file! The efficiency of the storage capacity of 'storage' is 0.001, although it should represent the ability of the storage to hold charge over time; check PR #676.

This should tell you to check the efficiency of your storage. If the efficiency of storages is only 0.01, it means that 99% of your stored energy is lost within one time step (self-discharge). So you have to adapt the value in your input file (from EPA). I dont think there is anything wrong with the EPA in this though.

ERROR:root:Energy system bus Electricity (DSO)_pdp has too few assets connected to it. The minimal number of assets that need to be connected so that the bus is not a dead end should be two, excluding the excess sink. These are the connected assets: Electricity_grid_DSO_consumption

This is fine, and always occurs. the is already issue about it, but basically you can ignore it.

WARNING:root:The calculation of the Total_excess per sector and their energy equivalent may currently be faulty. Please refer to issue #559

This is an open issue that we still have to fix, and then the warning will also go away. the user input has nothing to do with this.

WARNING:root:Attention, on bus Electricity there is excessive excess generation, totalling up to 18% of the inflows. The total inflows are 105113 and outflows 86049 It seems to be cheaper to have this excess generation than to install more capacities that forward the energy carrier to other busses (if those assets can be optimized).

This is a valid warning for the user, as it might hint towards weird outputs. also tells the user what should be changed to influence the weird behaviour.

This should tell you to check the efficiency of your storage. If the efficiency of storages is only 0.01, it means that 99% of your stored energy is lost within one time step (self-discharge). So you have to adapt the value in your input file (from EPA). I dont think there is anything wrong with the EPA in this though.

The value in the EPA file (the json2MVS.json) for "capacity" is 0.001

The value in the EPA file (the json2MVS.json) for "capacity" is 0.001

You mean this, right?

  "energy_storage": [
    { 
      "capacity": {
        "efficiency": {
          "unit": "factor",
          "value": 0.001
        },
      }
    }
  ],

We need to tell them to change it in their benchmark test, as it is not correct anymore. It should also not be in our benchmark test in the MVS anymore.