Skip to content

allow cached datasets to undergo futher selection#294

Merged
matthewlchambers merged 2 commits intorecursive-refacfrom
cache_select_fields
Nov 3, 2022
Merged

allow cached datasets to undergo futher selection#294
matthewlchambers merged 2 commits intorecursive-refacfrom
cache_select_fields

Conversation

@bl-young
Copy link
Copy Markdown
Contributor

@bl-young bl-young commented Nov 3, 2022 •
edited
Loading

Proposal to allow cached datasets to be further filtered for a specific activity set. Should not impact existing methods with cached datasets.

See following simplified example:

sources_to_cache:
  EIA_MECS_Energy:
    year: 2014
    selection_fields:
      Class: Energy
      Unit: Trillion Btu
      Description:
          - Table 2.2
          - Table 3.2
    estimate_suppressed: !script_function:temp_data_source_functions estimate_suppressed_mecs_energy
    clean_fba: !script_function:temp_data_source_functions clean_mecs_energy_fba
    clean_fba_w_sec: !script_function:temp_data_source_functions clean_mapped_mecs_energy_fba
    attribution_method: proportional
    attribution_source:
      Employment_national_2018: # replace with 2014 when available
        data_format: FBS
        year: 2018

source_names:
  EPA_NEI_Nonpoint:
    geoscale: state
    year: 2017
    activity_to_sector_mapping: SCC
    fedefl_mapping: NEI
    activity_sets:
      industry_combustion_coal:
        selection_fields:
          PrimaryActivity: !from_index:NEI_Nonpoint_2017_asets.csv industry_combustion_coal
        attribution_method: proportional
        attribution_source:
          EIA_MECS_Energy:
            selection_fields:
              Flowable: Coal

@matthewlchambers
Copy link
Copy Markdown
Contributor

matthewlchambers commented Nov 3, 2022 •
edited
Loading

@bl-young Just pushed (to cache_select_fields branch) a tweak that allows us to treat the cached FBS just like we would any other FBS that we're using as an attribution source, including but not limited to selection. Thoughts?

@bl-young
Copy link
Copy Markdown
Contributor Author

bl-young commented Nov 3, 2022

That makes sense to me - so that could include any additional data suppression or cleaning or something like that. I guess the limitation for the user is that you can only add but not take away (which makes sense...). I can't think of any use cases right now but don't see a reason not to include the function.

FYI I expect this would be quite handy for use of EIA_MECS in the ghg methods as well.

@matthewlchambers
Copy link
Copy Markdown
Contributor

matthewlchambers commented Nov 3, 2022

That makes sense to me - so that could include any additional data suppression or cleaning or something like that. I guess the limitation for the user is that you can only add but not take away (which makes sense...). I can't think of any use cases right now but don't see a reason not to include the function.

FYI I expect this would be quite handy for use of EIA_MECS in the ghg methods as well.

I expect so as well. I'll go ahead and merge this then.

@matthewlchambers matthewlchambers merged commit a3cc951 into recursive-refac Nov 3, 2022
@matthewlchambers matthewlchambers deleted the cache_select_fields branch November 3, 2022 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@matthewlchambers matthewlchambers Awaiting requested review from matthewlchambers

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bl-young @matthewlchambers