Skip to content

Keep preprocessor variables, drop unneeded columns in chooser table #792

@jpn--

Description

@jpn--

In components where the chooser table is copied and/or merged with other data tables (e.g. interaction-simulate and interaction-sample-simulate), we would like the option to copy ONLY the required columns/variables. Doing so should greatly reduce memory requirements when there are many un-used columns.

There are two potential approaches to this:

  • Manual. Allow the user to manually specify columns to keep or columns to drop. This has already been implemented in a handful of particularly problematic components, but having a more generic / widely applicable interface for this capability would be better.
  • Automatic. Scan the specification file and have the program decide in advance of copy/merge what columns will be needed.

Some concerns and complications:

  • Tracing. It may be desirable for an analyst to have access to all variables in a trace file, not just the retained variables.
  • Non-static variable names. It might be difficult to extract all variable name references, especially if some variable names are constructed programmatically ("on the fly") inside the model spec file, instead of appearing as a literal string.
  • Estimation mode. It is desirable for an analyst to have access to all variables in estimation mode.

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureNew feature or requestPerformanceChanges that improve performance

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions