Release 2.5.0

Co-authored-by: Richard Smythe <[email protected]>

* placeholder

* issue with NumPy 2.4.x update

* change list to numpy array for numba

* Fix/ci issues test (#1883)

* test with limited numba version

* try and alter oasis_exec_monitor.sh script

* change error log checking in bash.py

* pep8

* update chardet code

* revert log file warning checks

* revert previous commit

* test adding warning redirect sooner for gulmc

* does bash fail with redirect added

* Revert "does bash fail with redirect added"

This reverts commit 733fe2f.

---------

Co-authored-by: SkylordA <[email protected]>
Co-authored-by: Anish Kothikar <[email protected]>

* Optimize il_inputs and improve gul/il_inputs documentation

Performance:
- Add optimized path in get_il_input_items when no layered inputs exist
- Skip expensive split/concat operations when all rows have layer_id=0
- Still perform deduplication to remove premature layering
- Achieves ~23% speedup (832s -> 638s on 30M row dataset)

Documentation:
- Rewrite docstrings for get_il_input_items and get_gul_input_items
- Add section comments explaining processing flow
- Document key concepts: layered vs non-layered merging, premature
  layering removal, agg_id assignment, disaggregation

* pep8

* remove unused import

…ynamic footprint (#1869)

* gulmc Vectorize RNG inner loops

* Optimize GULMC CDF cache keys and improve documentation

Performance optimizations for the GULMC loss computation pipeline:

- Replace 5-element tuple CDF cache keys with int64 composite keys
  (eff_cdf_id << 32 | discriminator), eliminating per-item tuple
  allocation and multi-element hashing in the hot loop (~23% reduction
  in loss computation phase).
- Assign sequential eff_cdf_id per (areaperil, vulnerability) group
  during reconstruct_coverages, avoiding areaperil_id (uint64) in keys.
- Hoist cached_vuln_cdfs allocation out of the per-event loop (allocate
  once with np.empty, reuse across events).
- Replace per-event Numba Dict creation with pre-allocated array lookups
  for group_id -> rng_index mapping in reconstruct_coverages (O6).
- Vectorize inner loops in Latin Hypercube and Mersenne Twister RNG (O3).
- Add comprehensive docstrings to gen_empty_vuln_cdf_lookup, cache_cdf,
  compute_event_losses, and reconstruct_coverages.
- Add technical README.md for the gulmc module.

* gulmc Eliminate  Copy in Output Writing

* remove implementation history from readme

* set perms using arange

* Refactor fmpy variable naming conventions

Improve readability and consistency of variable names in the FM
loss computation module:

- Use _i suffix for indices/iterators (e.g., profile_i, layer_i, val_i)
- Use _count suffix for counts/lengths (e.g., children_count, node_val_count)
- Add descriptive prefixes for context (e.g., profile_, layer_, node_)
- Rename EXTRA_VALUES to EXTRA_SIDX_COUNT for clarity

Files modified:
- common.py: EXTRA_VALUES -> EXTRA_SIDX_COUNT
- compare.py: Updated import
- compute_sparse.py: Renamed loop variables and parameters
- back_allocation.py: Renamed allocation variables
- stream_sparse.py: Renamed stream handling variables
- manager.py: Renamed event loop counter

* Add comprehensive documentation comments to fmpy computation

Add module-level docstrings and detailed function comments explaining:
- FM computation architecture and algorithm flow
- CSR-like sparse storage model for memory efficiency
- Bottom-up level traversal with aggregation and back allocation
- Profile application (per-layer vs cross-layer)
- Back allocation rules and extras handling
- Stream I/O format and parsing state machine

This improves code maintainability by documenting the complex
financial module computation logic.

* Add FM documentation: technical details and user guide

TECHNICAL.md:
- Detailed CSR-inspired sparse storage format
- Data structure documentation (nodes, compute_idx, extras)
- Complete computation flow with algorithms
- Aggregation and back allocation explanations
- Stream I/O format specification
- Memory management and performance tips
- Debugging guidance

README.md:
- Quick start examples (Python API and CLI)
- Configuration options (allocation rules, net loss, memory)
- Input file requirements
- Output format explanation
- Supported financial terms
- Performance optimization tips
- Troubleshooting guide
- Architecture overview

* fixup

* fixup

* pep8

* update stable versions

* update linked packages

…at (#1864)

* Refactor extract_financial_structure to use NumPy arrays and CSR format

Replace all Numba typed Dict and List objects with memory-efficient
NumPy arrays and CSR (Compressed Sparse Row) format.

Changes:
- Replace node_layers dict with node_layers_arr (1D array)
- Replace node_cross_layers dict with node_cross_layers_arr (1D array)
- Replace node_to_output_id nested dict with output_id_arr (2D array)
- Replace programme_node_to_layers dict with layer_source array
- Replace parent_to_children dict with children_indptr/children_data CSR
- Replace child_to_parents dict with parents_indptr/parents_data CSR
- Replace programme_node_to_profiles dict with profiles_indptr/profiles_data CSR
- Replace has_tiv_policy dict with is_tiv_profile array

Implementation approach:
- Use two-pass algorithms (count then fill) to build CSR structures
  directly without intermediate dicts
- Use node_level_start[level] + agg_id for flat node indexing
- Add CSR-based helper functions: get_all_children_csr, get_all_parent_csr,
  get_tiv_csr

Add documentation describing the 11-phase data transformation pipeline.

…plete (#1875)

* wip - save anlaysis settings during file gen

* move settings gen to computation step

* make sure root anlaysis settings same as output

* move rundir/analysis_settings -> rundir/input/analysis_settings

* added tests for footprint layer and moved logs to main

* clean

* hidden logger bug

---------

Co-authored-by: Sam Gamble <[email protected]>

* Write binary files directly, bypass CSV intermediary

Eliminate the CSV intermediary step for GUL, IL/FM, and RI input files.
Previously the pipeline wrote DataFrames to CSV, then converted CSV to
binary at runtime. Now binary files are written directly from DataFrames
during preparation, with CSV output only when intermediary_csv=True
(for debugging).

Preparation changes:
- Refactor csvtobin converters to decouple CSV reading from binary
  writing: extract df_to_ndarray(), amplifications_write_bin(), and
  complex_items_write_bin() as reusable functions.
- write_gul_input_files(): replace per-column pop/prepare dicts with a
  unified files_write_info dict that drives binary+CSV output.
- get_il_input_items(): write FM binary files directly, rename pandas
  dtype dicts to avoid shadowing numpy dtypes, return .bin paths.
- write_files_for_reinsurance(): write RI binary files directly using
  df_to_ndarray().tofile().
- Thread intermediary_csv parameter through GenerateFiles, RunExposure,
  and all preparation functions.

Execution changes:
- Replace csv_to_bin/\_csv_to_bin with move_bin() which relocates
  pre-built .bin files to the run output directory.
- Update IL/RI detection in GenerateLossesDir and
  GenerateLossesDeterministic to check for .bin files (not just .csv).
- Update _check_each_inputs_directory to accept either .csv or .bin.
- Remove unused step_flag from deterministic loss commands.
- Use np.memmap(mode='r') instead of np.fromfile in load_as_ndarray
  and load_as_array for demand-paged binary reads.

Test updates:
- Remove CsvToBin test class and associated test helpers.
- Update test_generate_files expected paths from .csv to .bin.

…n jpids for multiprocessing (#1889)

Co-authored-by: Sam Gamble <[email protected]>