OasisLMF Changelog - 2.5.1

• #1885 - race condition with join-summary-info in generated bash script
• #1870 - Improve time and memory performance for big Portfolio
• #1870 - Improve time and memory performance for big Portfolio
• #1865 - latest docker tag repaced with 2.5
• #1866 - Fix typos and incorrect CLI docstrings
• #1869 - perf: Optimize GULMC performance - 4.5% faster standard, 76% faster dynamic footprint
• #1870 - Improve time and memory performance for big Portfolio
• #1872 - fix/lecpy_outofbounds
• #1874 - Update package readme
• #1821 - save combined analysis settings in outputs
• #1876 - Write binary files directly, bypass CSV intermediary
• #1879 - fix/oidc_header_fix
• #1882 - Package updates to either numpy or numba cause loss generation to fail.
• #1884, #1887 - Logs in main and tests in footprint_layer.py

OasisLMF Notes

fix/join-summary-info-bash - (PR #1889)

moves join-summary-info commands to end of kat/aal/lec, gives them own jpids for multiprocessing
closes #1885

Summary - (PR #1862)

Optimize get_gul_input_items and get_il_input_items functions for improved performance and add comprehensive documentation.

Changes

• Optimized DataFrame operations in get_gul_input_items to reduce memory usage and improve speed
• Optimized merge and groupby operations in get_il_input_items
• Added detailed docstrings explaining the processing flow, data structures, and key concepts

Performance Results

Big Dataset (disaggregation=False)

Initial locations: ~100k | GUL output rows: ~3.4M

Partial Dataset (disaggregation=True)

Initial locations: ~15k | GUL output rows: ~30M

Summary

• get_gul_input_items: 2.3x to 3.2x faster
• get_il_input_items: 1.18x to 1.44x faster

The disaggregation path shows larger improvements (3.2x for GUL, 1.44x for IL).

latest docker tag repaced with 2.5 - (PR #1865)

2.5 is now the latest set of images from 2.5.x oasis stable, same with 2.4 --> 2.4.x when they release

Refactor fmpy naming conventions and add comprehensive documentation - (PR #1871)

Summary

This PR improves the readability, maintainability, and documentation of the Financial Module (fmpy) computation code through systematic variable naming refactoring and comprehensive documentation additions.

fix/lecpy_outofbounds - (PR #1872)

The next_returnperiod_idx value isn't checked against the length of returnperiods before trying to access it.
This is an issue with write_ept, write_ept_weighted, write_psept, write_psept_weighted.

Update package readme - (PR #1874)

Update readmes supported versions

enhancement/root analysis_settings.json not updated when run is complete - (PR #1875)

• make output-dir/analysis_settings.json match output-dir/output/analysis_settings.json

fix/oidc_header_fix - (PR #1879)

Sets "authorization" header in request for access_token, was missing when auth_type was token.
required for OasisLMF/OasisPythonUI#49 to work with authentik

OasisLMF Changelog - 2.4.12

• #1824 - Wrong insured loss with type 2 terms and disaggregation
• #1155, #1807 - improve dtype coherence between files gen and pytools
• #1861 - CI Fix - testinstall ods/piwind from last stable, not main
• #1843 - support for pandas 3, fix BIPOI
• #1846 - Fix FM stream output: validate index range when writing special sidx
• #1816 - very low Insurance Loss values for portfolio with account level terms on version 2.4
• #1851 - Bashgen fix - swap rl for ri when running summaypy
• #1837 - Cyber FM fields not accounted for in oasis lmf profiles

OasisLMF Notes

fix: Wrong insured loss with type 2 terms and disaggregation - (PR #1825)

Use the correct term (type instead of code) to filter what term is an absolute value that needs to be dis-aggregated

improve dtype coherence between files gen and pytools - (PR #1827)

Incoherence between files generation and pytools load file dtype could lead to issue for example when loading file using np.loadtxt. This change unify the data type by making sure both side use the same source.

CI Fix - testinstall ods/piwind from last stable, not main - (PR #1861)

Need to test with prev versions not current development

fix BIPOI default value, provide support for pandas 3.0 - (PR #1843)

• fix the BIPOI 365d default value ignored
• provide multiple fix to support pandas v3.0
  note 1: those issue are grouped together as a better handling of BIPOI is needed to correctly support pandas v3.
  note 2: need Vinul great odstools fixes to fully function with pandas 3.0

Fix FM stream output: validate index range when writing special sidx - (PR #1846)

When outputting losses in the FM stream, special sidx values are written even when their loss is 0. Before writing, we check if the current sidx matches the special sidx we want to output.

The bug: The sidx value being checked could belong to the next node. IIf that value happens to match the special sidx we're looking for, we end up with an out of logical bound index. This causes the normal sample writing loop to never terminate, resulting in:

  File "/usr/local/lib/python3.12/dist-packages/oasislmf/pytools/fm/stream_sparse.py", line 220, in load_event
    if loss_val[node_loss_start + i_index]:
       ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 1425164 is out of bounds for axis 0 with size 1425164

To fix this, we add a check to make sure the sidx value we are looking at is part of the node logical range

fix IL for ptf with empty Policy layer level and account terms - (PR #1817)

Fix an issue on the prev_agg_id used for account terms for portfolio with no Policy layer level.

Bashgen fix - swap rl for ri when running summaypy - (PR #1851)

Fix for running "rl_output": true in analysis settings with "summarypy": true

support for cyber risk financial module - (PR #1852)

This PR adds support for cyber risk financial module (FM) terms to the OasisLMF framework, enabling the modeling of cyber insurance policies.

Added policy-level deductible and limit terms for 10 new cyber coverage types:

• NPBI (CoverageTypeID: 7) - Non-Physical Business Interruption
• CBI (CoverageTypeID: 8) - Contingent Business Interruption
• DIAS (CoverageTypeID: 9) - Data & Information Asset Security
• EXT (CoverageTypeID: 10) - Extortion
• FIN (CoverageTypeID: 11) - Financial
• INRE (CoverageTypeID: 12) - Incident Response
• LIA (CoverageTypeID: 13) - Liability
• REG (CoverageTypeID: 14) - Regulatory
• ENO (CoverageTypeID: 15) - Errors & Omissions
• CYB (CoverageTypeID: 16) - General Cyber

Updated exposure run logic to handle use cases without location files (e.g., cyber, marine), where exposures are defined at account level only.

OasisLMF Changelog - 2.5.0

• #1811 - add environment variables and coerce into integers
• #1809 - Improve PlatformBase login flow and debug messages
• #1813 - Fixed merging analyses computation_settings on the platform
• #1815 - Changing non OASIS_ env vars
• #1816 - very low Insurance Loss values for portfolio with account level terms on version 2.4
• #1818 - use acc_id and loc_id from merge and agg_id generation in fm files generation
• #1822 - fixing docs to correctly give way of working
• #1823 - Added unit tests for exposure data preparation functions
• #1824 - Wrong insured loss with type 2 terms and disaggregation
• #1826 - Enhancement/platform roe
• #1155, #1807 - improve dtype coherence between files gen and pytools
• #1830 - fix date format in csvs
• #1832 - katpy help text and cli doesn't allow parquet
• #1828 - Remove ktools binaries from python package
• #1831 - haz_cdf_empty dimension issue
• #1840 - perfomance improvement in lookup
• #1843 - support for pandas 3, fix BIPOI
• #224 - Optimise FM
• #1846 - Fix FM stream output: validate index range when writing special sidx
• #1837 - Cyber FM fields not accounted for in oasis lmf profiles
• #1853 - fm financial terms supported document misleading
• #1857 - fixup parquet key file exposure run
• #1860 - CI Fix - test to use new m settings loader
• #1223 - gulmc fails if vulnerability functions are numbered in non-contiguous way

OasisLMF Notes

Environment Variable Overrides for min/max_bloc_size - (PR #1811)

Add environment variables to override the min/max_bloc_size in the BasicKeyServer

fix/platform_login - (PR #1812)

change platformbase login logic and debug messages for failed login attempts
closes #1809

Fixed merging analyses computation_settings on the platform - (PR #1813)

The computation_settings sections in analyses and model settings are not pickup and merged by the platform.
Fixed so it works the same way as MDK

fix IL for ptf with empty Policy layer level and account terms - (PR #1817)

Fix an issue on the prev_agg_id used for account terms for portfolio with no Policy layer level.

use proxy acc_id and loc_id instead of LocNumber, AccNumber and PortNumber to speed up FM files generation - (PR #1818)

fix: Wrong insured loss with type 2 terms and disaggregation - (PR #1825)

Use the correct term (type instead of code) to filter what term is an absolute value that needs to be dis-aggregated

improve dtype coherence between files gen and pytools - (PR #1827)

Incoherence between files generation and pytools load file dtype could lead to issue for example when loading file using np.loadtxt. This change unify the data type by making sure both side use the same source.

fix date format in csvs - (PR #1830)

Switches all to YYYY-MM-DD
required for OasisLMF/ODS_Tools#158

fix/katpy_parquet - (PR #1833)

adds parquet option to katpy cli.py, amends help text. refactors code slightly
closes #1832

Refactored package and removed C++ binaries - (PR #1836)

• Moved to using pyproject.toml over setup.py (and updated CI to match)
• Dropped non-ORD outputs as options
• Pytools is now the main kernel option, ktools binaries are not built into the package
• Removed params that enabled pytools (now the only option)
• Removed 'legacy_stream' option
• Removed / Updated GenBash tests
• renamed parameters, anything using ktools- is now kernel-, so from --ktools-alloc-rule-il to --kernel-alloc-rule-il

fix size of haz_cdf_empty to match the intensity bin size - (PR #1838)

set the correct base size for haz_cdf_empty so it match the number of intensity bins.

Lookup performance improvements
Replaces row-by-row iteration with single vectorized pandas operations

fix BIPOI default value, provide support for pandas 3.0 - (PR #1843)

• fix the BIPOI 365d default value ignored
• provide multiple fix to support pandas v3.0
  note 1: those issue are grouped together as a better handling of BIPOI is needed to correctly support pandas v3.
  note 2: need Vinul great odstools fixes to fully function with pandas 3.0

feature/extra_oedversion_sources - (PR #1844)

Adds option to set oed version for schema from analysis settings and from exposure files.

Order of priority

CLI  # First
Oasis config file (oasislmf.json)
analysis_settings.json
exposure files (oedversion column)
ODS_Tools version  # Last

NOTE for users: Using OED Specifications below 3.1.0 will likely cause an error due to different schema formats.

closes OasisLMF/ODS_Tools#224

Fix FM stream output: validate index range when writing special sidx - (PR #1846)

When outputting losses in the FM stream, special sidx values are written even when their loss is 0. Before writing, we check if the current sidx matches the special sidx we want to output.

The bug: The sidx value being checked could belong to the next node. IIf that value happens to match the special sidx we're looking for, we end up with an out of logical bound index. This causes the normal sample writing loop to never terminate, resulting in:

  File "/usr/local/lib/python3.12/dist-packages/oasislmf/pytools/fm/stream_sparse.py", line 220, in load_event
    if loss_val[node_loss_start + i_index]:
       ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 1425164 is out of bounds for axis 0 with size 1425164

To fix this, we add a check to make sure the sidx value we are looking at is part of the node logical range

support for cyber risk financial module - (PR #1852)

This PR adds support for cyber risk financial module (FM) terms to the OasisLMF framework, enabling the modeling of cyber insurance policies.

Added policy-level deductible and limit terms for 10 new cyber coverage types:

• NPBI (CoverageTypeID: 7) - Non-Physical Business Interruption
• CBI (CoverageTypeID: 8) - Contingent Business Interruption
• DIAS (CoverageTypeID: 9) - Data & Information Asset Security
• EXT (CoverageTypeID: 10) - Extortion
• FIN (CoverageTypeID: 11) - Financial
• INRE (CoverageTypeID: 12) - Incident Response
• LIA (CoverageTypeID: 13) - Liability ...

OasisLMF Changelog - 2.3.20

• #1824 - Wrong insured loss with type 2 terms and disaggregation
• #1846 - Fix FM stream output: validate index range when writing special sidx
• #1863 - fm: use correct term disaggreagtion filter (#1825) ~ backport

OasisLMF Notes

fix: Wrong insured loss with type 2 terms and disaggregation - (PR #1825)

Use the correct term (type instead of code) to filter what term is an absolute value that needs to be dis-aggregated

Fix FM stream output: validate index range when writing special sidx - (PR #1846)

When outputting losses in the FM stream, special sidx values are written even when their loss is 0. Before writing, we check if the current sidx matches the special sidx we want to output.

The bug: The sidx value being checked could belong to the next node. IIf that value happens to match the special sidx we're looking for, we end up with an out of logical bound index. This causes the normal sample writing loop to never terminate, resulting in:

  File "/usr/local/lib/python3.12/dist-packages/oasislmf/pytools/fm/stream_sparse.py", line 220, in load_event
    if loss_val[node_loss_start + i_index]:
       ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 1425164 is out of bounds for axis 0 with size 1425164

To fix this, we add a check to make sure the sidx value we are looking at is part of the node logical range

fm: use correct term disaggreagtion filter (#1825) ~ backport - (PR #1863)

... Release notes description / summary
... Any text between these two tags will be automatically pulled into the platform release notes

OasisLMF Changelog - 2.4.11

• #1816 - very low Insurance Loss values for portfolio with account level terms on version 2.4
• #1818 - use acc_id and loc_id from merge and agg_id generation in fm files generation
• #1811 - add environment variables and coerce into integers
• #1813 - Fixed merging analyses computation_settings on the platform

OasisLMF Notes

fix IL for ptf with empty Policy layer level and account terms - (PR #1817)

Fix an issue on the prev_agg_id used for account terms for portfolio with no Policy layer level.

use proxy acc_id and loc_id instead of LocNumber, AccNumber and PortNumber to speed up FM files generation - (PR #1818)

Environment Variable Overrides for min/max_bloc_size - (PR #1811)

Add environment variables to override the min/max_bloc_size in the BasicKeyServer

Fixed merging analyses computation_settings on the platform - (PR #1813)

The computation_settings sections in analyses and model settings are not pickup and merged by the platform.
Fixed so it works the same way as MDK

OasisLMF Changelog - 2.4.10

• #1793 - clearer step param help info
• #1794 - read coverages and lossfactors functions have faulty logic for stdin
• #1684 - Validate output OED fields against portfolio early in the model run
• #1803 - Port blocked issues
• #1798 - bug/summary-info files not grouping by fields from accounts file
• #1801 - allow selection of oed backend dtype (pyarrow of numpy)
• #1804 - Build_merge accepts only csv arguments
• #1800 - bug/BIPOIType and BIWaitingPeriodType loaded incorrectly

OasisLMF Notes

Fix read coverages and lossfactors functions for stdin - (PR #1795)

Currently these functions wrongly check for an existing file before reading data, even when stdin is selected. Refactors functions to avoid this happening

Feature/analysis_settings_oed_fields_check - (PR #1796)

adds prelim check to GenerateFiles to check if oed_fields in analysis_settings are valid
Tries to solve #1684

fix/summary-info files not merging on acc file fields - (PR #1799)

• fix issue with merging acc_idx when generating summary-info in input generation

Add parameter oed_backend_dtype to allow selection of dtype when loading OED - (PR #1801)

Pandas now support two types of backend when loading csv files, pyarrow and numpy. this PR allow users to select which backend dtype ODS_tools should use

fix type of BIWaitingPeriodType and BIPOIType - (PR #1805)

OasisLMF Changelog - 2.4.9

• #1775 - Fix: Giving df engine to exposure
• #1781 - Choice of OED schema
• #1782 - Revamp/il by level
• #8 - PiWind model run issues
• #1784 - CI fix pip-comple error
• #1785 - Lookup arg 'complex_lookup_config_fp' has unexpected value

OasisLMF Notes

rewrite insurance loss fm file generation to process level one at a time - (PR #1782)

• write il fm file one level at a time to reduce memory consumption and improve performance
• fix a rare issue on fmpy layer calculation where extra info for min and max deductible were incorrect when setting new layers

part of work needed for OasisLMF/OasisPlatform#1082

adds option to check analysis_settings for join_summary_info - (PR #1783)

Allows join_summary_info via analysis_settings.json as well as the previous oasislmf.json file.

CI fix pip-comple error - (PR #1784)

• Pinned pip package on failing test
• Fixed PostgreSQL docker image mount
• OasisLMF/OasisPiWind#198

Fixed bug when loading complex_lookup_config_fp - (PR #1790)

See: #1785

OasisLMF Changelog - 2.3.19

• #1784 - CI fix pip-comple error
• #1776 - fix simple step policies
• #1788 - bypass amplification.bin load in pla when uniform factor is provided (backport)

OasisLMF Notes

CI fix pip-comple error - (PR #1784)

• Pinned pip package on failing test
• Fixed PostgreSQL docker image mount
• OasisLMF/OasisPiWind#198

fix simple step policies - (PR #1776)

Fix an issue in need_account_aggregation step that remove step id from the aggregation key thus removing step policies.

bypass amplification.bin load in post loss amplification when uniform factor is provided - (PR #1788)

When a uniform factor is provided, amplification.bin load is not needed. This PR remove this step and allow user to provide a uniform factor without the need to create the amplification.bin file

OasisLMF Changelog - 2.4.8

• #1762 - adds check for full data buffer with read_buffer functions for fm and…
• #1763 - Add areaperil_id filter for FootprintParquet
• #1765, #1766 - Enhancement/add settings to lookup and hooks
• #1768 - trailing slash missing for settings_template endpoint
• #1769 - Port eve to pytools
• #1771 - add cyber perils to hardcoded AA1 list of perils
• #1773 - Fix: websocket does not support context manager
• #1747 - Progress bar

OasisLMF Notes

Bug fix for fm and gul bintocsv - (PR #1762)

There was no check for full data buffer which causes idx out of range error.

Add areaperil_id filter for FootprintParquet - (PR #1763)

Add areaperil_id filter when querying data from FootprintParquet. This is help reduce the data transfer needed when using parquet over S3.

add settings information to lookup and hooks - (PR #1764)

• make settings information available in lookup and hooks.

• use OasisDataManager in Built-in lookup.

fix/trailing slash missing for setting_templatesendpoint - (PR #1767)

• trailing slash missing when no ID provided
• add tests to check for trailing slash

feature/evepy - (PR #1770)

• Implement eve in python as a new cli tool evepy
• Maintains the same behaviour as eve with a round robin shuffle by default and the following command line options:
• • -n : no shuffle
• • -r : randomise using Fisher Yates algorithm and stream output whilst randomising
• • -R: randomise using builtin shuffle

Also support input and output file selection using the -i and -o flags respectively.

add cyber perils to hardcoded AA1 list of perils - (PR #1771)

Allow exposure run to work without the need to specify cyber perils in --model-perils-covered

OasisLMF Changelog - 2.4.7

• #1706 - improve BI calculations based on duration of interruption
• #1749 - PLA amplification file should not be required when uniform factor is given
• #1758 - add analysis settings template endpoint to platform_api
• #1737 - pytools output cli should support pipe operations ( | , < , > )

OasisLMF Notes

feature/BI improvement - fmp - (PR #1745)

A new fm level SiteItem applies BIWaitingPeriod and BIPOI (period of indemnity).

• This new level is applied before the SiteCoverage level.
• Output loss from this level is calculated as output_loss = min(input_loss - BIWaitingPeriod, BIPOI).
• For multi peril the terms are in accordance with the alloc_rule_il, by default allocation will occur proportional to the input loss.

bypass amplification.bin load in post loss amplification when uniform factor is provided - (PR #1756)

When a uniform factor is provided, amplification.bin load is not needed. This PR remove this step and allow user to provide a uniform factor without the need to create the amplification.bin file

add settings template endpoint to api client - (PR #1757)

• add a interface for the /models/{model_pk}/setting_templates/ to the platform api client

adds stdin functionality for eltpy and pltpy - (PR #1751)

can use pipe operations for input summary files

Current command example

eltpy -E csv -i summary.bin -s selt.csv -m melt.csv -q qelt.csv

What is now possible

eltpy -E csv -i - -s selt.csv -m melt.csv -q qelt.csv < summary.bin  # Use '-' as placeholder for stdin
eltpy -E csv -s selt.csv -m melt.csv -q qelt.csv < summary.bin  # Pipe in without placeholder
cat summary.bin | eltpy -E -i - csv -s selt.csv -m melt.csv -q qelt.csv  # Use '-' as placeholder for stdin
cat summary.bin | eltpy -E csv -s selt.csv -m melt.csv -q qelt.csv  # Pipe in without placeholder