Skip to content
This repository was archived by the owner on Dec 20, 2024. It is now read-only.

Global Encoder-Processor-Decoder graph#9

Merged
theissenhelen merged 46 commits intodevelopfrom
feature/global-version
Jul 5, 2024
Merged

Global Encoder-Processor-Decoder graph#9
theissenhelen merged 46 commits intodevelopfrom
feature/global-version

Conversation

@JPXKQX
Copy link
Copy Markdown
Member

@JPXKQX JPXKQX commented Jun 27, 2024 •
edited
Loading

Description

Python package to build encoder-processor-decoder graph (with CLI support).

It implements 2 types of building nodes:

  • From NPZ files
  • From Zarr datasets (generated with anemoi-datasets)

The nodes may include some additional attributes:

  • corresponding to the area (or point density) associated with each node.

It implements 2 types of building edges:

  • using nearest neighbours
  • using neighbours in a radius

The edges can include some additional attributes:

  • edge direction
  • edge length

Both node & edge attributes support different normalization methods: l1, l2, unit-max, unit-std.

Added unit tests for all the functionality implemented.

The CLI follow anemoi-dataset syntax:

anemoi-graphs create recipe.yaml my_graph.pt

Issue ticket number and link

Issues #1, #2, #3, #4

This work would not have been possible without @theissenhelen @JesperDramsch and the rest of the AIFS team. Thanks!

This was linked to issues Jun 27, 2024
[Feature] Global Graph Generation #1
Closed
[Feature] Basic implementation of Node weighting #2
Closed
[Feature] Edge attributes, e.g. directionality #3
Closed
[Feature] Graph generation scripts and CLI #4
Closed
@JesperDramsch JesperDramsch added this to the First Release milestone Jun 27, 2024
@JesperDramsch JesperDramsch added the enhancement New feature or request label Jun 27, 2024
mchantry
mchantry reviewed Jun 28, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner reviewed Jul 5, 2024
View reviewed changes
sahahner
sahahner approved these changes Jul 5, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@sahahner sahahner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good from my side. I am looking forward to using it.

@theissenhelen theissenhelen merged commit 3609681 into develop Jul 5, 2024
@theissenhelen theissenhelen deleted the feature/global-version branch July 5, 2024 15:36
JesperDramsch added a commit that referenced this pull request Jul 8, 2024
@JPXKQX @theissenhelen
@sahahner @JesperDramsch
Global Encoder-Processor-Decoder graph (#9)
9231b56 
* feat: Initial implementation of global graphs

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
theissenhelen pushed a commit that referenced this pull request Jul 8, 2024
@JPXKQX
Fix (global version) (#12)
c57f797 
* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs + fixes

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* remove dataclass from attribute classes
JesperDramsch pushed a commit that referenced this pull request Jul 8, 2024
@JPXKQX @JesperDramsch
Fix (global version) (#12)
7bf0d23 
* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs + fixes

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* remove dataclass from attribute classes
theissenhelen added a commit that referenced this pull request Jul 25, 2024
@theissenhelen
6 generate graphs from icosahedral meshes (#11)
746ea2b 
* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>

* add cli command

* Ignore .pt files

* run pre-commit

* docstring + log erros

* initial tests

* feat: initial version of AttributeBuilder

* refactor: separate into node edge attribute builders

* feat: edge_length moved to edges/attributes.py

* remove __init__

* bugfix (encoder edge lengths) + refector

* feat: support path and dict for `config` argument

* fix: error

* refactor: naming

* fix: pre-commit

* feat: builders icosahedral

* feat: Add icosahedral graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* refactor: remove create_shere

* feat: Icosahedral edge builder

* feat: hexagonal graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* feat: hexagonal builders

* fix: AOI not implemented yet

* fix: abstractmethod and renaming

* chore: add dependencies

* test: add tests for trimesh

* test: add tests for hex (h3)

* fix: imports

* fix: output type

* refactor: delete unused file

* refactor: renaming and positioning

* feat: ensure src and dst always the same

* fix: imports

* fix: edge_name not supported

* test: add tests for TriIcosahedralEdges

* fix: assert missing for Hexagonal edges

* test: hexagonal edges

* fix: avoid same name

* fix: imports

* fix: conflicts

* update tests

* Include xhops to hexagonal edges

* docs: update docstrings

* fix: update attribute name

* refactor: rename multiscale nodes

* refactor: rename icosahedral nodes

* improve: clarity of function

* improve: function syntax

* refactor: simplify resolution assignment

* refactor: improve variable naming in icosahedral graph generation

* more comments

* refactor: naming

* refactor: separate into functions

* refactor: remove unused code

* refactor: remove unused options and rename

* doc: clarify cells and nodes

* fix: add return statements

* homogeneize: tri & hex edges

* naming: xhops to x_hops

* naming: x_hops in docstring

* docstring

* blank lines

* fix: add return statement

* simplify icoshaedral edges

* feat: select edge builder method based on ico node type

* test: adjust tests to new MultiScaleEdges

* docs: improve docstrings

* fix: remove LAM filtering for icosahedral, leave for next PR

* h3 (v4) not supported

---------
theissenhelen added a commit that referenced this pull request Sep 11, 2024
@theissenhelen @JPXKQX
@JesperDramsch @gmertes @iainrussell @roxas6path @anaprietonem @pre-commit-ci @github-actions
Latest release 0.3.0 (#47)
17c2847 
* Fix (global version) (#12)

* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs + fixes

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Sara Hahner <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* remove dataclass from attribute classes

* fix: config nodes name

* 6 generate graphs from icosahedral meshes (#11)

* Global Encoder-Processor-Decoder graph (#9)

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* fix: attributes as torch.float32

* new test: attributes must be float32

* fix typo

* Homogeneize base builders

* improve test docstrings

* homogeneize (name as class attribute)

* new input config

* new default

* feat: Initial implementation of global graphs

Co-authored by: Mario Santa Cruz <[email protected]>

* add cli command

* Ignore .pt files

* run pre-commit

* docstring + log erros

* initial tests

* feat: initial version of AttributeBuilder

* refactor: separate into node edge attribute builders

* feat: edge_length moved to edges/attributes.py

* remove __init__

* bugfix (encoder edge lengths) + refector

* feat: support path and dict for `config` argument

* fix: error

* refactor: naming

* fix: pre-commit

* feat: builders icosahedral

* feat: Add icosahedral graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* refactor: remove create_shere

* feat: Icosahedral edge builder

* feat: hexagonal graph generation

Co-authored-by: Mario Santa Cruz <[email protected]>

* feat: hexagonal builders

* fix: AOI not implemented yet

* fix: abstractmethod and renaming

* chore: add dependencies

* test: add tests for trimesh

* test: add tests for hex (h3)

* fix: imports

* fix: output type

* refactor: delete unused file

* refactor: renaming and positioning

* feat: ensure src and dst always the same

* fix: imports

* fix: edge_name not supported

* test: add tests for TriIcosahedralEdges

* fix: assert missing for Hexagonal edges

* test: hexagonal edges

* fix: avoid same name

* fix: imports

* fix: conflicts

* update tests

* Include xhops to hexagonal edges

* docs: update docstrings

* fix: update attribute name

* refactor: rename multiscale nodes

* refactor: rename icosahedral nodes

* improve: clarity of function

* improve: function syntax

* refactor: simplify resolution assignment

* refactor: improve variable naming in icosahedral graph generation

* more comments

* refactor: naming

* refactor: separate into functions

* refactor: remove unused code

* refactor: remove unused options and rename

* doc: clarify cells and nodes

* fix: add return statements

* homogeneize: tri & hex edges

* naming: xhops to x_hops

* naming: x_hops in docstring

* docstring

* blank lines

* fix: add return statement

* simplify icoshaedral edges

* feat: select edge builder method based on ico node type

* test: adjust tests to new MultiScaleEdges

* docs: improve docstrings

* fix: remove LAM filtering for icosahedral, leave for next PR

* h3 (v4) not supported

---------

* [feature] add changelog (#21)

* ci: changelog check

* feat: add changelog

* fix: on develop

* Skip no-commit-to-branch in code QA (#17)

* Clean nodes after building the graph (#23)

* feat: clean graph of unneeded attributes after creation

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* Make graph filename argument optional (#24)

* hotfix: make graph saving optional

Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>

* doc: update CHANGELOG

Co-authored-by: Jesper Dramsch <[email protected]>

* chore: add rtd pr preview

* HEALPix nodes (#16)

[feature] HEALPix node builder

Co-authored-by: theissenhelen <[email protected]>

* 25 integrate reusable workflows (#26)

* ci: add public pr label

* ci: add downstream-ci workflow

* ci: add ci-config

* fix: pre-commit

* docs: update changelog

* ci: switch main on downstream-ci

* ci: run ci on push on main

* Hotfix/clean graph (#28)

* Add binary dependencies to ci-config.yml (#29)

* Add binary dependencies to ci-config.yml

* docs: update changelog

---------

Co-authored-by: theissenhelen <[email protected]>

* ci: inherit pypi publish flow (#27)

* ci: inherit pypi publish flow

Co-authored-by: Jesper Dramsch <[email protected]>

* docs: update changelog

* ci: only downstream-ci for changes in src and tests

---------

Co-authored-by: Jesper Dramsch <[email protected]>

* Bugfix/31 ensure latlon input numpy array (#32)

* #31 Ensure lat lon values passed as numpy arrays

* fix formatting

* Updated CHANGELOG.md

* Remove bug in the graph cleaning function (#33)

* fix: remove the bug in the graph cleaning function

Co-authored-by: Jesper Dramsch <[email protected]>

* docs: update changelog

---------

Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Gert Mertes <[email protected]>

* Move to reusable workflows for QA and Testing (#34)

* ci: move to reusable workflows for QA and Testing

* docs: changelog add ci reusable workflow

* ci: downstream ci ignore docs

* ci: remove anchor, unsupported by github

* ci: add changelog release updater (#35)

* ci: changelog updater PR (#36)

* ci: correct inputs typo

* ci: fix changelog updater

* docs: ci fixes changelog

* ci: premissions

* ci: typo

* Ci/fix pr permission (#37)

* ci: add pull-requests permissions

* ci: add on workflow dispatch

* ci: remove 3.9 tests (#39)

* ci: remove 3.9 tests

* ci: remove testing of python 3.9

* dics: update changelog

* Fix support for Python 3.9 (#38)

* fix: support py3.9

Co-authored-by: Jesper Dramsch <[email protected]>

* fix: update & homogeneize changelog

* fix: update tests to support py39

* fix: style

* ci: rollback 3.9 tests

* refactor: annotations not used in tests

---------

Co-authored-by: Jesper Dramsch <[email protected]>

* Ci/fix_double_notes (#40)

* ci: avoid duplication of release notes

* ci: try write-all

* ci: fix permissions

* docs: update changelog

* ci: fix indent

* Inspection tools (#22)


Co-authored-by: Ana Prieto Nemesio <[email protected]>
Co-authored-by: Helen Theissen <[email protected]>

* [pre-commit.ci] pre-commit autoupdate (#41)

updates:
- [github.com/psf/black-pre-commit-mirror: 24.4.2 → 24.8.0](psf/black-pre-commit-mirror@24.4.2...24.8.0)
- [github.com/astral-sh/ruff-pre-commit: v0.4.6 → v0.6.2](astral-sh/ruff-pre-commit@v0.4.6...v0.6.2)
- [github.com/tox-dev/pyproject-fmt: 2.1.3 → 2.2.1](tox-dev/pyproject-fmt@2.1.3...2.2.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#43)

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.2 → v0.6.3](astral-sh/ruff-pre-commit@v0.6.2...v0.6.3)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* First version of documentation (#18)

Documentation web page
---------
Co-authored-by: Helen Theissen <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>

* [Changelog] Update to 0.3.0 (#44)

* [create-pull-request] automated change

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: theissenhelen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

---------

Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Jesper Dramsch <[email protected]>
Co-authored-by: Gert Mertes <[email protected]>
Co-authored-by: Mario Santa Cruz <[email protected]>
Co-authored-by: Iain Russell <[email protected]>
Co-authored-by: Rilwan (Akanni) Adewoyin <[email protected]>
Co-authored-by: Ana Prieto Nemesio <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: theissenhelen <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@JesperDramsch JesperDramsch Awaiting requested review from JesperDramsch JesperDramsch is a code owner

@mchantry mchantry Awaiting requested review from mchantry mchantry is a code owner

2 more reviewers

@theissenhelen theissenhelen theissenhelen left review comments

@sahahner sahahner sahahner approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@theissenhelen theissenhelen

@JPXKQX JPXKQX

Labels

enhancement New feature or request

Projects

None yet

Milestone

First Release

6 participants

@JPXKQX @floriankrb @mchantry @theissenhelen @sahahner @JesperDramsch