Skip to content

Automate ASDF schema updates#631

Merged
Cadair merged 36 commits intoDKISTDC:mainfrom
SolarDrew:asdf-automation
Oct 16, 2025
Merged

Automate ASDF schema updates#631
Cadair merged 36 commits intoDKISTDC:mainfrom
SolarDrew:asdf-automation

Conversation

@SolarDrew
Copy link
Contributor

@SolarDrew SolarDrew commented Sep 17, 2025

Deals with all the fiddly nonsense of changing a dozen things every time we have to bump a schema version.

Handles updating an existing schema:

$ python tools/update_schema.py dataset
$ git status
On branch asdf-automation
Your branch is up to date with 'origin/asdf-automation'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   dkist/io/asdf/converters/dataset.py
	modified:   dkist/io/asdf/entry_points.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	dkist/io/asdf/resources/manifests/dkist-1.6.0.yaml
	dkist/io/asdf/resources/schemas/dataset-1.3.0.yaml

no changes added to commit (use "git add" and/or "git commit -a")
$ git diff
diff --git a/dkist/io/asdf/converters/dataset.py b/dkist/io/asdf/converters/dataset.py
index 26eec70..f92d119 100644
--- a/dkist/io/asdf/converters/dataset.py
+++ b/dkist/io/asdf/converters/dataset.py
@@ -5,9 +5,11 @@ from asdf.extension import Converter
 
 class DatasetConverter(Converter):
     tags = [
+        "asdf://dkist.nso.edu/tags/dataset-1.3.0",
         "asdf://dkist.nso.edu/tags/dataset-1.2.0",
         "asdf://dkist.nso.edu/tags/dataset-1.1.0",
         "asdf://dkist.nso.edu/tags/dataset-1.0.0",
+        "tag:dkist.nso.edu:dkist/dataset-0.4.0",
         "tag:dkist.nso.edu:dkist/dataset-0.3.0",
         "tag:dkist.nso.edu:dkist/dataset-0.2.0",
         "tag:dkist.nso.edu:dkist/dataset-0.1.0",
diff --git a/dkist/io/asdf/entry_points.py b/dkist/io/asdf/entry_points.py
index e5f90c6..02c48ad 100644
--- a/dkist/io/asdf/entry_points.py
+++ b/dkist/io/asdf/entry_points.py
@@ -39,6 +39,7 @@ def get_extensions():
     dkist_converters = [FileManagerConverter(), DatasetConverter(), TiledDatasetConverter()]
     wcs_converters = [VaryingCelestialConverter(), CoupledCompoundConverter(), RavelConverter(), AsymmetricMappingConverter()]
     return [
+        ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.6.0", converters=dkist_converters),
         ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.5.0",
                                    converters=dkist_converters),
         ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.4.0",

dkist/io/asdf/resources/manifests/dkist-1.6.0.yaml is a copy of dkist-1.5.0.yaml with all the appropriate version numbers bumped; likewise dkist/io/asdf/resources/schemas/dataset-1.3.0.yaml increments dataset-1.2.0.yaml.

Also handles creating a new schema:

$ python tools/update_schema.py new-schema
$ git status
On branch asdf-automation
Your branch is up to date with 'origin/asdf-automation'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   dkist/io/asdf/converters/__init__.py
	modified:   dkist/io/asdf/entry_points.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	dkist/io/asdf/converters/new_schema.py
	dkist/io/asdf/resources/manifests/dkist-1.6.0.yaml
	dkist/io/asdf/resources/schemas/new_schema-0.1.0.yaml

no changes added to commit (use "git add" and/or "git commit -a")
$ git diff
diff --git a/dkist/io/asdf/converters/__init__.py b/dkist/io/asdf/converters/__init__.py
index 6493e43..74e0eef 100644
--- a/dkist/io/asdf/converters/__init__.py
+++ b/dkist/io/asdf/converters/__init__.py
@@ -3,3 +3,4 @@ from .file_manager import FileManagerConverter
 from .models import (AsymmetricMappingConverter, CoupledCompoundConverter,
                      RavelConverter, VaryingCelestialConverter)
 from .tiled_dataset import TiledDatasetConverter
+from .new_schema import NewSchemaConverter
\ No newline at end of file
diff --git a/dkist/io/asdf/entry_points.py b/dkist/io/asdf/entry_points.py
index e5f90c6..5b15102 100644
--- a/dkist/io/asdf/entry_points.py
+++ b/dkist/io/asdf/entry_points.py
@@ -6,7 +6,7 @@ import importlib.resources as importlib_resources
 from asdf.extension import ManifestExtension
 from asdf.resource import DirectoryResourceMapping
 
-from dkist.io.asdf.converters import (AsymmetricMappingConverter, CoupledCompoundConverter,
+from dkist.io.asdf.converters import (NewSchema, AsymmetricMappingConverter, CoupledCompoundConverter,
                                       DatasetConverter, FileManagerConverter, RavelConverter,
                                       TiledDatasetConverter, VaryingCelestialConverter)
 
@@ -36,9 +36,10 @@ def get_extensions():
     """
     Get the list of extensions.
     """
-    dkist_converters = [FileManagerConverter(), DatasetConverter(), TiledDatasetConverter()]
+    dkist_converters = [FileManagerConverter(), DatasetConverter(), TiledDatasetConverter(), NewSchema()]
     wcs_converters = [VaryingCelestialConverter(), CoupledCompoundConverter(), RavelConverter(), AsymmetricMappingConverter()]
     return [
+        ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.6.0", converters=dkist_converters),
         ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.5.0",
                                    converters=dkist_converters),
         ManifestExtension.from_uri("asdf://dkist.nso.edu/manifests/dkist-1.4.0",

In this case any necessary new files are created as templates.

$ cat dkist/io/asdf/converters/new_schema.py 
from asdf.extension import Converter


class NewSchemaConverter(Converter):
    tags = [
        "asdf://dkist.nso.edu/tags/new_schema-0.1.0"
        "tag:dkist.nso.edu:dkist/new_schema-0.1.0"
    ]

    def from_yaml_tree(cls, node, tag, ctx):
        # Construct and return an object of the appropriate type from the asdf yaml

        return

    def to_yaml_tree(cls, node, tag, ctx):
        # Construct and return yaml tree to represent the object being serialised
        tree = {}

        return tree
$ cat dkist/io/asdf/resources/schemas/new_schema-0.1.0.yaml
%YAML 1.1
---
$schema: "http://stsci.edu/schemas/yaml-schema/draft-01"
id: "asdf://dkist.nso.edu/schemas/new_schema-0.1.0"

title: |
  # Title for this object
description:
  # Description of this object

type: object
properties:
  # Properties of this object

required: [] # Properties required byt his object
additionalProperties: # [true|false]

TODO:

  • Get the latest version number from the git history instead of current files, so that you can increment multiple schemas without having to make a new manifest for each one.
  • Automate committing the changes, which will presumably run the pre-commit stuff and fix import orderings, linting etc. Decided against this.

@SolarDrew SolarDrew marked this pull request as ready for review September 30, 2025 13:52
@SolarDrew SolarDrew requested a review from Cadair October 15, 2025 08:56
@Cadair Cadair merged commit 052c819 into DKISTDC:main Oct 16, 2025
28 checks passed
@SolarDrew SolarDrew mentioned this pull request Jan 29, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants