Skip to content

[BUG] Rollback of controller fails when older binary lacks DOWN files for newer schema versions #1881

New issue
New issue
Closed
Bug
#1963
Closed
[BUG] Rollback of controller fails when older binary lacks DOWN files for newer schema versions#1881
Bug
#1963
Assignees
onematchfox

Description

@onematchfox
onematchfox
opened on May 18, 2026

📋 Prerequisites

  • I have searched the existing issues to avoid creating a duplicate
  • By submitting this issue, you agree to follow our Code of Conduct
  • I am using the latest version of the software
  • I have tried to clear cache/cookies or used incognito mode (if ui-related)
  • I can consistently reproduce this issue

🎯 Affected Service(s)

Controller Service

🚦 Impact/Severity

Blocker

🐛 Bug Description

When the kagent controller is downgraded to a previous version, the older controller fails to start if the newer version applied schema migrations that are unknown to the older version. This was observed when attempting to rollback after a failed upgrade from 0.9.0 to 0.9.3´. The .down.sql files for those migrations are embedded only in the newer container. The older container can neither roll the schema back nor start against the newer schema, resulting in a crash loop.

🔄 Steps To Reproduce

  1. Deploy application v0.9.0 — embeds migrations 000001–000003. Schema reaches version 3.
  2. Upgrade to v0.9.3 — embeds migrations 000001–000004. Schema reaches version 4.
  3. v0.9.3 is found to be broken; downgrade back to v0.9.0.
  4. v0.9.0 starts against a DB at schema version 4. Its embedded source only knows versions 1–3.

🤔 Expected Behavior

Rolling back to a previous version is a standard recovery operation (e.g. when a new release introduces a regression) and should work reliably.

📱 Actual Behavior

No response

💻 Environment

No response

🔧 CLI Bug Report

No response

🔍 Additional Context

No response

📋 Logs



📷 Screenshots


No response


🙋 Are you willing to contribute?


    

  •  I am willing to submit a PR to fix this issue
    • 

Activity
  1. onematchfox
    added the
    Bug
    issue type on May 18, 2026
  2. github-project-automation
    added this to  kagent kanbanon May 18, 2026
  3. github-project-automation
    moved this to Backlog in  kagent kanbanon May 18, 2026
  4. iplay88keys commented on May 18, 2026 
    @iplay88keys
    iplay88keys
    on May 18, 2026
    Contributor
    @onematchfox Thanks for raising this issue. Can you provide more context on the failed upgrade aspect? On each crash loop, the Kagent controller applies all down migrations before crashing, but without scaling to 0 before downgrading, the migrations would just run again.
    

    I think we will need to have a good solution in these cases, but in the meantime, you should be able to manually apply the down migrations with golang-migrate directly:
    

      

    1. Install golang-migrate
      2. 

    2. Stop the Kagent controller:
      3. 

    

    kubectl -n kagent scale deploy/kagent-controller --replicas=0
    

      

    1. Roll back the core track:
      2. 

    

    migrate \
  -source "github://kagent-dev/kagent/go/core/pkg/migrations/core#v0.9.3" \
  -database "postgres://kagent:[email protected]:5432/kagent?sslmode=require&x-migrations-table=schema_migrations" \
  goto 3
  5. onematchfox commented on May 19, 2026 
    @onematchfox
    onematchfox
    on May 19, 2026
    ContributorAuthor
    

    Can you provide more context on the failed upgrade aspect?
    

    

    Sorry, this is misleading. The upgrade didn't fail as such... I upgraded to v0.9.3 successfully to test the new version but then decided to rollback to v0.9.0 and was unable to do so.
    

    Thanks for the tip on the workaround. Will bear that in mind if this comes up again before we find an automated way of achieving this. On that, do you have any thoughts on how we might do this? It doesn't seem like golang-migrate has a good solution to this. The only thing I could really come up with was to store the down scripts in a database table when applying the migrations so that they could be retrieved and used by any version.
  6. iplay88keys commented on May 19, 2026 
    @iplay88keys
    iplay88keys
    on May 19, 2026
    Contributor
    

    Sorry, this is misleading. The upgrade didn't fail as such... I upgraded to v0.9.3 successfully to test the new version but then decided to rollback to v0.9.0 and was unable to do so.
    

    

    Ah, ok. I was worried that we had broken something on the upgrade.
    

    

    On that, do you have any thoughts on how we might do this? It doesn't seem like golang-migrate has a good solution to this. The only thing I could really come up with was to store the down scripts in a database table when applying the migrations so that they could be retrieved and used by any version.
    

    

    That is one approach I've been considering. I was also thinking that the kagent cli could have the migrations embedded so that you can run kagent migrate down or something similar. That would of course require the newer version of the cli to be used, but could be an option.
  7. onematchfox commented on Jun 3, 2026 
    @onematchfox
    onematchfox
    on Jun 3, 2026
    ContributorAuthor
    

    I was also thinking that the kagent cli could have the migrations embedded so that you can run kagent migrate down or something similar. That would of course require the newer version of the cli to be used, but could be an option.
    

    

    Yeah, this doesn't really work well in a GitOps environment or if someone in the cluster decided to kubectl rollout undo 😞
    

    Another alternative would be to adopt the expand/contract pattern and thereby lift the constraint that old versions cannot run against newer schemas. But, this does require engineering discipline as breaking schema changes need to be staged across multiple versions as well as some form of support guarantee by the maintainers along the lines of "rollback is supported across N minor versions" for example.
  8. onematchfox commented on Jun 4, 2026 
    @onematchfox
    onematchfox
    on Jun 4, 2026
    ContributorAuthor
    

    Another alternative would be to adopt the expand/contract pattern
    

    

    I see this is already documented as having been adopted in https://github.com/kagent-dev/kagent/blob/main/.claude/skills/kagent-dev/references/database-migrations.md#backward-compatible-schema-changes so maybe we can just not return an error if we're running an older binary against a newer schema? Will put a PR together for this in the meantime.
  9. iplay88keys commented on Jun 5, 2026 
    @iplay88keys
    iplay88keys
    on Jun 5, 2026
    Contributor
    I think this approach would work as an incremental improvement step. We have some ideas for how to make the migration/rollback process more robust, but that'll be a rewrite of much of the migrator logic.
  10. iplay88keys
    added a commit that references this issue on Jun 17, 2026
    fix(controller): skip DB migration when DB schema is ahead of binary (#…
    fed2086
  11. iplay88keys
    linked a pull request that will close this issuefix(controller): skip DB migration when DB schema is ahead of binary #1963on Jun 17, 2026
  12. iplay88keys
    assigned 
    onematchfox
    on Jun 17, 2026
  13. iplay88keys
    closed this as completedon Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
Labels
No labels
No labels
Type
Bug
Projects
Milestone
No milestone
Relationships
None yet
Development
No branches or pull requests
Issue actions