RFC: Alembic Adoption for Database Migrations in ADK-Python

@achilleasatha

Status: Draft

Date: 2026-02-05

Related Issue: #3343 - Update database schema automatically

1. Summary

Adopt Alembic as the database migration framework for adk-python to enable automatic schema updates, migration-coordinated deployments with minimal service disruption, and proper rollback support for enterprise production environments.

2. Motivation

2.1 Current Problems

From GitHub issue #3343 and community feedback:

Manual migrations are not viable for production: Users must manually execute migration scripts when upgrading ADK versions. For enterprise deployments with multiple environments and multiple ADK-based agents per environment, translates to a lot of manual work, SSHing into servers and running scripts.
Breaking changes in minor versions: Schema changes in ADK 1.14.0, 1.17.0, and 1.19.0 broke deployments without clear migration paths or automation.
No support for Kubernetes deployments: Cannot run migrations via Helm hooks or init containers. SSH access to every pod is not scalable or feasible.
Race conditions: Multiple pods starting simultaneously could attempt migrations concurrently, causing failures. The current migration script provided does not offer a locking mechanism to protect from race conditions.
No rollback capability: Current system doesn't support downgrade migrations, making failed deployments difficult to recover.
Permission constraints: Service accounts may lack DDL execution permissions, requiring manual intervention.

3. Proposal

3.1 Overview

Integrate Alembic as the primary database migration framework with:

Automatic migrations on startup (feature-flagged with ADK_AUTO_MIGRATE_DB)
Distributed locking to prevent race conditions across multiple pods
Kubernetes Helm hook support for pre-deployment migrations
Rollback capability via Alembic's downgrade functionality
Comprehensive testing across PostgreSQL, MySQL, and SQLite (and/or any other DBs that ADK-python wants to support)

Responsibility model: ADK owns schema definition and migration logic; operators retain full control over when and how migrations are executed (via environment variables or Helm hooks).

3.2 Key Features

For Developers:

Generate migrations: adk migrate generate --message "add_column"
Alembic autogenerate compares SQLAlchemy models to database
Clear upgrade/downgrade paths with version tracking

For Operations:

Helm pre-install/pre-upgrade hooks run migrations before app deployment
Database locking (eg. PostgreSQL advisory locks)

For Enterprise:

Migration-coordinated deployments with proper synchronization
Audit logging of all migration events
Support for Cloud SQL Proxy, Workload Identity, and secret management
Comprehensive documentation with Helm chart examples

4. Design Principles

Safe by default: Auto-migration disabled by default (ADK_AUTO_MIGRATE_DB=false) to prevent unexpected schema changes
Backward compatible: Phased rollout over releases with parallel support (e.g., deprecation warnings on minor/patches, destructive operations only on major releases)
Well-tested: Integration tests with real databases (PostgreSQL, MySQL, SQLite)
Production-ready: Distributed locking, timeout handling, error recovery
Well-documented: Migration guides, Helm examples, troubleshooting docs

5. Technical Approach

5.1 Architecture

Application Startup
        ↓
DatabaseSessionService._prepare_tables()
        ↓
Check: ADK_AUTO_MIGRATE_DB?
        ↓ (true)
AlembicRunner.check_needs_migration()
        ↓ (yes)
Acquire DatabaseMigrationLock
        ↓
Run Alembic upgrade to "head"
        ↓
Alembic updates alembic_version table
        ↓
Migration script updates adk_internal_metadata.schema_version
        ↓
Release lock

5.2 Distributed Locking

Version Tracking: Alembic uses alembic_version table to track migration revisions. ADK additionally maintains adk_internal_metadata.schema_version as a higher-level compatibility layer for existing tooling and schema detection logic.

PostgreSQL: Advisory locks (non-blocking, session-scoped)

SELECT pg_try_advisory_lock(1234567890);

MySQL/SQLite: Table-based locks with expiration

Lock expires after 5 minutes (default, configurable via ADK_MIGRATION_TIMEOUT)
Automatic cleanup of stale locks
Note: Best-effort locking to prevent accidental concurrent migrations. Not a strict consensus mechanism. Clock skew or stale locks may affect correctness in edge cases.

SQLite: Migrations assume single-writer deployments (e.g., local development)

Behavior:

Instance A acquires lock → runs migration
Instances B-E wait → poll for completion every 2 seconds
All instances verify schema after migration completes

When is locking used?

Deployment Mode	Locking Needed?	Reason
Helm hook (K8s)	❌ No	Single Job pod runs migration before app pods start - sequential by design
Auto-migration (ADK_AUTO_MIGRATE_DB=true)	✅ Yes	Multiple app instances may start simultaneously and call `_prepare_tables()`

Use cases requiring locks:

Cloud Run: Multiple instances scaling up simultaneously with shared Cloud SQL
Docker Compose: Multiple service replicas sharing a database
K8s without Helm hooks: Deploying with ADK_AUTO_MIGRATE_DB=true on multiple pods
Local development: Running multiple instances for testing

5.3 Kubernetes Integration

Migration Job (Helm pre-install/pre-upgrade hook):

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  template:
    spec:
      containers:
      - name: migration
        command: ["python", "-m", "google.adk.cli.migration_entrypoint"]
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: adk-db-secret
              key: url

Application Deployment:

env:
- name: ADK_AUTO_MIGRATE_DB
  value: "false"  # Disabled when using migration Job

6. Migration Workflow

6.1 For Developers

Creating a new migration:

# 1. Generate migration template for session database schema
adk migrate generate --message "add_session_tags"

# 2. Review auto-generated migration file
# alembic/versions/004_add_session_tags_v2.py

# 3. Customize if needed (data transformations, etc.)

# 4. Test migration
pytest tests/unittests/sessions/migration/

# 5. Commit migration file
git add alembic/versions/004_add_session_tags_v2.py
git commit -m "feat(migration): add session tags column"

Applying migration:

# Local development (applies to session database schema)
adk migrate session

# Production (via Helm)
helm upgrade my-app ./chart --set image.tag=v1.26.0
# Migration runs automatically via pre-upgrade hook

6.2 For Operations

Migration happens automatically in the library - users don't write migration code or operate migration scripts. There are two deployment modes:

Mode 1: Helm Pre-Upgrade Hook (Recommended for Production/Kubernetes)

# Migration runs as separate Job BEFORE app pods start
# See section 5.3 for full example

# In application deployment, disable auto-migration
env:
  - name: ADK_AUTO_MIGRATE_DB
    value: "false"

Mode 2: Auto-Migration on Startup (Local dev, testing, simple deployments)

# Just set environment variable - no code needed
export ADK_AUTO_MIGRATE_DB=true

# Migration runs automatically when DatabaseSessionService initializes
# This happens in _prepare_tables() method inside the library

Monitoring:

Track migration duration (e.g., Prometheus histogram)
Alert on migration failures
Monitor schema version lag across environments

Rollback (automated via Helm pre-rollback hook):

⚠️ Warning: Downgrades are best-effort only. Not all schema or data migrations are reversible (e.g., dropping columns, destructive transformations). Operators must verify downgrade safety before relying on automated rollback in production.

# templates/migration-rollback-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-migration-rollback
  annotations:
    "helm.sh/hook": pre-rollback
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation
spec:
  template:
    spec:
      containers:
      - name: migration-rollback
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        command:
          - python
          - -m
          - google.adk.cli.migration_entrypoint
          - --db-url
          - $(DATABASE_URL)
          - --downgrade
          - "1"  # Number of migrations to rollback (should auto-calculate from app version)
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: {{ .Values.database.secretName }}
              key: url

Workflow:

# Deploy v1.26.0 (migration runs automatically)
helm upgrade my-app ./chart --version 1.26.0

# Rollback to v1.25.0 (DB downgrade runs automatically)
helm rollback my-app
# ↑ Helm pre-rollback hook downgrades DB before rolling back app

7. Testing Strategy

7.1 Test Coverage

Unit Tests (tests/unittests/sessions/migration/):

AlembicMigrationRunner functionality
DatabaseMigrationLock (PostgreSQL + table-based)
Individual migration scripts (upgrade/downgrade)
Schema version detection and synchronization

Integration Tests (tests/integration/sessions/):

Full migration paths (V0 → V1 → V2)
Auto-migration on startup
Concurrent migrations
Rollback scenarios
Data preservation through migrations

GitHub Actions Workflow (.github/workflows/test-migrations.yml):

strategy:
  matrix:
    db: [postgres:15, postgres:14, mysql:8.0]
    python: ["3.10", "3.11", "3.12"]

services:
  postgres:
    image: ${{ matrix.db }}
    env:
      POSTGRES_PASSWORD: testpass
    ports:
      - 5432:5432

8. Backward Compatibility

8.1 Phased Rollout

ADK 1.25.0 (Parallel Support):

Add Alembic support
Keep existing manual migration scripts
Document both approaches
CLI supports both: adk migrate session (Alembic) and adk migrate session --legacy (manual)

ADK 1.26.0 (Soft Deprecation):

Alembic becomes default
Deprecation warnings for manual migrations

ADK 1.27.0 (Hard Deprecation):

Remove manual migrations from main codebase
CLI only supports Alembic

8.2 Migration Path for Existing Users

V1 databases (without Alembic):

# Bootstrap Alembic - stamps baseline revision without running migrations
adk migrate session --bootstrap-alembic

This stamps the database with 002_baseline_v1 revision, indicating it's already at V1 schema.

V0 databases (legacy pickle-based):

# Migrate V0 → V1 via Alembic
adk migrate session --upgrade

This runs the 003_v0_to_v1_migration.py script to convert pickle to JSON.

Automatic detection:
When ADK_AUTO_MIGRATE_DB=true, the system automatically detects schema version and bootstraps Alembic if needed (stamps appropriate baseline revision).

9. Documentation Plan

9.1 New Documentation

User Guides:

docs/migration_guide.md - Developer workflow for creating migrations
docs/helm_migration_guide.md - Kubernetes deployment patterns

Helm Chart Examples:

Simple PostgreSQL migration job
Cloud SQL with Workload Identity
Migration with health check init containers

Migration Template:
Enhanced alembic/script.py.mako with ADK-specific fields:

Database schema version
Compatible ADK versions
Description and data migration notes
Rollback considerations

9.2 Updated Documentation

src/google/adk/sessions/migration/README.md - Add Alembic workflow
Release notes for each version with migration instructions

10. References

GitHub Issue: #3343 - Update database schema automatically
Alembic Documentation: https://alembic.sqlalchemy.org/
LiteLLM Prisma Example: https://docs.litellm.ai/docs/proxy/prod#9-use-prisma-migrate-deploy
Helm Hooks: https://helm.sh/docs/topics/charts_hooks/
PostgreSQL Advisory Locks: https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS

@DeanChensj

@DeanChensj @wuliang229 - first draft RFC relating to #3343

I have started working on some of the implementation. This is a fair bit of work that might benefit from being split across a few PRs. I will let you digest and come back with thoughts.

A few things that are orthogonal:

Locking mechanism - this can be adopted regardless of what we use to manage migrations. It doesn't need to be part of this RFC but is probably a good feature to add. A native implementation of writing to an internal table to acquire the lock might be preferable as it will be agnostic.
We could add the SQLAlchemy ORM and alembic code foundation without doing the library integration, this will make it operable with the CLI and k8s via helm hooks and we can do the library integration in a follow up PR

Early steering would be very helpful. Support especially at later stages if you want to do more extensive integration testing across a matrix of PG, MySQL and other DB versions would be welcome. Similarly for documentation.

I think a fundamental problem here is that this is a library. Most people would not expect a library to own and maintain a DB schema, libraries mutating infra is a bit unconventional. I am not sure that a separate adk-migrator would serve the community better. ADK works with InMemory fine and persistence is optional (but practically a requirement for any serious enterprise application). Those can be managed separately. I am not arguing in favour of that they should. I think it creates a whole host of other problems re syncing versions, maintaining the separate packages and ensuring the data models stay aligned across etc.

But since we are expecting the library to define and manage the schema - it should be capable to do this fully on its own - which is what this RFC aims to address.

Raised draft PR #4408

@achilleasatha

@achilleasatha Thanks for the RFC, when you said "We could add the SQLAlchemy ORM and alembic code foundation without doing the library integration", do you mean Alembic library? I feel like Alembic is not a strict requirement here, we can have something similar and provide the migration functionality, if the future migrations are simple changes, which is what we aim to do.

@achilleasatha Thanks for the RFC, when you said "We could add the SQLAlchemy ORM and alembic code foundation without doing the library integration", do you mean Alembic library? I feel like Alembic is not a strict requirement here, we can have something similar and provide the migration functionality, if the future migrations are simple changes, which is what we aim to do.

Not quite, sorry that wasn't very clear. I was thinking we could have alembic manage the migrations via scripts & helm hooks and not be part of the app src. So when you run import adk at runtime it doesn't auto-migrate (if enabled). Essentially the changes in the database_session_service

That could have been implemented in a 2nd pass. It wasn't that much work so I ended up doing both in 1 pass. It is a fair bit of chance so if you prefer to roll that back or split it in 2 PRs I am happy to do so.

I do feel very strongly about alembic being part of this. If not alembic pick any other tool and stick with it (but alembic is truly the go-to here in the python ecosystem, it's battle-tested, robust, has a big community).

As I commented in my draft PR it's unconventional that a library manages and mutates things like a DB schema. If you are making the conscious architectural decision here that this is how you want ADK to operate - it needs to be able to self-manage this functionality FULLY.

if the future migrations are simple changes, which is what we aim to do

The reality of the matter is you do not know this. The changes outlined in this RFC and my PR give you tools to manage this going forward regardless of what happens. I agree with you that the intent is to always make small (additive) only changes. But things move fast, big refactors and re-architectures are sometimes required. New feature requests and requirements come in.

Isn't this the exact reason after all why we are having this conversation?

There needs to be a robust process and tooling in place to deal with change.

Change is the only constant. Embrace change, do not fight it 🙏

For context, the OpenAI agents python sdk suffers from the same problem. https://openai.github.io/openai-agents-python/ref/extensions/memory/sqlalchemy_session/

I feel like they are getting away with it because they have a much simpler schema (2 tables acting as JSON blobs).

I think there's an implicit assumption there that because it's so simple if they need to make breaking changes users will just recreate it.

I think it's only a matter of time until they have to follow suit.

@wuliang229

Hi @wuliang229 have we made any progress on this and the associated PR?

Even a small status update letting us know that this is under consideration is helpful

@achilleasatha

@achilleasatha The team is reviewing this proposal internally. This aligns with what the team wants and we should get back with an update soon.

@wuliang229

@wuliang229 just wanted to follow up on this. we’re quite interested in this as well since it aligns closely with our use cases and would help standardize migrations on our end.

Are there any updates or timelines you can share on the internal review? Happy to help with feedback, testing, or contributions if that would help move things forward.

Thanks!

Is there any progress on this? I am shocked that it's almost June 2026 and we're still asking for a way to reliably migrate the database in production systems for a framework like this. The number of errors that we come across because we discovered that once again the db has changed and we need to go run manual SQL commands to migrate our databases is both long and old. I have been in issues dating back to mid/late 2025 with this exact issue. It makes me scared to update ADK and we often delay because we're worried about more db changes that will come up.

There is a one off command already to migrate sessions from 1.22 I believe, why didn't we double down then and make an actual migration system?

Is there any progress on this? I am shocked that it's almost June 2026 and we're still asking for a way to reliably migrate the database in production systems for a framework like this. The number of errors that we come across because we discovered that once again the db has changed and we need to go run manual SQL commands to migrate our databases is both long and old. I have been in issues dating back to mid/late 2025 with this exact issue. It makes me scared to update ADK and we often delay because we're worried about more db changes that will come up.

There is a one off command already to migrate sessions from 1.22 I believe, why didn't we double down then and make an actual migration system?

No traction whatsoever unfortunately... I practically did the work, raised a PR with a proposed implementation that would address this and 🦗🦗🦗

@DeanChensj

Hi @DeanChensj and @wuliang229.

I'm hoping that tagging you will bring your attention to this critical RFC. I have pulled your usernames from the following existing issues that have come up in the past where you both have appeared to be code owners:

Upgraded from 1.17 to 1.19 and getting a SQL error when I try to open a session column events.input_transcription does not exist #3681
Update database schema automatically #3343

This has been a longstanding issue with critical implications for production deployments of ADK. In these issues there is talk of a migration command, but it doesn't seem like there seems to be one other than a one off command for migrating sessions in the past. In issue #3343 there was a request for an RFC and draft PR which appears to be presented here by @achilleasatha but remains unaddressed for a few months.

Is there a reason that database migrations are not being accepted or prioritized for this repo? This is a big deal which on more than one occasion has broken my production environment and I'm positive I'm not alone. In fact, the reason this is on my radar again is because we missed another db change that threw errors in production just recently.

What do we need to do to move this forward so we can ship updates reliably without fear of database changes breaking our production environments?

Thanks for your help.

Bumping again here - still a massive pain to date.

Navigation Menu

Repository navigation

RFC: Alembic Adoption for Database Migrations in ADK-Python #4387

1. Summary

2. Motivation

2.1 Current Problems

3. Proposal

3.1 Overview

3.2 Key Features

4. Design Principles

5. Technical Approach

5.1 Architecture

5.2 Distributed Locking

5.3 Kubernetes Integration

6. Migration Workflow

6.1 For Developers

6.2 For Operations

7. Testing Strategy

7.1 Test Coverage

8. Backward Compatibility

8.1 Phased Rollout

8.2 Migration Path for Existing Users

9. Documentation Plan

9.1 New Documentation

9.2 Updated Documentation

10. References

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC: Alembic Adoption for Database Migrations in ADK-Python #4387

Description

1. Summary

2. Motivation

2.1 Current Problems

3. Proposal

3.1 Overview

3.2 Key Features

4. Design Principles

5. Technical Approach

5.1 Architecture

5.2 Distributed Locking

5.3 Kubernetes Integration

6. Migration Workflow

6.1 For Developers

6.2 For Operations

7. Testing Strategy

7.1 Test Coverage

8. Backward Compatibility

8.1 Phased Rollout

8.2 Migration Path for Existing Users

9. Documentation Plan

9.1 New Documentation

9.2 Updated Documentation

10. References

Activity

achilleasatha commented on Feb 5, 2026

achilleasatha commented on Feb 7, 2026

wuliang229 commented on Feb 9, 2026

achilleasatha commented on Feb 9, 2026

achilleasatha commented on Feb 9, 2026

achilleasatha commented on Feb 17, 2026

wuliang229 commented on Feb 20, 2026

zepto-tanishqp commented on Mar 23, 2026

ryanovas commented on May 27, 2026

achilleasatha commented on May 27, 2026

ryanovas commented on May 28, 2026

achilleasatha commented on Jun 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions