Teams building chains on Substrate are struggling with migrating their nodes and runtimes. We want to improve the migration experience to reduce necessary effort and time spent as well as reduce the amount of effort it takes to migrate Substrate based chains.

💔 Status Quo

Currently Parity core developers write migrations in Substrate for Kusama and Polkadot. They are then removed within one to a few weeks of being introduced (in order to keep the codebase clean). Chain developers building on Substrate either a) apply these migrations the same way in a timely fashion or b) need to fork Substrate to modify and adjust the migrations or reintroduce them if they have been removed in the meantime.

Substrate users need to go into Substrate's PR listing and filter for runtime migrations in order to find the migrations.

🛑 Current Problems

• Users are confused about migrations:
  • where to find them (different locations, formatting, buried in history)
  • when to migrate (on new Substrate release? at a fixed interval?)
  • "Will this upgrade work for my chain?"
• Users need to do lots of manual work for migrations.
  • Especially when deciding to upgrade later than Kusama the manual work compounds with the confusion.
• Users have trouble testing migrations.
• Users cannot roll back migrations.

👓 Vision

We want Substrate users to be able to migrate their chains correctly as easily as possible.

As a result, we want migrations to be:

• Correct: Migrations should leave the chain in a correct state after the migration.
  • Testable: It should be possible to determine whether they run correctly.
  • Composable: Running several migrations one after another should not corrupt state.
• Discoverable: Migrations should be discoverable when users want to apply them or use them as inspiration for their own migration.
• Documented: Migrations should list assumptions so that users can verify whether they will work for their chain.
• Reusable: Migrations (especially simple ones) should be reusable so they can be applied to many chains and don't need to be rewritten.
  • Easy and Automatic: Simple migrations should just work™ without a lot of manual integration.

❎ Non-Goals

Complicated Migrations ≠ Easy and Automatic

We cannot make complicated migrations or migrations affecting many parts of the runtime effortless. There will always be manual work necessary for these cases. We can only provide guidance.

Maintaining Old Migrations

We will not maintain old migrations. Migrations target the transition from one version to another and will thus be outdated once applied. We can only keep documentation and code archives around for (very) old migrations.

✅ Proposed Solutions

• Pallet Storage Version Unified Pallet Storage Version substrate#6888
  • The storage version of a pallet should be stored in a standardized format.
• Testing Tool
  • There should be a tool that allows running end-to-end tests of runtime migrations to determine whether they run correctly.
  • @seunlanlege is working on a test runner addressing this here.
• Migrations in Release Notes add runtime migrations to release notes/changelog substrate#6875
  • We should mention migrations in changelogs and release notes so that people are aware of them and can integrate them.
• Migration Guide (draft doc for collaboration)
  • We could have more documentation on how to migrate runtimes successfully