Skip to content

feat: add nulls option to createIndex#1489

Merged
Shinigami92 merged 9 commits into
salsita:mainfrom
votingworks:create-index-nulls-clause
Aug 13, 2025
Merged

feat: add nulls option to createIndex#1489
Shinigami92 merged 9 commits into
salsita:mainfrom
votingworks:create-index-nulls-clause

Conversation

@jonahkagan

@jonahkagan jonahkagan commented Aug 4, 2025

Copy link
Copy Markdown
Contributor

Closes #1470

Adds an optional nulls option to the createIndex function with two possible values: 'distinct' and 'not distinct'. If provided, adds a NULLS DISTINCT or NULLS NOT DISTINCT clause to the index creation command.

Note that in the original issue I quoted from some docs saying that there were two other nulls clause possibilities: NULLS FIRST and NULLS LAST, but those actually are a separate option that is part of the sort order specifier, so I didn't include them (since there's no option for sort order in createIndex yet).

Adds an optional `nulls` option to the `createIndex` function with two possible values: `'distinct'` and `'not
distinct'`. If provided, adds a `NULLS DISTINCT` or `NULLS NOT DISTINCT` clause to the index creation command.
@jonahkagan
jonahkagan requested a review from Shinigami92 as a code owner August 4, 2025 22:11
@Shinigami92 Shinigami92 added the c: feature Request for new feature label Aug 11, 2025
@github-actions

github-actions Bot commented Aug 11, 2025

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🟢 Lines 92.54% (🎯 90%)
⬆️ +0.01%
3279 / 3543
🟢 Statements 92.54% (🎯 90%)
⬆️ +0.01%
3279 / 3543
🟢 Functions 96.66% (🎯 90%)
🟰 ±0%
261 / 270
🟢 Branches 91.23% (🎯 85%)
⬆️ +0.05%
874 / 958
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/operations/indexes/createIndex.ts 100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
100%
🟰 ±0%
Unchanged Files
src/db.ts 83.18% 88.88% 87.5% 83.18% 90-92, 122, 125-137, 177-179
src/index.ts 100% 100% 100% 100%
src/logger.ts 100% 100% 100% 100%
src/migration.ts 76.07% 86.76% 68.75% 76.07% 149-152, 173-189, 231-237, 296-337, 402-403, 429-431, 439-440, 457-460, 489-490
src/migrationBuilder.ts 96.19% 93.33% 100% 96.19% 833-837, 976-981
src/migrationOptions.ts 100% 100% 100% 100%
src/pgType.ts 100% 100% 100% 100%
src/runner.ts 80.33% 67.79% 100% 80.33% 189, 208-217, 229-230, 282-283, 325-328, 337-341, 354, 366-369, 392, 403, 433-446, 449-452, 472-474, 483, 485-494
src/sqlMigration.ts 85.36% 100% 80% 85.36% 54-60
src/operations/sql.ts 100% 100% 100% 100%
src/operations/casts/createCast.ts 100% 100% 100% 100%
src/operations/casts/dropCast.ts 100% 100% 100% 100%
src/operations/casts/index.ts 100% 100% 100% 100%
src/operations/domains/alterDomain.ts 100% 100% 100% 100%
src/operations/domains/createDomain.ts 100% 100% 100% 100%
src/operations/domains/dropDomain.ts 100% 100% 100% 100%
src/operations/domains/index.ts 100% 100% 100% 100%
src/operations/domains/renameDomain.ts 100% 100% 100% 100%
src/operations/domains/shared.ts 100% 100% 100% 100%
src/operations/extensions/createExtension.ts 100% 100% 100% 100%
src/operations/extensions/dropExtension.ts 100% 100% 100% 100%
src/operations/extensions/index.ts 100% 100% 100% 100%
src/operations/extensions/shared.ts 100% 100% 100% 100%
src/operations/functions/createFunction.ts 95.52% 94.44% 100% 95.52% 71-73
src/operations/functions/dropFunction.ts 100% 100% 100% 100%
src/operations/functions/index.ts 100% 100% 100% 100%
src/operations/functions/renameFunction.ts 100% 100% 100% 100%
src/operations/functions/shared.ts 100% 100% 100% 100%
src/operations/grants/grantOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/grantOnTables.ts 100% 100% 100% 100%
src/operations/grants/grantRoles.ts 100% 100% 100% 100%
src/operations/grants/index.ts 100% 100% 100% 100%
src/operations/grants/revokeOnSchemas.ts 100% 100% 100% 100%
src/operations/grants/revokeOnTables.ts 100% 100% 100% 100%
src/operations/grants/revokeRoles.ts 100% 100% 100% 100%
src/operations/grants/shared.ts 95.45% 85.71% 100% 95.45% 71
src/operations/indexes/dropIndex.ts 100% 100% 100% 100%
src/operations/indexes/index.ts 100% 100% 100% 100%
src/operations/indexes/shared.ts 88% 86.95% 100% 88% 23, 33-36, 48
src/operations/materializedViews/alterMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/createMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/dropMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/index.ts 100% 100% 100% 100%
src/operations/materializedViews/refreshMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedView.ts 100% 100% 100% 100%
src/operations/materializedViews/renameMaterializedViewColumn.ts 100% 100% 100% 100%
src/operations/materializedViews/shared.ts 100% 87.5% 100% 100%
src/operations/operators/addToOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/createOperator.ts 100% 100% 100% 100%
src/operations/operators/createOperatorClass.ts 100% 83.33% 100% 100%
src/operations/operators/createOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/dropOperator.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/dropOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/index.ts 100% 100% 100% 100%
src/operations/operators/removeFromOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorClass.ts 100% 100% 100% 100%
src/operations/operators/renameOperatorFamily.ts 100% 100% 100% 100%
src/operations/operators/shared.ts 85% 75% 100% 85% 24-25, 38
src/operations/policies/alterPolicy.ts 100% 100% 100% 100%
src/operations/policies/createPolicy.ts 100% 100% 100% 100%
src/operations/policies/dropPolicy.ts 100% 100% 100% 100%
src/operations/policies/index.ts 100% 100% 100% 100%
src/operations/policies/renamePolicy.ts 100% 100% 100% 100%
src/operations/policies/shared.ts 100% 100% 100% 100%
src/operations/roles/alterRole.ts 100% 100% 100% 100%
src/operations/roles/createRole.ts 100% 75% 100% 100%
src/operations/roles/dropRole.ts 100% 100% 100% 100%
src/operations/roles/index.ts 100% 100% 100% 100%
src/operations/roles/renameRole.ts 100% 100% 100% 100%
src/operations/roles/shared.ts 98.07% 76.92% 100% 98.07% 78
src/operations/schemas/createSchema.ts 100% 100% 100% 100%
src/operations/schemas/dropSchema.ts 100% 100% 100% 100%
src/operations/schemas/index.ts 100% 100% 100% 100%
src/operations/schemas/renameSchema.ts 100% 100% 100% 100%
src/operations/sequences/alterSequence.ts 94.73% 80% 100% 94.73% 23
src/operations/sequences/createSequence.ts 100% 100% 100% 100%
src/operations/sequences/dropSequence.ts 100% 100% 100% 100%
src/operations/sequences/index.ts 100% 100% 100% 100%
src/operations/sequences/renameSequence.ts 100% 100% 100% 100%
src/operations/sequences/shared.ts 78.57% 68.75% 100% 78.57% 41, 43-44, 49-50, 63-64, 69-70
src/operations/tables/addColumns.ts 100% 80% 100% 100%
src/operations/tables/addConstraint.ts 100% 100% 100% 100%
src/operations/tables/alterColumn.ts 89.33% 76.47% 100% 89.33% 69, 76-83
src/operations/tables/alterTable.ts 100% 100% 100% 100%
src/operations/tables/createTable.ts 89.39% 77.27% 100% 89.39% 51-55, 92-93
src/operations/tables/dropColumns.ts 100% 100% 100% 100%
src/operations/tables/dropConstraint.ts 100% 100% 100% 100%
src/operations/tables/dropTable.ts 100% 100% 100% 100%
src/operations/tables/index.ts 100% 100% 100% 100%
src/operations/tables/renameColumn.ts 100% 100% 100% 100%
src/operations/tables/renameConstraint.ts 100% 100% 100% 100%
src/operations/tables/renameTable.ts 100% 100% 100% 100%
src/operations/tables/shared.ts 83.15% 81.81% 80% 83.15% 157-158, 211-215, 241-242, 261-262, 287-288, 291-298, 301-302, 439-464
src/operations/triggers/createTrigger.ts 86.25% 68.18% 100% 86.25% 53-54, 66-67, 70-71, 74-77, 113
src/operations/triggers/dropTrigger.ts 100% 100% 100% 100%
src/operations/triggers/index.ts 100% 100% 100% 100%
src/operations/triggers/renameTrigger.ts 100% 100% 100% 100%
src/operations/triggers/shared.ts 100% 100% 100% 100%
src/operations/types/addTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/addTypeValue.ts 100% 100% 100% 100%
src/operations/types/createType.ts 100% 100% 100% 100%
src/operations/types/dropType.ts 100% 100% 100% 100%
src/operations/types/dropTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/index.ts 100% 100% 100% 100%
src/operations/types/renameType.ts 100% 100% 100% 100%
src/operations/types/renameTypeAttribute.ts 100% 100% 100% 100%
src/operations/types/renameTypeValue.ts 100% 100% 100% 100%
src/operations/types/setTypeAttribute.ts 100% 100% 100% 100%
src/operations/views/alterView.ts 100% 100% 100% 100%
src/operations/views/alterViewColumn.ts 100% 100% 100% 100%
src/operations/views/createView.ts 100% 100% 100% 100%
src/operations/views/dropView.ts 100% 100% 100% 100%
src/operations/views/index.ts 100% 100% 100% 100%
src/operations/views/renameView.ts 100% 100% 100% 100%
src/operations/views/shared.ts 100% 66.66% 100% 100%
src/utils/PgLiteral.ts 90.47% 100% 80% 90.47% 44-45
src/utils/createSchemalize.ts 100% 100% 100% 100%
src/utils/createTransformer.ts 100% 100% 100% 100%
src/utils/decamelize.ts 100% 100% 100% 100%
src/utils/escapeValue.ts 100% 100% 100% 100%
src/utils/formatLines.ts 100% 100% 100% 100%
src/utils/formatParams.ts 100% 100% 100% 100%
src/utils/formatPartitionColumns.ts 100% 100% 100% 100%
src/utils/getMigrationTableSchema.ts 100% 100% 100% 100%
src/utils/getSchemas.ts 100% 100% 100% 100%
src/utils/identity.ts 100% 100% 100% 100%
src/utils/index.ts 100% 100% 100% 100%
src/utils/intersection.ts 100% 100% 100% 100%
src/utils/makeComment.ts 100% 100% 100% 100%
src/utils/quote.ts 100% 100% 100% 100%
src/utils/stringIdGenerator.ts 100% 100% 100% 100%
src/utils/toArray.ts 100% 100% 100% 100%
src/utils/types.ts 100% 100% 100% 100%
Generated in workflow #2408 for commit 7e745de by the Vitest Coverage Report Action

@Shinigami92 Shinigami92 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test/migrations/095_index_nulls.js with applicable content

@jonahkagan
jonahkagan requested a review from Shinigami92 August 11, 2025 17:59
@jonahkagan

Copy link
Copy Markdown
Contributor Author

I added the migration test file. It looks like there's an integration test failure in CI (looks like an outdated snapshot maybe?).

I haven't yet been able to figure out how to run the integration tests locally - I may be missing some dependencies since I'm getting the following error:


 FAIL   integration  test/integration/db-config-it.spec.ts > node-pg-migrate config file and env fallback (PG 17)
Error: Could not find a working container runtime strategy
 ❯ getContainerRuntimeClient node_modules/testcontainers/src/container-runtime/clients/client.ts:63:9
 ❯ PostgreSqlContainer.start node_modules/testcontainers/src/generic-container/generic-container.ts:87:20
 ❯ PostgreSqlContainer.start node_modules/@testcontainers/postgresql/src/postgresql-container.ts:49:43
 ❯ Module.setupPostgresDatabase test/integration/utils.ts:71:10
     69|   databaseName: string = 'node_pg_migrate'
     70| ): Promise<StartedPostgreSqlContainer> {
     71|   return await new PostgreSqlContainer(containerImage)
       |          ^
     72|     .withUsername('ubuntu')
     73|     .withPassword('ubuntu')
 ❯ test/integration/db-config-it.spec.ts:36:21

If it's just a quick snapshot update it may be quicker for someone with a working development environment to do it rather than to try to help unblock me.

@brenoepics

brenoepics commented Aug 11, 2025

Copy link
Copy Markdown
Collaborator

I added the migration test file. It looks like there's an integration test failure in CI (looks like an outdated snapshot maybe?).

I haven't yet been able to figure out how to run the integration tests locally - I may be missing some dependencies since I'm getting the following error:


 FAIL   integration  test/integration/db-config-it.spec.ts > node-pg-migrate config file and env fallback (PG 17)
Error: Could not find a working container runtime strategy
 ❯ getContainerRuntimeClient node_modules/testcontainers/src/container-runtime/clients/client.ts:63:9
 ❯ PostgreSqlContainer.start node_modules/testcontainers/src/generic-container/generic-container.ts:87:20
 ❯ PostgreSqlContainer.start node_modules/@testcontainers/postgresql/src/postgresql-container.ts:49:43
 ❯ Module.setupPostgresDatabase test/integration/utils.ts:71:10
     69|   databaseName: string = 'node_pg_migrate'
     70| ): Promise<StartedPostgreSqlContainer> {
     71|   return await new PostgreSqlContainer(containerImage)
       |          ^
     72|     .withUsername('ubuntu')
     73|     .withPassword('ubuntu')
 ❯ test/integration/db-config-it.spec.ts:36:21

If it's just a quick snapshot update it may be quicker for someone with a working development environment to do it rather than to try to help unblock me.

Hmm, maybe testcontainers/testcontainers-node#913 or testcontainers/testcontainers-node#793 ?

Are you using nektos/act or something like that? But yeah, I think it is related to snapshots.

EDIT:

Yeah, it is the snapshot, do you want me to push the updated snapshot? @jonahkagan

image

@jonahkagan

Copy link
Copy Markdown
Contributor Author

@brenoepics thanks for offering! Looking at the snapshot diff you posted, I actually realized that the test migration was missing the unique option. I updated the migration and also added a guard to prevent that from happening again. See eaea60b for details.

If you wouldn't mind posting the result of the snapshot after that change, I can validate it before you push it. Thanks for offering to help!

@brenoepics

Copy link
Copy Markdown
Collaborator
image

It seems like maybe Postgresql filters out the default value "DISTINCT",
so it's not showing up in the snapshot output.
@jonahkagan

Copy link
Copy Markdown
Contributor Author

Hm, I still don't see the "NULLS" clause in the snapshot. I wonder if Postgres filters it out if you pass in the default value ("DISTINCT"). I'm gonna try changing it to "NOT DISTINCT": 5cd4572

@brenoepics

Copy link
Copy Markdown
Collaborator
### MIGRATION 095_index_nulls (UP) ###
CREATE TABLE "t095" (
  "id" integer NOT NULL
);
CREATE UNIQUE INDEX "t095_id_unique_index" ON "t095" ("id");
INSERT INTO "public"."pgmigrations" (name, run_on) VALUES ('095_index_nulls', NOW());

@jonahkagan

Copy link
Copy Markdown
Contributor Author

Ok, hmm, still not there. Last idea that I have is to remove the not null constraint from the column we're adding the index to: f12ca6e

@brenoepics

Copy link
Copy Markdown
Collaborator

Actually I think I missed a build on the second run, it seems to be good now:

### MIGRATION 095_index_nulls (UP) ###
CREATE TABLE "t095" (
  "id" integer
);
CREATE UNIQUE INDEX "t095_id_unique_index" ON "t095" ("id") NULLS NOT DISTINCT;
INSERT INTO "public"."pgmigrations" (name, run_on) VALUES ('095_index_nulls', NOW());

@jonahkagan

Copy link
Copy Markdown
Contributor Author

Awesome! That looks great!

@brenoepics

brenoepics commented Aug 11, 2025

Copy link
Copy Markdown
Collaborator

Last thing, nulls on createIndex is not available on PG 13 and 14. See https://www.postgresql.org/docs/14/sql-createindex.html and https://www.postgresql.org/docs/17/sql-createindex.html

Error executing:
CREATE UNIQUE INDEX "t095_id_unique_index" ON "t095" ("id") NULLS NOT DISTINCT;
                                                            ^^^^

syntax error at or near "NULLS"

maybe we can do something like we did at https://github.com/salsita/node-pg-migrate/blob/14cd48a6bf2a9aaac78b4d24a5142e11ee583bf3/test/migrations/093_alter_column_expression.js

All other versions are working fine now.

@jonahkagan

Copy link
Copy Markdown
Contributor Author

Updated with a version guard. Thanks for the pointer to the example

@Shinigami92 Shinigami92 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 nicely done
I'm glad that I requested to add a migration test 😅

@brenoepics could you also do an extra review so I don't overlook something?
Then I need to do a release for 9.0.0-alpha.1 as we are currently busy building everything together for the v9.0.0 release

@brenoepics brenoepics left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Shinigami92
Shinigami92 added this pull request to the merge queue Aug 13, 2025
Merged via the queue into salsita:main with commit 1c81b16 Aug 13, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: feature Request for new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support "NULLS" clause in createIndex

3 participants