Skip to content

Commit 5ef7b1b

Browse files
mdb-admdb-ad
authored
DRIVERS-2888 Support QE with Client.bulkWrite (#1770)
1 parent de3696e commit 5ef7b1b

File tree

7 files changed

+509
-54
lines changed

7 files changed

+509
-54
lines changed

‎source/client-side-encryption/limits/limits-encryptedFields.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"fields": [
3+
{
4+
"keyId": {
5+
"$binary": {
6+
"base64": "LOCALAAAAAAAAAAAAAAAAA==",
7+
"subType": "04"
8+
}
9+
},
10+
"path": "foo",
11+
"bsonType": "string"
12+
}
13+
]
14+
}

‎source/client-side-encryption/limits/limits-qe-doc.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"foo": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
3+
}

‎source/client-side-encryption/tests/README.md‎

Lines changed: 31 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -563,10 +563,13 @@ First, perform the setup.
563563
2. Using `client`, drop and create the collection `db.coll` configured with the included JSON schema
564564
[limits/limits-schema.json](../limits/limits-schema.json).
565565

566-
3. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
566+
3. If using MongoDB 8.0+, use `client` to drop and create the collection `db.coll2` configured with the included
567+
encryptedFields [limits/limits-encryptedFields.json](../limits/limits-encryptedFields.json).
568+
569+
4. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
567570
[limits/limits-key.json](../limits/limits-key.json)
568571

569-
4. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)
572+
5. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)
570573

571574
Configure with the `local` KMS provider as follows:
572575

@@ -578,27 +581,27 @@ First, perform the setup.
578581

579582
Using `client_encrypted` perform the following operations:
580583

581-
1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }`.
584+
1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }` into `coll`.
582585

583586
Expect this to succeed since this is still under the `maxBsonObjectSize` limit.
584587

585588
2. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
586-
`{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` Note:
587-
limits-doc.json is a 1005 byte BSON document that encrypts to a ~10,000 byte document.
589+
`{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` into
590+
`coll`. Note: limits-doc.json is a 1005 byte BSON document that encrypts to a ~10,000 byte document.
588591

589592
Expect this to succeed since after encryption this still is below the normal maximum BSON document size. Note, before
590593
auto encryption this document is under the 2 MiB limit. After encryption it exceeds the 2 MiB limit, but does NOT
591594
exceed the 16 MiB limit.
592595

593-
3. Bulk insert the following:
596+
3. Use MongoCollection.bulkWrite to insert the following into `coll`:
594597

595598
- `{ "_id": "over_2mib_1", "unencrypted": <the string "a" repeated (2097152) times> }`
596599
- `{ "_id": "over_2mib_2", "unencrypted": <the string "a" repeated (2097152) times> }`
597600

598601
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
599602
[command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
600603

601-
4. Bulk insert the following:
604+
4. Use MongoCollection.bulkWrite insert the following into `coll`:
602605

603606
- The document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
604607
`{ "_id": "encryption_exceeds_2mib_1", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }`
@@ -608,15 +611,34 @@ Using `client_encrypted` perform the following operations:
608611
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
609612
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
610613

611-
5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>`.
614+
5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>` into `coll`.
612615

613616
Expect this to succeed since this is still (just) under the `maxBsonObjectSize` limit.
614617

615618
6. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
616-
`{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }`
619+
`{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }` into
620+
`coll`.
617621

618622
Expect this to fail since encryption results in a document exceeding the `maxBsonObjectSize` limit.
619623

624+
7. If using MongoDB 8.0+, use MongoClient.bulkWrite to insert the following into `coll2`:
625+
626+
- `{ "_id": "over_2mib_3", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`
627+
- `{ "_id": "over_2mib_4", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`
628+
629+
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
630+
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
631+
632+
8. If using MongoDB 8.0+, use MongoClient.bulkWrite to insert the following into `coll2`:
633+
634+
- The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
635+
`{ "_id": "encryption_exceeds_2mib_3", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`
636+
- The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
637+
`{ "_id": "encryption_exceeds_2mib_4", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`
638+
639+
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
640+
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).
641+
620642
Optionally, if it is possible to mock the maxWriteBatchSize (i.e. the maximum number of documents in a batch) test that
621643
setting maxWriteBatchSize=1 and inserting the two documents `{ "_id": "a" }, { "_id": "b" }` with `client_encrypted`
622644
splits the operation into two inserts.

‎source/crud/bulk-write.md‎

Lines changed: 25 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -459,7 +459,7 @@ class BulkWriteResult {
459459
* The results of each individual write operation that was successfully performed.
460460
*
461461
* This value will only be populated if the verboseResults option was set to true.
462-
*/
462+
*/
463463
verboseResults: Optional<VerboseResults>;
464464

465465
/* rest of fields */
@@ -553,7 +553,9 @@ The `bulkWrite` server command has the following format:
553553
}
554554
```
555555

556-
Drivers MUST use document sequences ([`OP_MSG`](../message/OP_MSG.md) payload type 1) for the `ops` and `nsInfo` fields.
556+
If auto-encryption is not enabled, drivers MUST use document sequences ([`OP_MSG`](../message/OP_MSG.md) payload type 1)
557+
for the `ops` and `nsInfo` fields. If auto-encryption is enabled, drivers MUST NOT use document sequences and MUST
558+
append the `ops` and `nsInfo` fields to the `bulkWrite` command document.
557559

558560
The `bulkWrite` command is executed on the "admin" database.
559561

@@ -645,13 +647,6 @@ write concern containing the following message:
645647

646648
> Cannot request unacknowledged write concern and ordered writes
647649
648-
## Auto-Encryption
649-
650-
If `MongoClient.bulkWrite` is called on a `MongoClient` configured with `AutoEncryptionOpts`, drivers MUST return an
651-
error with the message: "bulkWrite does not currently support automatic encryption".
652-
653-
This is expected to be removed once [DRIVERS-2888](https://jira.mongodb.org/browse/DRIVERS-2888) is implemented.
654-
655650
## Command Batching
656651

657652
Drivers MUST accept an arbitrary number of operations as input to the `MongoClient.bulkWrite` method. Because the server
@@ -672,8 +667,10 @@ multiple commands if the user provides more than `maxWriteBatchSize` operations
672667

673668
### Total Message Size
674669

675-
Drivers MUST ensure that the total size of the `OP_MSG` built for each `bulkWrite` command does not exceed
676-
`maxMessageSizeBytes`.
670+
#### Unencrypted bulk writes
671+
672+
When auto-encryption is not enabled, drivers MUST ensure that the total size of the `OP_MSG` built for each `bulkWrite`
673+
command does not exceed `maxMessageSizeBytes`.
677674

678675
The upper bound for the size of an `OP_MSG` includes opcode-related bytes (e.g. the `OP_MSG` header) and
679676
operation-agnostic command field bytes (e.g. `txnNumber`, `lsid`). Drivers MUST limit the combined size of the
@@ -727,6 +724,12 @@ was determined.
727724

728725
Drivers MUST return an error if there is not room to add at least one operation to `ops`.
729726

727+
#### Auto-encrypted bulk writes
728+
729+
Drivers MUST use the reduced size limit defined in
730+
[Size limits for Write Commands](../client-side-encryption/client-side-encryption.md#size-limits-for-write-commands) for
731+
the size of the `bulkWrite` command document when auto-encryption is enabled.
732+
730733
## Handling the `bulkWrite` Server Response
731734

732735
The server's response to `bulkWrite` has the following format:
@@ -857,6 +860,15 @@ When a `getMore` fails with a retryable error when attempting to iterate the res
857860
entire `bulkWrite` command to receive a fresh cursor and retry iteration. This work was omitted to minimize the scope of
858861
the initial implementation and testing of the new bulk write API, but may be revisited in the future.
859862

863+
### Use document sequences for auto-encrypted bulk writes
864+
865+
Auto-encryption does not currently support document sequences. This specification should be updated when
866+
[DRIVERS-2859](https://jira.mongodb.org/browse/DRIVERS-2859) is completed to require use of document sequences for `ops`
867+
and `nsInfo` when auto-encryption is enabled.
868+
869+
Drivers requiring significant changes to pass a bulkWrite command to libmongocrypt are recommended to wait until
870+
[DRIVERS-2859](https://jira.mongodb.org/browse/DRIVERS-2859) is implemented before supporting automatic encryption.
871+
860872
## Q&A
861873

862874
### Is `bulkWrite` supported on Atlas Serverless?
@@ -928,6 +940,8 @@ error in this specific situation does not seem helpful enough to require size ch
928940

929941
## **Changelog**
930942

943+
- 2025-08-13: Removed the requirement to error when QE is enabled.
944+
931945
- 2025-06-27: Added `rawData` option.
932946

933947
- 2024-11-05: Updated the requirements regarding the size validation.

‎source/crud/tests/README.md‎

Lines changed: 1 addition & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -607,40 +607,7 @@ Execute `bulkWrite` on `client` with `largeNamespaceModel`. Assert that an error
607607
Assert that `error` is a client error. If a `BulkWriteException` was thrown, assert `BulkWriteException.partialResult`
608608
is unset.
609609

610-
### 13. `MongoClient.bulkWrite` returns an error if auto-encryption is configured
611-
612-
This test is expected to be removed when [DRIVERS-2888](https://jira.mongodb.org/browse/DRIVERS-2888) is resolved.
613-
614-
Test that `MongoClient.bulkWrite` returns an error if the client has auto-encryption configured.
615-
616-
This test must only be run on 8.0+ servers. This test must be skipped on Atlas Serverless.
617-
618-
Construct a `MongoClient` (referred to as `client`) configured with the following `AutoEncryptionOpts`:
619-
620-
```javascript
621-
AutoEncryptionOpts {
622-
"keyVaultNamespace": "db.coll",
623-
"kmsProviders": {
624-
"aws": {
625-
"accessKeyId": "foo",
626-
"secretAccessKey": "bar"
627-
}
628-
}
629-
}
630-
```
631-
632-
Construct the following write model (referred to as `model`):
633-
634-
```javascript
635-
InsertOne {
636-
"namespace": "db.coll",
637-
"document": { "a": "b" }
638-
}
639-
```
640-
641-
Execute `bulkWrite` on `client` with `model`. Assert that an error (referred to as `error`) is returned. Assert that
642-
`error` is a client error containing the message: "bulkWrite does not currently support automatic encryption". If a
643-
`BulkWriteException` was thrown, assert `BulkWriteException.partialResult` is unset.
610+
### 13. *Removed*
644611

645612
### 14. `explain` helpers allow users to specify `maxTimeMS`
646613

0 commit comments

Comments
 (0)