Skip to content

JSON datatype support#2558

Merged
divang merged 61 commits intomainfrom
user/divang/json-datatype-support
Jul 21, 2025
Merged

JSON datatype support#2558
divang merged 61 commits intomainfrom
user/divang/json-datatype-support

Conversation

@divang
Copy link
Copy Markdown
Contributor

@divang divang commented Dec 6, 2024

Description

This pull request introduces support for the JSON datatype in the Microsoft JDBC Driver for SQL Server. With these changes, applications using the driver can now read from and write to SQL Server columns of type JSON, enabling seamless integration with semi-structured data.
Below are some of the example scenarios:

  1. Insertion of JSON Data into Table
@Test
public void insertJsonData() throws SQLException {
    String insertSql = "INSERT INTO " + AbstractSQLGenerator.escapeIdentifier(tableName) + " (data) VALUES (?)";
    String json = "{\"name\":\"John\",\"skills\":[\"Java\",\"SQL\"]}";
    try (PreparedStatement pstmt = connection.prepareStatement(insertSql)) {
        pstmt.setString(1, json);
        pstmt.executeUpdate();
    }
}
  1. Select JSON Data from Table
@Test
public void getJsonData() throws SQLException {
    String query = "SELECT data FROM " + AbstractSQLGenerator.escapeIdentifier(tableName);
    try (PreparedStatement stmt = connection.prepareStatement(query);
         ResultSet rs = stmt.executeQuery()) {
        while (rs.next()) {
            String json = rs.getString("data");
            System.out.println("JSON: " + json);
        }
    }
}
  1. Stored Procedure Call with JSON IN/OUT
@Test
public void testJsonStoredProcedureInputOutput() throws SQLException {
    createProcedure();
    String call = "{call " + AbstractSQLGenerator.escapeIdentifier(procedureName) + "(?, ?)}";
    try (CallableStatement cstmt = connection.prepareCall(call)) {
        String inputJson = "{\"key\":\"value\"}";
        cstmt.setString(1, inputJson);
        cstmt.registerOutParameter(2, microsoft.sql.Types.JSON);

        cstmt.execute();
        String outputJson = cstmt.getString(2);
        System.out.println("Output JSON: " + outputJson);
    }
}
  1. Bulk Copy from CSV to Table with JSON Column
@Test
public void testBulkCopyJsonFromCSV() throws Exception {
    String dstTable = RandomUtil.getIdentifier("BulkJsonTest");
    String fileName = filePath + jsonInputFile;
    try (Connection con = getConnection();
         Statement stmt = con.createStatement();
         SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(con);
         SQLServerBulkCSVFileRecord fileRecord = new SQLServerBulkCSVFileRecord(fileName, encoding, ",", false)) {
        // Create the destination table
        stmt.executeUpdate("CREATE TABLE " + dstTable + " (c1 BIT, c2 nchar(50), c3 JSON)");
        fileRecord.addColumnMetadata(1, "c1", java.sql.Types.BIT, 0, 0);
        fileRecord.addColumnMetadata(2, "c2", java.sql.Types.NCHAR, 10, 0);
        fileRecord.addColumnMetadata(3, "c3", microsoft.sql.Types.JSON, 0, 0);
        fileRecord.setEscapeColumnDelimitersCSV(true);
        bulkCopy.setDestinationTableName(dstTable);
        bulkCopy.writeToServer(fileRecord);
    }
}

Key Changes

  • Type System and Metadata

    • Added JSON to the type system (enums such as TDSType, SSType, JDBCType, and their categories).
    • Updated metadata handling to detect, map, and propagate JSON types correctly.
  • Connection and Feature Negotiation

    • Implemented feature negotiation for JSON support in SQLServerConnection, including versioning and server capability detection.
    • Extended the TDS protocol handling and connection handshake to advertise and recognize JSON support.
  • Bulk Copy and Data Table Support

    • Modified SQLServerBulkCopy, SQLServerBulkCSVFileRecord, and SQLServerDataTable to handle JSON columns in bulk operations.
    • Ensured proper streaming, encoding, and null handling for JSON columns during bulk inserts and exports.
  • Parameter and Value Handling

    • Updated parameter setting, reading, and writing logic to process JSON values seamlessly.
    • Ensured correct behavior for both application-supplied and database-returned JSON data.
  • Test and Build Integration

    • Added JSONTest exclusion tags in build.gradle and pom.xml for targeted testing.
    • Enabled grouping and exclusion of JSON-specific tests for various build profiles.

Motivation

The addition of JSON datatype support allows Java applications to leverage SQL Server’s native JSON features directly through the JDBC driver. This enhancement addresses modern data requirements by enabling efficient storage, retrieval, and manipulation of semi-structured JSON data within relational workflows.

Testing

  • Comprehensive unit and integration tests were added for JSON columns.
  • Tests ensure correct data round-trip, null handling, and edge-case coverage.

For details on all code changes, see the full diff.

@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 6, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 32 lines in your changes missing coverage. Please review.

Project coverage is 51.55%. Comparing base (87f0553) to head (f8d3c0b).
Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java 0.00% 11 Missing ⚠️
...rc/main/java/com/microsoft/sqlserver/jdbc/dtv.java 12.50% 6 Missing and 1 partial ⚠️
.../microsoft/sqlserver/jdbc/SQLServerConnection.java 75.00% 3 Missing and 2 partials ⚠️
...n/java/com/microsoft/sqlserver/jdbc/DataTypes.java 87.50% 2 Missing ⚠️
...n/java/com/microsoft/sqlserver/jdbc/Parameter.java 0.00% 2 Missing ⚠️
...oft/sqlserver/jdbc/SQLServerBulkCSVFileRecord.java 0.00% 2 Missing ⚠️
...oft/sqlserver/jdbc/SQLServerPreparedStatement.java 33.33% 1 Missing and 1 partial ⚠️
...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2558      +/-   ##
============================================
+ Coverage     51.50%   51.55%   +0.05%     
- Complexity     4050     4067      +17     
============================================
  Files           149      149              
  Lines         34136    34236     +100     
  Branches       5700     5718      +18     
============================================
+ Hits          17581    17651      +70     
- Misses        14076    14103      +27     
- Partials       2479     2482       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@divang divang marked this pull request as draft December 10, 2024 20:23
@lilgreenbird
Copy link
Copy Markdown
Contributor

/azp run public-mssql-jdbc.windows

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@azure-pipelines
Copy link
Copy Markdown

No pipelines are associated with this pull request.

@lilgreenbird
Copy link
Copy Markdown
Contributor

/azp run public-mssql-jdbc.linux

@lilgreenbird
Copy link
Copy Markdown
Contributor

/azp run CI-MacOS

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

1 similar comment
@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

Comment thread src/main/java/microsoft/sql/Types.java
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/dtv.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/dtv.java
Copy link
Copy Markdown
Contributor

@lilgreenbird lilgreenbird left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to add tests for datatype conversions. See PR for sql_variant for example

@divang divang self-assigned this Feb 12, 2025
@divang divang marked this pull request as ready for review February 12, 2025 09:54
David-Engel
David-Engel previously approved these changes Feb 13, 2025
@saurabh500
Copy link
Copy Markdown

Can you update the description with a list of high level areas that have been modified in this PR ?

Comment thread src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerParameterMetaData.java Outdated
David-Engel
David-Engel previously approved these changes Feb 14, 2025
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/DataTypes.java
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCSVFileRecord.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/dtv.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/tvp/TVPTypesTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/tvp/TVPTypesTest.java
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/unit/statement/RegressionTest.java Outdated
@Ananya2 Ananya2 force-pushed the user/divang/json-datatype-support branch from 352dd3a to 7b23fc8 Compare July 7, 2025 07:03
@Ananya2 Ananya2 modified the milestones: 13.2.0, 13.1.1 Jul 8, 2025
@Ananya2 Ananya2 requested a review from David-Engel July 9, 2025 05:40
Copy link
Copy Markdown
Collaborator

@David-Engel David-Engel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review (21/31)

Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerConnection.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerResource.java Outdated
Comment thread src/main/java/com/microsoft/sqlserver/jdbc/SQLServerResource.java Outdated
Copy link
Copy Markdown
Collaborator

@David-Engel David-Engel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another partial review 23/31

Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/datatypes/JSONFunctionTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/jdbc/resultset/ResultSetTest.java Outdated
Comment thread src/test/java/com/microsoft/sqlserver/testframework/AbstractTest.java Outdated
@divang divang merged commit a7fe77a into main Jul 21, 2025
18 of 19 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Closed/Merged PRs in MSSQL JDBC Jul 21, 2025
muskan124947 added a commit that referenced this pull request Jul 30, 2025
)

* JSON datatype support (#2558)

* JSON datatype implementation.

Co-authored-by: lilgreenbird <[email protected]>
Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>
Co-authored-by: Muskan Gupta <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>

* Add new trusted AKV URLs for FR and DE (#2708)

* Add new trusted AKV URLs for FR and DE

* Update SQLServerColumnEncryptionAzureKeyVaultProvider.java

* Enhance code coverage for SQLServerConnectionPoolProxy

* Updated test to fix failures

---------

Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: lilgreenbird <[email protected]>
Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>
Co-authored-by: Mahendra Chavan <[email protected]>
Ananya2 added a commit that referenced this pull request Jul 31, 2025
* JSON datatype support (#2558)

* JSON datatype implementation.

Co-authored-by: lilgreenbird <[email protected]>
Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>
Co-authored-by: Muskan Gupta <[email protected]>
Co-authored-by: Ananya Garg <[email protected]>

* Add new trusted AKV URLs for FR and DE (#2708)

* Add new trusted AKV URLs for FR and DE

* Update SQLServerColumnEncryptionAzureKeyVaultProvider.java

* Updates for 13.1.1 preview release (#2715)

* Updates for 13.1.1 preview release

* Fixed PR link

* Removed unnecessary space

* Enhance test coverage for SQLServerSQLXML file.

---------

Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: lilgreenbird <[email protected]>
Co-authored-by: Divang Sharma <[email protected]>
Co-authored-by: Muskan Gupta <[email protected]>
Co-authored-by: Mahendra Chavan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Closed/Merged PRs

Development

Successfully merging this pull request may close these issues.

9 participants