Addax 6.0.7 has released! 🎉

We are pleased to announce version 6.0.7, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.
Below are the key updates in this release:

📜 Changes

8618d02 [maven-release-plugin] prepare for next development iteration
6877e07 (tag: 6.0.7) [maven-release-plugin] prepare release 6.0.7
0d53b54 [bugfix][lib][rdbms] fix #1380 Operation not allowed after ResultSet closed failure observed in writer initialization
f6666f6 [chore][plugin][mysqlreader] Remove unused import statement from MysqlReader.java
efa6c62 [improve][plugin] Enhance job template configuration with additional parameters and improved formatting
5bd62aa [bugfix][plugin][elastcisearchreader] Fix ES query serialization and avoid adding type to search path (#1379)
58b9eae [chore][update][3rd] Update dependency versions in pom.xml
1adb9db [improve][plugin][mongodbwriter] add supports for wildcard column (#1377)
0e2134f [improve][plugin][mongodbreader] add supports for wildcard column (#1376)
544eefc [chore][github] add pull request template to standardize contributions
c70da5f [improve][plugin][mongodbwriter] implement nested field handling for better data structure support
b6972a4 [maven-release-plugin] prepare for next development iteration

Addax 6.0.6 has released! 🎉

We are pleased to announce version 6.0.6, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.
Below are the key updates in this release:

📜 Changes

b6972a4 (HEAD -> master) [maven-release-plugin] prepare for next development iteration
6928b57 (tag: 6.0.6) [maven-release-plugin] prepare release 6.0.6
bb9935c (origin/master, origin/HEAD) [bugfix][core] ensure proper validation when updating task group communication by checking for null previous value
7e93665 [improve][core] enhance thread safety and performance by using final maps and ThreadLocal for DecimalFormat
6529ac4 [chore][3rd] update CodeQL action to v4 and bump dependency versions in pom.xml
538b793 [chore][plugin][hdfswriter] suppress log4j warnings by using log4j-over-slf4j bridge
9c31179 [refactor][lib][rdbms] optimize SQL logging, bytes handling, object creation, and record building
9dcf8cb [feature][plugin][streamwriter] add hex preview formatting for binary columns and improve console output
b54b542 [feature][plugin][mongodbreader] Support Nested Field Extraction in MongoDBReader Plugin via Dot Notation (#1360)
345df81 [chore][3rd] update json-smart to version 2.6.0 and fastjson2 to version 2.0.59
12e8389 [bugfix][plugin][hdfswriter] set scale before precision for ORC decimal to avoid invalid precision/scale error
f530037 [improve][core] improve date compatibility for BSD/macOS and support custom log file name in addax.sh
b675390 [maven-release-plugin] prepare for next development iteration

Addax 6.0.5 has released! 🎉

We are pleased to announce version 6.0.5, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.
Below are the key updates in this release:

Highlight

The new Server module provides HTTP interfaces for submitting and managing data collection tasks. Users can submit JSON job configurations via POST, and the server executes tasks asynchronously, returning a unique task ID. Progress and results can be queried using this ID.

Features

• RESTful API for task submission and status query
• Configurable maximum concurrent tasks (default: 30)
• Integrated with Addax core Engine for direct job execution
• Supports command-line and environment variable concurrency settings
• Startup/shutdown script with background mode support

📜 Changes

482146c [maven-release-plugin] prepare for next development iteration
6f6a2f7 [maven-release-plugin] prepare release 6.0.5
e97f70c [improve][server] Refactor AddaxServer and TaskService for improved readability and maintainability
a0d6b9a [feature][add][server] add minimal HTTP server module for job submission and status query (#1359)
fab780e [chore][docs] add related projects section to README files
a6a7ca8 [feature][plugin][hdfswriter] add automatic path creation option in HDFS writer configuration
667b22c [bugfix][plugin][streamwriter] resolve console log output inconsistency between terminal and log files
7aef269 [improve][core] Enhance exception handling by including JDBC URL and query SQL in error messages
64d5378 [chore][3rd] update dependencies to updated version
ada7ebc [chore][core] remove deprecated methods
668e8fb [chore][maven] remove unnecessary excludes from package.xml dependencies
d1571b3 [chore][core] Add 'RANK' keyword to SQL constant definitions
00b18c0 [improve][lib][rdbms] Add support for column name quoting in rdbms writer
158c184 [bugfix][plugin][s3writer] fix #1344, handle null columns in record2String with continue to prevent duplicate and assertion errors
a3efd3b [chore][core] add totalBytes to job statistics and update documentation
380cbe7 [bugfix][plugin][hivereader] add hadoop-shaded-protobuf dependency for enhanced compatibility
357a855 [improve][core][script] enable Java security to support JDK 21+
6047869 [improve][plugin][mongodbreader] use MongoRowConverter for BSON document processing
5704a3e [improve][plugin][excelwriter] Improved Excel write performance by generating XML files directly (#1341)
4a09681 [chore][plugin][datareader] replace deprecated RandomStringUtils.randomAlphanumeric with insecure variant
a02defe [chore][github][action]: update actions/setup-java and actions/upload-pages-artifact to latest versions
e5816e8 [maven-release-plugin] prepare for next development iteration

Addax 6.0.4 has released! 🎉

We are pleased to announce version 6.0.4, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.

Below are the key updates in this release:

📜 Changes

c402a61 [chore][docs] update mkdocs navigation for improved readability and localization
e3a1390 [refactor][lib][storage] enhance FileHelper and StorageWriterUtil with improved error handling, validation, and documentation
a4c6ed6 [chore][lib][rdbms] enhance JavaDoc comments across multiple classes for improved clarity and understanding
e621b75 [chore][core] improve comments and formatting in FilterTransformer and RetryUtil
d1ac643 [chore][core] improve Communication class with deep-copy constructor and message cloning
f772a62 [chore][lib][rdbms] refactor SQL query formatting for SQLite and Sybase cases
e65ff3f [chore][core][doc] enhance JavaDoc comments for better clarity and understanding
6bd212b [chore][lib][rdbms] replace List with Set for numeric JDBC types in GetPrimaryKeyUtil
9f939b5 [chore][github][action] Bump actions/checkout from 4 to 5 (#1332)
5b5bb62 [chore][3rd] update dependency versions in pom.xml
c6f5317 [chore][github][issue] update default version in NEW-ISSUE.yml to 6.0.3
3b2bcf2 [maven-release-plugin] prepare for next development iteration

Addax 6.0.3 has released! 🎉

We are pleased to announce version 6.0.3, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.
Below are the key updates in this release:

📜 Changes

35ad873 (origin/master, origin/HEAD) [chore][docs] chanege the default directory
82414b0 [feature][lib][rdbms] support reading SQL from external files in querySql configuration (#1331)
2d5a780 [chore][docs] restore deleted docs
87c0215 [bugfix][plugin][cassandrareader] fix missing netty dependencies , fix #1329
2de7f97 [chore][github][issue-template] update version references and improve log output instructions in NEW-ISSUE.yml
240418f [refactor][doc][json] streamline JSON configuration by consolidating connection parameters and enhancing readability
4487de3 Merge branch 'master' of github.com:wgzhao/Addax
700acee [bugfix][lib][rdbms] SingleTableSplitUtil#buildStringSplitPointQuery should distinguish oracle and other rdbms to generate statement
110361e [chore][github][action] update GitHub Actions to latest versions for improved performance and security
e97928a [chore][docs] update plugin development and transformer documentation for clarity and consistency
a32b6b3 [chore][docs] update language links in mkdocs.yml for correct routing
8af9790 Add bilingual documentation support with English as default language (#1321)
95e04e4 Merge branch 'master' of github.com:wgzhao/Addax
368297b [chore][plugin][starrockswriter] improve logging format and enhance error handling
d0296c3 [chore][core] correct capitalization in job format upgrade comment in ConfigParser.java
9b24e6d [chore][core] simplify a switch statement in LoadUtil.java for improved readability
0ff7383 [feature][core] add common date constants and parameters to addax.sh for enhanced job configuration
dccc07c [chore][update][3rd]: update dependency versions in pom.xml
9c9b370 [chore][plugin][kuduwriter] simplify KuduHelper constructors and enhance KuduWriterTask with modern switch expressions
4e0852e [maven-release-plugin] prepare for next development iteration

Addax 6.0.2 has released! 🎉

We are pleased to announce version 6.0.2, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.

Below are the key updates in this release:

Highlight

• update Hadoop version to 3.3.6 (#1306)
• migrate from Apache HttpClient to Java 11 HttpClient (#1305)
• add hidden file filter to listStatus method in DFSUtil

📜 Changes

46ef26e (origin/master, origin/HEAD) [feat][plugin][hdfsreader] add hidden file filter to listStatus method in DFSUtil
d3afd05 [chore][docs][hivereader] add version compatibility information for HiveReader
dfb77d6 [bugfix][core][build] increase pom.xml read limit to capture a project version correctly
0539fbe [upgrade][dependencies] update Hadoop version to 3.3.6 (#1306)
02304c5 [chore][license][update] add Apache License 2.0 header to missing files
e33e74b [chore][plugin][dependencies] update log4j and zstd versions in pom.xml
0d343c6 [improve][plugin][httpreader] improve JSON body handling in POST requests
55b9198 [refactor][core] simplify job resource handling by removing HTTP request logic
d0dbbc3 [bugfix][core] fixed job file URL parsing to support case-insensitive protocols
32d42c5 [chore][plugin][icebergwriter] clean up pom.xml by removing unused dependencies and properties
bbb3e64 [chore][plugin][hdfsreader] fix indentation in pom.xml
56678f4 [chore][plugin][hivereader] remove unnecessary whitespace in pom.xml
ab24da9 [chore][plugin][dependencies] update Avro to 1.12.0 and ORC to 1.9.5
a9a6322 [chore][plugin][icebergwriter] update error message for unsupported file format
0647617 [bugfix][plugin][icebergwriter] fix error when write orc format table (#1304)
fdc8934 [chore][plugin][httpreader] remove unused commons-net dependency from pom.xml
2657f08 [refactor][plugin][httpreader] migrate from Apache HttpClient to Java 11 HttpClient (#1305)
5349776 [refactor][core] switch garbage collector from G1GC to ZGC for improved performance
fbaab28 [refactor][core] replace Apache HttpClient with Java 11 HttpClient for improved HTTP handling
50fd555 [maven-release-plugin] prepare for next development iteration

Addax 6.0.1 has released! 🎉

We are pleased to announce version 6.0.1, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.

Below are the key updates in this release:

1. Add support for range data type for PostgreSQL (#1283)
2. Add support for kerberos with kudu (#1290, #1292)
3. Add support for geometry data type for MySQL (#1298, #1299)

📜 Changes

2405b5c [bugfix][plugin][elasticsearchreader] adds missing Apache HttpClient dependency
57bb867 [chore][dependencies] update jackson, redis.replicator, woodstox, and sqlite JDBC versions
682dd90 [feature][plugin][mysqlwriter] Add support for GEOMETRY data type handling (#1299)
f1e4d31 [feature][plugin][mysqlreader] Add support for GEOMETRY data type handling (#1298)
ab65c64 [chore][dependencies] update versions for druid, jansi, jsch, ngdbc, and iceberg
3b01026 [feature][plugin][kudureader] Add support for kerberos with kudu (#1292)
9a7a25a [improve][plugin][writer] enhances conflict handling by filtering out conflict columns in an update statement
b1f993f [chore][plugin][kuduwriter] simplify KuduHelper initialization and improve code comments
b0caf7a [feature][writer][kuduwriter] Add support for kerberos with kudu (#1290)
5fbada0 [refactor][plugin][postgresql] clean up error handling and improve code readability
6f95151 [add][plugin][postgresql] Add support for the range data type (#1283)
fc5128f [chore][update][dependencies] bump versions for paimon, gson, httpclient, clickhouse, and databend
f3111d7 [bugfix][plugin][hdfs] add exclusion for hadoop-client-api in protobuf-java dependency
545aa24 [chore][docker] update build script to use correct addax version directory
8fad050 [maven-release-plugin] prepare for next development iteration

Addax 6.0.0 has released! 🎉

We are pleased to announce version 6.0.0. This update includes significant improvements, most notably the transition from JDK 1.8 to JDK 17. Below is a summary of key changes and enhancements in this release:

Key Highlights

1. Migration to JDK 17
   • Improved Performance: With JDK 17's performance optimizations, users will experience faster execution and reduced resource consumption.
   • Enhanced Security: JDK 17 introduces stronger security mechanisms, offering additional protection against potential vulnerabilities.
   • Expanded Language Features: Developers can now utilize the latest Java language features such as:
     • Switch Expressions: Simplify switch logic for better code readability.
     • Text Blocks: Streamline the handling of multi-line strings.
     • Records: Create immutable data structures with minimal boilerplate code.
2. Compatibility Updates
   • Refactored the application codebase to remove deprecated APIs and ensure full compatibility with JDK 17.
   • Updated dependency libraries to their JDK 17-compatible versions.
3. Optimizations
   • Improved memory management and garbage collection efficiency with JDK 17's G1 GC enhancements.

Upgrade Notes

1. Java Runtime Environment Requirement:

   • This version requires JDK 17 or newer to run. Users currently running JDK 1.8 will need to upgrade their Java runtime environment before deploying this version.

2. Breaking Changes:

   • Some deprecated features and APIs from JDK 1.8 may no longer function. Ensure custom integrations or extensions are updated accordingly.

3. Backward Compatibility:

   • Older versions of the software and configurations may require minor adjustments due to the upgraded platform. Refer to the compatibility guide in our documentation for detailed instructions.

📜 Changes

da81f0f (tag: 6.0.0) [maven-release-plugin] prepare release 6.0.0
f0e5114 (origin/master, origin/HEAD) [refactor][github][action] remove create-release to avoid creating duplicate label
0c6d619 [chore][github][action] Update maven-publish.yml to trigger on version tags and refine upload condition
0982b26 [chore][core] Refactor comments and logging statements for clarity
0b1ee2c [bugfix][plugin][streamwriter] Fixed issue where stdout resource being closed prevented the logger from writing to the console.
13bc07a [chore][docs] change jdk to 17
70b6b1c [chore][docs] Add iceberg logo
7ac68c2 [chore][core] Add reserved keyword system
e0c2555 [improve][plugin][s3writer] refine pom to reduce the target size
9749d3d [improve][plugin][s3writer] Rename formatwriter package to writer for consistency
8400e24 [add][writer][s3writer] Add support for orc and parquet storage format (#1275)
cb1964d [chore][plugin][icebergwriter] satisfy code style: Improve code formatting and consistency in IcebergHelper and IcebergWriter classes
67a1f0a [chore][plugin][icebergwriter] Add Apache License header to IcebergHelper and IcebergWriter classes
fbbd0ea [add][plugin][icebergwriter] add write plugin icebergwriter (#1208)
330b2fd [feature][core] Optimize garbage collection settings for JDK 17
17843b5 [chore][doc] Update runtime requirements in README files to specify JDK 17
ea54c93 Optimzie for jdk17 (#1273)
afbf8b8 [feature][plugin][s3reader] Add support for path-style access configuration in S3 client
e4c4bed [feature][docker] Add Docker Build and Push steps to CI workflow
381951d [maven-release-plugin] prepare for next development iteration

Known bugs:

hdfsreader and hdfswriter plugins

if you encounter the following errors:

2025-06-04 13:56:02.341 [       job-0] ERROR Engine               - java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
        at com.wgzhao.addax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:68)
        at com.wgzhao.addax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:90)
        at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:642)
        at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:280)
        at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:126)
        at com.wgzhao.addax.core.Engine.start(Engine.java:62)
        at com.wgzhao.addax.core.Engine.entry(Engine.java:116)
        at com.wgzhao.addax.core.Engine.main(Engine.java:145)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.ctc.wstx.io.InputBootstrapper
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        ... 8 more

you can delete the $ADDAX_HOME/plugin/reader/hdfsreader/libs/hadoop-client-api-3.2.4.jar and $ADDAX_HOME/plugin/writer/hdfswriter/libs/hadoop-client-api-3.2.4.jar

Addax 5.1.2 has released! 🎉

We are pleased to announce version 5.1.2, which focuses on improving stability and performance by addressing several bugs and implementing minor code refinements.
Below are the key updates in this release:

Highlights

Now, you can use JDK 17 to run Addax

📜 Changes

0ff910b (tag: 5.1.2) [maven-release-plugin] prepare release 5.1.2
d5ca390 [bugfix][lib][rdbms] Quote column name for split key to avoid reserved word
1a0c789 [feature][update] Enhance JDK support and update runtime requirements to JDK 17
27df774 [feature][update] Upgrade Project Compilation to JDK 17 (#1269)
da61691 [feature][rdbms] Add Sybase ASE support for primary key and unique key retrieval in GetPrimaryKeyUtil (#1265)
862b0dd [feature][rdbms] Add support for sqlite with autoPk item (#1264)
39e7a21 [chore][core] Revert Simplify needChannelNumber calculation by removing redundant checks
500c090 [chore][update][3rd] Update dependency versions in pom.xml
bf3b274 [improve][doc] Enhance autoPk section with detailed behavior and logging information
b509e75 [refactor][core] Simplify needChannelNumber calculation by removing redundant checks
a182da7 [improve][doc] Add notes on character set considerations for autoPk and splitPk usage
9425c7c [chore][lib][storage] Update CSVParser instantiation to replace deprecated methods with updated API
4cfa2d8 [improve][svg] Refactor SVG files for improved readability and consistency
bed068c [feature][github][ci] Update changelog generation to use latest release tag and enhance release notes
2b41adb [feature][docker] Add version output step to docker-push.yml for tagging images
5bc426e [maven-release-plugin] prepare for next development iteration

Addax 5.1.1 has released! 🎉

We are excited to share the routine patch release of version 5.1.1, focused on incremental improvements to ensure a smoother and more reliable experience. This version primarily addresses a range of minor bugs and incorporates subtle code enhancements for better system performance.

Bug Fixes:

• [addax-rdbms] Fix calculateStringBetween() method to avoid generating backslash characters that lead to invalid SQL conditions. (#1258)
• [hbase20xsqlreader] Fix null values in convertPhoenixValueToAddaxColumn method
• [hdfswriter] Fix missing parquet-avro dependency with exclusions in pom.xml

📜 Changes

• 31d0ee0 [feature][issue-template] Update default version to 5.1.0 and adjust version options in NEW-ISSUE.yml
• 8eaad4c [chore][deps] Bump com.google.protobuf:protobuf-java from 3.25.1 to 3.25.5 to fix CVE-2024-7254
• 72a6013 [bugfix][lib][rdbms] Fix calculateStringBetween() method to avoid generating backslash characters that lead to invalid SQL conditions. (#1258)
• 8869217 Merge branch 'master' of github.com:wgzhao/Addax
• ca2dee4 [chore][deps] Update some library versions in pom.xml
• 76903b4 [bugfix][reader][hbase20xsqlreader] Handle null values in convertPhoenixValueToAddaxColumn method
• 4f06f38 [chore][deps][3rd]: Update poi and databend JDBC driver versions in pom.xml
• 50bce26 [chore][github][ci] Update Docker actions to latest versions in workflow
• 69fc89e [bugfix][plugin][hdfswriter] Add missing parquet-avro dependency with exclusions in pom.xml
• 8aa639a [feature][ci] Add GitHub Actions workflow for building and pushing Docker images
• 8c27ccd [chore][depends] Fix Authorization Bypass Through User-Controlled Key vulnerability
• 17826ac [chore][depends] Fix vulnerable to SQL Injection via line comment generation
• 93d2129 [chore][depends] Fix Parquet Avro Vulnerability to Arbitrary Code Execution
• 360c98a [chore][depends] Update dependency versions in pom.xml for httpcore, joda-time, ognl, influxdbClient, and spotbugs-maven-plugin
• a8f467d [improve][docker] Simplify Dockerfile by using official Maven and OpenJDK images and add support for Aliyun mirrors
• e347749 [improve][core] update shebang in addax.sh and adjust COPY command in Dockerfile
• 6a97a9b [maven-release-plugin] prepare for next development iteration