2025-10-14 Version 2.12.2

Security

• Require full TLS trust in OpenSearch plugins by default unless insecure is configured. Fixes CVE-2025-62371 / GHSA-43ff-rr26-8hx4. (#6171)
• Use standard TLS when downloading the geoip database from an HTTP URL. Fixes GHSA-3xgr-h5hq-7299. (#6167)
• Use the TLS protocol identifier. Fixes GHSA-28gg-8qqj-fhh5. (#6166)
• Updated Netty to 4.1.125 to resolve CVE-2025-55163, CVE-2025-58057, CVE-2025-58056 (#6085, #5998)
• Updated commons-lang to 3.18.0 to resolve CVE-2025-48924 (#6085)
• Updated BouncyCastle to 1.81 to resolve CVE-2025-8916 (#6085)

Maintenance

• Update the bundled JDK to 17.0.16. (#6172)
• Require the smoke tests to pass before attempting to promote. (#6086)

2025-08-05 Version 2.12.1

Bug Fixes

• Service Map does not rotate with multiple workers (#5901)
• Fixes a regression in core where @SingleThread annotated processors are only running the last instance. (#5904)
• geoip S3 download fails to handle existing files during download (#5898)
• geoip local file fails to handle existing files during download (#5899)

Security

• CVE-2025-46762 - Parquet 1.15.2 (#5923)
• CVE-2025-48734 - commons-beanutils 1.11.0 and Checkstyle 10.26.1 (#5923)
• CVE-2024-57699 - json-smart 2.5.2 (#5923)
• CVE-2025-24970 - Netty 4.1.123 (#5923)
• CVE-2025-27817 - Apache Kafka 3.9.1 and Confluent Kafka 7.9.1 (#5923)
• CVE-2024-7254 - protobuf-java 3.25.5 (#5924)
• CVE-2025-49146 - Postgresql JDBC driver 42.7.1 (#5936)
• CVE-2025-23206 - aws-cdk-lib 2.190.0 (#5925)
• CVE-2025-5889 - aws-cdk-lib 2.190.0 (#5925)
• CVE-2025-47273 - setuptools v78 (#5926)

Maintenance

• Updated the smoke tests scripts to use the end-to-end tests (#5913)

2025-06-26 Version 2.12.0

Breaking Changes

Features

• OTel telemetry unified source (#5596)
• Add SQS sink to Data Prepper (#5634)
• KDS cross account stream (#5687)
• Add otlp sink (for AWS X-Ray) (#5663)
• Adds the Modulus operator to Data Prepper expressions (#5685)
• Add API tokens as an Authc/z method for OpenSearch Sink (#5549)
• Add support for convering keys to lowercase/uppercase in RenameKeyProcessor (#5810)
• Add multi-line csv support (#5784)
• Add auto conversion option to convert_type processor (#5782)
• Add detect format processor (#5774)
• Add getEventType() expression function (#5686)

Enhancements

• Allow plugins to access the default pipeline role (#4958)
• Allow disabling metrics (#5431)
• Support enabling specific experimental plugins (#5675)
• Make opensearch source scroll timeout configurable and increase default value (#5679)

Bug Fixes

• [BUG] OpenTelemetry Spans are indexed using the span id causing collisions (#5370)
• [BUG] _default route no longer seems to exist in 2.11.0 (#5763)
• Fix kafka source with glue registry (#5765)

Security

• protobuf-3.19.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl: 1 vulnerabilities (highest severity is: 7.5) (#5802)
• urllib3-2.2.2-py3-none-any.whl: 2 vulnerabilities (highest severity is: 5.3) (#5801)
• protobuf-3.20.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl: 1 vulnerabilities (highest severity is: 7.5) (#5800)
• urllib3-1.26.19-py2.py3-none-any.whl: 2 vulnerabilities (highest severity is: 5.3) (#5799)

Maintenance

• Refactor maven downloading logic to be dynamic (#5826)

2025-04-24 Version 2.11.0

Breaking Changes

• The Docker task no longer runs as root. This may break existing Docker compose configurations. For example, mapping the .aws directory to the /root/.aws directory should map it to /.aws instead.

Features

• Support AWS Aurora/RDS PostgreSQL as source (#5309)
• Support SQS as a Source (#1049)
• Jira Connector - to seamlessly sync all the ticket details to OpenSearch (#4754)
• AWS Lambda as Processor and Sink (#4699)
• Support AWS Aurora/RDS MySQL as source (#4561)
• AWS Lambda as Sink (#4170)
• Integrate OpenSearch Ml-Commons into Data Prepper (#5509)

Enhancements

• Update OTel Protobuf specification (#5434)
• Data Prepper support for dynamic renaming of keys (#4849)
• S3 DLQ should pass expected bucket owner to PutObject request (#5498)
• AWS Secrets Manager Plugin does not support sts_header_overrides (#5475)
• Support configurable stream read constraints max length in the JSON input codec (#5466)
• Support reading S3 object metadata only (#5433)
• Kafka local AWS Glue registry support (#5377)
• Address Scale Items for Lambda Processor and Sink (#5031)
• Add support to skip remote peer forwarding based on configuration (#5127)
• Add index_types for OTel logs and metrics #3148 (#3929)
• Send RetryInfo on OTel timeouts to improve or clarify backpressure on OTel gRPC sources (#4294, #4119)
• Enhance Lambda processor to retry based on certain class of exception (#5340)
• Support multiple delete_when condition in delete_entries processor (#5315)
• Add additional index_types (#3148)
• Add experimental feature concept (#2695)

Bug Fixes

• NullPointerException on S3 Delete Event Due to Null Object Size (#5448)
• Index Template with flat_object type field fails during deserialization (#5425)
• DynamoDB source with acknowledgements expires frequently (#5412)
• Kinesis source doesn't pass the given polling retrieval config to underlying KCL (#5269)
• UTF-8 Character Encoding Issues in opensearchproject/data-prepper container (#5238)
• Validate that routes configured in the sink exist on startup of Data Prepper (#5106)
• Escaping of "/" in json pointers (#5101)
• DynamoDB Source doesn't support parsing data with Control Characters (#5027)

Security

• [docker] Run as non root (#5311)
• werkzeug-3.0.3-py3-none-any.whl: 2 vulnerabilities (highest severity is: 7.5) (#5122)

Maintenance

• Migrate existing plugins to use POJO configuration classes. (#5246)
• Support examples in documentation (#5077)
• End-to-end log test failing when index name has DateTime pattern suffix in OpenSearch Sink Config (#984)

2025-04-16 Version 2.10.3

Security

• Updates Parquet to 1.15.1. Fixes CVE-2025-30065. (#5597)

2024-12-09 Version 2.10.2

Bug Fixes

• FIX: missed exception in plugin error (#5105)

Security

• Fix otel_logs_source server configuration for getHttpAuthenticationService. Fixes CVE-2024-55886. (#5215)

2024-10-21 Version 2.10.1

Bug Fixes

• [BUG] Kinesis source is failing on startup (#5084)

2024-10-15 Version 2.10.0

Features

• Kafka source: support SASL/SCRAM mechanisms (#4241)
• OpenSearch Bulk API Source (#248)
• Support AWS Kinesis Data Streams as a Source (#1082)
• Support OpenTelemetry logs in S3 source (#5028)

Enhancements

• Use HTML in JsonPropertyDescription instead of Markdown (#4984)
• Variable drain time when shutting down via shutdown API (#4966)
• Make max connections and acquire timeout configurable on S3 sink client (#4949)
• Support BigDecimal data type in expressions (#4817)
• Caching implementation of EventKeyFactory (#4843)
• Json codec changes with specific json input codec config (#5054)

Bug Fixes

• [BUG] Close Opensearch RestHighLevelClient in OpenSearchClientRefresher on shutdown and initialization failure (#4770)

Security

• CVE-2024-6345 (High) detected in setuptools-68.0.0-py3-none-any.whl (#4940)
• CVE-2023-46136 (High) detected in Werkzeug-2.2.3-py3-none-any.whl (#4938)
• CVE-2024-34069 (High) detected in Werkzeug-2.2.3-py3-none-any.whl (#4938)
• CVE-2024-37891 (Medium) detected in urllib3-2.0.7-py3-none-any.whl (#4937)
• CVE-2024-35195 (Medium) detected in requests-2.31.0-py3-none-any.whl (#4939)
• CVE-2024-5569 (Low) detected in zipp-3.15.0-py3-none-any.whl ([#4936]#4936))

Maintenance

• Fixes and improvements for AbstractSinkTest (#5021)
• Update the test logging to include the failed assertion (#4987)

2024-08-28 Version 2.9.0

Features

• Support sets and set operations in Data Prepper expressions (#3854)
• Add startsWith expression function (#4840)
• Support default route option for Events that match no other route (#4615)
• Delete input for processors which expand the event (#3968)
• Dynamic Rule Detection (#4600)
• Kafka Source should support message headers (#4565)
• Aggregate processor : add option to allow raw events (#4598)
• Add support for start and end times in count and histogram aggregate actions (#4614)
• Add an option to count unique values of specified key(s) to CountAggregateAction (#4644)
• Flatten processor: option for keys wihout brackets (#4616)
• Modify Key Value processor to support string literal grouping (#4599)
• Make AWS credential management available in data-prepper-config.yaml (#2570)

Enhancements

• Support enhanced configuration of the Kafka source and buffer loggers (#4126)
• Update the rename_keys and delete_entries processors to use EventKey (#4636)
• Update the mutate string processors to use the EventKey. (#4649)
• OpenSearch Sink add support for sending pipeline parameter in BulkRequest (#4609)
• Add support for Kafka headers and timestamp in the Kafka Source (#4566)

Bug Fixes

• [BUG] Visibility duplication protection fails when using S3 source for large files and receiving 10 messages from SQS queue (#4812)
• [BUG] ChangeVisibilityTimeout call failure during pipeline shutdown. (#4575)
• [BUG] Service-map relationship should be created regardless of missing traceGroupName (#4821)
• [BUG] Unable to create stateful processors with multiple workers. (#4660)
• [BUG] Routes: regex doesn't work (#4763)
• [BUG] Grok plugin CLOUDFRONT_ACCESS_LOG pattern does not compile (#4604)
• [BUG] The user_agent processor throws exceptions with multiple threads. (#4618)
• [BUG] DynamoDB source export converts Numbers ending in 0 to scientific notation (#3840)
• Fix null document in DLQ object (#4814)
• Fix KeyValue Processor value grouping bug (#4606)

Security

• CVE-2024-6345 (High) detected in setuptools-68.0.0-py3-none-any.whl (#4738)
• CVE-2024-39689 (High) detected in certifi-2023.7.22-py3-none-any.whl (#4715)
• CVE-2024-5569 (Low) detected in zipp-3.15.0-py3-none-any.whl (#4714)
• CVE-2024-3651 (High) detected in idna-3.3-py3-none-any.whl (#4713)
• CVE-2024-35195 (Medium) detected in requests-2.31.0-py3-none-any.whl (#4562)
• CVE-2024-37891 (Medium) detected in urllib3-2.0.7-py3-none-any.whl (#4641)

Maintenance

• Removes Zookeeper from Data Prepper (#4707)
• Tools to generate User Agent strings in the performance-test project (#4620)