CoRB 2.6.0 Release Notes

Overview

CoRB 2.6.0 includes significant new features for job management, enhanced export capabilities, improved security, and a modernized web interface. This release adds support for restartable jobs, JSON file processing, export file splitting, and an interactive options builder UI.

Major New Features

Restartable Jobs (#65)

CoRB jobs can now be restarted from the point of interruption, avoiding reprocessing of already-completed URIs.

• New Option: RESTARTABLE - Enable restart capability by persisting job state locally
• New Option: RESTART-STATE-DIR - Configure where restart state files are stored (defaults to TEMP-DIR)
• Provides best-effort at-least-once restart semantics
• Restart files are automatically cleaned up after successful job completion
• URIs that failed processing are marked as processed and skipped on restart

Usage:

RESTARTABLE=true
RESTART-STATE-DIR=/path/to/restart/state

JSON File Loaders (#96)

New loader classes enable processing JSON files similar to existing XML file loaders.

• New Classes:
  • FileUrisJSONLoader - DOM-based JSON file processing
  • FileUrisStreamingJSONLoader - Memory-efficient streaming JSON processing
• New Options:
  • JSON-FILE - Path to JSON file for URI loading
  • JSON-NODE - XPath expression to select JSON values (default: /*)
  • JSON-METADATA - XPath to metadata that will be passed to modules
  • JSON-TEMP-DIR - Temporary directory for streaming loader
  • METADATA-TO-PROCESS-MODULE - Pass metadata to process module
  • LOADER-BASE64-ENCODE - Base64 encode file content in envelope

Example:

URIS-LOADER=com.marklogic.developer.corb.FileUrisStreamingJSONLoader
JSON-FILE=/path/to/data.json
JSON-NODE=/items/*
LOADER-USE-ENVELOPE=true

Export File Splitting (#287)

Export files can now be automatically split into multiple files based on size or line count thresholds.

• New Options:
  • EXPORT-FILE-SPLIT-MAX-LINES - Maximum lines per file before splitting
  • EXPORT-FILE-SPLIT-MAX-SIZE - Maximum file size before splitting (supports units: B, KB, MB, GB, TB)
  • EXPORT-FILE-AS-ZIP - Compress output files as ZIP
• Split files use numeric suffixes (e.g., export.txt, 001_export.txt, 002_export.txt)
• Automatic configuration of PreBatchUpdateFileTask and PostBatchUpdateFileTask when needed
• Supports EXPORT-FILE-TOP-CONTENT and EXPORT-FILE-BOTTOM-CONTENT with splitting

Example:

EXPORT-FILE-NAME=report.csv
EXPORT-FILE-SPLIT-MAX-SIZE=100MB
EXPORT-FILE-AS-ZIP=true

Options Builder Web UI

New interactive web interface for building CoRB job configurations.

• Build job configurations through a guided web form
• Download generated .properties file
• Launch jobs directly from the browser
• Organized by functional groups: Connection, Security, Input, Processing, Execution, Export, Monitoring, Retry, Advanced
• Search and filter options
• Field validation and help text
• Support for custom/additional properties

Access: Navigate to http://localhost:<port> on whatever port or range you specified when the Job Server is running and select "Options Builder" tab.

Enhanced Jasypt Support

Upgraded Jasypt decrypter with support for initialization vectors and configurable encrypters.

• New Options:
  • JASYPT-PROPERTIES-FILE - Path to Jasypt properties file (default: jasypt.properties)
  • JASYPT-STRING-ENCRYPTER - Custom encrypter class (default: StandardPBEStringEncryptor)
  • JASYPT-IV-GENERATOR - IV generator class (requires Jasypt 1.9.3+)
• Support for modern encryption algorithms with IV generators
• Examples: RandomIvGenerator, StringFixedIvGenerator

Example:

DECRYPTER=com.marklogic.developer.corb.JasyptDecrypter
JASYPT-IV-GENERATOR=org.jasypt.iv.RandomIvGenerator
JASYPT-STRING-ENCRYPTER=org.jasypt.encryption.pbe.StandardPBEStringEncryptor

Configurable Host Key Decrypter Cipher

HostKeyDecrypter now supports custom cipher configuration.

• New Option: HOST-KEY-DECRYPTER-CIPHER - Specify cipher transformation (default: AES)
• Supports simple algorithm names (e.g., AES) or full transformation strings (e.g., AES/GCM/NoPadding)
• Enables use of modern encryption modes like GCM

Example:

DECRYPTER=com.marklogic.developer.corb.HostKeyDecrypter
HOST-KEY-DECRYPTER-CIPHER=AES/GCM/NoPadding

Enhancements

Web UI Modernization

Complete rewrite of the Job Server dashboard using modern web technologies:

• Replaced AngularJS + Bootstrap + jQuery with AlpineJS (lightweight 15KB framework)

Test Framework Improvements

• Upgraded to JUnit 5.14.3
• Upgraded to Mockito 4.11.0
• Refactored test architecture to avoid System.exit() calls
• Improved test isolation and maintainability
• Added comprehensive test coverage for new features

Security Improvements

This release includes extensive security hardening based on security scan findings:

XML External Entity (XXE) Protection

• Disabled DTD and external entity resolution in XML parsers
• Applied secure XML processing configurations across all XML/XPath operations
• Protection against XXE injection attacks

Resource Management

• Ensured proper closing of streams and resources using try-with-resources
• Fixed potential resource leaks
• Thread-safe mutations and synchronization improvements

Secure Defaults

• Standardized character encoding to UTF-8 across the application
• Enhanced null safety checks throughout codebase
• Protected finalize() methods to prevent exploitation via constructor exceptions
• Serialization security improvements

Input Validation

• Enhanced parameter validation in Job Server endpoints
• Path traversal attack prevention
• Restricted file access patterns

Dependency Updates

Runtime Dependencies

• MarkLogic XCC: 11.3.1 → 12.0.0
• AlpineJS: 3.15.8 (new dependency, replaces AngularJS/Bootstrap/jQuery)
• Jasypt: Examples updated to 1.9.3

Build Dependencies

• Gradle: 8.14.4
• JUnit: 5.14.3
• Mockito: 4.11.0
• JetBrains Annotations: 26.1.0
• External Sorting in Java: Updated to latest version

Bug Fixes and Improvements

Code Quality

• Comprehensive Javadoc documentation added across all classes
• Better handling of edge cases and null values

Task Auto-Configuration

Automatic task configuration for common patterns:

• Auto-configure PreBatchUpdateFileTask when EXPORT-FILE-TOP-CONTENT is specified with ExportBatchToFileTask
• Auto-configure PostBatchUpdateFileTask when using:
  • EXPORT-FILE-BOTTOM-CONTENT
  • EXPORT-FILE-SPLIT-MAX-LINES
  • EXPORT-FILE-SPLIT-MAX-SIZE
  • EXPORT-FILE-AS-ZIP
  • EXPORT-FILE-SORT with ExportBatchToFileTask

Job Server Improvements

• Better handling of job lifecycle and context management
• Improved metrics collection and reporting
• Enhanced error handling and user feedback
• Support for running jobs directly from the web UI

Platform Compatibility

• Improved Windows compatibility for file handling and tests
• Better UTF-8 character encoding support
• Cross-platform path handling improvements

Breaking Changes

None. This release maintains backward compatibility with 2.5.x configurations.

Note: The EXPORT-FILE-AS-ZIP option now uses dash-separated naming for consistency. The underscore-separated variant is deprecated but still supported for backward compatibility.

• issue #253 If the value of EXPORT-FILE-NAME is empty or blank and no filename specified then it should not auto-magically enable the ExportBatchToFileTask.
• issue #256 FileUrisStreamingXMLLoader is dropping namespaces when documents use a default namespace (no prefix) running in Java 11+.

• OPTIONS_FILE assumes ISO 8859-1 encoding #239
  • use the system encoding by default and provide the ability to override with OPTIONS-FILE-ENCODING option.
• Add support for OAuth and MarkLogic Cloud authentication #243
• Provide an SSLConfig for standard SSL/TLS and option for custom TrustStore #245

What's Changed

• Bump com.marklogic.ml-gradle from 4.5.3 to 4.6.0 by @dependabot in #226
• Bump com.marklogic:marklogic-xcc from 11.0.3 to 11.1.0 by @dependabot in #233
• Catch IllegalArgumentException when attempting to decrypt unencrypted values by @hansenmc in #235
• Usage improvement by @hansenmc in #236

Full Changelog: v2.5.5...v2.5.6

What's Changed

• Renew content source by @hansenmc in #228
• Add usage options and move usage info before exceptions for errors on Manager.init() options by @hansenmc in #231
• Require PROCESS-MODULE to be configured for Export*ToFile by default by @hansenmc in #212
• Remove JAXB dependency to make Java upgrades and compatibility easier by @hansenmc in #214
• Auto-configure TwoWaySSL by @hansenmc in #217
• Expose execError with hasExecError() method by @hansenmc in #218
• Upgrade to GitHub-native Dependabot by @dependabot-preview in #158
• Bump org.sonarqube from 3.3 to 3.4.0.2513 by @dependabot in #190
• Bump com.github.kt3k.coveralls from 2.12.0 to 2.12.2 by @dependabot in #211
• Bump org.sonarqube from 3.4.0.2513 to 4.0.0.2929 by @dependabot in #207
• Bump com.marklogic:marklogic-xcc from 11.0.2 to 11.0.3 by @dependabot in #223
• Bump com.marklogic.ml-gradle from 4.5.2 to 4.5.3 by @dependabot in #224

Full Changelog: v2.5.4...v2.5.5

• #182 Add an explicit UTF-8 to getBytes when retrieving XdmItem, rather than rely upon OS defaults. This helps avoid issues with Windows clients and avoid having to explicitly set the character encoding with a system variable.
• #184 Introduced a delay in the shutdown of the Job Server, in order for the UI to have time to collect stats after a job has completed. Also added the exitCode to the metrics and display in the UI when a job completes.
• #186 Instantiate a new ContentSource object on error and replace if the resolved IP is different. This can help avoid issues with proxies or DNS with dynamic IP ranges when the original IP is decommissioned. This behavior is enabled by default, but can be disabled using the CONTENT-SOURCE-RENEW boolean option.
• #187 Evaluate shouldRetry() first, in order to allow for query retry for ServerConnectionException if message matches QUERY-RETRY-ERROR-MESSAGE

The following issues were addressed in this release:

• #97 Optionally return a different exit code when fail-on-error=false and there are errors. Allows for a different exit code to be configured with a new option EXIT-CODE-IGNORED-ERRORS which will be returned when there are errors and FAIL-ON-ERROR=false
• #149 SchemaValidateBatchToFileTask not appending content, overwriting with last item
• #161 Better diagnostics for easily detectable developer errors in QueryUrisLoader
• #171 Corb runs selector even when the job is started with COMMAND=STOP
• #173 Option UseConcMarkSweepGC deprecated. Changed to UseG1GC and a note to consider DISK-QUEUE to control memory consumption.
• #177 QueryUrisLoader uses Integer.parseInt to obtain total count. Changed to use Long.parseLong()
• Upgraded Angular.js and JQuery libraries, gradlew to 7.3.2, and various build plugins.
• Corrected an issue with shadowJar and the publishing process so that it does not append "-all" to jar filename. This should resolve marklogic/ml-gradle#597

• Ensure that custom inputs with empty string values are retained and set in modules
• Expose an option XML-SCHEMA-HONOUR-ALL-SCHEMALOCATIONS to control setting https://xerces.apache.org/xerces2-j/features.html#honour-all-schemaLocations, default enabled.

• #148
  • Change xcc.httpcompliant to be enabled by default
  • Consolidate logic for checking System properties and CoRB options and looking for normalized keys in both Snake_Case and Kebab-Case in a convenience method Options.findOption.
  • Ensure that MAX-OPTS-FROM-MODULE works if specified as original name MAX_OPTS_FROM_MODULE
• #149 Add the option to append content to resolve issue with schemaValidateBatchToFileTask overwriting and not appending content
• #151 Add examples to demonstrate how to construct and execute CoRB jobs

• #126 Error thrown when trying to test if password is URL encoded
• #130 Automatically use ExportBatchToFileTask when setting EXPORT-FILE-NAME
• #140 Update SSL config to use TLSv1.2