the ./flux copy command has an option --output-abort-on-write-failure that is always set to true even if it is not specified in the execution of the command.
This has the effect of any failed batches (maybe due to an intermittent 500 err being returned by the MarkLogic http endpoint) resulting in the copy command aborting even if the option is not specified.
From looking at the code in the Spark connector referencing this option, there is an option for failed batches to be retried and ultimately written to an archive specified by the --output-failed-documents-path option on retry exhaustion, it's just that this branch is never executable with the ./flux copy command.
Investigating the source code seems to suggest that this ternary operator needs to have the false branch set to "false" rather than null so that where it is referenced in the MarkLogic Spark connector can trigger retry/write to failed document archive behavior.
Fixing this will be helpful in situations where there is a batch failure towards the end of a ./flux copy of a large amount of documents. If 10 million documents are being copied and there is a failure in a batch processing the 9 millionth document, the entire copy command aborts and we need to attempt to copy all 10 million documents again.
If abort on batch failure is set to false and we specify a --output-failed-documents-path , the copy can complete and any failed batch documents in the --output-failed-documents-path can be retried independently and we avoid re copying 9 million documents.
the ./flux copy command has an option --output-abort-on-write-failure that is always set to true even if it is not specified in the execution of the command.
This has the effect of any failed batches (maybe due to an intermittent 500 err being returned by the MarkLogic http endpoint) resulting in the copy command aborting even if the option is not specified.
From looking at the code in the Spark connector referencing this option, there is an option for failed batches to be retried and ultimately written to an archive specified by the --output-failed-documents-path option on retry exhaustion, it's just that this branch is never executable with the ./flux copy command.
Investigating the source code seems to suggest that this ternary operator needs to have the false branch set to "false" rather than null so that where it is referenced in the MarkLogic Spark connector can trigger retry/write to failed document archive behavior.
Fixing this will be helpful in situations where there is a batch failure towards the end of a ./flux copy of a large amount of documents. If 10 million documents are being copied and there is a failure in a batch processing the 9 millionth document, the entire copy command aborts and we need to attempt to copy all 10 million documents again.
If abort on batch failure is set to false and we specify a --output-failed-documents-path , the copy can complete and any failed batch documents in the --output-failed-documents-path can be retried independently and we avoid re copying 9 million documents.