Skip to content

mariadb backup: BackupJob and RestoreJob end Succeeded when the operator Job failed #5195

Closed

Description

@myasnikovdaniil

Describe the bug

When the Job behind a mariadb-operator Backup or Restore fails, the MariaDB driver ends the BackupJob or RestoreJob as Succeeded. For a backup it also creates a Cozystack Backup in phase Ready that points to storage with no archive in it.

The driver maps Complete=True to success and handles failure only as Complete=False with reason JobFailed. Operator never emits that combination, when the Job has Failed=True it sets Complete to True with reason JobFailed and message Failed (SetCompleteWithJob in mariadb-operator 25.10.2). The comment in mariadbtypes/types.go (JobFailed -> Status=False) has it backwards.

Environment

Cozystack v1.7.0, mariadb-operator 25.10.2. The driver code is the same on main (4087218).

To reproduce

  1. Create a MariaDB app and run a BackupJob with cozy-default. It succeeds and <namespace>/<app>/backup.<timestamp>.gzip.sql appears in the bucket.
  2. Copy the strategy cozy-default-mariadb, change only spec.template.storage.s3.bucket to a bucket that does not exist, bind it from a new BackupClass.
  3. Run a BackupJob with that class.

Actual behaviour

Pod crash-loops with error pushing target backup ... Access Denied., the Job fails with BackoffLimitExceeded after about 3 minutes and the operator Backup shows

conditions:
- type: Complete
  status: "True"
  reason: JobFailed
  message: Failed

BackupJob goes to Succeeded with Ready=True, reason BackupCompleted. The Backup is created Ready, takenAt is the failure time and status.artifact.uri is under the missing bucket. Same when the dump itself fails (databases: [no_such_db] in the strategy gives mariadb-dump: Got error: 1049: "Unknown database"), so it is not only storage errors. No event and nothing in the driver log.

RestoreJob has the same shape: an operator Restore with Complete=True and reason JobFailed gives RestoreJob Succeeded, reason RestoreCompleted. A Plan on a schedule keeps producing "successful" backups with nothing behind them.

Expected behaviour

BackupJob ends Failed with the cause in the message and no Backup is created. RestoreJob ends Failed too.

The code is in internal/backupcontroller/mariadbstrategy_controller.go, lines 207 and 234 for backup and 573 and 588 for restore. Success branch never looks at cond.Reason and failure branch needs Status == False. I think Complete=True with JobFailed should be the terminal failure and success only for JobComplete.

PR #5015 edits the same function but not this mapping.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    on Nov 4, 2026
  2. self-assigned this
    on Nov 4, 2026
  3. scooby87 commented on Nov 12, 2026

    @scooby87
    Contributor

    Fixed by #5207.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugtriage/needs-triageIndicates an issue needs triage by a maintainer

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions