Repository navigation
fix(targetTime): clarify timestamp formats and UTC interpretation - #8937
Conversation
|
❗ By default, the pull request is configured to backport to all release branches.
|
|
Was looking at this, and have a concern. The code still says RFC3339. See type RecoveryTarget struct {
<------->
// The target time as a timestamp in the RFC3339 standard
// +optional
TargetTime string `json:"targetTime,omitempty"`And in the func ConvertToPostgresFormat(timestamp string) string {So, if the documentation is indeed wrong, and only timestamps are to be allowed, shouldn't the code change too? |
jsilvela
left a comment
There was a problem hiding this comment.
I think the issue with the targetTime in the document is the missing Z at the end.
The targetTime does admit RFC3339, RFC3339Micro, and timestamp.
That said, the code for handling is a bit convoluted and could use some updating.
| @@ -351,7 +351,7 @@ spec: | |||
| kind: VolumeSnapshot | |||
| apiGroup: snapshot.storage.k8s.io | |||
| recoveryTarget: | |||
| targetTime: "2023-07-06T08:00:39" | |||
| targetTime: "2023-06-11 08:00:39.00000+02" | |||
There was a problem hiding this comment.
Been digging a bit in the code.
The problem with the existing targetTime in the doc, "2023-07-06T08:00:39", is that it's not valid RFC3339. To have a zero timezone offset, you need a final "Z".
This worked in a unit test.
recoveryTarget := RecoveryTarget{
TargetTime: "2023-07-06T08:00:39Z",
}
Expect(recoveryTarget.BuildPostgresOptions()).To(ContainSubstring("recovery_target_time = '2023-07-06 08:00:39.000000Z'"))
| @@ -385,8 +385,7 @@ recovery target. | |||
| Here are the recovery target criteria you can use: | |||
|
|
|||
| targetTime | |||
| : Time stamp up to which recovery proceeds, expressed in | |||
| [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) format. | |||
| : Time stamp up to which recovery proceeds, expressed in the form of a timestamp. | |||
There was a problem hiding this comment.
Forgot to add in the review. Timestamps are admitted, definitely, but so is RFC3339, see my previous comment on the missing Z.
There was a problem hiding this comment.
See test below :)
|
And, since @mnencia approved previously: I'd like your take on this. The webhook uses func ParseTargetTime at BUT, in func ConvertToPostgresFormat Why do we have the special format allowed in ParseTargetTime? |
|
After your comments @jsilvela, I tried again an unit test to confirm there is an issue, event with Z at the end of timestamp. First create a cluster with backup section using plugin : ---
apiVersion: v1
kind: Secret
metadata:
name: minio-creds
type: Opaque
data:
ACCESS_KEY_ID: bWluaW9hZG1pbg==
ACCESS_REGION: ZnItcGFy
ACCESS_SECRET_KEY: bWluaW9hZG1pbg==
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: cluster-alice
spec:
imageName: ghcr.io/cloudnative-pg/postgresql:17.5
instances: 1
storage:
size: 1Gi
# Backup properties
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: minio-store
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: minio-store
spec:
configuration:
destinationPath: s3://demo/pierrick
endpointURL: http://172.18.0.3:9000
s3Credentials:
accessKeyId:
name: minio-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: minio-creds
key: ACCESS_SECRET_KEY
region:
name: minio-creds
key: ACCESS_REGION
Create backup Create a cluster with bootstrap restore method apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: cluster-bob
spec:
imageName: ghcr.io/cloudnative-pg/postgresql:17.5
instances: 1
storage:
size: 1Gi
bootstrap:
recovery:
source: backup-cluster-alice
targetTime: "2025-10-27T16:30:10Z"
externalClusters:
- name: backup-cluster-alice
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: minio-store
serverName: cluster-aliceThe bob cluster start with a recovery Pod : {
"level": "info",
"ts": "2025-10-27T16:32:59.56105952Z",
"msg": "Starting webserver",
"logging_pod": "cluster-bob-1-full-recovery",
"address": "localhost:8010",
"hasTLS": false
}
... but get this error : {
"level": "info",
"ts": "2025-10-27T16:33:03.221008034Z",
"msg": "Starting up instance",
"logging_pod": "cluster-bob-1-full-recovery",
"pgdata": "/var/lib/postgresql/data/pgdata",
"options": [
"start",
"-w",
"-D",
"/var/lib/postgresql/data/pgdata",
"-o",
"-c port=5432 -c unix_socket_directories=/controller/run",
"-t 40000000",
"-o",
"-c listen_addresses='127.0.0.1'"
]
}
{
"level": "info",
"ts": "2025-10-27T16:33:03.238768322Z",
"logger": "pg_ctl",
"msg": "waiting for server to start....2025-10-27 16:33:03.238 UTC [46] LOG: invalid value for parameter \"recovery_target_time\": \"2025-10-27 16:30:10.000000Z\"",
"pipe": "stdout",
"logging_pod": "cluster-bob-1-full-recovery"
}
{
"level": "info",
"ts": "2025-10-27T16:33:03.238788644Z",
"logger": "pg_ctl",
"msg": "2025-10-27 16:33:03.238 UTC [46] FATAL: configuration file \"/var/lib/postgresql/data/pgdata/custom.conf\" contains errors",
"pipe": "stdout",
"logging_pod": "cluster-bob-1-full-recovery"
}
{
"level": "info",
"ts": "2025-10-27T16:33:03.33110249Z",
"logger": "pg_ctl",
"msg": "pg_ctl: could not start server",
"pipe": "stderr",
"logging_pod": "cluster-bob-1-full-recovery"
}
|
|
@pchovelon huh! The conversion from RFC3339 to postgres was partial. I was expecting full or pass-through, from the code ... I'll try to zero on this tomorrow. |
|
OK, @pchovelon , pretty stumped. And, the string you show is a valid Postgres timestamp: So, may be a specific issue with |
|
According to postgres, if it's valid as a
|
|
Howdy @pchovelon . I've made some experiments.
If the RFC3339 timestamp input in the manifest is instead converted to use Your doc fix, IMO, while relevant, should be changed:
I've opened cloudnative-pg/machinery#167 to fix the conversion function. |
Great, thank you @jsilvela . Running after time here ... I try to answer asap. |
|
/test |
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20577814711 |
Signed-off-by: Pierrick Chovelon <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]> Signed-off-by: Pierrick <[email protected]>
Update the machinery dependency to include the fix that ensures RFC3339-like timestamps without timezone (e.g., "2023-07-06T08:00:39") are interpreted as UTC. The documentation has been updated to clarify that RFC 3339 timestamps without an explicit timezone suffix are interpreted as UTC. Related: cloudnative-pg/machinery#167 Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
All the changes requested by Jaime have been implemented.
Update documentation and API reference to clarify that targetTime accepts both RFC3339 and PostgreSQL timestamp formats. Document that timestamps without an explicit timezone are interpreted as UTC. Add link to PostgreSQL documentation for recovery_target_time and include a warning recommending users to always specify an explicit timezone to avoid ambiguity. Signed-off-by: Marco Nenciarini <[email protected]>
) Update the machinery dependency to v0.3.3 to ensure timestamps without timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC. The documentation has been updated to clarify that targetTime accepts both RFC 3339 and PostgreSQL timestamp formats. Closes #8928 Related: cloudnative-pg/machinery#167 Signed-off-by: Pierrick Chovelon <[email protected]> Signed-off-by: Pierrick <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Jaime Silvela <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 1a64853)
) Update the machinery dependency to v0.3.3 to ensure timestamps without timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC. The documentation has been updated to clarify that targetTime accepts both RFC 3339 and PostgreSQL timestamp formats. Closes #8928 Related: cloudnative-pg/machinery#167 Signed-off-by: Pierrick Chovelon <[email protected]> Signed-off-by: Pierrick <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Jaime Silvela <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 1a64853)
) Update the machinery dependency to v0.3.3 to ensure timestamps without timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC. The documentation has been updated to clarify that targetTime accepts both RFC 3339 and PostgreSQL timestamp formats. Closes #8928 Related: cloudnative-pg/machinery#167 Signed-off-by: Pierrick Chovelon <[email protected]> Signed-off-by: Pierrick <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Co-authored-by: Jaime Silvela <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 1a64853)
…39 format (#8981) Exercises the machinery fix (cloudnative-pg/machinery#168) that allows RFC3339 timestamps in recovery_target_time. The existing PITR snapshot test is extended to also restore with an RFC3339-formatted targetTime, ensuring both formats work end-to-end. Ref: #8937, cloudnative-pg/machinery#167 Signed-off-by: Jaime Silvela <[email protected]>
…39 format (#8981)%0A%0AExercises the machinery fix (cloudnative-pg/machinery#168) that allows%0ARFC3339 timestamps in recovery_target_time. The existing PITR snapshot%0Atest is extended to also restore with an RFC3339-formatted targetTime,%0Aensuring both formats work end-to-end.%0A%0ARef: cloudnative-pg/cloudnative-pg#8937: Jaime Silvela <[email protected]> cloudnative-pg/cloudnative-pg@refs/heads/main
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1) **Release date:** Feb 5, 2026 ##### Enhancements - Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 --> ##### Fixes - Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 --> - Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 --> - Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397)) - Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 --> - Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 --> - The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 --> - Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 --> - Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 --> - Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 --> - Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 --> - Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 --> - Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 --> - Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 --> - Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 --> - Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 --> - Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 --> - Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 --> - `cnpg` plugin: - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 --> - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
##### [\`v1.28.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.28.1) **Release date:** Feb 5, 2026 ##### Enhancements - Added support for Azure's `DefaultAzureCredential` authentication mechanism for backup and recovery operations. This can be enabled by setting `azureCredentials.useDefaultAzureCredentials: true` in the backup configuration, simplifying authentication in Azure environments without requiring explicit storage account keys or SAS tokens. ([#9468](cloudnative-pg/cloudnative-pg#9468)) <!-- 1.27 1.25 --> ##### Fixes - Fixed validation of PostgreSQL extension names containing underscores (e.g., `pg_partman`, `pg_ivm`). Extension names with underscores are automatically sanitized to use hyphens for Kubernetes volume names while preserving the original name in mount paths. Webhook validation prevents naming conflicts after sanitization. Contributed by [@shusaan](https://github.com/shusaan). ([#9386](cloudnative-pg/cloudnative-pg#9386)) <!-- 1.27 --> - Fixed a critical issue where the `TimelineID` in the cluster status was not reset to 1 after a major version upgrade. Because `pg_upgrade` initializes a new timeline, keeping the old ID (e.g., timeline 2) caused replicas to attempt to restore incompatible history files from object storage, leading to fatal "requested timeline is not a child of this server's history" errors. ([#9830](cloudnative-pg/cloudnative-pg#9830)) <!-- 1.27 --> - Fixed an issue where stale TLS status fields in the `Pooler` were not cleared after being removed from the specification. This was particularly critical when upgrading to v1.28.0, where the `ServerTLS` field was repurposed, causing PgBouncer to use incorrect certificates and resulting in "unsupported certificate" errors that blocked all application connectivity. The operator now explicitly clears `ServerCA`, `ClientCA`, `ClientTLS`, and `ServerTLS` status fields when they are no longer configured. ([#9397](cloudnative-pg/cloudnative-pg#9397)) - Fixed a bug where replicas could enter a crash-loop by attempting to download timeline history files from future timelines. This occurred when stale files remained in the WAL archive from a previous cluster life, and replicas would incorrectly try to fetch them during recovery. ([#9650](cloudnative-pg/cloudnative-pg#9650)) <!-- 1.27 1.25 --> - Fixed a race condition in `replica_cluster` setups during designated primary transitions, preventing transient "no primary" states in the replica cluster. ([#9601](cloudnative-pg/cloudnative-pg#9601)) <!-- 1.27 1.25 --> - The backup controller now uses the unique instance session ID to detect instance manager restarts. This prevents the operator from incorrectly assuming a backup is still progressing if the underlying container has crashed and restarted, which previously led to orphaned backup objects. ([#9370](cloudnative-pg/cloudnative-pg#9370)) <!-- 1.27 --> - Fixed a validation gap in Azure object store configurations where the `storageAccount` was not required when using explicit credentials (such as a storage key or SAS token). The operator now enforces that a storage account name is provided in these cases and that `connectionString` is mutually exclusive with other authentication parameters. ([#9604](cloudnative-pg/cloudnative-pg#9604)) <!-- 1.27 1.25 --> - Optimized the deletion path so the operator begins cleaning up resources immediately when a cluster is marked for deletion. This significantly reduces the time a cluster remains in `Terminating` status while waiting for internal reconciliation loops. ([#9555](cloudnative-pg/cloudnative-pg#9555)) <!-- 1.27 1.25 --> - Fixed an issue where replication slots were not properly dropped from replicas when the feature was disabled or the cluster was reconfigured. This ensures that unused slots do not cause WAL build-up on the primary. ([#9381](cloudnative-pg/cloudnative-pg#9381)) <!-- 1.27 1.25 --> - Fixed an issue where `imagePullSecrets` were not added to the `ServiceAccount` created for the `Pooler`. Previously, these secrets were applied to the Deployment but not the SA, which caused image pull failures in restricted environments using certain security policies. ([#9427](cloudnative-pg/cloudnative-pg#9427)) <!-- 1.27 1.25 --> - Added a check to verify ownership before the operator deletes a `PodMonitor`. This prevents the operator from accidentally deleting manually managed monitoring resources that happen to share a name with expected CNPG resources. Contributed by [@juliamertz](https://github.com/juliamertz). ([#9340](cloudnative-pg/cloudnative-pg#9340)) <!-- 1.27 1.25 --> - Fixed a bug where `pg_stat_archiver` metrics would continue to report stale data on standby instances after a switchover. The exporter now skips these metrics on standbys, as PostgreSQL only provides valid archiver stats on the primary. ([#9411](cloudnative-pg/cloudnative-pg#9411)) <!-- 1.27 1.25 --> - Clarified the interpretation of timestamp formats for recovery `targetTime`. Timestamps provided without an explicit timezone are now consistently interpreted as UTC. Contributed by [@pchovelon](https://github.com/pchovelon). ([#8937](cloudnative-pg/cloudnative-pg#8937)) <!-- 1.27 1.25 --> - Fixed backup status updates to prevent "resource has been modified" errors during concurrent updates. ([#9551](cloudnative-pg/cloudnative-pg#9551)) <!-- 1.27 1.25 --> - Fixed event reporting to use the correct pod name when a backup pod is not found. ([#9552](cloudnative-pg/cloudnative-pg#9552)) <!-- 1.27 1.25 --> - Improved performance of scheduled backup operations for clusters with a very high number of historical backups. ([#9489](cloudnative-pg/cloudnative-pg#9489)) <!-- 1.27 1.25 --> - Fixed error handling when removing finalizers on `Database` objects. ([#9431](cloudnative-pg/cloudnative-pg#9431)) <!-- 1.27 1.25 --> - `cnpg` plugin: - Updated the `status` command to display "Disabled" when the `skipWalArchiving` annotation is present on a cluster. This replaces confusing "starting up" or "unknown" states when WAL archiving is intentionally bypassed. ([#9709](cloudnative-pg/cloudnative-pg#9709)) <!-- 1.27 1.25 --> - Fixed the `logs --follow` command to continue polling for new pods instead of exiting prematurely when all current log streams complete. ([#9599](cloudnative-pg/cloudnative-pg#9599)) <!-- 1.27 1.25 -->
Update the machinery dependency to v0.3.3 to ensure timestamps without timezone (both RFC3339 and PostgreSQL formats) are interpreted as UTC.
The documentation has been updated to clarify that targetTime accepts both RFC 3339 and PostgreSQL timestamp formats.
Closes #8928
Related: cloudnative-pg/machinery#167