[Bug]: If managed user creation/alteration fails for whatever reason, postgres logs the user's password #9799

Activity

1. lgyurci

   assigned

   gbartolini

   on Jan 26, 2026
2. lgyurci

   added

   triagePending triagePending triage

   on Jan 26, 2026
3. dosubot commented on Jan 26, 2026

   

   dosubotBot

   on Jan 26, 2026 – with DosuBot

   Last edited by dosubot

   More actions

   This is a real security risk: CloudNativePG logs SQL statements from PostgreSQL exactly as they are executed, including any passwords in queries like ALTER ROLE ... PASSWORD ... if an error occurs. There is no built-in mechanism in CloudNativePG or its logging configuration to redact or suppress sensitive data from these logs, and the PostgreSQL log settings are fixed and cannot be changed by users in CloudNativePG. The operator collects logs in CSV format from PostgreSQL and outputs them in JSON, but does not filter or modify the query text—so any password in the SQL will appear in the logs if the query fails or is logged for any reason. This behavior is not documented or addressed in existing issues or documentation (reference, reference).

   Best practice is to strictly limit access to logs and avoid exposing them to anyone who should not see sensitive data. If you need to mitigate this, you may want to propose a feature to redact passwords in logs or investigate PostgreSQL-level options, but currently there is no workaround in CloudNativePG.

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
4. lgyurci commented on Jan 26, 2026

   

   lgyurci

   on Jan 26, 2026

   Author

   More actions

   @dosu bad bot, we have multiple good (or at least better) options here than "strictly limit access to logs", which is an absurd statement to make. Logs are usually stored in plain-text, so they will always be vulnerable to someone staring at my screen if I'm reading them. Random log parsers can also leak this password to unimaginable places, so the "best practice" definitely isn't "keeping logs safe". Instead, I would suggest:

   1. Use a prepared statement
   2. Changing the loglevel for the user alterating transaction, which in theory is possible, like

   BEGIN;
   SET LOCAL log_min_error_statement = 'PANIC'; 
   ALTER ROLE alice PASSWORD 'supersecret';
   COMMIT;
   

   3. Do the hashing beforehand, and only put the hash into the creation/alteration query, like

   ALTER ROLE alice PASSWORD 'SCRAM-SHA-256$4096:...:...:...';
   

   I'm by no means a postgresql expert, so the above solutions could carry other security implications that I'm not aware of, or could not be possible at all.
   Also, arguing with an AI on an online forum was not on my agenda today...

   👍React with 👍1Reacted by Pierrick
5. dosubot commented on Jan 26, 2026

   

   dosubotBot

   on Jan 26, 2026 – with DosuBot

   Last edited by dosubot

   More actions

   You're right that there are better ways to mitigate password leakage in logs than just restricting log access. Here’s how your suggestions map to what’s possible in CloudNativePG and PostgreSQL:

   • Prepared statements: CloudNativePG does not use prepared statements for DDL like ALTER ROLE—it sends raw SQL via the Go database/sql API with the pgx driver. In PostgreSQL, prepared statements are not supported for DDL commands (including ALTER ROLE), so this approach wouldn’t prevent passwords from appearing in logs and isn’t implementable in this context [code reference].

   • Changing log level with SET LOCAL log_min_error_statement: CloudNativePG does not currently wrap role management operations in a transaction that changes the log level, nor does it issue SET LOCAL commands before running DDL. While this is theoretically possible in PostgreSQL, the operator does not implement it, and there’s no evidence in the code or docs that this is supported or planned. It would require significant changes to how SQL is generated and executed [code reference].

   • Pre-hashing passwords: This is fully supported and is the only currently available mitigation. If you provide a pre-hashed password (MD5 or SCRAM-SHA-256) in the Kubernetes Secret, CloudNativePG will use the hash directly in the SQL, so the plaintext password never appears in the query or logs. This is documented and works as you described [docs].

   So, if you want to avoid password leakage in logs, the actionable solution right now is to always use pre-hashed passwords in your Secrets. The other options aren’t supported due to PostgreSQL and operator limitations. If you think log redaction or dynamic log level changes should be added, it would require a feature proposal and architectural changes.

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
6. ardentperf commented on Jan 31, 2026

   

   ardentperf

   on Jan 31, 2026

   Contributor

   More actions

   CNPG wrapping role management with log_min_error_statement is an interesting idea - this might work

   👍React with 👍1Reacted by Jaime Silvela
7. jsilvela commented on Feb 9, 2026

   

   jsilvela

   on Feb 9, 2026

   Last edited by jsilvela

   Contributor

   More actions

   This is a very good find, and a good bug report @lgyurci
   I think storing the pass in SCRAM-SHA will be the way to go. I'll try to get a PR for this out this week.
   EDIT: on reconsideration I too think log_min_error_statement is the better idea.

   👍React with 👍1Reacted by Lázár György
8. jsilvela commented on Feb 10, 2026

   

   jsilvela

   on Feb 10, 2026

   Contributor

   More actions

   For the same reason you outlined, this is a problem for the roles created as part of cluster bootstrap (app and perhaps the superuser), so I'm going to apply the same fix with log_min_error_statement.
   I have a PR going (linked above). I'm doing a couple of tests more, but I think it's almost ready.
9. jsilvela commented on Feb 11, 2026

   

   jsilvela

   on Feb 11, 2026

   Contributor

   More actions

   @lgyurci the PR is ready. I have a question about your issue report. The root cause seems to be the role operation was executed in a read-only transaction:

   cannot execute ALTER ROLE in a read-only transaction

   There are very few places in the codebase where a read-only transaction is used.
   What was going in your cluster at the time of this error?
   Do you have a repro for the issue?
10. jsilvela

    added

    security 👮

    on Feb 11, 2026
11. github-project-automation

    added this to Security Mapon Feb 11, 2026
12. github-project-automation

    moved this to Todo in Security Mapon Feb 11, 2026
13. lgyurci commented on Feb 11, 2026

    

    lgyurci

    on Feb 11, 2026

    Last edited by lgyurci

    Author

    More actions

    I even made another issue for it: #9800

    EDIT: thanks for your work so far!

    👍React with 👍1Reacted by Jaime Silvela
14. jsilvela commented on Feb 11, 2026

    

    jsilvela

    on Feb 11, 2026

    Contributor

    More actions

    Ah, right! I saw that other issue but had not made the connection.
    The fix in the PR would fix that other issue, but I think a separate fix might be in order to avoid executing the role reconciler in replica clusters. Ought to be a relatively easy fix, I'll see if I can find time soon.

    👍React with 👍1Reacted by Lázár György
15. jsilvela commented on Feb 11, 2026

    

    jsilvela

    on Feb 11, 2026

    Last edited by jsilvela

    Contributor

    More actions

    Direct verification of the fix on a replica cluster, duplicating your reported conditions:

    {"level":"info","ts":"2026-02-11T13:57:39.049732289Z",
      "logger":"postgres","msg":"record","logging_pod":"cluster-replica-tls-1",
      "record":{"log_time":"2026-02-11 13:57:39.049 UTC",
        "user_name":"postgres","database_name":"postgres","process_id":"156","connection_from":"[local]",
        "session_id":"698c8ad3.9c","session_line_num":"1",
        "command_tag":"ALTER ROLE","session_start_time":"2026-02-11 13:57:39 UTC",
        "virtual_transaction_id":"26/2","transaction_id":"0",
        "error_severity":"ERROR",
        "sql_state_code":"25006",
        "message":"cannot execute ALTER ROLE in a read-only transaction",
        "application_name":"cnpg-instance-manager","backend_type":"client backend",
        "query_id":"0"}
    }
16. armru

    assigned

    armru

    and

    mnencia

    on Feb 11, 2026
17. mnencia

    closed this as completedin #9950on Feb 21, 2026
18. github-project-automation

    moved this from Todo to Done in Security Mapon Feb 21, 2026
19. cnpg-bot

    added 3 commits that reference this issue on Feb 21, 2026

    fix(roles): prevent postgres from leaking role passwords in logs (#9950)

    ...

    This issue will close once commit ec52b0a is merged into the 'main' branch.

    ec52b0a

    fix(roles): prevent postgres from leaking role passwords in logs (#9950)

    ...

    This issue will close once commit 49f1cc3 is merged into the 'main' branch.

    49f1cc3

    fix(roles): prevent postgres from leaking role passwords in logs (#9950)

    ...

    This issue will close once commit dfca895 is merged into the 'main' branch.

    dfca895
20. pull

    added a commit that references this issue on Feb 21, 2026

    fix(roles): prevent postgres from leaking role passwords in logs (clo…

    ...

    This issue will close once commit 983bc6b is merged into the 'main' branch.

    983bc6b