[Feature][Audit] Add query_audit_always_on to keep query audit from being disabled at runtime #70122

Background

Some deployments must audit every query: compliance rules require a record of each statement in fe.audit.log and in __internal_schema.audit_log, and require that the database's administrators cannot quietly switch that off. Doris treats ADMIN as trusted, and today query audit is something ADMIN, and partly any user, can turn off or weaken at runtime.

Current behavior

• SET GLOBAL enable_audit_plugin = false stops the audit table from receiving events.
• ADMIN SET FRONTEND CONFIG ("audit_event_log_queue_size" = "0"), or /api/_set_config with persist=true, drops every event for both fe.audit.log and the table.
• audit_plugin_max_sql_length = 0 reduces each recorded statement to a truncation marker, and audit_plugin_max_insert_stmt_length = 0 drops even the table name of INSERT ... VALUES. A negative audit_plugin_max_sql_length drops every event except INSERT ... VALUES, because the truncation throws.
• The batching variables (audit_plugin_max_batch_bytes, audit_plugin_max_batch_interval_sec), audit_plugin_load_timeout and the dispatch timeouts (query_audit_log_timeout_ms, be_report_query_statistics_timeout_ms, workload_runtime_status_thread_interval_ms) can hold events in memory or make loads drop them.
• enable_prepared_stmt_audit_log is a session variable, off by default, so any user's server-side prepared executions (COM_STMT_EXECUTE) leave no record.
• In fe.conf, skip_audit_user_list drops the events of the users in it, and an audit_log_modules without query stops fe.audit.log from recording queries.

Expected behavior

An operator can enable a mode in which no database account, root included, can turn query audit off or hollow it out through SQL, the FE HTTP config interface or session settings; only someone who can edit the FE's configuration files and restart it can.

Proposed design

A new FE config query_audit_always_on, default false, not runtime-mutable. While it is on, every statement the FE receives is audited (failed statements and server-side COM_STMT_EXECUTE included) to both sinks, and no protected parameter may be weaker than its factory default: the switches stay on, a limit whose smaller values drop events has its default as the minimum, a wait whose larger values hold events back has it as the maximum, and the batching knobs are fixed.

• QueryAuditPolicy (fe-common) defines the rules once. Readers use the effective value of each variable, since old values also come from the image, the edit log, forwarded sessions and existing sessions.
• SET, SET_VAR, UNSET and runtime config changes reject a weaker value with an error, so the attempt fails and is audited itself; UNSET VARIABLE ALL keeps such a variable instead of failing.
• At startup, conflicting values from fe.conf and fe_custom.conf are replaced in memory with a WARN (skip_audit_user_list is ignored, query is added to audit_log_modules, out-of-bound configs get their bound) instead of refusing to start.
• A master that takes over rewrites persisted non-compliant global values through the global variable edit log, so SHOW VARIABLES agrees on every FE.
• A prepared execution forwarded to the master is audited only by the FE that received it, as COM_QUERY is.

Alternatives considered: making only enable_audit_plugin read-only, which leaves the other knobs and persisted values in place; enforcing on read only, under which a rejected change appears to succeed and leaves no record; and smaller floors such as 4096 bytes for the SQL length, for which there is no objective threshold.

Impact

Users who need audit that administrators cannot turn off, in both shared-nothing and cloud mode. FE only. With the switch off nothing changes, except that a negative audit_plugin_max_sql_length counts as 0 and is rejected by SET.

Reproduction or evidence

SET GLOBAL enable_audit_plugin = false;
ADMIN SET FRONTEND CONFIG ("audit_event_log_queue_size" = "0");
SET GLOBAL audit_plugin_max_sql_length = -1;  -- drops every event except INSERT ... VALUES

Scope and non-goals

In scope: the parameters above, and audit of failed prepared executions. Out of scope: code execution inside the FE (INSTALL PLUGIN, JDBC driver_url), tampering with records already written (for example TRUNCATE on the audit table), reliable delivery (a full queue, a failed load or an FE crash can still lose events), enable_bdbje_debug_mode, load and slow-query audit, and external audit plugins.

Acceptance criteria

• With the switch on, each change above is rejected with an error naming query_audit_always_on, and the attempt appears in the audit table as a failed statement.
• Every COM_STMT_EXECUTE is audited once, also when forwarded to the master.
• Conflicting values in the config files are overridden at startup with a WARN.
• ADMIN SET FRONTEND CONFIG ("query_audit_always_on" = "false") fails with is not mutable.
• With the switch off, behavior is unchanged except the negative-length fix.

Risks and compatibility

No metadata, image, Thrift or Protobuf change; the master's rewrite uses the existing global variable edit log. Each FE enforces its own setting, so every FE must be configured. With the switch on, clients that set enable_prepared_stmt_audit_log = false get an error, prepared executions raise audit volume, and clusters that had shortened the recorded statements record up to 2 MB per statement.

Related

• The threat model (threat-model.md) gains a property for this mode (§4.8 property 13, M20), pending maintainer agreement.
• [Enhancement](audit log) Add print audit log sesssion variable #38419 and [Enhancement](audit log) Set print audit log sesssion variable default value to false #38865 added enable_prepared_stmt_audit_log, [feature](audit) add new FE config to skip audit for certain user #38310 added skip_audit_user_list.