Skip to content

Add AI Request Logging experiment with observability dashboard#149

Draft
Jameswlepage wants to merge 9 commits intoWordPress:developfrom
Jameswlepage:feature/ai-request-logging
Draft

Add AI Request Logging experiment with observability dashboard#149
Jameswlepage wants to merge 9 commits intoWordPress:developfrom
Jameswlepage:feature/ai-request-logging

Conversation

@Jameswlepage
Copy link
Copy Markdown
Contributor

@Jameswlepage Jameswlepage commented Dec 19, 2025

ai-logs

What?

Adds a comprehensive AI Request Logging experiment that provides observability for all AI operations:

  • Real-time logging of every AI request (provider, model, tokens, duration, cost estimate)
  • React-powered admin dashboard under Settings → AI Request Logs
  • REST API endpoints for programmatic access
  • Configurable retention period and automatic cleanup

Why?

AI operations are opaque—users have no visibility into what requests are being made, how much they cost, or whether they're succeeding. This experiment provides the observability surface needed for debugging, cost tracking, and usage analysis.

How?

Architecture uses the decorator pattern to wrap the SDK's HTTP transporter:

  • Logging_Integration wraps the transporter on wp_loaded/admin_init using the public setHttpTransporter() API
  • Logging_Http_Transporter decorates requests, capturing timing and delegating to Log_Data_Extractor
  • Log_Data_Extractor parses request/response payloads with 4 filter hooks for extensibility:
    • ai_request_log_providers - customize provider detection
    • ai_request_log_context - filter context data
    • ai_request_log_tokens - custom token extraction
    • ai_request_log_kind - request type detection
  • AI_Request_Log_Manager coordinates schema, repository, and cost calculation
  • AI_Request_Cost_Calculator estimates costs based on provider pricing tables

Supports 14 providers out of the box: OpenAI, Anthropic, Google, Fal, Cloudflare, Groq, Grok, HuggingFace, DeepSeek, Ollama, OpenRouter, Azure, Cohere, and Mistral.

Testing Instructions

  1. Enable the AI Request Logging experiment in Settings → AI → Experiments
  2. Navigate to Settings → AI Request Logs and toggle "Logging enabled"
  3. Trigger an AI-powered feature (e.g., Title Generation, Excerpt Generation)
  4. Verify the log table populates with request details (provider, model, tokens, duration)
  5. Test filtering by provider, status, and date range
  6. Verify "Purge logs" clears all entries
  7. Disable the experiment and confirm no new logs appear
Open WordPress Playground Preview

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a comprehensive AI Request Logging experiment that provides observability for AI operations through real-time logging, a React-powered admin dashboard, REST API endpoints, and configurable retention settings. The implementation uses the decorator pattern to wrap the SDK's HTTP transporter and supports 14 AI providers out of the box.

Key changes:

  • Logging infrastructure with database schema, repository pattern, and cost estimation
  • React-based admin dashboard with DataViews for log viewing and filtering
  • REST API endpoints for programmatic access to logs and settings
  • Automatic cleanup based on retention period and row limits

Reviewed changes

Copilot reviewed 46 out of 47 changed files in this pull request and generated no comments.

Show a summary per file
File Description
webpack.config.js Adds webpack entry point for AI request logs admin bundle
src/admin/hooks/usePersistedView.ts Custom hook wrapping @wordpress/views for persisting DataViews state
src/admin/components/provider-icons.tsx Provider icon component mapping for displaying AI provider logos
src/admin/components/icons/* SVG icon components for various AI providers
src/admin/ai-request-logs/index.tsx Main React app component with state management and API integration
src/admin/ai-request-logs/components/* UI components for logs table, summary cards, settings panel, and modal
src/admin/ai-request-logs/style.scss Comprehensive styles for the admin dashboard
includes/helpers.php Helper functions for experiment checking, icon loading, and timeout configuration
includes/bootstrap.php Bootstrap integration for logging initialization
includes/Logging/* Core logging infrastructure (Manager, Schema, Repository, HTTP Transporter)
includes/Experiment_Loader.php Registers the AI Request Logging experiment
tests/Integration/Includes/Logging/* Integration tests for log manager functionality

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@sethrubenstein
Copy link
Copy Markdown

This is great. Building something like this was at the top of my to-do list for January, before I start developing more AI abilities for our staff.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeffpaul
Copy link
Copy Markdown
Member

@jeffpaul
Copy link
Copy Markdown
Member

  • Reminder for us to add a screenshot gif and entry in the readme.txt for this feature before merging/releasing.

@JasonTheAdams JasonTheAdams removed their request for review December 21, 2025 05:58
@jeffpaul jeffpaul moved this from Needs review to In discussion / Needs decision in WordPress AI Planning & Roadmap Jan 7, 2026
@jeffpaul jeffpaul modified the milestones: 0.2.0, 0.3.0 Jan 7, 2026
@jeffpaul
Copy link
Copy Markdown
Member

jeffpaul commented Feb 2, 2026

Trying to open PR for review to see if that'll trigger playground generation

@jeffpaul jeffpaul marked this pull request as ready for review February 2, 2026 18:20
@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 2, 2026

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: Jameswlepage <[email protected]>
Co-authored-by: jeffpaul <[email protected]>
Co-authored-by: sethrubenstein <[email protected]>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@jeffpaul jeffpaul marked this pull request as draft February 2, 2026 18:21
@jeffpaul jeffpaul added the props-bot Manually triggers Props Bot to ensure the list of props is up to date. label Feb 3, 2026
@jeffpaul
Copy link
Copy Markdown
Member

jeffpaul commented Feb 3, 2026

In working on a demo video to highlight this feature on make/ai, my first reaction is that the new screen should likely be under Tools and not Settings in wp-admin.

@jeffpaul jeffpaul modified the milestones: 0.3.0, 0.4.0 Feb 3, 2026
@jeffpaul jeffpaul modified the milestones: 0.4.0, Future Release Feb 4, 2026
@mathetos
Copy link
Copy Markdown

Adding the plugin/theme as a source of the API call would be really important here as well. It's one thing to see that an API use happened, it's another – especially in a common WP environment – to know how or where it was triggered and how to STOP it from continuing (thinking of excessive spend as a problem to solve).

@Jameswlepage Jameswlepage force-pushed the feature/ai-request-logging branch from 3731f8f to f93095e Compare March 25, 2026 19:52
@Jameswlepage
Copy link
Copy Markdown
Contributor Author

I force-pushed feature/ai-request-logging with a current-develop refresh of this PR.

This was handled as a port onto the current develop branch, not a conflict-only rebase. The original branch was too old relative to the plugin’s current architecture, so the work now sits on top of the current feature system and admin app structure.

What changed in this refresh:

  • moved the screen under Tools -> AI Request Logs
  • switched the feature to the current feature/loader setup
  • added the colored plugin icon in the header
  • updated provider metadata handling for the current admin/provider model
  • added request source attribution for plugin/theme/core callers in the HTTP logging layer
  • added integration coverage for feature registration, Tools menu wiring, REST access, and summary period handling
  • fixed the new admin app’s strict TypeScript issues and got npm run typecheck passing
  • aligned the summary API with the periods already supported by the repository (minute, hour, day, week, month, all)
  • removed the misleading request-kind filter wiring that was pointing at operation_pattern
  • removed dead request-log UI/backend surface that wasn’t actually wired (/logs/stats, exposed max_rows REST setting, unused tooltip component)
  • changed schema maintenance to a schema-version gate so normal requests no longer keep probing the table definition on every load
  • removed unused timer state from the logging manager
  • updated the experiment doc in docs/experiments/ai-request-logging.md

Verification on this branch:

  • composer lint
  • composer phpstan
  • npm run lint:js
  • npm run typecheck
  • npm run build

Manual smoke check:

  • opened the screen in a real browser via WordPress Playground on WP trunk (latest Playground was still on 6.9.4, while the plugin now requires 7.0)

Remaining limitation here:

  • the PHP test suite still needs the normal WordPress test environment. Direct local PHPUnit fails because the WP test constants/bootstrap are not present outside wp-env, and npm run test:php is still blocked until Docker/wp-env is available.

If reviewers want, I can follow this with a fresh screenshot/GIF of the updated screen now that the branch is current.

@Jameswlepage
Copy link
Copy Markdown
Contributor Author

CleanShot 2026-03-25 at 17 10 47@2x

@Jameswlepage
Copy link
Copy Markdown
Contributor Author

Jameswlepage commented Mar 25, 2026

Superseded by the next comment with the correctly formatted CI update.

@Jameswlepage
Copy link
Copy Markdown
Contributor Author

Pushed 666459e on top of the CI-fix pass.

This follow-up does two things:

  • adds direct-access guards to the new AI request logging PHP files so Plugin Check doesn’t hard-fail on them
  • updates the Example Experiment integration test to explicitly fire rest_api_init after feature registration, which matches the current REST server lifecycle and should unblock the PHP matrix that was returning 404 for /ai/v1/example

The new run is in progress. The fast checks already restarted cleanly, and dependency-review, PHPCS, and PHPStan are green on the new head.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 25, 2026

Codecov Report

❌ Patch coverage is 33.35488% with 1031 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.80%. Comparing base (32c5e2d) to head (12cfe83).

Files with missing lines Patch % Lines
includes/Logging/AI_Request_Log_Repository.php 28.67% 296 Missing ⚠️
includes/Logging/Log_Data_Extractor.php 20.46% 206 Missing ⚠️
includes/Admin/Provider_Metadata_Registry.php 0.00% 167 Missing ⚠️
includes/Logging/Logging_Http_Transporter.php 0.00% 125 Missing ⚠️
includes/Logging/AI_Request_Log_Page.php 19.11% 55 Missing ⚠️
includes/Logging/AI_Request_Log_Manager.php 40.00% 42 Missing ⚠️
...ncludes/Logging/REST/AI_Request_Log_Controller.php 84.58% 37 Missing ⚠️
...eriments/AI_Request_Logging/AI_Request_Logging.php 35.18% 35 Missing ⚠️
includes/Logging/AI_Request_Log_Schema.php 73.95% 25 Missing ⚠️
includes/Logging/AI_Request_Cost_Calculator.php 4.16% 23 Missing ⚠️
... and 2 more
Additional details and impacted files
@@              Coverage Diff              @@
##             develop     #149      +/-   ##
=============================================
- Coverage      57.85%   49.80%   -8.05%     
- Complexity       615     1020     +405     
=============================================
  Files             46       57      +11     
  Lines           3165     4712    +1547     
=============================================
+ Hits            1831     2347     +516     
- Misses          1334     2365    +1031     
Flag Coverage Δ
unit 49.80% <33.35%> (-8.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Jameswlepage
Copy link
Copy Markdown
Contributor Author

Jameswlepage commented Mar 26, 2026

Pushed follow-up fix at 073052f.

This tightens the request logs screen in two places:

  • The main logs table now constrains its width correctly, so it no longer pushes the right-hand settings sidebar off the page.
  • *:models discovery calls are now classified as metadata instead of falling through to text, and the Operations filter is now a persisted multi-select that excludes those model-discovery requests by default unless you opt in.

Also rebuilt the request logs assets and added a narrow regression test for the extractor classification path.

Local checks:

  • composer lint
  • npm run lint:js
  • npm run typecheck
  • npm run build

Jameswlepage and others added 2 commits March 26, 2026 15:17
- Replace hand-rolled table, filters, search, and pagination with
  @wordpress/dataviews for the Request Logs screen
- Bundle @wordpress/dataviews and @wordpress/ui (not registered in
  WP 7.0-RC1) via custom DependencyExtractionWebpackPlugin config
- Copy DataViews CSS into build/ and enqueue from PHP with fallback
  to wp-dataviews style handle when available
- Filter unregistered script dependencies in Asset_Loader to prevent
  wp_enqueue_script notices
- Expand AI_Request_Cost_Calculator from 3 providers / 25 models to
  8 providers / 50+ models (adds DeepSeek, Mistral, Cohere, Groq,
  Grok, Ollama; adds o-series reasoning models, Claude 4.5/4.6 IDs)
- Add Azure → OpenAI provider alias for cost lookup
- Return $0.00 for local providers (Ollama) instead of null
- Fix provider tooltip: click-to-toggle instead of hover so links
  are reachable
- Compact operation cell to max 2 rows (code + kind inline)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix PHPCS array arrow alignment in cost calculator (phpcbf auto-fix)
- Fix PHPStan: cast filemtime() to string for wp_enqueue_style version
- Fix ESLint a11y: add role, tabIndex, onKeyDown to provider cell
- Fix ESLint prettier: auto-formatted multiline expressions
- Fix ESLint import: suppress no-extraneous-dependencies for webpack
  build-tool requires (dependency-extraction-webpack-plugin,
  copy-webpack-plugin)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Copy link
Copy Markdown
Member

@jeffpaul jeffpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please left align the Log retention days portion with the rest of the text on the settings page
  2. Let's remove the Open the... text and link and then update the second description sentence to View detailed logs at Tools > AI Request Logs and link on the AI Request Logs text.
Screenshot 2026-03-27 at 10 35 00 AM
  1. Not seeing the logo on the upper left of the AI Request Logs page
Screenshot 2026-03-27 at 10 37 50 AM

@mateuswetah
Copy link
Copy Markdown

Could someone please provide me a working built version of this? I tried on Playground but it seems that ""https://github.com/WordPress/ai/releases/download/ci-artifacts/pr-149-12cfe8353575986d2fb62b26c0b4533c4c3f7546.zip" is not available.

I'm interesting on seeing how it behaves against a similar feature that I was implemented in a another plugin. We're using the CoreAI methods so I believe our requests should be logged there as well and this UI is awesome!

@jeffpaul
Copy link
Copy Markdown
Member

@mateuswetah you can pull a built ZIP via the Build Plugin ZIP action: https://github.com/WordPress/ai/actions/runs/23660730731

@mateuswetah
Copy link
Copy Markdown

Just did a few tests here and it is really nice!

image

I've got two questions:

  • I'm using the CoreAI client in auto mode for the model selection and I have the OpenAI connector configured. Since my request is routed to the OpenAI provider, the "Source" ends up being the ai-provider-for-openai (plugin). Would it be possible (and would it make sense) to have a way to identify that first the call came from my Plugin?
  • Is it just me or the filters aren't being clearable? I click the "Reset" button and the "x" in the filter tag and nothing happens :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

props-bot Manually triggers Props Bot to ensure the list of props is up to date.

Projects

Status: In discussion / Needs decision

Development

Successfully merging this pull request may close these issues.

6 participants