Skip to content

Help wanted: test API-first content workflows across nine CMS platforms #3115

Open

Description

@esokullu

We need help testing the nine CMS content workflows added in #3114 against real installations and real model runs. Automated coverage exercises packaged skills, permissions, synthetic HTTP fixtures and local browser modules; it does not establish that authentication, custom editors and publishing work on every live CMS.

Testing just one CMS, one browser and one workflow is useful. Please report both successes and failures, including cases where WebBrain correctly falls back to the editor because the API is unavailable.

Use a build from the latest main containing #3114, following the local installation instructions. Use a site/environment you control and disposable test content. See CMS API-first scope and limitations for supported APIs, authentication and fallback behavior. You do not need to buy a plan, install integrations or change account permissions to participate.

We are looking for reports on:

CMS Areas to check
Ghost Posts versus pages, Lexical/cards, tags/authors, draft updates and web-only publishing without sending a newsletter.
Drupal Existing node bundles, JSON:API availability, session/CSRF authentication, body formats, taxonomy, language and moderation.
Joomla Articles/categories, existing API-token access, intro/full-text preservation, language, checkout and workflow-controlled publication.
Webflow Collection fields/references, staged versus live items, CMS locale selection and per-item publication.
Shopify Articles/pages and existing blogs, Admin GraphQL access, hidden drafts, HTML/media/tags and publication state.
Wix Blog Draft-to-post identity, Rich Content, categories/tags, language and updating an existing published post through its linked draft.
Strapi Installed major version, Content API access, v5 document IDs, explicit draft status, locales, relations and components.
Contentful Space/environment selection, complete localized fields/metadata, version conflicts and publishing the intended entry.
Sanity Project/dataset/schema, draft/published identity, Portable Text/references, revision guards and Actions API publication.

Please try the applicable steps below. Unsupported features and unavailable API credentials are useful findings too.

  1. Create and verify a draft. Use an existing content type with representative formatted text, a link, existing media and existing taxonomy where supported. Confirm the saved record exists, remains unpublished and retains its native content structure.
  2. Edit the same record. Ask for a title-only or small body change. Confirm its ID, untouched content, relations and locale are preserved. If editing published content, verify that WebBrain does not silently unpublish it to create a draft.
  3. Publish one verified item, if you intend to make the test content public. Check both the CMS state and the public URL. Report differences caused by caching, a separate frontend or access restrictions. This workflow should not publish an entire site, deploy a frontend or send email.
  4. Check permissions and fallback. Ask mode must remain read-only, including blocking GraphQL/query POSTs. An otherwise usable API should request /allow-api only when needed and respect refusal. Missing credentials, insufficient rights or unsupported editor fields should lead to an appropriate UI fallback or a clear blocker. Note whether Strict Secret Handling is enabled.
  5. Check recovery when practical in a test environment. Report expired credentials, concurrent edits/version conflicts, partial publication or interrupted requests. WebBrain should reconcile the existing record before retrying and avoid duplicate drafts, repeated writes or false success claims.
  6. Check skill loading. In Mid/Full, the relevant full recipe should load on demand and clear on a new task; short catalog summaries remain available. Compact has no skill loader. Generic admin/Studio paths and Webflow marketing pages should not inject CMS adapter notes.

For each report, please include:

  • CMS name/version, hosted or self-hosted, and the relevant content type/workflow.
  • WebBrain version or commit, Chrome/Firefox version and OS.
  • Model/provider, prompt tier (Compact/Mid/Full), Ask/Act/Dev mode, Strict Secret Handling and API-permission state.
  • Authentication method and role/scope names if known — never the credentials themselves.
  • The exact task prompt, minimal reproduction steps, expected result and actual result.
  • Whether the API or UI route was used, what saved/published state you verified, and any visible error text or HTTP status.
  • Any site-specific constraints, such as localization, moderation, custom fields/editors or a separately deployed frontend.

Screenshots or a short screen recording would be especially helpful, if possible: show the initial editor state, WebBrain's relevant actions/permission or error messages, the saved draft or updated record, and the final publication state/public page. Redact tokens, cookies, personal data and private content. Raw traces can contain secrets; only share inspected, sanitized excerpts.

Please comment here with your results. For a reproducible bug, feel free to open a separate issue and link it here. We will use this issue to collect evidence across all nine CMS platforms and both browsers.

Activity

  1. esokullu commented on Sep 30, 2026

    @esokullu
    CollaboratorAuthor

    Coverage for this issue lives in docs/cms-api-first.md, which landed with #3114 and is in the current 38.0.0 build. The "Supported service contracts" table gives, per platform, the write API and content scope it targets plus the auth each one needs and where it stops: Ghost Admin posts and pages with updated_at collision checks, Drupal JSON:API with entity and workflow rights (cookie auth also needs the session CSRF token), Joomla Web Services v1 behind an existing X-Joomla-Token, Webflow Data API v2 with per-item publish, Shopify versioned GraphQL Admin where even read-only queries are gated POSTs, Wix Blog v3, Strapi v5 Content REST where status=draft is essential, Contentful CMA because Delivery and Preview keys cannot write, and Sanity Content Lake mutations with revision guards.

    The same page documents the parts that are easy to misread as supported when they aren't: no schema creation, no plugin or theme installs, no role changes, no commerce data, no newsletters, no whole-site publishing or front-end deploys. "The API supports it" and "this browser session can authenticate to it" are separate conditions, and Strict Secret Handling often forces the editor path because a REST nonce can reach the model conversation and raw traces.

    If a live test contradicts that table, treat the table as the defect and open a separate issue linked from here. A CMS version, plan, or custom editor that changes what is reachable is worth reporting even when the fallback behaved correctly, since the table should say so.

  2. molonlav3 commented on Oct 8, 2026

    @molonlav3

    One CMS, one workflow is a well-sized task. I run MusedIn, where agents take testing tasks and report what they saw. Agents there mostly can't hold a live CMS login, so this fits best if you supply a throwaway test site in the task. Success and failure reports both come back as public posts. Posting it takes a minute at https://musedin.com/post (or POST /api/job); say what done looks like, e.g. a log or a pass/fail. I can't promise anyone picks it up, and reports come back as public posts on the task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions