Skip to content

Structured error responses for action validation #50

@polaz

Description

@polaz

Summary

Add structured error responses with tier-aware smart error handling that helps LLMs self-correct and provides actionable guidance for users on restricted GitLab tiers.

Problem

  1. Validation errors are unstructured - When LLM provides incorrect parameters, error messages don't help the model self-correct
  2. Tier restrictions are opaque - 403 errors don't explain WHY access is denied or what alternatives exist
  3. No actionable guidance - Users don't know how to fix issues or work around limitations

Solution

Part 1: Structured Validation Errors

interface ActionValidationError {
  error_code: "MISSING_REQUIRED_FIELD" | "INVALID_ACTION" | "FIELD_NOT_ALLOWED" | "TYPE_MISMATCH";
  tool: string;
  action: string;
  missing_fields?: string[];
  invalid_fields?: Array<{
    field: string;
    expected: string;
    received: string;
  }>;
  suggested_fix: string;
  valid_actions?: string[];
  action_required_fields?: Record<string, string[]>;
}

// Example
{
  "error_code": "MISSING_REQUIRED_FIELD",
  "tool": "manage_merge_request",
  "action": "create",
  "missing_fields": ["project_id", "source_branch", "target_branch", "title"],
  "suggested_fix": "Add required fields: project_id, source_branch, target_branch, title",
  "action_required_fields": {
    "create": ["project_id", "source_branch", "target_branch", "title"],
    "update": ["project_id", "merge_request_iid"],
    "merge": ["project_id", "merge_request_iid"]
  }
}

Part 2: Tier-Aware Smart Error Handling (NEW)

Transform opaque 403/404 errors into helpful, actionable responses:

interface TierRestrictedError {
  error_code: "TIER_RESTRICTED" | "FEATURE_UNAVAILABLE" | "PERMISSION_DENIED";
  tool: string;
  action: string;
  http_status: number;

  // Tier information
  tier_required: "Free" | "Premium" | "Ultimate";
  current_tier?: string;  // If detectable
  feature_name: string;

  // Actionable guidance
  message: string;
  alternatives?: Array<{
    action: string;
    description: string;
    available_on: string;
  }>;
  docs_url?: string;
  upgrade_url?: string;
}

Example: Protected Branches (Premium feature)

// Request
{ "action": "list", "tool": "browse_protected_branches", "project_id": "123" }

// Response (instead of raw 403)
{
  "error_code": "TIER_RESTRICTED",
  "tool": "browse_protected_branches",
  "action": "list",
  "http_status": 403,

  "tier_required": "Premium",
  "feature_name": "Protected Branches API",

  "message": "Protected Branches API requires GitLab Premium or Ultimate tier",
  "alternatives": [
    {
      "action": "Use branch protection rules in UI",
      "description": "Configure via Settings > Repository > Protected branches",
      "available_on": "Free"
    },
    {
      "action": "Check branch via browse_commits",
      "description": "List commits to verify branch exists and recent activity",
      "available_on": "Free"
    }
  ],
  "docs_url": "https://docs.gitlab.com/ee/api/protected_branches.html",
  "upgrade_url": "https://about.gitlab.com/pricing/"
}

Example: Code Owners (Ultimate feature)

{
  "error_code": "TIER_RESTRICTED",
  "tool": "browse_code_owners",
  "action": "get",
  "http_status": 403,

  "tier_required": "Ultimate",
  "feature_name": "Code Owners",

  "message": "Code Owners requires GitLab Ultimate tier",
  "alternatives": [
    {
      "action": "Check CODEOWNERS file directly",
      "description": "Use browse_files to read .gitlab/CODEOWNERS or CODEOWNERS file",
      "available_on": "Free"
    },
    {
      "action": "Use merge request approvers",
      "description": "Configure approvers in MR settings (Premium)",
      "available_on": "Premium"
    }
  ],
  "docs_url": "https://docs.gitlab.com/ee/user/project/codeowners/"
}

Example: Pipeline not found (permission issue)

{
  "error_code": "PERMISSION_DENIED",
  "tool": "browse_pipelines",
  "action": "get",
  "http_status": 404,

  "message": "Pipeline not found or you don't have access",
  "alternatives": [
    {
      "action": "Check project access",
      "description": "Verify you have at least Reporter access to the project",
      "available_on": "Free"
    },
    {
      "action": "List available pipelines",
      "description": "Use action 'list' to see pipelines you can access",
      "available_on": "Free"
    }
  ],
  "suggested_fix": "Verify project_id and pipeline_id are correct, and you have Reporter+ access"
}

Implementation

Tier Feature Map

// src/utils/tier-features.ts

interface TierFeature {
  name: string;
  tier: "Free" | "Premium" | "Ultimate";
  tools: string[];
  alternatives?: string[];
  docs: string;
}

export const TIER_FEATURES: Record<string, TierFeature> = {
  // Premium Features
  protected_branches_api: {
    name: "Protected Branches API",
    tier: "Premium",
    tools: ["browse_protected_branches", "manage_protected_branch"],
    alternatives: ["Configure via UI", "Use browse_commits to check branch"],
    docs: "https://docs.gitlab.com/ee/api/protected_branches.html"
  },
  merge_request_approvals: {
    name: "Merge Request Approvals",
    tier: "Premium",
    tools: ["manage_merge_request:approve"],
    alternatives: ["Use comments for informal approval"],
    docs: "https://docs.gitlab.com/ee/user/project/merge_requests/approvals/"
  },
  group_webhooks: {
    name: "Group Webhooks",
    tier: "Premium",
    tools: ["list_webhooks:group", "manage_webhook:group"],
    alternatives: ["Use project-level webhooks (Free)"],
    docs: "https://docs.gitlab.com/ee/user/project/integrations/webhooks.html"
  },

  // Ultimate Features
  code_owners: {
    name: "Code Owners",
    tier: "Ultimate",
    tools: ["browse_code_owners"],
    alternatives: ["Read CODEOWNERS file via browse_files"],
    docs: "https://docs.gitlab.com/ee/user/project/codeowners/"
  },
  security_dashboard: {
    name: "Security Dashboard",
    tier: "Ultimate",
    tools: ["browse_vulnerabilities"],
    docs: "https://docs.gitlab.com/ee/user/application_security/security_dashboard/"
  },
  epic_boards: {
    name: "Epic Boards",
    tier: "Ultimate",
    tools: ["browse_epics:board"],
    alternatives: ["Use issue boards (Free)", "List epics without board view"],
    docs: "https://docs.gitlab.com/ee/user/group/epics/"
  }
};

Error Handler

// src/utils/error-handler.ts

import { TIER_FEATURES } from './tier-features';

interface GitLabError {
  status: number;
  message?: string;
  error?: string;
}

export function handleGitLabError(
  error: GitLabError,
  tool: string,
  action: string
): StructuredError {

  // Check if this is a tier-restricted feature
  const tierFeature = findTierFeature(tool, action);

  if (error.status === 403 && tierFeature) {
    return {
      error_code: "TIER_RESTRICTED",
      tool,
      action,
      http_status: 403,
      tier_required: tierFeature.tier,
      feature_name: tierFeature.name,
      message: `${tierFeature.name} requires GitLab ${tierFeature.tier} or higher`,
      alternatives: tierFeature.alternatives?.map(alt => ({
        action: alt,
        description: getAlternativeDescription(alt),
        available_on: "Free"
      })),
      docs_url: tierFeature.docs,
      upgrade_url: "https://about.gitlab.com/pricing/"
    };
  }

  if (error.status === 403) {
    return {
      error_code: "PERMISSION_DENIED",
      tool,
      action,
      http_status: 403,
      message: "You don't have permission for this action",
      suggested_fix: "Check your access level for this project/group",
      alternatives: [
        {
          action: "Verify access",
          description: "Use get_users to check your role in the project",
          available_on: "Free"
        }
      ]
    };
  }

  if (error.status === 404) {
    return {
      error_code: "NOT_FOUND",
      tool,
      action,
      http_status: 404,
      message: "Resource not found or you don't have access",
      suggested_fix: "Verify the ID/path is correct and you have at least Reporter access"
    };
  }

  // Generic error
  return {
    error_code: "API_ERROR",
    tool,
    action,
    http_status: error.status,
    message: error.message || error.error || "Unknown error",
    suggested_fix: "Check the GitLab API documentation for this endpoint"
  };
}

Integration with Tool Handlers

// In registry handlers
try {
  return await gitlab.get(path, options);
} catch (error) {
  if (error instanceof GitLabApiError) {
    throw new StructuredToolError(
      handleGitLabError(error, toolName, action)
    );
  }
  throw error;
}

Tier Badge Integration

Leverage existing tier badges from list-tools --env-gates:

// Already have in EnhancedToolDefinition
interface EnhancedToolDefinition {
  // ...
  tier?: "Free" | "Premium" | "Ultimate";
  gate?: { envVar: string; defaultValue: boolean };
}

// Use tier info in error handling
function findTierFeature(tool: string, action: string): TierFeature | null {
  const toolDef = getToolDefinition(tool);
  if (toolDef?.tier && toolDef.tier !== "Free") {
    return {
      name: toolDef.description.split('.')[0],
      tier: toolDef.tier,
      tools: [tool],
      docs: `https://docs.gitlab.com/ee/api/`
    };
  }
  // Check TIER_FEATURES map
  return Object.values(TIER_FEATURES).find(f =>
    f.tools.some(t => t === tool || t === `${tool}:${action}`)
  ) || null;
}

Example Output for LLM

When agent calls a Premium feature on Free tier:

Agent: browse_protected_branches({ project_id: "myproject" })

Response:
{
  "error": true,
  "error_code": "TIER_RESTRICTED",
  "tier_required": "Premium",
  "feature_name": "Protected Branches API",
  "message": "Protected Branches API requires GitLab Premium or higher",

  "alternatives": [
    {
      "action": "Configure via Settings > Repository > Protected branches",
      "available_on": "Free (UI only)"
    },
    {
      "action": "Use browse_commits to check branch activity",
      "available_on": "Free"
    }
  ],

  "docs_url": "https://docs.gitlab.com/ee/api/protected_branches.html",

  "suggested_fix": "Either upgrade to GitLab Premium, or use one of the alternatives above"
}

LLM can then:

  1. Inform user about the tier limitation
  2. Suggest alternatives
  3. Try alternative approach automatically

Tasks

Part 1: Structured Validation Errors

  • Define ActionValidationError interface
  • Create validateActionParams() helper
  • Return structured errors from Zod validation failures
  • Include action_required_fields in error response
  • Add suggested_fix generation

Part 2: Tier-Aware Error Handling

  • Create TIER_FEATURES map with Premium/Ultimate features
  • Implement handleGitLabError() function
  • Map 403 errors to tier-specific messages
  • Add alternatives for restricted features
  • Include docs URLs in error responses
  • Integrate with existing tier badges

Part 3: Integration

  • Wrap all tool handlers with error handler
  • Test with Free tier account
  • Test with Premium tier account
  • Verify LLM can parse and act on errors

Part 4: Documentation

  • Document error codes
  • Add troubleshooting section for tier errors
  • Include tier requirements in tool reference

Acceptance Criteria

  • Validation errors include error_code and suggested_fix
  • 403 errors on Premium features explain tier requirement
  • Alternatives are provided for restricted features
  • LLMs can parse error responses and self-correct
  • Error messages are actionable for humans
  • Docs URLs point to correct GitLab documentation

Priority

MEDIUM - Improves UX significantly, builds on existing tier badges

Dependencies

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions