Repository navigation
SDK docs: advanced ai-evaluation pages - #520
Merged
Merged
Conversation
- FrameworkEvaluator: blocking, non-blocking, distributed execution modes - Backends: ThreadPool, Celery, Ray, Temporal (with config examples) - Resilience: circuit breakers, rate limiting, retries, presets - Custom evaluations: @custom_eval decorator, simple_eval, EvalBuilder - All code tested against ai-evaluation SDK - Nav: placed after Running Evaluations, before Metrics Reference
- Auto-generate evaluation pipelines from app descriptions - 7 pre-built templates (customer_support, rag_system, code_assistant, content_moderation, agent_workflow, healthcare, financial) - Pipeline customization: add/remove/set_threshold/enable/disable - Export to YAML/JSON for CI/CD - AppAnalyzer with LLM-powered classification - All code tested against ai-evaluation SDK
Pipeline: architect review → humanizer → 5-persona → code tested Content: - 13 guard models (API + local) with type/speed/auth table - Multi-model voting with 4 aggregation strategies - Screening methods: input/output/retrieval + async + batch with examples - screen_retrieval example for RAG chunk filtering - 14 local scanners with descriptions - ScannerPipeline: composition, parallel, fail-fast, default factory - Scanner configs: TopicRestriction, Language, Regex constructors - GuardrailsConfig: 7 fields documented with defaults - ScannerConfig: 15 fields documented with defaults and thresholds - SafetyCategory: per-category actions (block/flag/redact/warn) - fail_open behavior explained, timeout_ms scope clarified Review fixes: - Removed unused asyncio import from async example - Added FI_API_KEY comment to quick example - Added screen_retrieval code example for RAG users
New sections: - Gateway: GuardrailsGateway factory methods (with_openai, with_local_model, with_ensemble, auto), screening sessions (sync + async context managers) - Backend Discovery: discover_backends(), get_backend_details() - Local Model Setup: HuggingFace IDs, VRAM requirements, env vars for VLLM/Ollama for all 9 local models (LlamaGuard, WildGuard, ShieldGemma, Granite Guardian, Qwen3Guard) - Scanner Result Types: ScanResult (8 fields), ScanMatch (6 fields), PipelineResult (6 fields) — complete field reference - Writing Custom Backends: BaseBackend extension pattern with example Page now covers 100% of the Guardrails module API surface.
TopicRestrictionScanner kwargs, RegexScanner needs RegexPattern objects, SafetyCategory requires name arg, Gateway imports, VRAM figures, model count.
setup_tracing(), auto-instrumentation, auto-enrichment, cost tracking, span processors, 13+ exporter backends. All imports verified.
LocalEvaluator, HybridEvaluator, OllamaLLM, metric registry, routing logic. All code tested, 6 review issues fixed.
Code security: 15 AST detectors, 4 eval modes, judges, benchmarks. Architect + humanizer reviewed, 10 issues fixed.
Added cards and routing table for distributed, autoeval, guardrails, local/hybrid, otel, code security. Fixed icons to use valid icon map.
- Add FITracer custom spans section (decorators, lambdas, convenience methods) - Add FISpan methods reference table with mime_type params - Add EvalTags section (Python, TypeScript, C#) - Add PII redaction details (6 detected types) - Add suppress tracing (Python, C#) - Expand register() to 4-language tabs with full param tables - Expand context helpers to 4-language tabs - Expand TraceConfig to 4 tabs including TypeScript per-instrumentor pattern - Expand cleanup section with Java/C# shutdown - Add Java Spring Boot application.yml config - Add Java Traced* wrapper pattern for 25+ integrations - Fix EvalTagType.OBSERVATION_SPAN_TYPE -> OBSERVATION_SPAN - Fix TypeScript batch default (false not true) - Fix broken /docs/tracing/auto-overview links -> /docs/tracing/auto - Add UNKNOWN span kind to table
hadarishav
approved these changes
Mar 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All code snippets tested against
ai-evaluation==1.0.1. Each page went through docs-architect review with fixes applied.Test plan