Commit f7d63e5

title: "Components of Observability"

Observability in LLM-based applications relies on a structured framework that captures execution details at different levels of granularity. Each request follows a well-defined path, where **individual operations are recorded, grouped into execution flows, and organized for broader analysis.** This structured approach enables teams to **track model performance, debug failures, and optimize system efficiency.**

A Span represents a single operation within an execution flow, recording input-output data, execution time, and errors. Each span provides insight into specific steps such as:

- LLM Calls – Capturing model invocation, prompt processing, and response generation.

- Retrieval Operations – Logging queries made to external databases or indexes.

- Tool Executions – Tracking API calls and function invocations.

- Error Handling – Recording failures, timeouts, and system issues.

Spans provide fine-grained visibility into each operation, allowing teams to identify where delays, errors, or inefficiencies originate.

A Trace connects multiple spans to represent the full execution flow of a request. It provides a structured view of how different operations interact within an LLM-powered system. Traces help teams:

- Analyze dependencies between retrieval, inference, and tool execution.

- Identify performance bottlenecks by measuring latency across spans.

- Debug unexpected behaviors by tracing execution paths from input to output.

For instance, a trace for an AI-driven search system may include:

1. A retrieval span fetching relevant documents.

2. An LLM span generating a response.

3. A tool execution span calling an external API.

By correlating these spans within a trace, teams can reconstruct the entire request flow, making it easier to analyze system behavior and optimize workflows.

A Project provides a structured way to manage multiple traces, ensuring observability is organized across different applications, use cases, or deployments. Projects allow teams to:

- Segment and categorize observability data for different LLM-powered applications.

- Compare model versions to track improvements in accuracy and performance.

- Filter and analyze execution trends across multiple traces.

For example, an organization might maintain separate projects for:

- Customer Support AI – Handling traces related to automated support queries.

- Content Generation AI – Managing traces for LLM-powered writing assistants.

- Legal AI Assistant – Tracking execution flows for contract analysis tasks.

By structuring observability in this way, teams can effectively monitor, compare, and optimize LLM-powered applications at scale.

title: "Understanding Observability"

As LLMs transition from experimentation to production, ensuring their reliability, fairness, and efficiency becomes critical. The Observe feature is designed to provide AI teams with real-time insights, evaluation metrics, and diagnostic tools to monitor and improve LLM-based applications.

This feature goes beyond simple monitoring, it enables teams to trace model behaviour, detect anomalies, measure AI performance, and diagnose issues such as hallucinations, inconsistencies, and inefficiencies.

By leveraging automated scoring, structured evaluation criteria, and historical trend analysis, Observe helps AI teams fine-tune LLM performance, debug failures, and optimize models for long-term reliability.

​

The Observe feature is built with five core objectives that help AI teams track, diagnose, and optimize LLM behaviour in production environments:

1. Real-Time Monitoring

Track LLM-generated responses, system telemetry, and model behaviour in live applications.

Visualise AI operations with structured trace logs and session analysis.

​

2. Ensuring Model Reliability

Detect unexpected hallucinations, misinformation, or irrelevant outputs.

Identify task completion failures and ambiguous AI responses.

​

3. Improving Model Accuracy & Alignment

Apply predefined evaluation templates to measure coherence, accuracy, and response quality.

Automate scoring based on performance benchmarks and structured criteria.

​

4. Accelerating Debugging & Problem-Solving

Pinpoint issues by analysing traces, sessions, and response deviations.

Use structured logs and failure patterns to diagnose and fix model inefficiencies.

​

5. Monitoring Bias & Fairness

Evaluate AI responses for ethical risks, safety concerns, and compliance adherence.

Apply bias-detection metrics to maintain responsible AI behaviour.

​

**1. LLM Tracing & Debugging**

Observability starts with LLM Tracing, which captures every input-output interaction, system response, and processing time in an LLM-based application.

- **Trace Identification** – Assigns a unique trace ID to every AI response for tracking and debugging.

- **Response Auditing** – Logs input queries, AI-generated responses, and execution times.

- **Error Detection** – Highlights failed completions, latency issues, and incomplete outputs.

​

**2. Session-Based Observability**

LLM applications often involve multi-turn interactions, making it essential to group related traces into sessions.

- **Session IDs** – Cluster multiple interactions within a single conversation or task execution.

- **Conversation Analysis** – Evaluate how AI performs across a sequence of exchanges.

- **Performance Trends** – Track how AI evolves within a session, ensuring consistency.

​

**3. Automated Evaluation & Scoring**

Observe provides structured evaluation criteria to score AI performance based on predefined metrics.

- **Evaluation Templates** – Predefined models for coherence, completeness, and user satisfaction.

- **Scoring System** – Uses quantitative metrics to assess response effectiveness.

- **Pass/Fail Flags** – Automatically detect responses that fall below a quality threshold.

- **Real-Time Evaluations** – Apply automated scoring to AI-generated responses as they occur.

- **Custom Criteria** – Define organization-specific evaluation metrics to tailor observability to unique use cases.

​

**4. Historical Trend Analysis**

Observability is not just about real-time monitoring—it also involves tracking model behaviour over time.

- **Performance Trends** – Compare past vs. present AI behaviour to measure improvement.

- **Cross-Model Comparisons** – Analyze different versions of an LLM to assess enhancements.

- **Statistical Insights** – Apply standard deviation, percentiles, and response distributions to detect long-term anomalies.

​

**5. Automated Issue Detection & Alerts**

To ensure AI systems remain functional, Observe enables automated issue detection and alerting.

- **Live Monitoring** – Observe token consumption, processing delays, and response failures in real time.

- **Threshold-Based Alerts** – Notify users if error rates or latency exceed safe limits.

- **Workflow Automation** – Automatically flag and log problematic interactions for further analysis.

By providing a comprehensive observability framework, Observe empowers AI teams to build more reliable, fair, and high-performing LLM applications in production environments.

title: "Spans"

Spans are the fundamental units of tracing in observability frameworks, providing structured, event-level data for monitoring, debugging, and performance analysis. A span represents a discrete operation executed within a system, capturing execution timing, hierarchical relationships, and metadata relevant to the operation’s context.

They are aggregated into traces, which collectively depict the flow of execution across various system components. This document provides an in-depth technical analysis of spans, their attributes, classifications, and their role in system observability.

A span consists of multiple attributes that encapsulate its execution details. These attributes can be categorized into the following sections:

- **Identification and context** provide the span's unique ID, trace ID, and optional parent span ID, establishing hierarchical relationships. It may also include a project reference for system-wide organization.

- **Execution details** define the operation recorded, including a descriptive name, span type (e.g., function call, API request, database query), and input/output data. If an operation fails, error metadata captures failure details like error codes, messages, and stack traces.

- **Timing and performance** track execution efficiency through start and end timestamps, latency measurement, and resource usage, such as computational cost or token consumption for LLM-related spans.

- **Metadata and custom attributes** provide additional context via tags, annotations, and JSON-based extensible fields. Execution environment details, including host machine, service instance, and deployment version, further enrich observability.

Spans are categorized based on the type of operation they capture. This classification ensures structured trace analysis and aids in performance monitoring.

- **Tool Spans**

It tracks operations executed by external tools or functions. It captures essential details, including the tool’s name, description, parameters, and performance metrics, enabling comprehensive monitoring of tool interactions.

- **Chain Spans**

It represents individual steps in a sequential workflow where data flows through multiple interconnected operations. It facilitates the visualization and analysis of execution pipelines, helping optimize process efficiency and detect bottlenecks.

- **LLM Spans**

It captures interactions with large language models, recording input prompts, generated completions, token usage, and invocation parameters. These spans provide insights into model performance, response times, and computational costs.

- **Retriever Spans**

It logs data retrieval operations, such as querying a database or fetching documents from an index. It stores search parameters and results, ensuring traceability and facilitating performance assessment of retrieval mechanisms.

- **Embedding Spans**

It tracks text-to-vector transformations used in machine learning applications. It records embedding vectors, associated model metadata, and processing details, supporting efficient monitoring of vectorization processes.

- **Agent Spans**

It documents actions performed by autonomous agents, including decision-making logic and tool interactions. It captures the rationale behind an agent’s choices, providing transparency into automated workflows and AI-driven decision processes.

- **Reranker Spans**

It logs result reordering or ranking adjustments based on specific scoring criteria. It retains input documents and their updated rankings, facilitating analysis of ranking models and relevance optimization.

- **Unknown Spans**

It serves as a fallback for operations that do not fit predefined span types. It ensures that all observed activities are recorded, even when their category is not explicitly defined.

- **Guardrail Spans**

It monitors compliance and enforce safety rules within a system. It captures validation results, applied policies, and compliance status, ensuring adherence to predefined operational constraints.

- **Evaluator Spans**

It represents assessment activities conducted to measure system performance or model effectiveness. It tracks evaluation metrics, scoring data, and feedback, supporting the continuous improvement of models and workflows.

title: "Observe LangChain Application"

This guide provides a detailed walkthrough of setting up and using Observe in your LangChain projects.

Install the FutureAGI package, which includes the Observe framework.

pip install futureagi

This package provides the essential tools for tracing, monitoring, and evaluating LangChain applications.

Set up your environment with the necessary API keys for OpenAI and FutureAGI. These credentials are crucial for:

- Accessing OpenAI's language models.

- Authenticating with FutureAGI's observability services.

- Securely transmitting telemetry data.

import os

os.environ["OPENAI_API_KEY"] = "your-openai-api-key"

os.environ["FI_API_KEY"] = "your-futureagi-api-key"

os.environ["FI_SECRET_KEY"] = "your-futureagi-secret-key"

The trace provider is the core component that enables observability.

- Creates a new project in the Observe platform

- Establishes a connection for sending telemetry data

- Sets up versioning for tracking changes over time

- Initializes OpenTelemetry integration for distributed tracing

from fi.integrations.otel import register

from fi.integrations.otel.types import ProjectType

trace_provider = register(

project_type=ProjectType.OBSERVE,

project_name="langchain_demo",

project_version_name="v1"

)

Configure automatic instrumentation for your LangChain components. The instrumentation:

- Automatically captures all LangChain operations

- Tracks prompt templates and their parameters

- Monitors model calls and responses

- Records timing information for each step

- Enables detailed tracing of the chain execution

from fi.integrations.otel import LangChainInstrumentor

from langchain_core.prompts import ChatPromptTemplate

from langchain_openai import ChatOpenAI

LangChainInstrumentor().instrument(tracer_provider=trace_provider)

prompt = ChatPromptTemplate.from_template("{x} {y} {z}?").partial(x="why is", z="blue")

chain = prompt | ChatOpenAI(model_name="gpt-3.5-turbo")

Run your chain with full observability enabled:

def run_chain():

try:

result = chain.invoke({"y": "sky"})

print(f"Response: {result}")

except Exception as e:

print(f"Error executing chain: {e}")

if __name__ == "__main__":

run_chain()

During execution:

- Each step of the chain is automatically traced

- Prompts and completions are logged

- Errors are captured and recorded

- Performance metrics are collected

- All data is sent to your Observe dashboard

With the instrumentation setup from SDK, all telemetry data flows into Observe's UI, where you can interact with real-time dashboards, apply filters, and analyze performance.

LLM Tracing is pivotal for obtaining granular insights into your application's interactions with language models. After adding a project to the observability platform, users can access a detailed view of trace information, which includes:

- **Node Type**: Specifies the type of operation (e.g., input handling, model inference).

- **Trace ID**: A unique identifier that allows you to track individual interactions.

- **Input/Output**: Displays the input provided to the model along with its output.

You can also view the evaluation metrics for your traces here. You can setup the evals from the [Evals & Tasks](#8-evals-and-tasks) section. These metrics quantify response quality, allowing you to apply evaluations that compute key statistics, such as count, average, and standard deviation—and visualize insights through interactive graphs.

Sessions enable to group and manage related tracing activities, providing comprehensive insights into the interaction history between users and AI. This includes:

- **Timeframe Filtering**: You can filter sessions by specific time intervals, making trend analysis straightforward.

- **Detailed Interaction Histories**: Access a chronological log of interactions within a session, crucial for understanding conversation flows in applications like chatbots.

- **Contextual Analysis**: View complete session data to analyze user behavior and model response patterns effectively.

This section automates model evaluation and monitoring, streamlining performance analysis by enabling you to define sampling rates, apply filters, set evaluation criteria, and manage tasks. By specifying the percentage of data to evaluate, you can optimize resource efficiency while ensuring meaningful insights.

[Click here to learn about all the evals provided by FutureAGI.]( /future-agi/products/evaluation/eval-definition/overview)

Filters allow you to narrow evaluations to specific data subsets based on custom criteria, providing a targeted approach to analysis. Evaluation criteria can be established to assess model performance comprehensively, incorporating key metrics that reflect accuracy, consistency, and relevance.

With features like detailed LLM Tracing, session-centric data analysis, and automated evaluation tasks, Observe empowers teams to maintain high-quality, reliable AI systems.

title: "Using Interface"