dMCP

Opportunistically Discover and Utilize Remote MCP Servers in Your Agents

Start Up:

Simply run:

docker compose up

This spins up 3 containers: Client, Website, and MCP Server -- each with its own startup command.

Dynamic MCP Discovery Event Flow

This document describes the complete event flow of a dynamic Model Context Protocol (MCP) agent that discovers and integrates new tools at runtime by detecting MCP servers through HTTP headers.

Architecture Overview

The system consists of three components:

1. Client Agent - A LangChain-based agent that can dynamically discover and integrate MCP servers
2. MCP Server - Provides job search and application tools via the MCP protocol
3. Web Server - Serves job listing pages with MCP discovery headers

Event Flow

Phase 1: Initial Agent Invocation

The user initiates a job search:

User Query: "i want to get a job at TechCorp. Please look for jobs online. 
I am a software engineer, open to anything. What jobs can I apply for?"

The agent starts with two base tools:

• bing-search-tool - Mock web search functionality
• web-crawler-tool - Extracts content and HTTP headers from URLs

Step 1: Web Search

The agent calls bing-search-tool with the query:

{'query': 'TechCorp jobs TechCorp careers "TechCorp" software engineer jobs...'}

This returns two results:

• http://website:8000/first-page - Software Engineer job posting
• http://website:8000/second-page - Job market article

Step 2: Web Crawling & MCP Discovery

The agent calls web-crawler-tool on the first URL:

{'url': 'http://website:8000/first-page'}

The crawler extracts:

• Page content (job requirements, benefits, etc.)
• HTTP headers, including the critical mcp_url header

Found MCP URL in headers: http://mcp:8000/mcp
Discovered new MCP server: http://mcp:8000/mcp

The web server's /first-page endpoint includes this custom header:

response.headers["mcp_url"] = "http://mcp:8000/mcp"

Phase 2: Agent Reconstruction with New Tools

Once the MCP server is discovered, the system:

1. Registers the MCP server in the client's server configuration
2. Recreates the agent with expanded toolset
3. Loads MCP tools from the newly discovered server

New MCP Tools Discovered! Recreating Agent
Loaded 2 tools from 1 MCP server(s)
  - search_jobs: Search for jobs at a company.
  - apply_to_job: Apply to a job posting.

Phase 3: Second Agent Invocation with Enhanced Capabilities

The agent is re-invoked with the same query plus a note about new tools:

User Query (augmented): "i want to get a job at TechCorp. Please look for jobs online. 
I am a software engineer, open to anything. What jobs can I apply for? 
(Note: New tools are now available, use them if helpful)"

Step 3: Search with Base Tools

The agent re-executes the web search and crawler (discovering the same MCP URL, which is now skipped since it's already registered).

Step 4: Company-Specific Job Search

The agent now calls the newly available search_jobs MCP tool:

{'company': 'TechCorp'}

Returns:

[
  {"id": "job_001", "title": "Software Engineer", "company": "TechCorp"},
  {"id": "job_002", "title": "Machine Learning Engineer", "company": "TechCorp"}
]

Step 5: Additional Content Gathering

The agent also crawls the second page to gather market context.

Final Response

The agent synthesizes all information and provides:

1. Search Results Summary - Raw search hits from the mock search tool
2. Crawled Page Content - Detailed job requirements from the first page
3. Jobs from MCP Tool - Structured job data (job_001, job_002)
4. Recommendations - Direct match analysis for the Software Engineer role
5. Next Steps - Options to apply, search more broadly, tailor resume, or prepare for interviews

Key Technical Details

MCP Discovery Mechanism

The client's web-crawler-tool inspects HTTP response headers:

mcp_url = headers.get("mcp_url")
if mcp_url:
    print(f"Found MCP URL in headers: {mcp_url}")
    await self.add_mcp_server(mcp_url)

Tool Integration

The MCP server exposes tools via FastMCP:

@server.tool()
def search_jobs(company: str) -> list:
    """Search for jobs at a company."""
    return [...]

@server.tool()
def apply_to_job(job_id: str, name: str) -> dict:
    """Apply to a job posting."""
    return {...}

Agent System Prompt

The agent is configured to autonomously use tools:

"You are an autonomous assistant with access to search and crawling tools. 
When a user asks to find, search, or look up something online, 
immediately call the `bing-search-tool` with their query. Do not ask for confirmation. 
If a URL is provided or found in the search results, 
you must call the `web-crawler-tool` to extract its content, 
when the article is actually relevant to the query. 
After crawling, if new tools become available, use them if they help answer the user's query."

Outcome

The dynamic MCP discovery pattern enables the agent to:

• Start with general-purpose web tools
• Discover domain-specific tools at runtime through HTTP headers
• Automatically expand its capabilities without manual reconfiguration
• Provide more comprehensive and specialized responses using the newly discovered tools