Graphiti MCP Server vs Aletheia Evaluation¶

This FAQ explains the differences between using the Graphiti MCP server directly (e.g., with Claude Desktop) versus using Aletheia's evaluation framework, and when to use each.

The Question¶

"I discovered the Graphiti MCP server. How does it compare to Aletheia? What's the difference between Claude Desktop answering questions via MCP versus Aletheia's evaluation pipeline?"

Quick Answer¶

• Graphiti MCP Server: A retrieval and analytics interface that lets an LLM access your knowledge graph. The LLM synthesizes answers using its own judgment.
• Aletheia: A controlled evaluation framework that forces grounded answers with programmatic verification and quality metrics.

Example: Complex Aviation Query¶

To illustrate the differences, consider this question:

"What incidents in Barcelona or Madrid involved Boeing aircraft and resulted in emergency declarations (MAYDAY or PAN-PAN)?"

Graphiti MCP Server (Aletheia Fork)¶

Available Tools¶

The MCP server exposes 15 tools across 6 groups:

Self-Describing Connectors¶

Each MCP server instance introspects its own graph at startup via DomainProfile, generating domain-specific:

• Tool descriptions — the LLM sees aviation-specific guidance when connected to the aviation graph
• Entity catalogs — types, counts, and sample entities
• MCP resources — graphiti://domain_summary, graphiti://entity_catalog, graphiti://relationship_types

Four domain connectors are configured in Aletheia: Aviation Safety, Safety Recommendations, Airworthiness Directives, and Terrorist Organizations.

How Claude Desktop Answers¶

User: "What incidents in Barcelona or Madrid involved Boeing aircraft
       and resulted in emergency declarations (MAYDAY or PAN-PAN)?"

Claude Desktop (via MCP):
1. Calls search(query="Barcelona Madrid Boeing MAYDAY PAN-PAN")
2. Optionally calls run_cypher() for structured analytics
3. Optionally calls explore_node() to traverse from a known entity
4. Uses parametric knowledge + returned results to synthesize answer

MCP Limitations for Evaluation¶

Aletheia Evaluation Framework¶

Pipeline Architecture¶

Question
    │
    ▼ [1. Query Analysis]
    Detect patterns: "Barcelona OR Madrid", "Boeing", "MAYDAY"
    → Generate SearchFilters (node_labels, edge_types)
    │
    ▼ [2. Graphiti Search]
    Multi-method search with:
    - Cosine similarity + BFS traversal
    - RRF reranking
    - Optional community search
    - Custom search filters
    │
    ▼ [3. Evidence Transformation]
    transform_search_results(results) → EvidenceUnit[]
    Each unit has: eid, subject_name, predicate, object_name, fact
    │
    ▼ [4. Context Building]
    Atomic evidence units with citation IDs
    Optional COMMUNITY_CONTEXT section
    │
    ▼ [5. LLM Answer Generation]
    Structured JSON with mandatory citations
    │
    ▼ [6. Grounding Verification]
    Programmatic check that answer entities appear in cited evidence
    │
    ▼ [7. RAGAS Evaluation]
    Quality metrics: precision, recall, faithfulness, similarity
    │
    ▼ [8. Grounding Report]
    Pass rates, rejection reasons, diagnostics

Example Evidence Output¶

EVIDENCE:
[E1]
subject: Occurrence-2024-0157-EU
predicate: OCCURRED_AT
object: Barcelona Airport
fact: The aviation incident Occurrence-2024-0157-EU occurred at Barcelona Airport
source: aviation_safety

[E2]
subject: Occurrence-2024-0157-EU
predicate: INVOLVED_AIRCRAFT
object: Boeing 737-800
fact: Boeing 737-800 registration EC-LUT was the aircraft involved
source: aviation_safety

Grounding Verification¶

Aletheia programmatically verifies that:

1. All cited evidence IDs are valid
2. Entities mentioned in the answer appear in the cited evidence
3. INSUFFICIENT_CONTEXT is returned when evidence is lacking

Three grounding modes control strictness:

Side-by-Side Comparison¶

When to Use Which¶

Use MCP Server¶

• Interactive exploration of knowledge graphs
• Conversational memory for agentic workflows
• Structured analytics via Cypher queries
• Quick answers where auditability is less critical
• Building AI agents that need persistent memory
• Schema discovery and ontology browsing

Use Aletheia Evaluation¶

• Measuring retrieval system quality
• Ensuring answers are grounded in evidence
• Preventing parametric knowledge leakage
• Generating quality metrics for system improvements
• Production systems requiring auditability
• Comparing search configurations (baseline comparisons)

Integration¶

The MCP server and Aletheia are complementary:

1. Build your knowledge graph with Aletheia (parsers, episode builders, schema inference)
2. Evaluate retrieval quality with Aletheia's RAGAS pipeline
3. Serve the graph to LLMs via MCP connectors for interactive use
4. Analyze the graph with Cypher queries via the MCP server

The same graph database powers both — Aletheia builds and evaluates it, MCP serves it.

Summary¶