When Your Ontology Is Too Big for Your LLM

Executive Summary

• The Problem: When using AI to query large enterprise knowledge graphs, providing the full ontology as context to the LLM becomes slow, expensive, and surprisingly inaccurate.
• The Trap: As the ontology grows, the model struggles to find the relevant concepts in a sea of noise, a classic long-context retrieval problem. Performance degrades just when you need it most.
• The Solution: A “retrieve-and-expand” pipeline. First, use semantic search to find a small set of relevant entities. Then, use the graph’s structure to deterministically add the local context needed for a correct query.
• The Payoff: This approach keeps costs low and performance high, regardless of ontology size. It’s a more robust, scalable way to build reliable knowledge-graph QA systems.

Introduction

LLMs have opened a powerful new frontier: turning natural-language questions into executable queries over structured data. By grounding an AI agent in a knowledge graph’s formal ontology, we can get answers that are not just eloquent but also factually correct and traceable.

But what happens when the ontology, the semantic map of your enterprise, grows too large for the model to read? At enterprise scale, ontologies can span hundreds of thousands of concepts. Forcing an LLM to find the handful of relevant entities for a single question in that massive context is like asking someone to find a specific address by giving them a phone book for the entire country.

It’s inefficient, expensive, and, as we’ll see, it breaks. This article explores why this “full-ontology prompting” approach fails and presents a more intelligent, scalable alternative.

The Breaking Point: Why Full-Ontology Prompting Fails at Scale

The simplest approach to knowledge graph QA is to give the model the user’s question and the entire ontology, asking it to identify the relevant entities and generate a query. This works for small, self-contained schemas. But in a real enterprise, where ontologies are large, modular, and ever-growing, this pattern hits a wall.

There are three main reasons for this failure:

• Rising Costs & Latency: The more tokens you feed the model, the more you pay, and the longer you wait. With large ontologies, every single question becomes a slow and expensive operation.
• Declining Accuracy: More context does not mean better understanding. Models struggle to pinpoint a few relevant items in a long document filled with noise. This is a well-known weakness of long-context models. As the ontology grows, the model’s ability to recall the correct entities for the query actually decreases.
• Increased Ambiguity: Large, modular enterprise ontologies often contain similar concepts in different domains. As more modules are added, a question can acquire plausible but incorrect interpretations, confusing the model and leading to wrong answers.

Our research confirms this across multiple models and benchmarks. As we increased the size of the ontology context, retrieval quality consistently dropped while latency shot up.

Full-ontology prompting is a useful baseline for demos, but it’s a fragile foundation for a production system. It gets slower, more expensive, and less reliable just as your enterprise schema begins to scale.

A Better Way: Retrieve First, Then Recover Structure

Instead of asking the model to re-read the entire map for every trip, we can give it just the relevant section. Our alternative is a two-step “retrieve-and-expand” pipeline that uses the ontology according to its structure.

This approach dramatically changes the cost and performance profile. The cost of answering a question remains low and predictable, even as the ontology grows to massive scale.

Here’s how it works:

Step 1: Retrieve with Semantic Search First, we index the entire ontology once.

Each class and property is embedded based on its name, labels, and descriptions. When a user asks a question, we use fast, low-cost vector search to find a small set of candidate entities whose descriptions are semantically similar to the user’s wording.

This step reliably finds the main concepts in the question. However, semantic search alone isn’t enough. It often misses two types of essential entities:

• Generic properties with little unique descriptive text (e.g., hasValue, hasName).
• Connecting properties that describe the graph’s structure rather than business language.

Step 2: Expand to Recover Local Structure This is where we leverage the power of the graph.

Starting from the entities found in the retrieval step, we run two deterministic expansion steps that traverse the ontology’s structurewithout another expensive model call.

• Local Expansion (E1): We add all the datatype properties directly attached to the retrieved entities.
• Graph Expansion (E2): We traverse the graph further to include multi-hop relationships (object properties) and the classes they connect to.

The result is a compact, relevant schema “snippet” that contains the entities from semantic search plus the surrounding structural context needed to build a correct query. We give this small, targeted context to the LLM.

Why This Matters: A More Robust Division of Labor

This retrieve-and-expand pipeline establishes a more robust division of labor:

• Use language models for what they do best: Interpreting the nuance of human language in the user’s question.
• Use the knowledge graph for what it does best: Providing explicit, reliable, and traversable structure.

By separating these concerns, we build a system that is not only more efficient but also more reliable. Across our enterprise benchmarks, this pipeline consistently delivers a high-recall schema context at a fraction of the cost of full-ontology prompting.

Trade-offs and Limits: The Challenge of Ambiguity

This method isn’t a silver bullet. While it dramatically improves performance, some challenges remain. The hardest questions are those that are inherently ambiguous, where the user’s intent cannot be resolved by either semantic similarity or graph structure alone.

For example, if a user asks about “services” but the ontology contains “Connected Services,” “Financial Services,” and “Internal Services,” the system may still struggle to pick the right one without a clarifying dialogue. Deterministic expansion recovers structure, but it cannot invent intent that is absent from the question.

Conclusion: If You Remember One Thing…

The limiting factor in enterprise knowledge-graph QA is not the LLM’s ability to write SPARQL. It’s the ability to efficiently and accurately identify the small set of schema entities a question requires.

Don’t treat your ontology as a single, massive document to be force-fed into a prompt. Treat it as a semantic system. Use its structure to your advantage.

By combining fast semantic retrieval with deterministic graph expansion, you can create a system that is cheaper, faster, and more reliable as it scales. This is how you move from impressive demos to robust, enterprise-grade analytics. Use language models to interpret language, and use the knowledge graph to recover structure. That is the key to building QA systems that win.

If you want to go deeper: our 12-page whitepaper lays out the full methodology, including benchmark results.