← back
Your LLM Ran Out of Knowledge — Now What?
Original: Your LLM Ran Out of Knowledge — Now What?
Takeaway
For low-knowledge domains, encode expert heuristics as a rule library and route prompts to the right rule set before reasoning — wisdom rules in, reasoning model on top.
Summary
- For domains with little written-down expertise (corporate negotiations, risk and security management) LLMs lack training data even if they reason well
- Proposed pattern: give LLMs explicit heuristics (must do / must not do, provable rules) per problem domain — like onboarding a smart but inexperienced graduate
- Demo backend matches user prompts to one of several heuristic sets (e.g. negotiations, geopolitics) via a tool-use-style LLM classifier — falls back to a long-winded match because tool use wasn't reliable enough
- Standardizes scenarios into a fixed format before passing to a planning/reasoning model (Anthropic + Nous's World-Sim) for added realism
- Result: structured output following the rules with traceable references to which heuristic was applied — easy to edit rule files per scenario
heuristicsreasoningdomain-knowledge
Original description
LLMs are great when there is sufficient training material, but what do we do when there is little or no knowledge to train on? We demo a novel framework that enables reasoning LLMs to operate in domains with limited training data by combining domain-specific heuristics, explicit rules, and real-world constraints. This allows us to guide AI decision-making in specialized fields like corporate negotiations, geopolitics, etc, where trainable knowledge is scarce. This methodology opens new possibilities for AI applications in technical domains where traditional training data is scarce. Github - https://github.com/agsheves/OracReasoningEngine