Case study · toolwitness.ai
Check the tool.
Show the evidence.
I built toolwitness.ai, a public directory that checks AI tools before publishing them. Each record shows what was checked and what remains unknown.
The problem
AI agents use tools called MCP servers, skills and hooks. They are easy to publish and hard to trust. Publishing a tool does not mean anyone has checked it first.
What I built
A scheduled pipeline finds tools, gathers evidence and publishes only records that pass automated checks and independent AI review.
The lab starts MCP servers in locked Docker containers with no network access and a read-only file system. It checks whether they start and return their tool list. Skills and hooks get source-file inspection; those records do not claim the tool was run.
OpenAI Codex checks the claims against the evidence. Grok, from a different AI vendor, gives a second opinion. Missing evidence or a failed review blocks publication.
Checks that are tested too
Automated checks compare file fingerprints, tool versions, dates and claims with saved evidence. They also check the lab's isolation settings and reject unsupported promises of safety.
Each check is itself tested by planting a known fault and confirming it is caught. Tests change evidence, remove claim sources and weaken isolation settings. Other tests disable publication rules and confirm the test suite detects the fault.
Results so far
138 published records in the committed snapshot dated : 29 offline MCP listings and 109 source inspections.
These are records of limited checks, not a guarantee of safety. An offline listing shows startup and a tool list; it does not show tool calls or a full AI task.