devLap

I Used an AI Agent to Test an Open-Source TypeScript Tool and Found a Real Bug

typescript dev.to

A maintainer recently invited me to test AgentInspect, a local-first toolkit for debugging and testing TypeScript AI-agent trajectories.

I was interested, but a proper review could easily turn into hours of setup, testing, documentation, and reproduction work. Instead of choosing between a superficial comment and a large manual audit, I tried a third option: an AI-assisted black-box test with clear boundaries.

The result was a reproducible bug report that the maintainer confirmed and fixed.

The testing rule that mattered most

The AI agent was instructed to behave like a new external user.

During the first pass, it could use the public README, npm package, user-facing documentation, and documented CLI. It could not inspect the source code, existing issues, pull requests, or internal tests.

That constraint mattered. If the agent had read the implementation first, it might have worked around confusing behavior and missed the actual onboarding experience.

The test covered a fresh pnpm and ESM setup, the first useful trace, intentional trajectory failures, CLI diagnostics, and larger synthetic traces. All test data was local and synthetic.

The bug appeared in combined search filters

A search using both --name and --status returned a run that matched only the status filter.

For example:

pnpmexecagent-inspectsearch--dir.agent-inspect--namedefinitely_not_matching_xyz--statuserror--json
Enter fullscreen mode Exit fullscreen mode

The run had an error status, but its name did not match definitely_not_matching_xyz. The output still included it because the run-level matcher treated status as an independent match reason.

The expected behavior was simple: when several structured filters are supplied, each filter should narrow the result set.

Human review was still essential

The agent also produced several observations that initially sounded like bugs but were really documentation questions, design choices, or usability suggestions.

Before reporting anything, I separated:

  • observed behavior from assumptions;
  • confirmed mismatches from preferences;
  • reproducible defects from possible product decisions;
  • measured facts from claims that the evidence could not support.

That review prevented a noisy issue containing unrelated suggestions. I sent the maintainer a concise summary and asked which finding represented unintended behavior.

The maintainer confirmed the combined-filter behavior as the clearest functional issue, supplied acceptance criteria, and asked for one focused GitHub issue. The issue was subsequently fixed.

A reusable workflow for AI-assisted open source testing

This experience gave me a workflow I plan to reuse:

  1. Define a strict black-box phase. Do not let the agent inspect the implementation before testing the public experience.
  2. Use an isolated project and synthetic data. Never expose client code, credentials, private prompts, or production traces.
  3. Require exact evidence. Record the OS, runtime, package version, commands, exit codes, expected result, and actual result.
  4. Preserve failed attempts. Setup friction is useful feedback when it is documented accurately.
  5. Review every conclusion. AI can run tests and organize evidence, but it can also overstate severity or infer an undocumented expectation.
  6. Report one confirmed problem at a time. A small reproducible issue is easier to review and more likely to produce a useful fix.

What I learned

AI agents can reduce the repetitive work involved in open-source testing, especially environment capture, command execution, negative tests, and report drafting.

The valuable contribution, however, is not the volume of generated notes. It is the final judgment about what the evidence actually proves.

In this case, automation made a thorough first pass practical. Human review turned the output into a focused report. Clear communication gave the maintainer enough information to confirm and fix the problem.

That is a much better outcome than asking an AI agent to scan a repository and generate as many issues as possible.

Connect with Me

If you found this article helpful, let's connect and discuss modern development workflows!

Source: dev.to

arrow_back Back to Tutorials