Pentest-Agent — building an LLM-aware red-team assistant

10/15/2025

Why build Pentest-Agent?

Manual reconnaissance is repetitive and error-prone. Chaining tried-and-true tools into repeatable, auditable workflows speeds up triage and frees attention for the creative bits of red-teaming. The goals were simple and stubborn:

  • Human-readable plans so humans can audit and stop anything suspicious.
  • Pluginized design so new scanners or payloads can be dropped in safely.
  • Structured outputs (JSON) so downstream automation, reporting, and ticketing are trivial.
  • Play nice with existing tools (nmap, masscan, nikto, gobuster) instead of reinventing them.

High level architecture

Components

  • Core Orchestrator (async) — schedules cycles, asks plugins for proposals, executes approved actions, logs an audit trail.
  • Plugins — focused modules that implement propose_actions() and execute(action) (e.g., scanner, parser, reporter).
  • LLM Adapter — converts structured data into prompts and ingests LLM responses into structured plans. Design principle: LLMs assist planning, they don't execute dangerous commands.
  • Runner script — CLI entrypoint (example: agent_runner.py) to wire pieces together and boot the loop.
  • Persistence / Audit — append-only logs for every action and decision for postmortem and compliance.
[Runner] -> [Orchestrator] -> {Plugins}
                   ↓
               [LLM Adapter]
                   ↓
                [Audit DB / Files]

Tech choices & why

  • Python 3.11+ for a rich ecosystem and excellent async support.
  • asyncio to handle IO-bound tasks (scans, HTTP requests) without blocking.
  • nmap for discovery — mature, scriptable, and easy to parse.
  • Plugin architecture to isolate capabilities and reduce blast radius.
  • LLM (optional) only for analysis and suggestions; default runs should be deterministic and auditable.

Example: runner and main loop

A minimal runner that wires things up and ensures the first action is reconnaissance:

#!/usr/bin/env python3
import argparse
import asyncio
import sys

from lib.app import PentestAgent

async def main():
    parser = argparse.ArgumentParser(prog="agent_runner")
    parser.add_argument("target", help="hostname or IP to target")
    parser.add_argument("--cycles", type=int, default=3, help="max agent cycles")
    args = parser.parse_args()

    agent = PentestAgent()
    if not agent.set_target(args.target):
        print("Invalid target:", args.target)
        sys.exit(1)

    agent.load_plugins()
    await agent.run_plan_loop(initial_data_type="nmap", max_cycles=args.cycles)

if __name__ == "__main__":
    asyncio.run(main())

Make the initial data type explicit (initial_data_type="nmap") so the orchestration won't guess what to do first.

Plugins: recommended responsibilities

  • nmap_plugin — run nmap, parse XML output, emit structured host/service findings.
  • vuln_parser — correlate services to CVEs or feed to vulnerability scanners.
  • http_enum — perform web content discovery, follow redirects, gather endpoints.
  • reporter — format final results into JSON summaries and human-readable reports.

Each plugin should be small, testable, and explicit about capability (e.g., can_exploit: false).

Structured outputs: force structure, avoid raw text

Convert raw tool output into JSON before feeding it to the LLM. Example:

{
  "hosts": [
    {
      "ip": "10.0.0.5",
      "open_ports": [
        {"port": 22, "service": "ssh", "banner": "OpenSSH 7.9"},
        {"port": 80, "service": "http", "banner": "nginx"}
      ]
    }
  ]
}

Prompt the LLM with explicit instructions: "Given this JSON, recommend up to 3 follow-up non-destructive checks and return JSON." Then validate the output and convert to actions.

Important implementation notes

Running blocking tools safely

Use asyncio.create_subprocess_exec() or asyncio.to_thread() for blocking subprocesses. Blocking the event loop with a synchronous subprocess.run() will stall your orchestrator.

Plugin dependency graph

Early versions executed plugins in arbitrary order. Fix: have plugins declare produces: ["nmap_json"] and consumes: ["nmap_json"] so the orchestrator can schedule tasks in dependency order.

LLM safety

Never accept raw shell commands from an LLM to execute. LLMs can recommend commands, but always validate them against a safe whitelist or translate them into structured executor operations.

Logging & audit

Add timestamps, plugin names, and unique action IDs. Keep logs append-only and machine-readable (JSONL) for replay and compliance.

Lessons learned (aka the pain points)

  • Async + subprocesses are fiddly. Properly manage processes and timeouts.
  • Order matters. Define explicit data flows between plugins.
  • LLMs hallucinate confidently. Always validate LLM output.
  • Test with canned outputs. Unit-test parsers with saved nmap XML.
  • Rate limit. Don't accidentally DDoS a target because a plugin got aggressive.

Security & ethics (say it like it is)

This project automates offensive actions. Use it responsibly:

  • Only test assets you own or have explicit permission to test. Unauthorized testing is illegal.
  • Destructive actions are disabled by default. Require explicit enabling and stronger auth for exploit-capable plugins.
  • Keep an audit trail. Record who ran the agent, the target, and the actions.
  • Treat captured secrets and PII as sensitive. Do not store them in public logs.
  • Rate-limit and throttle to avoid unintended service disruption.

Include a SECURITY.md in your repo and adhere to it.

How I test it locally

  • Build disposable labs with Vagrant, Docker Compose, or local VMs (Metasploitable, OWASP Juice Shop).
  • Unit-test plugins using canned nmap XML and expected JSON outputs.
  • Integration tests run a fake plugin that proposes deterministic actions and assert the audit trail matches.

Roadmap (practical)

Short term

  • Add masscan plugin for large-range recon.
  • Harden nmap parsing and add rate-limit config.
  • Implement dry-run executor to validate plans without executing.

Medium term

  • RBAC + signed configs to control destructive plugins.
  • Integrate with issue trackers (Jira/GitHub) for auto-ticketing.
  • Add a web UI for human approval of LLM plans.

Long term

  • Multi-agent choreographies for coordinated red-team playbooks.
  • Formal, auditable "playbook" language (YAML) for SOC-approved automation.

Quick start

  1. Clone the repo.
  2. Create .env (LM settings are optional).
  3. Install: pip install -r requirements.txt (use a venv).
  4. Run: python agent_runner.py 10.0.0.5 --cycles 2 (replace target).
  5. Inspect ./logs/ for JSONL audit output.

Example config

agent:
  initial_data: nmap
  max_cycles: 4
plugins:
  - name: nmap_plugin
    enabled: true
    safe_mode: true
  - name: http_enum
    enabled: true
  - name: exploit_plugin
    enabled: false  # off by default for safety

Contributing

  • Fork -> branch -> PR. Keep changes focused.
  • Include tests for parsers and orchestrator behavior.
  • Document security implications in PR descriptions.

Closing thoughts

Automation is an act of both laziness and rigor: lazy because you remove repetitive tasks, rigorous because automation forces you to formalize decisions. If your tool is dangerous, make it auditable and require human intent for anything that can cause harm. Build small, test often, and keep the logs noisy.

Comments