AI Harness & Eval
Building production-grade agent infrastructure, reproducible sandbox evaluation frameworks, and deterministic reinforcement learning environments.
Production Agent Harnesses
At Intuitive, I architected the end-to-end agent harness powering hierarchical task planning, sub-agent delegation, and parallel tool execution. I delivered the infrastructure to safely deploy agentic capabilities across internal applications.
Durable Agent Loops
Owned the complete agent loop architecture including automatic retries, budget caps, stop conditions, and checkpointed recovery for long-horizon task execution.
Credential Isolation
Executed dynamic code in short-lived Docker sandboxes utilizing seccomp profiles, read-only rootfs, and egress allowlists, brokering Vault-issued tokens.
CI Evaluation Pipelines
Built eval suites powered by production traces. By establishing ground-truth scoring and LLM-as-judge graders, every prompt and model upgrade was rigorously gated.
Project Apollo: RL Environments
Evaluating autonomous agents requires frameworks that go beyond static QA datasets. With Project Apollo, I developed reinforcement learning environments and sandboxed execution harnesses designed specifically to probe multi-step agentic LLM capabilities.
RL Environments for Eval
Built challenging task seeds, complex reward functions, and programmatic verifiers to score deep, multi-step tool use, including defenses against agentic reward hacking.
Sandboxed Eval Harness
Architected sandbox-isolated agentic tasks utilizing Inspect and Harbor, perfectly decoupling the underlying simulation environments from the evaluation logic for reproducible runs.
Eval Design Methodology
Integrated current research patterns to construct evaluations grading time-based decisions, dependency resolution, red-herring distractions, and rollback behavior.
Tooling & Developer Experience
Shipped a customized task-seed CLI and detailed trace viewer enabling teammates to rapidly author, replay, inspect, and debug entirely new evaluation environments.
Sequel2SQL: Alignment & RLEF
In my capstone project sponsored by Microsoft, I developed Sequel2SQL, an agentic LLM framework utilizing AST-based segment parsing to translate natural language to optimized SQL.
Standard fine-tuning produces functionally correct but computationally disastrous queries. To solve this, I designed a deterministic RLEF (Reinforcement Learning from Execution Feedback) environment to align models using execution metrics rather than brittle subjective judging.
View Microsoft Capstone Repository
LLM Alignment & DPO
Aligned local LLMs utilizing the Hugging Face TRL library via Direct Preference Optimization on custom RLEF datasets, teaching index utilization and hash joins.
Deterministic Rewards
Scored agent outputs on live database execution metrics via EXPLAIN ANALYZE instead of relying on unpredictable LLM-as-judge signals.
Anti-Reward Hacking
Implemented literal matching checks, empty result checks, and true hardware I/O analysis to prevent the agent from artificially cheating the evaluation loop.