This episode dives into Agent-as-a-Judge, a new method for evaluating the performance of AI agents. Unlike traditional methods that focus only on final results or require human evaluators, Agent-as-a-Judge provides step-by-step feedback during the agent’s process. This method is based on LLM-as-a-Judge but tailored for AI agents' more complex capabilities.To test Agent-as-a-Judge, the researchers created a dataset called DevAI, which contains 55 realistic code generation tasks. These tasks include user requests, requirements with dependencies, and non-essential preferences. Three code-generating AI agents—MetaGPT, GPT-Pilot, and OpenHands—were evaluated on the DevAI dataset using human evaluators, LLM-as-a-Judge, and Agent-as-a-Judge. The results showed that Agent-as-a-Judge was significantly more accurate than LLM-as-a-Judge and much more cost-effective than human evaluation, taking only 2.4% of the time and costing 2.3% of human evaluators.The researchers concluded that Agent-as-a-Judge is a promising, efficient, and scalable method for evaluating AI agents and could eventually lead to continuous improvement of both AI agents and the evaluation system itself.
https://arxiv.org/pdf/2410.10934