Accuracy-Cost Tradeoff on HumanEval

arXiv Logo Paper link for "AI agents that matter"
GitHub Logo Full reproducibility materials for the paper titled "AI agents that matter"

Input ($/1M tokens)

Output ($/1M tokens)

GPT-4
GPT-3.5
Llama-3-8B
Llama-3-70B
Log Scale

Legend

Complex agent
✕
Baseline agent
Zero-shot model