Introducing GitBench
GitBench is our attempt to see how well LLMs handle git-specific tasks. We put together a suite of 17 benchmark categories with a total of 204 specific test cases. We ran these benchmarks against 43 (and counting) different models with all of their supported effort/reasoning levels.