GitHub put Copilot Evals into public preview on 17 November 2026. An eval, in GitHub’s words, is “a repeatable test for AI behavior”. It is aimed at teams who have set up Copilot with their own skills, custom agents and MCP tools (connectors that let an agent use other software) and want to know whether those setups actually work.
What the changelog says
- What it checks. Both the outcome and the path the agent took to reach it.
- How you create one. Ask Copilot in plain language, or use the
/evalcommand. - How you run one. You approve each run, see which checks passed or failed, then compare runs from before and after a change.
- Why rerun. AI output varies between runs, so GitHub recommends rerunning the same eval after changes to catch regressions.
- Availability. Public preview on Copilot Individual, Business and Enterprise. Install through the VS Code extension, the Copilot CLI plugin or the GitHub Copilot app. It is built on Vally, a local eval tool.
Why it matters
Most teams adopting AI coding tools tune them by feel: a prompt gets tweaked and everyone hopes it improved. Evals turn that into testing. The same shift happened to ordinary software decades ago, and it is what separates a demo from something you can depend on.
So what
If you are commissioning AI features or an AI-built product, ask your supplier how they know the AI parts keep working after a change. A good answer names a test suite, not a feeling. Our AI product work in services and AI-assisted development both treat testing as part of the build. If you are planning one, start a project.