Two announcements landed two days apart in the same week, from two different platforms, aimed at the same problem: engineering leaders can’t see what their AI coding tools are actually costing or delivering.
On 20 July, Amazon CloudWatch launched Coding Agent Insights, pulling OpenTelemetry metrics from Claude Code, OpenAI Codex, and GitHub Copilot into CloudWatch’s existing operational dashboards. It lets an engineering leader set proactive alerts on token spend, correlate agent adoption with commit throughput and pull request velocity, work out which model gives the best cost-to-output ratio, and decide which teams should get expanded access based on actual usage data rather than requests.
On 22 July, GitHub shipped its own answer: a Copilot Metrics Impact Dashboard that buckets licensed users into adoption cohorts — Phase 1 (code-first), Phase 2 (agent-first), Phase 3 (multi-agent or Copilot app) — plus a Passive segment for people who have a licence and aren’t using it. It tracks pull requests merged per user, merge velocity, and lines of code per day against those cohorts, with an “adoption multiplier” comparing engaged users against passive ones.
So what
Six months ago, rolling out Claude Code or Copilot across a team was a judgement call — you bought seats, hoped for the best, and measured success anecdotally. That’s no longer how the vendors expect this to work. Both of these ship as management tooling: token budgets, adoption phases, cost-to-output ratios, unused-licence flags. AI coding tool spend has moved from a discretionary line item approved on faith to a budget line that’s expected to be justified with data, the same way cloud infrastructure spend already is.
For founders and CTOs deciding how deeply to lean into AI-assisted development — inside their own team or when evaluating a development partner — this is worth noting as a maturity marker. If a vendor or partner can’t tell you which teams or projects are getting genuine throughput gains from AI tooling versus which have unused licences sitting idle, they’re behind where the tooling itself now sits. It’s also a good question to ask any partner running your build: how do they measure whether their AI-assisted workflow is actually faster, not just busier. That’s exactly the kind of engineering discipline we bring to AI-assisted development work, or get in touch if you want to talk through how a partner should be instrumenting this.