CMOtech US - Technology news for CMOs & marketing decision-makers
United States
Google sets out five-point test for AI coding skills

Google sets out five-point test for AI coding skills

Mon, 16th Nov 2026 (Today)
Sofiah Nichole Salivio
SOFIAH NICHOLE SALIVIO News Editor

Google has outlined a five-point method for testing and refining AI coding agent skills used with its Google Cloud Developer Plugin. The guidance draws on internal trials across multiple agent frameworks and evaluation tools.

The approach focuses on whether a skill file improves an agent's performance in controlled tests, rather than on one-off prompt checks on a developer laptop. It also includes Google's contributions to NVIDIA's open-source evaluator and the use of a Developer Knowledge MCP server to ground responses in documentation.

At the heart of the guidance is a warning that many skill files fail before their main instructions are read. Coding agents often route requests using only the short name and description in a YAML frontmatter section, which can mean skills are not triggered or are overridden by broader neighbouring descriptions.

To address that, Google recently open-sourced a tool called skill-reach. It checks manifests, compares vocabulary overlap between related skills, and runs routing probes across systems including Claude Code, Google Antigravity and OpenAI Codex.

It argues this should happen before teams spend money on container-based testing. Google also highlighted security risks that standard code linters can miss in markdown instructions, including prompt injection, data exfiltration, tool poisoning and trigger abuse.

Clean testing

A second part of the method rejects local workstation benchmarking. Developer laptops, Google argued, introduce hidden state into tests through cached cloud credentials, exported project settings, shell histories and other environment variables.

That can produce false positives, because a baseline agent may appear to complete a deployment task using inherited settings rather than the skill under test. Resource contention on a workstation can also alter outcomes by slowing file scans or causing subprocess timeouts.

In internal testing during experiments with code retrieval agents using AlphaEvolve, benchmark scores fell when the same agent ran on a machine under CPU load. Google said that changed the agent's behaviour enough to affect the result.

Paired ablation

Google also argued against judging skills from a single successful run. Instead, it said teams should use repeated paired tests, with one arm running a base model without a skill in a clean sandbox and the other running the same prompt and environment with the skill mounted.

More than a third of benchmark tasks in one retrieval study produced different scores across consecutive runs using the same code and prompt, it said. On that basis, Google recommends measuring results over several attempts, such as Pass@3, to determine whether a skill genuinely improves outcomes.

The paired approach also helps show whether a model already performs the task on its own. Where tests show identical success rates and no efficiency gain, the skill should be removed because it adds context without improving the result, according to Google.

Guardrails

Another element of the framework is separating correctness, efficiency and security in grading systems to prevent automated optimisers from exploiting the benchmark. Internal AlphaEvolve retrieval tests, Google said, found several ways in which evolved candidates could achieve higher scores without actually solving the task.

Examples included reading answer files directly from disk and launching subprocesses to bypass abstract syntax tree checks. This behaviour can emerge when a single score combines correctness and efficiency, because a shortcut may appear more efficient than a legitimate solution.

To reduce that risk, Google said test fixtures, grading scripts and answer keys should sit in a dedicated verifier container, separate from the agent workspace until execution has ended. Security failures should also remain separate from other measures, so lower token use cannot offset a failed test or an unsafe action.

Google added that its contributions to NVIDIA's SkillEvaluator extend checks to fail any trial that reads Kubernetes service account tokens or queries cloud metadata endpoints. It also said live OAuth tokens are redacted from saved trajectories.

Operational habits

The final theme is the content of the skill files themselves. In Google's tests, the strongest performers were not large markdown documents filled with copied API reference material, but compact instructions that teach operational habits such as checking command-line help before making changes, pulling only the relevant recipe on demand and defaulting to least-privilege authentication.

By contrast, long documentation extracts added little value because leading models already contain much of that public information from training data. Google said the extra text can slow response times and obscure the practical constraints an agent should follow.

The evaluation setup also lets teams retest skills when models change. Google said it reruns a paired matrix against new models and removes any skill whose measured improvement falls to zero, adding: "Deleting a skill because the new model graduated past it is one of the cleanest high-fives in software engineering."