Rolling out AI in the SDLC: a pilot you can measure

The company bought an AI licence for every developer, or it turned an AI reviewer on in every repository on the same Monday. A month later the lead cannot name one thing that got better. Review comments sit unread. Someone pasted a database password into a chat to explain a bug. Generated tests assert that a function returns the value the function already returns, including the bug. The lead now spends the afternoon reading model output that used to be a ten-minute pass over the change.

A licence is a purchase. A step in the process is a place where a named person expects a named result. The team bought the first and skipped the second.

1. What the first month actually shows

Acceptance criteria in the ticket were thin, so people asked the model to rewrite the ticket. The model invented a second behaviour. The author shipped the first one. Reviewers then argued with a list nobody had agreed.

Pull requests grew a comment about names and about possible edge cases that named no case. Developers learned to resolve the thread without reading it. The human review still happened, only later, and the lead read both the change and the comment.

Test generation looked busy. The new file called the function and expected the value it already returns. The pipeline stayed green. A bug found after release was the same bug, and the new test now guards that bug instead of catching it.

One person pasted production logs, an access token, and a customer email into a vendor chat, because the prompt box was the fastest place to think. The repository has no line about what the model may see. Chat history is not the company's storage, and the person who pasted the log cannot delete the vendor's copy.

None of this fails a job. The invoice for the licences is the only number anyone can point at. A feeling that AI is now "in the process" is not a measurement.

The lead's week shows the cost plainly. Monday used to be a pass over the change. It is now a pass over the change, plus a pass over comments that repeat the linter, plus a test file that stays green while the bug is still there. The licence did not remove a review. It added a second text the same person has to distrust.

2. Why the tool never became a step

The pipeline already had stages: a ticket with acceptance lines, a pull request, a review against the checklist the team already uses, tests on the changed code, release notes, and a tag. The model sat beside that pipeline. It was not inside one stage, so nobody could say which result it owed the team.

Nobody owns the step. When a comment is wrong, there is no person whose job is to say so and to change the rule. "Help the developers" is not a rule. A model that may rewrite acceptance criteria, approve its own tests, and comment on every file will do all three. The team then treats every output as optional, because some of it is wrong, and the optional pile hides the one comment that was right.

Nobody wrote the baseline before the licences arrived. Review wait time, bugs found in the week after a release, and tasks returned because an acceptance line was missing were already knowable. They were not written down. After a month the only comparison is a feeling, and the feeling is that there is more to read.

Prompts and settings live in a browser profile or in a private note. The next developer does not have them. The lead cannot review a change to the prompt the way they review a change to the pipeline, because the prompt is not a file in the repository.

A check that stays green when the model never answered is a separate failure, already described in AI integration in the SDLC: when a pull-request check fails open. This note does not repeat that gate. Whether a person should still read the path a change will touch is the neighbouring question in ten signs your site needs a code review before the next feature.

3. One stage, one repository, two to four weeks

Pick one stage where the pain is already visible, and leave every other stage alone until this one moves a number.

Four stages are enough to choose from, and the team picks one. Acceptance criteria on the ticket, before anyone writes code: the model may propose lines, and a person approves the lines that stay. Review of a pull request against the checklist the team already uses: the model may say which checklist line the change does not cover, and it may not invent a new checklist. Gaps in tests on the changed code: the model may name a branch or a case with no test, and it may not add a test that only echoes the current return value. Release notes for the tag: the model may draft notes from the merged tickets, and a person edits them before they go out.

Write three lines in the repository before the pilot starts. What the step does, as one sentence the team can read aloud: on this repository, the check lists checklist items the change does not cover. What it must not do: it does not approve the pull request, it does not rewrite the ticket, and it does not comment on names. Who approves: a named role, the author or the reviewer, not a vague "the team". The author fixes a real gap, or the reviewer marks the flag as wrong, with a name and a date.

Write the data rule in the same file. The model may read the pilot repository and the ticket text. It may not read production data, customer records, passwords, access tokens, or database credentials. Name the repositories that are in the pilot, and name the ones that stay out. If a prompt needs a log line, a person strips secrets, customer emails, and tokens before that line leaves the laptop. A vendor chat is not a place to paste a production dump just this once.

Run the pilot for two to four weeks on that one repository. Do not switch the same job on in every service on the first Monday. A second repository is a second pilot, after the first one has numbers.

Measure on the day the pilot starts, then measure the same way on the day it ends. Two or three numbers are enough. How long a pull request waits for the first human review. How many bugs the team finds in the week after a release, on that repository. How many tasks come back because an acceptance line was missing or a checklist item was skipped. A note from the start can read like a blank form: median wait for first review, 18 hours; bugs in the week after the last two releases, 4; tasks returned for a missing acceptance line this month, 6. Those figures are a shape, not a result from a project. Your numbers will differ. A tool that moves none of them is a narrator. Turn it off, or narrow what the step is allowed to say. Do not open a second stage to make the licence feel worth the invoice.

Keep the prompt, the checklist the model sees, and the job settings in the repository, under version control, next to the code they judge. A change to the prompt is a pull request with an author. A developer who joins next month reads the same file the model reads. A prompt that exists only in one person's chat cannot be reviewed, and it cannot be rolled back.

Make the check fail closed, or label it as advice in the comment itself. A required job that stays green on a model error has not checked the change. An advisory comment has to say that it cannot block the merge, so the team does not treat a paragraph as a gate. How a green timeout gets built is the other note. The rule on this page is smaller: an optional paragraph is not a quality gate.

Extend only after one number has moved and people still read the flags. The next stage gets the same three lines, a new baseline, and its own two to four weeks. Licences for people who do not touch the pilot repository can wait until a second repository repeats the result.

• One stage, one repository, two to four weeks.
• A written line for what the step does, what it must not do, and who approves.
• A written line for what the model may see: no production data, no passwords, no access tokens, and the repositories that are in.
• Two or three numbers, written down before the pilot, and the same numbers at the end.
• The prompt, the checklist, and the job settings live in the repository.
• The check fails closed, or the comment says it is advice and cannot block the merge.
• The next stage waits until this one has a number that moved.

4. What one session leaves on the page

One session maps the pipeline the team already runs and picks the first stage from a pain the lead can point at, not from a vendor menu. The page you leave with names the stage, the three lines, the data rule, the repository, the start and end dates, and the numbers you will compare. The same session can put the first check in the repository, with the prompt in version control and the job set to fail closed or marked as advice.

That page is the rollout. It is not a new product, and it is not a chat widget on the site. Services cover this pass on a codebase you already ship. Before the rollout turns into noise, a technical audit and code review of the code you run today tells you where the real gaps are. Selected work keeps each result next to the operation that produced it. If you want the page written with you, send the stage that hurts, the checklist the team already uses, and whether the current AI job can block a merge. Leave production data, passwords, and access tokens out of the message.