AI pull-request checks that fail open: how to fix them

The pull request already carries an AI comment, and the required check is green. The comment talks about names. The ticket asked for a partial refund that writes a ledger row. The diff never touches that ledger. Friday's tag still goes out, because the release owner trusts the green check and does not open the comment. The integration looks installed. It did not hold the change.

This belongs on the delivery path, not in a chat on the site. The work is already moving: a ticket, a pull request, a pipeline, a release. A useful check names a gap a person can verify before merge. A check that stays green when the model never answered is decoration on the pipeline. A louder model does not turn that decoration into a gate, and a visitor chatbot is a different product.

1. Green can mean the model never ran

Fail-open is a plain outcome, and it shows up in the software delivery lifecycle exactly where owners hope an AI integration will watch quality. The job is supposed to ask a model whether this diff misses something the team already requires. The model times out, the token quota is empty, or the vendor returns an error. The job then exits as success, or the job is not required, so a red comment does not stop the merge. The pipeline shows the same green either way. The release owner reads green as "someone looked." The log says nobody got an answer.

One merged pull request is enough to see it. Open the job log for a change that landed this week. If the log says the model failed, timed out, or returned nothing, and the job is still green, the integration fails open. If the comment is red and merge is still allowed, the comment is a note. A note is not a gate. Both shapes let the same gap leave with the tag.

A skipped test sits next to that silence. The repository used to require a job. The same diff that the model praises for style marks that job as allowed to fail, or it deletes the test file. The AI comment does not mention the skip. The pipeline stays calm. The release loses the only automatic witness it had, and the green check still says the change was reviewed.

2. Why a person cannot gate every diff

A tech lead cannot read every diff before the release and still run the week. That limit is why the reviewer was bought. Manual review fails as volume. It does not fail because the lead has no judgment. The dangerous substitute is a comment the team is free to ignore. The hour did not move onto a gate. It moved onto a paragraph people open after the merge, if they open it at all.

The owner meets the miss after the tag. A refund path the ticket named is absent for customers. A query expects a column the release never added, because the migration file was not in the diff. The story in the channel is that AI reviewed the pull request. The job log says the model did not answer, or it answered about formatting. The gap was a delivery fact. The check was not aimed at facts.

3. What the check may flag

Start from three facts the team already treats as required. The acceptance line on the ticket. The tests the repository already requires. The release items you refuse to cut a tag without: a feature flag left at the safe default, a migration present when queries changed, a rollback note beside the tag. The check may say which of those is missing. It does not get to invent a fourth rule about style, tone, or how clever the diff looks.

The handoff from ticket to pull request is the first fact. Put one acceptance line beside the diff, in the words already on the ticket: a partial refund writes a ledger row. The check may say that sentence is not reflected in the changed files. A person confirms that in a minute, or points at a shared package the diff does not show. That confirmation is the catch. The gate held a real gap, or a person recorded why the flag was wrong.

Here is the same rule when it actually stops a release. The acceptance line is on the ticket. The diff changes a controller and does not change the ledger write. The model returns that the line is missing. The job is required, so merge stops. The author adds the write, or the release owner records an exception with a name, a time, and a link. That is a check. The green timeout from the week before was not a check. It was a label.

A remark that the names could be cleaner does not belong in that required job. Style may stay a comment. It must not be what turns the check red, or the team learns that red means taste and starts clearing red the way they clear any inbox. A noisy note that trains people to ignore a real miss is the neighboring failure, written up separately as when AI quality checks only add noise.

The pipeline is the second fact. The check may say a previously required test job is skipped, allowed to fail, or removed in this diff. It may say its own model call returned no answer. Those two lines are enough for CI. Both are visible in the job list and the log. They do not need a score, and they do not need a paragraph about architecture.

The release is the third fact. When someone creates the tag, the check may say a named checklist item is still open. The item has to be a line the team already uses. An open item is a gap the release owner can verify. A closed item stays quiet. If every tag grows a long comment, the check became a second inbox, and people will dismiss it, including the morning it was right.

4. What stays with a person

A fair flag is not a product verdict. The release owner decides whether the missing line matters for this tag, or whether the behavior lives in code the diff does not show. The model will miss a ledger write that sits in a package. It will flag a line the team removed from the ticket yesterday and forgot to edit. Both mistakes are ordinary. They stay useful while a person can mark them. They turn sour when pay or a public score depends on a week with no flags.

An exception needs a name and a time. The note beside the pull request can read: accepted by the release owner, the write lives in the shared package, with the date and a link. The gate keeps that note. It does not silently start passing every future comment that mentions a ledger. A permanent ignore is how next month's real gap leaves under the same label.

Product judgment stays with a person. Whether this refund should exist, whether one account may run with the flag on, whether Friday's tag can wait: the model does not own those calls. The check can show that a required line is missing. The conversation about shipping anyway belongs to the owner, in writing, with a time. A site chat does not take that conversation, and neither does a score on the pull request.

5. One repository, one week, two numbers

Use one repository that will cut a release this week. Do not roll the check out to every service on Monday. Make one job required. Limit what that job may say to the three facts: the acceptance line is missing from the diff, a required test is no longer required, or the model returned no answer. Every other remark stays a comment that cannot block merge.

Count the merges where the model returned no answer and the pipeline stayed green. That count belongs at zero. If it is not zero, the job fails open, and the integration is not a check yet. Then count the notes a person confirmed as a real gap that would have shipped. If that count is zero and the comments are only about style, the tool is a narrator. A narrator stays off the required list.

While you count, the check follows the pull request. It does not replace the tests the repository already runs. It does not talk to visitors. The people this gate is for are the ones who merge and the ones who own the release.

• The AI job is required, and a timeout or a model error fails the job instead of painting it green.
• A comment that cannot block merge is not called a quality gate.
• The required job may name a missing acceptance line, a skipped test, or an open release item, and nothing else.
• A person records an exception with a name, a time, and a link. The job does not learn to pass that class of note forever.
• No site chat is added as the AI integration for delivery.

When to ask for help

You can read one job log on your own. Ask for help when the AI job is green on an error, when a red comment cannot block merge, or when you cannot separate a required fact from a style remark. Ask when a vendor folded a site chat into the same offer as the pull-request check. Those are different products. The chat talks to people who have not opened a ticket. The check is about a change that is about to ship.

Services cover that gate on a repository you already ship. If your pull-request check stays green after a model timeout, the AI integration service rebuilds the job so it fails closed instead of passing silently. Selected work includes this kind of delivery path as an operation, not as a slogan. If you write, name the check, say what the job does when the model times out, and name one release that went out green with a gap you can point at. Leave customer records and repository secrets out of the first message.