Where Generative AI Services Belong in Delivery

We spent a year treating this as a product decision. Which model, which vendor, which use case. The more useful framing turned out to be narrower. Where in our existing delivery process does this actually change something, and who owns it there.

Answering that produced better results than any model comparison we ran.

Inside engineering. Our engineers use assistants daily for tests, routine code, and understanding unfamiliar parts of the system. Review load went up and typing went down, which is the correct trade.

Inside the product, narrowly. A summary that saves scrolling. Plain language search. Sensible defaults on a repeated action. None of it impressive and all of it reducing steps for a user.

Everything else we tried either did not survive evaluation or produced enthusiasm without measurable effect.

This is the area that surprised me most.

Generating tests from a specification works well for the obvious cases and needs review. Maintaining selectors after a design change is genuinely faster. Both are real time savings.

What changed more is what we test. A system whose output varies between runs cannot be asserted with a pass or fail check.

Our test automation services partner had to build something different. Standard test automation services assume a deterministic answer. A scored evaluation set with a threshold, run in the pipeline, treated as a gate. A drop in score blocks the release the way a failing test does.

That took a fortnight to build and it is the reason we can change prompts without discovering the consequences from users.

Generative AI services deployed outside your delivery process become a separate system with separate rules, and that is how estates fragment.

Ours ship through the same pipeline as everything else. Same review, same environments, same release process, with two additions.

The evaluation set runs as a gate. And token usage is recorded per change, so a prompt edit that improves answers slightly and doubles cost gets discussed rather than merged quietly.

Teams engaging devops consulting for SaaS products should raise this specifically. The tooling question is whether your pipeline can run a scored evaluation and act on the result, which many cannot without work.

Our first attempt looked for places to apply the technology. That produced a list and very little else.

The second looked at where our teams already lost time. Writing tests. Reading long case histories. Searching for something in documentation nobody had structured.

Three of those became features. The list from the first attempt produced none.

Generative AI services deliver most when pointed at a task somebody already does repeatedly, rather than at a capability somebody wants to demonstrate.

An answer is a product of the model, the prompt, and the retrieved content.

All three are versioned and pinned. When any moves, we run the evaluation set against both and look at what changed.

We learned this when a provider updated a model underneath us and our answers shifted with no deployment on our side. Nothing had broken. Nothing alerted. Quality had simply moved.

The failure I have seen most often in other organisations.

A generative feature is built by whoever was enthusiastic, ships, and then degrades slowly because nobody is watching the right signals. It does not fail. Retrieval quality drops, latency creeps, refusal rates move.

Ours has a named owner per feature, the same as any service. They watch grounding rate, validity, latency at the tail, and cost per outcome.

Without that, generative AI services produce a demonstration that becomes a liability on roughly a nine month cycle.

●      What task does this replace, and how long does it take today?

●      Can a reviewer check the output in under a minute?

●      What happens if it is wrong and nobody notices?

●      What does it cost per successful outcome at expected volume?

●      Who owns it after launch, by name?

The third question does most of the work. It separates a summary that wastes five minutes from a record change nobody catches for a month.

Who should scope this?

The team that owns the process, with engineering. Generative AI services scoped by a technology function alone tend to answer engineering questions rather than the ones the business asked.

Build or buy?

Buy the models and the plumbing. Build the workflows and the connectors to your own systems. The workflow is where both the value and the risk live.

Where does the delivery tooling need to change?

Two places. A scored evaluation gate, and cost recorded per change. Teams buying devops consulting for SaaS should ask about both specifically.

How do we test something non deterministic?

Scored evaluation rather than assertion. Your test automation services partner should be able to build that, and many have not been asked to yet.

Should this sit with a separate team?

Not permanently. A central team for the shared platform and the controls, with product teams owning their own features, has worked better for us than either extreme.

What is the most common mistake?

Treating it as a model project. Generative AI services are systems work with a model inside them, and teams who understand that ship sooner.

The cost nobody puts in the business case.

Reviewing a sample of output monthly is somebody's standing job. So is revalidating when a model, prompt, or document set changes.

That is an operational cost, not a project one. Any proposal presenting generative AI services as an implementation with an end date has left it out.

We fund it annually, like audit, and it is the reason our features still work rather than having quietly degraded.

We stopped asking what we could build with this and started asking where in our delivery process it changes something.

That produced two answers inside engineering, three inside the product, and a much longer list of things we decided not to do.

It also made the ownership obvious, which is the part that determines whether any of it still works a year later.