We went from early LLM experiments to broad use of chats and coding agents across a large engineering organization at Raiffeisenbank. This article is based on Raiffeisenbank’s experience adopting LLM tools and coding agents across roughly 500 engineers. On the adoption dashboards, everything looked healthy: people opened the tools, tried them, came back, brought examples, showed demos, and tech leads helped their teams learn.
The flow metrics did not move for a long time.
Lead Time did not go down in a systemic way. Throughput did not grow enough to be visible at the level of the engineering organization. Defect Rate did not give a clear signal that quality had improved.
This is the core lesson from that case: counting AI users is useful. It is no longer enough. At some point the real question changes. How many real work items pass through their stages with agent support? Where does the agent enter the workflow? How many items move from idea to outcome with agent support across most of the path?
Two ways to read this article
If you came for Agentic Engineering, jump to “What Agentic Engineering has to change”. The main point starts there: the unit to accelerate is not the AI user. It is the path of the work item through the system.
If you want the context, start from the beginning. The first sections explain why growing MAU for LLMs and coding agents looked good and still did not move Lead Time or Throughput.
Why bring AI into engineering teams at all
It is easy to introduce AI as a fashionable tool: give people access, run a training session, collect the first enthusiastic stories, and draw a chart of active users. After a month, it already looks like adoption is happening.
For an engineering organization, that is not enough.
In software delivery, the important question is whether a work item moves faster from “we will do this” to production. Users, clients, business stakeholders, and regulators do not see the number of prompts. They see speed, quality, and predictability of change.
So we looked at three flow metrics.
Lead Time — how much time passes from the moment a team takes a work item into delivery until it reaches the customer.
Throughput — how many work items a team actually delivers in a given period.
Defect Rate — whether quality gets worse along the way.
The expectation from AI was simple: if it really helps the engineering system, Lead Time should go down or Throughput should go up, while Defect Rate stays under control.
This frame matters. An adoption dashboard shows that people have started touching the tool. Delivery metrics show whether the system of work has changed. Business outcomes explain why the effort exists in the first place: faster change, lower operational effort, safer work with risk, and faster validation of product hypotheses.
First attempt: LLM chats
The first step was obvious. Give people access to LLM tools, explain the value, run training, show useful scenarios.
AI adoption is still a change in habits: people need to understand the reason, try the tool, learn how to use it, and reinforce the new way of working.
It worked at the level of usage. People started using AI for writing, explanations, idea generation, small questions, and first drafts. Monthly active users grew. Regular users appeared. Inside the organization, it became easier to talk about AI without treating it as a toy for enthusiasts.
The flow metrics did not change.
A simple formulation emerged: chat was more like tourism than delivery work.
A person enters, looks around, asks a question, gets value, and returns to the old workflow. They may save 15 minutes, understand an unfamiliar topic faster, write an email without friction, or sketch a document.
The work item in the system continues along the same route: analysis, development, testing, review, approvals, release, support.
The LLM lived next to the work. It barely entered the path to production.
So MAU for LLM tools became a good signal of interest and habit. It did not become a signal that the engineering system was getting faster.
Second attempt: coding agents
The next hypothesis looked stronger: AI should enter the place where code, tests, documentation, and pull requests appear.
Coding agents are closer to real engineering work. They can search through the codebase, prepare a first version of a change, write tests, update a README, help with the terminal, explain inherited code, and gather context for a pull request.
We moved in that direction.
The best formats were the ones where people worked on a real task immediately: a real ticket from the tracker, a small group, the participant in the driver’s seat, an expert nearby as a navigator, a tech lead showing the scenario on their own task, and a team regularly sharing findings in internal AI Labs.
Hackathons worked well as the first funnel. They gave reach, lowered fear, and helped many people enter the tool quickly. Workshops worked deeper: each participant came with their own task and went through setup, prompts, errors, clarifications, and a result. That is how muscle memory started to form.
Coding agents began to enter the workday of some engineers. People stopped treating the agent as a separate toy. Local stories appeared: code showed up faster, tests were written earlier, documentation was updated with less resistance.
The flow metrics still did not move systemically.
That was the uncomfortable part. We had taken a more relevant step, closer to engineering work. We had created habits for part of the organization. We had good local stories.
The whole route of the work item to production barely moved.
Where the real constraint was
Motivation was not the main constraint. People understood AI well enough to start using it. The constraint was the design of the workflow.
Developers adopted agents faster. Testers moved at a different speed. DevOps and analysts used agents in different scenarios and with different intensity. One part of the flow became faster, while the other parts continued to work in the old way.
Coding-agent adoption was uneven across roles, so the flow accelerated locally instead of changing as one system.
Lead Time rarely consists of development alone. In a living organization, a work item waits for requirement clarification, review, a test environment, a release window, business validation, a neighboring team, or a risk decision.
Imagine an ordinary product change: adjusting a fee calculation rule.
A developer with an agent finds the right places in the code faster, drafts the change, writes tests, and prepares a pull request. Their part goes from two days to half a day.
Then the item waits two days for clarification from an analyst. Then it gets stuck on a test environment. Then it waits for review. Then it moves to the next release window. Then the business asks for one more check because the rule touches money.
In a local metric, the developer became faster. In the Lead Time of the work item, almost nothing changed.
Short version: the accelerated stage simply delivered the item to the next place where it had to wait. The speed of the system is not defined by the fastest person. It is defined by the place where work waits.
That is why local acceleration in development quickly runs into requirements, testing, review, release, or business validation.
The adoption dashboard starts lying when it becomes the whole picture
You still need the adoption dashboard. Without it, you do not know whether the tool reached people. MAU, regular usage, the share of coding-agent users, tech-lead activity, workshop attendance — all of these are useful signals.
The problem starts when the dashboard is read as the result.
User growth shows that people opened the door. It does not show that delivery changed. Regular usage says a habit is forming. It does not show where waiting in the flow decreased. The number of model calls shows activity. It does not answer which business goal moved.
After the first waves of adoption, we started looking at AI through the path of the work item: where it appears, what it does, and which part of the work it actually changes.
That created the next measurement layer. We became interested in work items: in which items agents actually worked, which stages they touched, where their involvement ended, and which part remained a manual bottleneck.
This changes the conversation. The team stops arguing about general enthusiasm for AI and starts looking at the route of the item. Did the agent participate in analysis? Did it help collect requirements? Did it check constraints? Did it find the right places in the code? Did it write tests? Did it prepare documentation? Did it help with checks? Is there a trace in a pull request, ticket, test report, README, or another work artifact?
This is the shift from counting tool usage to seeing agent coverage across the stages of a work item. Now we can see where AI is really inside the work, not only who has opened the tool.
What Agentic Engineering has to change
After two attempts, the next hypothesis became clearer.
If you want to reduce Lead Time, the agent cannot only accelerate an individual specialist. It has to help the work item move through the system.
This is where Agentic Engineering begins.
For me, Agentic Engineering starts when the agent becomes a participant in the flow of the work item. It gathers context from the ticket, code, and documentation; proposes a change; runs checks; writes tests; updates the README, documentation, or API description; and prepares the next step for a human.
The human keeps the frame: goal, constraints, quality criteria, risk decisions, and accountability for consequences.
That boundary matters. An agent can propose options quickly. The responsibility for the engineering decision remains with the human and the team.
The metaphor is simple: agents can be fast helpers, but they need a goal, context, constraints, and a way to check the result.
In this logic, Agentic Engineering changes the route of the work item. Fewer empty handoffs between stages. More prepared context. More automatic checks. More artifacts appearing in the place where work happens, without a separate reminder.
Now we count work items, not only users
The next practical shift for us is to count work items where agents work inside the flow.
The earlier question was simpler: how many people use AI? Now the question is more precise: how many work items passed through analysis, development, testing, documentation, release preparation, or validation with agent support?
The management focus shifts.
When we count people, we manage tool adoption. When we count work items, we start managing the delivery system. In the first case, we see who opened AI. In the second, we see where AI entered the work, which stage it accelerated, and where the item still waits for a human, an environment, a check, or a decision.
For example, a work item can be agentic only at the development stage. The agent helped find the right code, prepare the change, write tests, and assemble a pull request. That is useful. Then the item may continue on the old route: waiting for requirement clarification, a test environment, review, a release window, or business validation.
Another item may pass through more stages with an agent: analysis, specification, code change, tests, documentation, and part of the checks. Now there is a chance to influence Lead Time because the agent works on several parts of the route and helps the item keep moving.
So the depth of agent involvement matters more than the fact of usage: one stage, two stages, several stages, or almost the whole path to outcome.
That is how we see where agent use remains a habit of a single role and where it starts changing how work moves through the system.
For a management conversation, this is more useful than general MAU. If a team has many AI users and almost all work items get help only in coding, the next bottleneck will probably be in requirements, testing, review, or business validation. If the share of items with agent support across several stages grows, there is a stronger reason to expect movement in flow metrics.
The logic becomes: the adoption dashboard shows entry, stage coverage shows AI entering real work, multi-stage agent support shows a change in the route, and Lead Time, Throughput, Defect Rate, and business outcomes answer whether any of it mattered.
Why this leads to AI-first and AI-native teams
In this logic, the interest in AI-first and AI-native teams becomes easier to understand. I use both terms here for teams that design their workflow around agents rather than adding agents on top of an existing process.
They are often described as small groups of two or three people who use agents to take responsibility for the whole path of a work item: from understanding the problem to release and observing the outcome in production.
There is plenty of hype around this. The rational part is simple.
If a large share of Lead Time is spent on handoffs between stages, a small team with end-to-end responsibility reduces those handoffs. There are fewer transfers between people, less waiting, faster coordination, and more responsibility for the outcome as a whole.
AI-native teams are a hypothesis about the flow of work: fewer handoffs, more end-to-end responsibility, faster feedback.
In this model, the team tries to hold the whole path of the work item: from understanding to implementation, checks, and observing the outcome in production.
The test is concrete: does end-to-end responsibility combined with agentic work lead to lower Lead Time, higher Throughput, and stable quality?
We had such a pilot: a small team, a newer area with limited inherited code, end-to-end responsibility, and working with agents as part of the delivery workflow. The first month went into habit formation. People learned to work at the level of specifications, use agent frameworks, and split the work into roles and checks.
At first, the team did not become dramatically faster in Lead Time. But with a smaller group, it delivered a comparable amount of work, then started delivering more. Throughput grew and quality stayed stable. The sample was small, so we treated this as a signal rather than proof. That is still important: the first effect of an AI-first approach may appear as the ability of a smaller team to hold and increase the flow of outcomes before Lead Time falls.
Then the constraints start.
The cost of the approach
On a slide, a small AI-native team looks almost perfect. In a real organization, it quickly runs into the system.
The first constraint is cognitive load. End-to-end responsibility for the whole development cycle puts pressure on people. They need to understand the work item, code, tests, release, risks, domain constraints, and consequences. If a person moves beyond their role without support, they get more work instead of more leverage.
The second constraint is validation. Agents generate faster than people can check. Without tests, observability, canary releases, rollback, quality criteria, and clear constraints, speed turns into a queue of manual review.
The third constraint is domain knowledge. A business expert is often the only carrier of how the process really works: where exceptions exist, where mistakes are dangerous, and why a formally correct solution may be wrong in real life. If the whole flow constantly depends on one person, that person becomes the next bottleneck.
The fourth constraint is neighboring teams and shared processes. A fast team still lives inside an organization. Its work runs into security, architecture review, data, infrastructure, a shared release process, and units with a different level of AI maturity.
The flow starts waiting again.
Accelerating one team helps only until the work item reaches a slower external stage.
The operating harness matters more than heroics
If agentic work depends only on strong individuals, it does not scale well.
You need a harness: rules, context, skills, templates, checks, knowledge base, quality criteria, safe scenarios, and clear boundaries of autonomy. Some people call this an AI engineering platform. The name is secondary. The point is that an agent needs a working environment where it can take context, perform the task, and check the result in a repeatable way.
Experts do not disappear in this model. They spend less time answering the same questions manually and more time turning their knowledge into rules, examples, checks, and context for agents.
A tester helps the agent work with a test matrix, quality criteria, and risks. An analyst turns domain knowledge, exceptions, and constraints into context the agent can use. A DevOps engineer describes a safe infrastructure scenario so the agent does not improvise its way through the terminal.
In role-level agentic work, the human does not disappear from the process. They get extra hands and remain responsible for the quality standard.
This is how the human role changes. The person becomes the holder of the quality standard in their part of the system and gradually teaches the agent to do more work inside that standard.
What to check before an experiment
Agentic Engineering is a poor fit for a command like “from Monday, we work as AI-native.” A better starting point is flow diagnosis.
Before the experiment, look at the whole path of the work item: from the decision to do it to production and outcome validation. Where does the item wait? Where is context lost? Where is an expert needed? Where is the check fully manual? Where can the team already work with agents? Where do tests, observability, and safe deployment exist?
Then choose a narrow hypothesis. For example: “we will increase the share of work items where the agent helps with code, tests, and documentation.” Or: “we will test whether waiting in analysis decreases when the agent gathers context from the ticket, wiki, and code in advance.” Or: “we will measure how many items pass through several stages with agent support.”
Then the experiment becomes testable. It has an activity metric, a flow metric, and a link to a business goal.
Where this leads
After several waves of adoption, the lesson became clear: AI usage by itself does not accelerate the path of a work item to production.
LLM tools create personal value. Coding agents strengthen developers and individual roles. Agentic Engineering starts changing the route of the work item through the system.
This does not mean every organization urgently needs small AI-native teams. The working hypothesis is calmer: AI has to enter the places where work actually waits, loses context, or passes through manual checks.
So the next measurement layer for us is the work item. How many items received agent support at one stage? How many passed through several stages with agents? How many came close to an end-to-end path? Where did the agent stop? Where did the human become a quality control point, and where did the human turn into a queue of manual review?
After that, the conversation about Agentic Engineering becomes concrete.
The question is: which part of the work item’s path did we change — and did it become faster, better, or more useful for the business?
Agentic Engineering does not start with the agent. It starts with making the path of the work item visible.