Goldman’s shift on AI highlights the gap between prototype and production that every bank has

News broke last week that Goldman Sachs has pulled Claude Code and GitHub’s AI coding tools back from its traders and bankers. Under internal guidance updated in October, the tools now sit largely with engineers and the strats who work between the desk and technology. The bank says the change followed a review of productivity and risk management.

It is a sharp turn. As recently as July, Goldman’s CIO was describing a model in which business staff would prototype applications with AI and the bank’s engineers would refine them. Some traders built their own tools to watch client flows, model positions and create content materials. Several are reportedly frustrated by the reversal. I would be too.

Yet Goldman made the right call. A regulated institution ran an experiment, examined the results and weighed them against its risk appetite. That is the discipline we should expect from a bank. The rest of the industry should read the decision as a signal, not an outlier, because the forces behind it are present at every firm.

What happens next? If the industry concludes that only engineers can be trusted with AI-built software, every request from the desk returns to the engineering backlog it was trying to escape. The need that sent it to an AI coding assistant in the first place does not go away. It compounds.

The problem AI exposed was never the code

Strip away the headlines and the Goldman story is about capacity. Every capital markets CTO runs two books of work. One is the strategic book: cloud migration, core platform modernization, the move to T+1 in Europe, regulatory reporting, data programes. These are generally large, mandatory and rightly consume the best engineering talent.

The other book is the ever growing list of business-enabling solutions demanded across the bank. A desk needs a better view of axes and client flow. An operations team wants an exceptions workflow that replaces three spreadsheets and an inbox. A sales team needs a pricing tool for a new product. Each request is modest. Together they are enormous, and almost none of them will ever outrank the strategic programmes in a prioritisation meeting.

The business has always found ways around that queue. First it was spreadsheets, then VBA macros, RPA, Python notebooks and desktop dashboards. Each wave delivered real value and left behind an estate of tech debt and end-user computing that nobody fully owned. When those tools failed, the industry learned that a calculation can run perfectly and still be wrong. Each time, the answer was to lock the tool down. Each time, the demand simply moved to the next one.

AI coding assistants are the most powerful version of this pattern yet. A trader who can describe a workflow can have working code by the end of the afternoon. That is genuinely remarkable, and I would not want the industry to lose sight of it. But it means the long-tail that IT has never had capacity to serve can now produce software faster than IT can review it.

Easy to start, no easier to finish

AI makes it easier to start a software build, but in financial markets, the finish line, deploying an application into production, is where nearly all the cost and risk sit.

A prototype that works in a demo has answered one question: does it run? A production application at a regulated institution has to answer many more. Will it hold up under a volatile session’s message volumes? Does it leave the audit trail that MiFID II, SEC and FINRA rules require? Can a different engineer support it in two years? A general-purpose coding assistant brings none of that with it.

There is a deeper mismatch too. Large language models are probabilistic. They produce the most likely good answer, and for well-understood problems they do it brilliantly. Financial software rests on deterministic requirements. An access control enforces the right entitlement or it doesn’t. There is no such thing as probably compliant.

The pressure moves to the handover. What the business produces with AI are, in effect, prototypes, and hardening them falls to engineering. Engineers interviewed by Waters have described exactly that: output up, but review effort up with it. Spread across a whole institution, the hardening queue grows faster than any team can clear it. Most prototypes never reach production. The ones that do often arrive by a side door, which is the worst outcome of all

The shifting landscape for AI

The industry is already testing different ways to cope with the production challenges firms are encountering. Goldman narrowed who can use the tools. Some firms are going the other way, letting staff outside engineering build on a governed internal platform designed around its data, legal and compliance needs. Another approach keeps coding assistants with professional developers and holds them accountable for what they commit. These are different bets, but they share an insight: the question is not whether AI writes code, but who authorizes what that code does.

From where I sit, four conditions have to hold before business-built software comes anywhere near production.

First, the controls must already exist. Entitlements, audit, resilience and integration should come built in, tested once and reused, rather than reinvented each time someone asks an AI for an application.

Second, the AI’s role must be narrow. It should fill specific, bounded gaps, so the same request produces the same result instead of something new each time.

Third, IT must set the standards and decide what goes live. The business describes the need and validates the result. Technology owns the architecture, the approval and the off switch.

Fourth, nothing can be a black box. Every artifact, from requirements to tests to code, should be readable by any engineer on the team, in technologies the firm already runs, long after the original author has moved on.

Not everything belongs in this model. Some systems are so critical that they will and should stay with engineering. But a large share of the long tail can be served this way, and that is where the backlog lives.

Governance is the enabler, not the brake

For CTOs, the practical work starts with a few honest questions. How much of your long-tail backlog will never be prioritized under current capacity? How much software initiated by the business is already running outside your line of sight? Which applications genuinely need a bespoke engineering effort, and

which could be assembled from foundations you already trust? And when something built with AI goes wrong, is it clear today who is accountable?

The answers will differ by firm. The direction should not. Banning the tools leaves the need unmet and pushes it into the shadows. Handing them out freely moves risk from the business to the balance sheet. Neither is a strategy.

Goldman will not be the last bank to make this call, and I respect that it made one. But the firms that pull ahead will not be those that give their traders the most powerful tools, or those that take them away. They will be the ones that give the business a safe, governed way to build, without asking IT to compromise on the security, compliance and resilience that our markets depend on.

This commentary was first published in Waters Technology.