On September 24, 2026, Dataiku published its Global AI Confessions Report, CIO Edition, based on an online survey of 685 CIOs across eight countries, conducted by The Harris Poll from July 9 to 29, 2026. Among US respondents, 82% said they regret at least one major AI vendor or platform selection made in the past 18 months.
Regret is only half the story. Across the full sample, 75% of CIOs said they plan to use more or different models to improve AI continuity, and 74% are considering open-source or open-weight models as a hedge against future model unavailability. Eighty percent have conducted or plan to conduct AI dependency risk assessments. CIOs are already planning for a swap.
They have reason to. On October 14, 2026, OpenAI announced that GPT-4o snapshots, GPT-4o mini, GPT-4.1, and GPT-4.1 mini will shut down on January 14, 2027. On October 1, it had already scheduled gpt-5.1, gpt-5.3-codex and gpt-5.4-nano for retirement on April 1, 2027. If your product runs on one of them, your migration date is already on someone else's calendar.
The tooling around the models moves too. OpenAI's deprecation page shows its Evals platform going read-only on October 31, 2026, and shutting down on November 30, 2026. Its reusable prompts API and Agent Builder also shut down on November 30. Teams that kept their test sets and prompts inside that platform now have a migration project of their own.
A disclosure before we go further. Golabs designs and builds custom AI systems, so we benefit when companies decide to own more of their AI stack. Dataiku sells an AI platform, Accenture sells AI services, and Forrester sells research. We flag each source's interest where it applies, and we are explicit about where portability is not worth the cost.
What Does It Mean to Build AI That Survives a Model Swap?
It means designing your AI so the model is a replaceable part, not the foundation. Prompts, test sets, data pipelines, and business logic live in systems you own, and you control every model call. With 82% of US CIOs regretting a recent AI vendor pick, we believe changing models quickly is an architecture decision you make early.
In our experience, many custom AI projects start the other way around. A team picks the strongest model available, builds directly against its API, tunes prompts until the demo works, and stores evaluation data in the vendor's console. That is a reasonable way to prove value quickly. It becomes a liability once the system reaches production and the model starts to change underneath it.
The change rarely arrives as a single event. A model is retired on a published schedule, a newer version answers the same prompt differently, a price moves, or a cheaper model becomes good enough for half your workload. Each one raises the same question: how much of your system must you rewrite to take advantage of it or survive it?
Why Do So Many CIOs Regret Their AI Vendor Choices?
The survey doesn't ask why, so what follows is our reading of the surrounding data, not Dataiku's finding. Two other numbers on the same page point clearly in one direction. Nearly three in four CIOs (72%) cannot consistently measure whether all their AI agents are delivering business outcomes.
In the same survey, 72% of CIOs say their AI budget is likely to be cut or frozen if performance targets aren't met by the end of 2026, rising to 87% among US respondents. When results are hard to prove, and budgets are on the line, a vendor choice made 18 months ago under different assumptions is easy to regret.
Accenture's September 10, 2026 report, "The CIO's Guide to AI Tokenomics," adds a cost angle. Drawing on a survey of 750 senior executives at companies with more than $1 billion in revenue and its own research, Accenture reports that 54% of AI requests are misrouted to a more capable, more expensive model than the task needs.
Accenture also found that 90% of the more than 9,000 tasks in its research do not require frontier-model capability. Its recommendation is to identify the roughly 10% of workloads that do, route the rest to mid-tier or open-weight models, and enforce that routing at the gateway. You can only do that if your architecture lets you choose a model per task.
What Does a Swap-Ready Custom AI System Look Like?
The pattern is not exotic, and none of it requires abandoning the vendor you use today. Most of it is ordinary software engineering discipline applied to a component that changes faster than most. These are the six practices we apply when we build tailored AI agents for clients.
Put a gateway between your applications and every model. Applications call your internal interface, and the gateway decides which provider and model to use. Routing rules, retries, logging, and cost tracking live in one place. When you retire a model, you change a configuration and rerun your tests instead of touching every application that depends on it.
Own your evals, and keep them outside the vendor's console. Your test sets, scoring rules, and acceptance thresholds are the most valuable asset in a model swap, because they tell you whether a new model is good enough. OpenAI's own Evals platform is scheduled to shut down on November 30, 2026. Store evals in your repository and run them in your own pipeline.
Keep prompts and agent logic in your codebase. Prompts are code. Version them, review them, and test them like code. OpenAI's guidance for its retiring reusable prompts API is to move prompt content into application code. Starting there removes one migration entirely, and it lets you keep model-specific prompt variants side by side when two models need different wording.
Route by task, not by habit. Classify workloads by what they actually need, such as reasoning depth, latency, cost ceiling, and data sensitivity, and assign a default model to each class. Accenture found that 54% of requests go to a more expensive model than needed, suggesting many organizations haven't done this yet. Our AI orchestrator is built around this idea.
Keep your data layer model-agnostic. Retrieval indexes, embeddings, document stores, and conversation history should not depend on one provider's proprietary format. Plan for re-embedding when you change embedding models, and budget the compute for it. The data you collect about what good answers look like belongs to you and should move with you.
Rehearse the swap before you need it. Once a quarter, run your full eval suite against a second model, ideally from a different provider or an open-weight family. You learn what breaks while nothing is urgent. Dataiku found 80% of CIOs have conducted or plan to conduct AI dependency risk assessments, and a rehearsal is a concrete version of one.
How Much Does Portability Cost, and When Is It Not Worth It?
Portability is not free, and anyone selling it as free is overselling. A gateway adds a component to run and secure. Evals take real effort to write and maintain. Supporting two models means testing twice. For a short-lived internal tool or a prototype, building directly against one vendor is often the right call, and we tell clients so.
There is also a capability trade-off. Some of the most useful features are vendor-specific, and designing for the lowest common denominator can leave value on the table. The goal is not to avoid every proprietary feature. It is to know which ones you depend on, wrap them behind your own interface, and decide in advance what you would do if they disappeared.
The calculus changes when a system is customer-facing, regulated, embedded in a core process, or expected to run for years. Those are the systems where a forced migration costs the most, and, in our view, where a regretted vendor choice is most expensive. Our rule of thumb is simple: the longer a system will live, the more of it you should own.
Hedging is spreading below the model layer as well. Forrester's 2027 technology predictions, published October 21, 2026, forecast that in 2027 a quarter of AI infrastructure RFPs will require non-NVIDIA accelerator options. That is a prediction, not a measurement, but it points the same way: buyers want room to switch at every layer.
What Should You Ask Your AI Development Partner?
If someone else is building your AI, portability belongs in the contract and the statement of work, not just the architecture diagram. Ask who owns the prompts, evals, and fine-tuned weights. Ask how the system would move to another provider and what that would cost. We cover the wider vetting process in how to choose an AI development partner.
Ask what happens when the build team leaves. A swap-ready system is only useful if your own people can run the swap, which means documentation, handover, and access to every repository and pipeline. Our post on who owns your AI agent after the vendor's engineers leave walks through that handover in detail.
Finally, ask for proof. A partner who claims a system is portable should be able to show the eval suite running against a second model. That one demonstration tells you more than any slide. It is also the clearest test of the principle we described in buy the model, build the moat: the model is rented, everything around it is yours.
Common Questions About Swapping AI Models
Why do companies regret their AI vendor choices?
Dataiku's September 2026 survey found 82% of US CIOs regret at least one major AI vendor or platform selection from the past 18 months, though it does not ask why. The same survey found that 72% of CIOs cannot consistently measure whether their AI agents deliver business outcomes, making early choices harder to defend when budgets tighten.
How often do AI models get retired?
Often enough to plan for. OpenAI's public deprecation page lists more than a dozen deprecation announcements in 2026 alone. On October 14, 2026, it announced that GPT-4o snapshots, GPT-4o mini, GPT-4.1, and GPT-4.1 mini will shut down on January 14, 2027. Any system built on a specific model should assume a migration date will arrive.
What is an LLM gateway?
An LLM gateway sits between your applications and AI model providers. Applications call the gateway, and it decides which model answers, applying routing rules, retries, logging, and cost controls in one place. It lets you change or add models through configuration instead of rewriting every application that uses AI.
Should we use open-weight models to avoid lock-in?
They can be part of the answer. Dataiku found that 74% of CIOs are considering open-source or open-weight models as a hedge against future model unavailability. Open-weight models give you more control over versions and hosting, but they add operational work. Stronger protection comes from an architecture that lets you test and switch between models.
If you are planning a custom AI build, or want an outside review of how locked in your current system is, talk to the Golabs team. We will start with your eval suite, and we will tell you where portability is not worth the cost.