Frontier Infrastructure

I've spent too much thinking about the dissonance between what AI platforms (Anthropic, OpenAI, Google, etc.) are saying about their business model and what they actually seem to be doing. This was before all the congressional lip service about AI regulation. – To be clear, I think this is a valid concern and should be addressed, I just don't see Congress actually doing anything about it.

With the possible exception of OpenAI, all the major platforms claim their goal is to build an AWS-style infrastructure for everything (and they do mean everything). This is a great goal and a solid business proposition (except that we have ~6 companies that are all promising the same thing and have roughly interchangeable models). However none of them seem to be actually taking the steps to meet this goal.

For infrastructure, the three main components that are needed are 1) Reliable Response, 2) Rapid Response, and 3) Reliable Billing.

Reliable Response

Evaluating Reliable Response for an LLM is more of a ballpark since the systems are, by definition, generative. You're never going to get the exact same answer for the same question twice (math excluded, maybe). So we're really asking, are these answers similar enough, within reason, that I can reliably predict what I'm going to get back?

And the answer is largely yes. There is still drift that happens over a seemingly arbitrary time period as a model ages. Stability and drift are non-trivial, especially in the case of something like Claude Code where velocity and token use can vary widely depending on how introspective/thorough a model becomes and how reliable. But for most business implimentations, it's fine.

Rapid Response

Evaluating Rapid Response is fairly straightforward: the more reasoning and research that's necessary, the longer it will take, but even that is getting shorter and shorter.

Reliable Billing

Evaluating Reliable Billing is tricky because billing is dependent on a wobbly card tower of choices: how many tokens are input? How many are output? How atomic is the context? How much reasoning does the problem require? How much over-reasoning and researching is the model performing?

In general, what I've seen in a business/API implementation is (in order): not many, slightly more, very atomic, not much, and generally more than a human would deem necessary unless explicit instructions are given.

So, it's really the Reliable Billing piece that does not seem to be being addressed by our model providers. This is somewhat ironic given that API calls are almost certainly the highest margin part of their business. It turns out, is a major problem for businesses at all scales. Enterprise clients are finding their employees either peg the needle on token consumption or don't use it at all (varying from employee to employee), and SMBs are mirroring the employee to employee pattern and finding that they are either becoming overly reliant on LLMs, to the point of them becoming a major cost center, or not using them at all.

The expected response from the model providers is, of course, it's still cheaper and faster than hiring N employees! Which; fair. The acceleration of expected deliverables ensures that competitive companies are not going to taper their usage any time soon. At least, not until their COO or CFO tells them they need to. Which creates a cascading set of problems. You don't miss your water 'til the well's run dry. And then what are you supposed to do? One hugely productive quarter followed by one significantly less productive as the quotas take hold.

But more to the point, the frontier model race is wildly unnecessary for most business applications. It may act as an enterprise sales pitch, but that is a sales pitch that’s more expensive than several Super Bowl commercials. In fact, it is difficult to think of business problems that existing models have not already solved.

Anything relating to writing or basic analysis has been solved, though still requires an editorial pass. Image and video generation still isn't fully solved, but it's pretty good at this point, certainly good enough for social media. Other use cases absolutely require an editorial pass. Ironically, I have yet to see a model reliably create coherent slide decks.

Cursor developed an effective, model-agnostic pattern wherein a user can use advanced models to develop plans or analyze existing code, but use its much faster and highly reliable coding model to actually create the code. This is method is relaible, fast, and cost-effective.

So why are the model providers focused on frontier development like it's a space race when what the vast majority of businesses need is faster, cheaper, more reliable, not deeper analysis? Most companies are not focused on Navier-Stokes level problems or really complicated curves but they are heavily impacted by each potentially breaking or cost increasing micro update to their model or interface.

We're getting a new step change model roughly every 6 months, breakthroughs in materials science, biotech and medicine, math, and computer science. All of which is really cool for humanity, but doesn't do a damn thing for the reliable backbone I need to create dependable business applications and features.

I humbly suggest that our model providers fork their dev teams, with one dedicated to this frontier research and development, while steering the majority of their company to focus on an actual infrastructure-centric product.

It's not as sexy because infrastructure is inherently not sexy. Infrastructure is reliable and as fast and as cheap as possible.