What Does AI Cost? No One Knows.
Why is it so difficult to get the engineering team to tell what AI tools cost?
Well, it’s not really about cantankerous engineers. (Most of the time, it’s not.)
It’s really about how different groups in an organization manage their work and the KPIs they set for themselves. DevOps teams are sensitive to encroachment. FinOps teams face criticism for their insistence on controlling costs for work that an engineering team deems essential.
DevOps and FinOps groups can collaborate, and they often do so in organizations that have structured approaches, enabling each group to meet its goals and align with the overall enterprise’s best interests.
A DevOps team manages a delivery pipeline to enable software deployments into production with velocity and efficiency. A FinOps team optimizes cloud spending and seeks value for the organization.
At FinOpsX last week in San Diego, the reality of AI opens a new maze that is infinitely more complex than managing traditional cloud costs and the value it brings.
DevOps KPIs do not align with the efforts of FinOps practitioners, said Robert Reeves, an independent consultant and one of the founders of Liquibase, a database change management company with an approach it calls “Database DevOps.”
As many of our readers know, KPIs for engineering include mean time to recovery, velocity and the number of pushes a team makes into production.
“That’s where engineering is, that’s how management looks at them, for KPIs,” Reeves told The New Stack. “And it’s not in alignment with what we see from our friends on the FinOps side of the house. So that’s why there’s that disconnect.
“Engineering is looking at the FinOps practitioner and saying, ‘Why are you telling me how to do my job? I won’t tell you how to do your job.’ And so that’s what the conflict is.”
Limited Visibility Into Costs
At FinOpsX, Reeves stood by the pool at the Marriott Marquis San Diego Marina, explaining that what’s to come doesn’t look pretty. FinOps teams have limited visibility into the tools and platforms used by engineering teams to integrate generative AI (GenAI) capabilities into their products.
“They’re both trying to do good things for the organization,” Reeves said. “But they’re going about it differently, and both sides are saying, ‘Oh, it’s so hard to work with the other ones.’ And now we have these new dynamics with the adoption of AI.”
Developer teams may receive more investment for developing AI. Still, FinOps teams often have limited visibility into the costs of these tools and platforms, as well as the value they deliver.
And while investments continue to roll in, the product team may not even charge more; instead, it sometimes absorbs the cost for strategic reasons.
The FinOps teams are looking at the engineering teams and saying, “Well, what models are you using?” Asking all these kinds of questions can get uncomfortable.
“ I think AI spend is a ticking time bomb,” Reeves said.
J.R. Storment, executive director of the FinOps Foundation, said in an interview before the conference that FinOps professionals should currently focus on data collection. It’s only by knowing the data that optimization occurs.
To its credit, the FinOps Foundation has developed a substantial body of information for teams entering the world of FinOps AI. For instance, it has created a framework to quantify AI and its impacts on the enterprise. The foundation announced a certification called “FinOps for AI.”
Reeves said the community knows the return on investment (ROI) spent on cloud spending. He credited the FinOps Foundation with releasing FinOps Open Cost and Usage Specification (FOCUS) and a standard data model to evaluate cloud spending across all providers. At FinOpsX, the Foundation announced the release of FOCUS version 1.2.
Pressure From the Corporate Suite
The executive team wants to capitalize on the opportunities it sees in AI. Product teams get the pressure, and they also see their competitors adding AI capabilities, fueling the race.
Additionally, companies are spending small fortunes on GPUs. One airline customer for one of the cloud services, according to a FinOpsX presentation, has engineers on three eight-hour shifts, 24 hours per day.
And that leaves the FinOps practitioner in a bind. They’ve finally reached the point where they can optimize cloud spending, but now comes AI and its onslaught.
“Now they have to worry about AI spend, whether it’s GPUs or LLM [large language model] usage, or whatever,” Reeves said.
Product teams are layering AI into the services that they offer. However, the engineering teams are not exactly transparent about the models they’re using.
Furthermore, cloud providers do not disclose their own internal stacks, platforms, tools and other components and capabilities that come with their AI offerings. They certainly don’t discuss their hardware configurations or what custom chips they use. It would be pretty unusual for them to discuss their training methodologies or data sources.
Still, the money flows for AI. Companies are clamoring so much that large consultancies are being asked for AI anything from customers who face their internal mandates. They want something AI. So, they utilize AI, an off-the-shelf service based on a standard LLM.
“And so you’ve got different parts of the organization that are spending a ton of money,” Reeves said. “It doesn’t seem to be everybody who wants to spend money on AI is getting their budget approved, and that is gonna cause problems because the business does not know which department has the best ROI for that spend.”
