What Developers Need To Build Successful AI Apps
Earlier this year, The New Stack’s Daryl Taft wrote that Java is rising as a first-choice language for building AI applications. Though Python continues to dominate AI use cases for data science, when it comes to core business applications, Java dominates.
Just as Spring improved the lives of enterprise Java developers by giving them a streamlined framework for building Java apps, and just as Spring Boot revolutionized the way enterprise devs build microservices, Spring AI is set to change the game again.
On the heels of the Model Context Protocol (MCP) Java SDK’s introduction, I spoke with Mark Pollack, a founder and original tech lead for Spring AI, an open source AI framework for Spring and Spring Boot, to learn about the world of AI engineering, frameworks and the MCP.
Read on for a summary of Mark’s thoughts from our conversation, edited for clarity.
What Is AI Engineering?
There’s been a common pattern: On one side, there are machine learning and data science people, and on the other side, there are software engineers. And now, I think, there’s a legitimate category between them called AI engineer — a term we’ve used consciously for at least 18 months. And Spring AI is an application framework for AI engineering.
ChatGPT came into existence and made some simple APIs that can be called by any dev in any language. Java developers know how to interact with APIs, and they are often far more complicated than model vendors’ APIs. Interacting with a model API is less complex; you just need to learn some AI terminology.
Once upon a time (as some of us remember), there weren’t database abstractions like JDBC (Java database connectivity) or ODBC (open database connectivity). Every database vendor did the same thing — but there was no standardization if you wanted to switch from one to the other. So you’d want a uniform API, a portable service abstraction.
Let’s say you have an application that might be enhanced by doing something with AI. If you’re in Java, in many use cases, you can use Spring AI to implement that idea with good maintainability, such as avoiding vendor lock-in.
If you want to try out every package from every research paper, you’ll be disappointed. We built Spring AI as a distillation of the most common things you need. If you’re trying to be the tip of the spear, you’ll be in Python. Organizations likely have an AI or data science working group that is Python-based, and those won’t go away. But the pie gets bigger.
About AI Frameworks
We believe AI framework competition is good for developers; there doesn’t have to be a consensus winner. Spring AI, LangChain and LangChain4j share similar goals. There are things each does better than the others.
Two years ago, the LangChain codebase wasn’t something we wanted to pull in. Something can be a great project and adopted by many people, but it is still very far from an object-oriented view that you would see with Java developers.
LangChain is the 800-pound gorilla that got people interested in AI frameworks, and LangChain4j exists. But we felt there needed to be a Spring-based AI framework as an extension of the Spring ecosystem.
We could have used a collection of other projects, but our mission has always been to provide Java developers with the best way to build their apps. We believed AI engineering was a key capability that we needed to own and have an opinion on.
How To Approach AI Engineering
Java developers looking to get started with AI engineering need a handful of fundamental abstractions to play.
If you want to analyze a corpus of corporate documents with a model that hasn’t been trained on it, you need a vector database. OK, there are 20 of them, and all of them have different APIs and their own query languages. No one wants lock-in.
If you look at these query languages, it’s hieroglyphics. Triple underscore, curly brace, dollar sign … Instead, we wrote our own small query grammar. It looks like simple SQL: Where this, then that. We translate that to the query language of the vector store. So we have a Spring concept called a portable service abstraction. In the beginning, that was about databases, but the same concept applies: Get up and running quickly with as little boilerplate as possible.
If you’re new to Spring, you’ll be shopping for a framework. But this is usually an organizational decision. We’re not typically talking to hobbyists. If they haven’t heard of Spring, most likely they’re not working in Java. If you’re not working in Java, use the language you’re most comfortable with and find a framework that does the same things.
MCP: Rocket Fuel for AI Apps?
If you look at the zeitgeist around the Model Context Protocol (MCP), everyone loves it. At the AI Engineer Summit, the most attended talk was “Building Agents with Model Context Protocol.” The major competitors were pretty quiet at first. They wanted to draw people into their own ecosystem. The old XKCD cartoon about standards could have played out here.
Source: xkcd, CC BY-NC 2.5
But in recent weeks, we’ve seen movement toward MCP from AWS, Microsoft and OpenAI. So maybe we’ve seen alignment already.
MCP is like rocket fuel for your AI apps. To be sure, not all MCP servers are created equally or will ever be used in enterprise applications. But there are many mature options that are. So it becomes the class library of tools. Developers are often given another tool for the toolbox; with MCP, it’s more like a giant toolbox.
It’s so easy to grab functionality off the shelf and incorporate it into your app. It could be written in Python, TypeScript, anything, because it’s just JSON-RPC on the wire. You can see why the AI community embraced it across the board.
What Devs Need To Succeed With AI Apps
We may be biased, but we firmly believe you will need a framework. If you don’t use Spring AI, you’ll get a raw SDK or REST model. The latter option will take an untenable amount of time once you start calling your tools. It’s complicated, it’s not standardized and ultimately it leads to undifferentiated heavy lifting that keeps your devs away from doing the thing they do best: writing code.
Platform engineers want to centralize their control over access to AI. They want to control costs, they want to provide access to multiple models and versioned models, and they want the ability to swap approved models in and out in their apps from a central location so that they can use operational controls, rate limiting, observability, service binding and registries.
You want to give developers predefined tools. You don’t want to ask the architecture committee at the bank every time you want to try a new tool. It’s a bank, so you can’t install anything you want. That’s where an AI-focused developer platform can help give you one central access point.
In theory, you can write all this yourself, but who wants to do that? Tanzu provides a higher starting place for AI apps. Where the line between homebuilt or platformed is drawn, different people have different opinions, but a developer platform meets a real need in the enterprise.
At the end of the day, I don’t get paid to collect GitHub stars: I get paid to build products that provide value to someone.
