Introducing AiKA: Backstage Portal AI Knowledge Assistant
Spotify is an extreme dogfooding organization. The audio streaming company invests heavily in its own internal developer experience, proving tooling like Backstage in-house before launching to new customers and even open source contributors.
Now Spotify is applying that ethos to AI-backed knowledge consolidation — and broadly sharing the challenges that go into building that.
Majd Salman and Jofre Mateu Matesanz, platform engineers from Spotify’s platform developer experience (PDX) team, presented AiKA — the AI knowledge assistant — to the greater public at KubeCon + CloudNativeCon Europe.
As a retrieval-augmented generation (RAG) based chatbot, AiKA relies on organizational context and data ecosystems to feed into a conversational search that references source material. As AI must not further interrupt flow, AiKA works within the integrated development environment, Slack, cross-organizational docs, and other developer must-haves.
AiKA is already used by more than a thousand Spotify employees in both technical and non-technical roles. Among 80% of Spotify developers, AiKA has become the preferred AI tool, the company says.
This week, Spotify started rolling out an alpha version of AiKA with select Backstage Portal Cloud customers and design partners. It’s expected to be released to the broader customer base later this year.
Measuring Barriers to Developer Productivity
“We focus on building tools and systems that help make Spotify engineers more productive,” Salman said, kicking off the talk.
Both speakers had spent the last eight years working on Spotify’s internal developer platforms — well before the Backstage internal developer portal was even conceived.
“As a platform department, we focus not only on measuring productivity, we are also concerned about what gets in the way,” Matesanz said.
“That’s why, in 2020, we started a quarterly engineering satisfaction survey to get data and trends on these productivity boosters and blockers across all of Spotify,” he said. “Since the very beginning, and for every run, we’ve seen how issues with finding information or missing documentation have been consistently in the top three” developer productivity blockers.
This isn’t an isolated complaint at this one company. On average, across the tech industry, developers lose one day a week to inefficiencies, including more than 30 minutes a day spent looking for things. The vast majority of companies are weighed down with knowledge fragmentation.
At Spotify, information is scattered across:
- More than 10,000 sites.
- More than 22,000 repositories.
- Slack is the company’s main communication pathway with 150+ channels.
- Other information sources, including HR data, requests for comments, architectural decision records, planning tools, code examples, dashboards and, of course, different qualities of documentation.
The Spotify culture of squads is famously grounded in self-organization and developer autonomy. Early on, this resulted in eight different teams building their own solutions. That may have been fine then, but now the streaming company has scaled to 600 squads.
“We had duplicated efforts,” Salman said. “There were fewer best practices. Infrastructure and knowledge weren’t being shared between solutions.”
When large language models (LLMs) started grabbing headlines three years ago with GPT 3.5, while they liked this new way of communicating, he said, it had very limited context windows that were hard to train from scratch or fine-tune — not to mention the frequent hallucinations.
Retrieval augmented generation (RAG) emerged as a pattern to address these limitations.
“RAG is a technique that allows you to do question and answering over your own private data,” Salman described.
You begin, he explained, by processing meaningful documentation. These docs are split into meaningful chunks, and then, using an embedding model, each chunk is converted into a vector representing its semantic meaning. These vectors are then stored in a vector database. Then when a user asks a question, that question query is also converted using the same embedding model.
This RAG process reduces data duplication and repeat questions.
“This approach is also more cost-effective, because you’re only dealing with a subset of information instead of using the whole knowledge base,” Salman said, which in turn reduces the carbon footprint. “It also addresses some of the hallucination issues by grounding responses in actual documentation.”
Also, since the RAG-based responses can be grounded with citations of where the answers came from, it further reduces hallucinations.
RAG Grounded Knowledge Search
At the 2023 Spotify Hack Week, several teams aimed to build AI chatbots to aid internal support. Eventually, AiKA — the AI knowledge assistant — became a consolidation of those efforts and was deployed to production internally that December.
First and foremost, Salman said, this shared knowledge platform plus the AiKA chatbot was grounded in some core principles:
- Trust and transparency: Verifying where the docs come from.
- Positive knowledge feedback loops: Continuous feedback to doc owners.
- Flexibility: Customizable experience with different interfaces within Backstage.
- Meet users where they are: Integrating with different interfaces, including Slack and Backstage.
“We believe there’s a strong synergy there, where better documentation leads to improved responses and better responses incentivize teams to update and maintain better documentation,” Salman said. “This leads to a positive feedback loop, and we wanted to tackle and support multiple experiences and be adaptable to various module needs.”
Thus, AiKA the artificial intelligence knowledge assistant was built. Integrated into Spotify’s own version of the Backstage developer portal, AiKA can retrieve information from key knowledge sources, including Slack support channels and organizational data.
“Then we can blend all these contexts together to generate useful answers,” Matesanz said.
Not wanting to just build for developers, the AiKA RAG relies on all of Spotify’s key internal documentation, including a Slackbot that allows personalized search for private conversations. It comes with an API and a Python library to build customizations on top.
Because AiKA is grounded in internal knowledge, it understands nuances that perhaps popular LLMs would miss. For example, MMA at Spotify stands for “managed monitoring and alerting,” not the popular acronym for “mixed martial arts” — AiKA knows the difference and can use that knowledge to teach new teammates this lingo.
Across those 600 teams, a newcomer can quickly discover who owns what via a conversation with AiKA.
“…we have an evaluation framework so we can evaluate the reason for things, like retrieval accuracy and answer quality.”
– Majd Salman, Spotify platform engineer
AiKA is also uses mainstream third-party LLMs. That means if a developer wants to ask, for example, a more general Python question, industry standards plus any Spotify-specific touches will be considered in the answer.
The most powerful feature so far, the pair said, is AiKA’s confidence-scoring model that outputs a confidence score — from 0 to 1 — of how confident it is that an answer will be helpful to the user, based on the question, the retrieved documents and the answer itself.
“And we have an evaluation framework so we can evaluate the reason for things, like retrieval accuracy and answer quality,” Salman said. “This helps us to move safer and make changes confidently, and all of this feeds back into our observability system,” based on OpenTelemetry.
Now 25% of Spotify employees and 86% of Spotify developers use AiKA weekly — with more than 1,000 daily users. The PDX team also receives regular requests from teams across the company for how to make their documentation available within AiKA. Ever the inner-sourcing company, AiKA closes that positive feedback loop by encouraging Spotify users to not only use, but to find and fix gaps in documentation.
How To Customize AiKA Backstage for Your Org
Once AiKA was proven within Spotify, it came time to offer it outside the company. Last week, AiKA started being rolled out to Spotify Portal for Backstage customers and design partners, to adapt AiKA to their organization’s needs.
At Spotify, the platform and developer experience team didn’t just dump all the company’s docs into AiKA — and they don’t recommend new AiKA adopters do either. Start intentionally small and build from there, focusing first on the highest quality documentation and your main channel.
Of course, as they found, quickly users will want to add their own content, which is a good sign of internal developer platform success. With this in mind, the PDX team built into AiKA an ingestion pipeline to allow teams to maintain their own data sources.
Just be careful, Salman warned, because this autonomy increases the likelihood of semantically similar but irrelevant answers coming in from other domains — like backend versus web versus security. The more domains, the more dilution. Within AiKA you can use a re-ranking system to give or reduce importance of documents.
“With the ability to filter the knowledge available, we were able to expand from being a general assistant to creating a platform capable of creating focused experiences.”
– Jofre Mateu Matesanz, Spotify platform engineer
“While re-ranking helps to entangle this mess a bit, getting rid of the background noise helps even more,” Salman said. Within the same embedding space, his team has enabled users to highlight “a specific region or a set of topics that are relevant to some discipline [because] we found that often when users ask a question, they have a specific context in mind.”
He gave the example that a backend engineer asking about deployments is unlikely to be interested in iOS deployments, so this discovery is automatically narrowed down to filter out noise.
“Having all this knowledge in the same vector space is very useful for discovery,” he continued, “but the ability to narrow it down and filter out noise unlocks new opportunities.”
With this in mind, AiKA users can turn on knowledge filters to set up customized AI assistants.
“With the ability to filter the knowledge available,” Matesanz said, “we were able to expand from being a general assistant to creating a platform capable of creating focused experiences.”
Goalie: A Slackbot for Support
Next, Spotify teams wanted to expand the AiKA application to include automation of support work, which led to the creation of the AiKA Goalie Bot, a fully customizable Slackbot.
Goalie Bot monitors Slack support channels that are in charge of handling everything from routine questions to longer troubleshooting, debugging sessions.
“We saw the opportunity to let AI take care of this more boring work, and let human goalies [support] handle these more interesting troubleshooting sessions,” Matesanz said. “When the bot sees an incoming question, the first thing is to decide if it’s capable or not to answer this question at this given point.”
This Slackbot follows the same confidence scoring as the main AiKA assistant, in order to decide if it’s capable of answering a question and if it’s even more capable then to perform the action on behalf of the user.
“If the confidence in the answer is Low, it’s just going to stop there and do nothing. If the confidence is Medium, it’s going to propose the generated answer to the human goalie and let them decide how to proceed,” he said. “Lastly, if the confidence is High, it’s automatically going to post an answer citation by citing all the sources as we mentioned, and save some time for everyone.”
Since interactions are particularly different per Slack per team, with varying technical depth and quality of documentation, this is built to be a highly customizable feature. With a declarative YAML approach to Slack channel configuration, each team is in charge of its own system. They are in charge of the default knowledge base, documentation, which Slack channels, and even custom data sources.
Importantly, each team determines its own confidence levels.
At Spotify, non-technical teams are also configuring their own Goalie Slackbots.
“Our approach to configuration without having to write a single line of code means that non-technical teams have been able to adopt this bot and benefit from it,” Matesanz said.
On average across Spotify, that has led to 30% of questions answered, which, he says, equates to thousands of hours.
Goalie is only currently available to Spotify employees, but could be considered for future product updates.
RAG-Based Lessons Learned (the Hard Way)
It’s not automatic though. A lot of consideration goes in up front.
“We believe that retrieval is the most important part,” Salman said. “Without finding the right documents and the right information, you can’t really provide accurate answers.”
Cast a really wide net to start your data search, he recommends, but then input quality over quantity. Even when the PDX team doubled the amount of documents in context, they didn’t see improved results because the relevancy during retrieval dropped off logarithmically.
“You may see a lot of improvements in the first five documents you provide, but then rapidly diminishing returns after that,” he warned, as “you often just end up filling the context window unnecessarily and every data source needs special consideration.”
This includes how information is presented. For example, if your data is in a graph structure, he said, “it needs to be flat and embedded in a useful way for question answering.”
Even a single Slack conversation can be embedded in several ways.
“You really need to try to anticipate what kinds of questions your users want to ask the data.”
– Salman
“You really need to try to anticipate what kinds of questions your users want to ask the data,” Salman said. “Perhaps one of the trickiest challenges is determining if something is a hallucination or if it’s bad data. And when you have an incorrect answer, is it because the LLM made it up or is it because it’s correctly using outdated information?”
This is why AiKA and Goalie Bot are built to emphasize citation, both for information verification and to notify documentation owners if things are going out of date.
Next up, internally and then for Backstage Portal Cloud customers, Matesanz and Salman’s team is working to build a semantic search across multiple knowledge sources, beyond pre-defined data sources.
In addition, “we’re working on enhanced reasoning,” Salman said. “So instead of users having to know which data sources to query, the system can deduce automatically — based on the query and context users are in — which data sources are relevant.”
And, unsurprisingly, they are exploring agentic AI capabilities to deal with multi-part questions that require information from different sources, including testing with real-time information.
For now, it’s all about rolling AiKA out to Spotify Portal Cloud customers that the company has the closest cooperation and tightest feedback loops with.
Since Aika translates to “love song” in Japanese (AI = love!) and “time” in Finnish, we can only hope that AiKA translates to better just-in-time communication and collaboration among development teams and beyond.
