How Expedia Group Moved From 21 Platform Stacks to 1
Mergers and acquisitions are often an impetus for change. It was no different when Expedia Group, the online travel tech company, decided to unify its 21 different travel brands and tech stacks, thousands of microservices, APIs, and bespoke libraries in a multi-year planned migration.
And it started all this in 2020, just before the worldwide pandemic that would bring the travel industry to a screeching halt. Guillermo Manzo and Giorgio Delle Grottaglie, as leaders in the developer experience organization, have seen it all over the last four years.
Expedia Group recently completed its migration project, so The New Stack sat down with the pair to learn about large-scale team and technology mergers, platform engineering, developer productivity and how to measure it all.
New Teams for DevEx and the Platform
For Expedia Group, the 2010s saw the acquisition of other travel aggregators and metasearch engine sites, like HomeAway, Travelocity, Orbitz and Venere. It didn’t see nearly as much merging.
The original strategy was to maintain a collection of different brands to help the company grow, which resulted in 21 isolated products, with 21 different cloud migrations, often with the same or very similar technical foundations.
As Peter Kern took over as CEO in 2020 he planned a three- to four-year digital transformation aimed to simplify the company and reduce duplicate work across brands.
But in March 2020, travel — and Expedia Group — were grounded by the Covid-19 pandemic.
Of course, a significant reduction in traffic due to a decline in customer deman is not a bad time to cut back on technical debt and to centralize core capabilities. It also helped that the company’s then-new CEO and CFO abstained from collecting a salary the first year of the pandemic, while other senior execs took a 25% pay cut.
That’s when Manzo’s developer experience (DevEx) team was created. Instead of pushing out new customer-facing features when demand for travel was low, Delle Grottaglie said, “the product and tech leadership team dedicated most of the engineering workforce to consolidating functions into core teams, regardless of brands.”
Thus Expedia Group’s platform team was born, as well as consolidated network, finance and frontend teams, eventually all working behind a single portal.
The DevEx team was also asked to standardize how different teams interacted with each other.
Early developer feedback highlighted that common pain points were documentation discovery and freshness, how disparate tooling was, and how onboarding took weeks. Early on, Manzo said his team heard that, “If you want to work with another team, and you push a [pull request], good luck!”
Despite the business’s shifting priorities due to the pandemic, the platform team was prioritized, he said, because “We were able to tie the core deliverables to top priority business objectives and support all these various different pieces of the transformation.”
The Backstage Front Door
Back in early 2020, each of the more than 21 Expedia Group brands had its own bespoke internal developer portal. Part of the DevEx team’s first task was to take ownership of the organization’s tooling and learn the different portals, tooling and APIs each brand relied on.
The product team researched and wrote an internal white paper that looked to understand what an internal developer portal is and what the business needs are. They discovered several teams were often using the same products in different ways, including an infant version of Spotify’s Backstage, various cloud platforms and some internally built portals.
“Think about the pandemic: the company was focused on cutting costs, finding new ways to work together, do less with more. Some of these [portals] got knocked down by default,” Manzo said, because they were not sustainable.
Expedia Group undertook the popular build-versus-buy debate, which ended in creating a proof of concept for and, by the end of 2020, becoming the 18th adopter of the now-leading open source internal developer portal template, Backstage.
“By the end of 2020, the product and tech org already had a full gameplan of basically replacing all portals, and centralizing all that in Backstage, focusing on onboarding — high among the key pain points at most companies,” Manzo said.
This wasn’t just about new hires, but also about onboarding the thousands of existing staff members to the new platform and processes. “Over three years,” he said, “the product and tech leadership team focused on getting everybody in the right place, in the right spots, using the right tool sets and, slowly but surely, identifying and simplifying our tech stack.”
As all these companies integrated and migrated their systems, the Backstage internal developer portal became the front door to the new Expedia Group ecosystem, with a series of chatbot interfaces, search and developer support.
Getting 21 Brands on Board
This in turn, should have reduced context switching by magnitudes at Expedia, by urging everyone onto the same golden paths.
But, while all these brands were being combined into one, there was more for any developer to know. Every developer needed to have some understanding of the frontend, platform and cloud infrastructure tooling, even when their roles didn’t focus on that. Expedia had six different compute platforms alone — plus a new one being designed to supersede the existing one.
In other words, the motivation to focus on the developer experience came down to “cognitive load, in addition to what a normal developer should have to deal with every day,” Delle Grottaglie said.
Adopting Backstage, he said, helped by “having just one single place to start looking for something. Whereas, on the other side, there was this big complexity added by this consolidation into single teams.”
During this transformation, the platform team was still responsible for the legacy architecture built in the early 2000s. The DevEx team faced the typical resistance to migration, with teams prioritizing features over migration and technical debt.
Like most platform engineering programs, success is found in persuading developers to come along, not forcing them to adopt. This tactic works with a combination of early success stories and measuring developer productivity — and then amplifying both.
“We paved roads, so developers don’t have to question what exists,” Manzo said, selling their internal developer customers on one-click infrastructure capabilities. “If you build an app on the new platform, you will get these things out of the box. Build on the old platform, you can’t deliver because you’re out of compliance, and you’re spending all your time getting back to compliant.”
Expedia Group worked to not only simplify the developer experience for devs, but for ops too. On the CI/CD side, Delle Grottaglie said, “Everyone had their own Jenkins instance, their own GitHub, their own [JFrog] Artifactory. All this was consolidated into single products, regardless of which team or brand you were working with.”
New Focus on ‘App-Centric Documentation’
“My team focuses on ensuring that the end-to-end developer journey is as efficient as possible, mainly around onboarding,” Manzo said. “We think about shipping on Day One, being able to create an application on the first day, reducing context switching and cognitive load through the developer portal Backstage, as well as our ChatOps platform, which is a strong supplement, especially nowadays with chat, AI and other capabilities to support developers.”
The Expedia ChatOps platform also brings together the whole suite of the core developer productivity toolset so the company can understand how productive their developers are.
A perennial top developer demand is better documentation — but devs rarely prioritize improving documentation themselves. But now that the migration project is finishing, Expedia Group’s DevEx team is turning its focus to docs.
“We are working to focus on app-centric documentation,” Manzo said. “We’ve created automated capabilities and quality controls in an effort to centralize docs within Backstage.
The DevEx team is also evaluating generative AI for docs-as-code and leveraging SlackBots as first-round technical support that automatically pairs requests with answers.
Early Results: Cloud Costs Down 10%
This February, then-CEO Kern announced that, after more than three years, the Expedia Group’s digital transformation is mostly complete.
This included eliminating dependencies on 76 different advertising agencies globally, Kern said in a recent all-hands meeting, by enabling Expedia Group to build a full-service, internal marketing, creative and media-buying team, with shared data and tooling.
In addition, he boasted, the company has already been able to build a new rental car service on Hotels.com in just weeks, instead of the usual months or years it would have taken before.
The company has estimated that this organization-wide transformation has helped cut cloud costs by over 10%. The Expedia Group tech and product team has grown from 30% to half of all staff. And to reduce competition with itself, the company are focused on three flagship brands — Expedia, Hotels.com and Vrbo.
The developer experience engineering team has grown over the last few years and is a subset of the whole DevEx team. That larger team supports the runtime and edge platforms and the CI/CD pipeline. Together they work to increase the productivity of more than 6,000 developers across the Expedia Group organization.
Developer experience wasn’t just in charge of engineering these changes, but in developing a culture of inner sourcing. By the third quarter of 2023, the DevEx team merged more than 500 pull requests from its internal community.
Measuring Continuous Platform Improvement
“All of 2023 was about getting that realignment,” Manzo said. “Understanding developer support became very, extremely important to onboarding more of the toolset.”
2024 is about measuring DORA metrics and developer productivity metrics to understand the effectiveness of the digital transformation, including launching a new semi-annual survey, as well as shorter and more frequent customer satisfaction (CSAT) surveys.
The hope is that pain points will noticeably shift over time — for instance, as documentation becomes updated and discoverable, it will drop in priority and creating a service search engine will jump in priority.
The next step in their productivity metrics puzzle is a way to visualize and measure the full end-to-end developer journey. The company isn’t looking to measure performance with this, but to use these metrics to understand how to improve developer productivity, gather more robust feedback, and deliver a suite of metrics for engineering leaders.
In the first quarter of this year, the team piloted a new developer survey to about 1,000 developers. The platform team, Manzo told The New Stack, is examining the results of the developer survey, trying to identify DevEx bottlenecks and areas that can be improved, before rolling it out to the whole engineering organization.
“And then, in parallel, the team has kicked off the engineering productivity program, which focuses on driving developer effectiveness and efficiency,” he said. “We have collaborated with a third-party platform to set up an end-to-end program to capture and measure the cycle time and various other input metrics, and set up automation opportunities to drive that productivity value add in response to not just a survey but other details that come out from the other subset programs.”
The company also organizes listening sessions, especially when teams have migrated to the new platform — because those teams become their priority customers during the transition. These are less structured, qualitative discussions to figure out how to refine the migration process in terms of the new documentation and user service capabilities needed.
The DevEx team also spends time every day in its support intake Slack channel, looking for questions raised and ideas for improvement.
As early adopters of Backstage, Expedia Group has also contributed back to the open source project’s community, particularly around proof of value metrics. The DevEx team also created and open sourced a series of Backstage plugins that allowed it to display DORA metrics for each Backstage service, for security analysis and vulnerability scanning, and for small surveying tools.
Expedia Group’s Internal Developer Portal Metrics
The developer experience team has several platform metrics it considered:
- Usage rate, measured via logins.
- Time to find documentation.
- The ability to create an application on Day One.
- Average time to create an app.
- Average time to resolution of internal support requests.
- Documentation relevancy.
- Search relevancy.
- Time to merge inner source contributions.
These leading metrics can be tied back to Expedia Group’s overall business goals, as well as they influence the platform team’s DORA metrics.
As the time it takes to find relevant documentation reduces, so should the organization’s mean time to recovery (MTTR), Manzo said.
“No one should be telling you that you’re going to be able to have zero incidents,” he said. “It should be, when you do have an incident, you have automation and documentation, everything you need right there on the spot, so that way you can do your job.”
To support a generative culture, the developer experience team publishes these metrics regularly within Backstage, Delle Grottaglie said: “We started a year and a half ago with publishing these metrics and this has truly helped with transparency and setting the right expectation.”
It also helps push the platform team toward continuous improvement and keeps the global support response rate high, across hundreds of support tickets per month.
This year is also all about driving down technical debt, including a Smash-a-thon, which had the whole engineering org fixing bugs.
Now that the major digital transformation is finished, shutting down legacy tooling is another top priority.
The organization is also looking at ways to collaborate with vendors for managed services, increased internal developer support, and ways to unlock developer productivity, Manzo said, in order to reallocate engineering priorities.
All geared, he said, toward “better usage of our developers, and also more streamlined efforts into driving company strategies that ultimately trickle down into the org-level strategies.”
