How TV 2 Prioritizes and Measures Developer Experience
LONDON — If you’re reading this, you’ve probably never heard of TV 2. Unless you’re among the 85% of Danes who spend an hour and ten minutes a day consuming content from the state-owned, commercially funded, largest broadcasting network in Denmark.
Like all long-standing telecommunications companies, TV 2 has grown from the flow TV of the 80s all the way through to streaming services and free apps. It’s another accidental tech company, but a tech company nonetheless. This means that this media tech company has to figure out not only how to adopt modern, cloud native technology and a DevOps mindset, but how to treat developers as its customers.
For Emma Dahl Jeppesen, her role as product manager over the last couple years has evolved from leading a DevOps team that was accidentally adding more pressure on developers, to a developer experience team that researches and builds self-service tooling and a new observability platform, all focused on reducing devs’ cognitive load.
Dahl Jeppesen’s team sits on the crucial border between tech and business, which she reflected on last week on stage at Eficode’s The DevOps Conference and in a follow-up interview with The New Stack. Read on for how her team developed hypotheses and then tested and measured their impacts in order to improve developer experience at TV 2.
More Than Technology Has to Evolve
“We’ve always been called a media company. Now we’re a media-tech company because flow TV is slowing dying out,” Dahl Jeppesen said during her fireside chat at The DevOps Conference. What is a state-owned company to do when it suddenly realizes Netflix is its most formidable competitor? “Now it’s a whole different strategy to make the business work and to pivot the business to ensure it stays relevant in 2024.”
TV 2 began, like many organizations, with a restructuring and a refocus on adopting new cloud technologies and DevOps. This led to the whole engineering organization quickly growing by about 350% from about 100 to now 450 engineers.
However, this digital transformation kicked off, Dahl Jeppesen said, with the DevOps team creating tools that they would hand over to the site reliability engineers [SREs], “and be like, ‘Here you go! Now do whatever you want with them.’ And it brought on a lot of cognitive load. We’re trying to do something a little bit different these days.”
She echoed the not-uncommon impetus for a shift in company culture — abnormally high turnover. “We were just that DevOps team that had too many tools, too many services, legacy environments. And we had a big cognitive load. And so, at some point, they all quit and the team was dissolved.” So she and the one engineer who remained pivoted from a DevOps to a developer experience team.
To Dahl Jeppesen, developer experience is everything “about how developers feel about their work. It’s about how productive they feel when they’re working.” She continued, “I think, from DevOps, we’ve learned about speed and quality feedback loops, but this takes it to a whole new level. It adds other stuff like flow state — how much are the developers able to keep a state of flow in their work? Also what’s really important is cognitive load, like how much cognitive load do they have when they’re working?”
She went onto say that developer experience — sometimes called DevEx — is also about “trying to take something that’s really complex, like tooling, and then wrapping it up in something nice that would think about that user experience and treat them [developers] as your users.”
This includes abstractions and a platform engineering mindset, which at TV 2 is achieved through the command line interface and an internal developer platform. They also work with a lot of Kubernetes and Jenkins for continuous integration. But just about every team had its own stack. This vast tooling landscape with seven different build lines and Kubernetes clusters, Dahl Jeppesen commented, is bad for business, plus it makes it hard to aggregate data about the developer experience. Now they are adopting a platform engineering mindset with a command line interface abstraction across the top, with access via an API.
Over the last two months, all the engineers and their supporting teams have also begun to work within a Team Topologies structure, with everyone organized around capabilities in about 15 application or stream-aligned teams with the developer experience team serving the whole org and abstractions at the domain level.
Throughout these changes, developers have always had the ability to push to production when they wanted, as long as the observability and monitoring is in place.
How to Measure Developer Experience
As the DevEx team has grown to include Dahl Jeppesen and four engineers serving 200 engineers, their work to understand and to measure developer experience has expanded.
“We started with DORA metrics, like a lot of other companies,” Dahl Jeppesen said, but they shut down that dashboard after a year and a half, “because the teams felt like they were being measured.” And she said that developers were gaming the metrics, meaning a lot of self-proclaimed “elite” teams weren’t in reality. DORA metrics had been brought in at TV 2 top-down, by executives who, she remarked, had maybe read part of the book Accelerate, but hadn’t connected the metrics to action.
Now, her team is working to implement DevEx metrics — which focus on measuring flow state, feedback loops and cognitive load — while working to be really transparent what they are measuring and why. Developer surveys have arisen as a low-effort approach for developers.
“Talk to your developers. Get to know them. Because they will tell you everything they hate and everything they love.” — Emma Dahl Jeppesen, TV 2
But while it’s simple for devs to fill them out, “survey design, to be honest, is really really hard,” she said, but also “I think doing something is better than doing nothing — ask your developers about their experience of things.”
Their surveying endeavor kicked off with asking a lot of questions about how devs felt about tooling, which uncovered that the TV 2 engineers already really loved GitHub Actions, signaling they should move away from Jenkins and toward the preferred CI pipeline.
“Talk to your developers. Get to know them. Because they will tell you everything they hate and everything they love,” Jeppesen advised. Before, “we had that ‘build it and they will come’ approach, which just doesn’t work. Because you need to treat developers as users.”
And once you know what their problems are, developer experience and platform engineering teams are much more able to fix those problems. Which is why she also doesn’t discount the power of the water cooler chat to catch what they really think.
TV 2 has adopted objectives and key results (OKRs) to align the engineering organization. Her team has the objective to standardize their continuous integration pipelines, and a key result supporting that is to get three developers onboarded and championing it.
Not only did she say that about 85% of developers complete their quarterly survey, but their engineer retention rate has increased.
Mapping Out the Developer Journey
Surveys, of course, are a very subjective way to measure the developer experience, which is why Dahl Jeppesen’s team combines that with systems data, like telemetry, for a balance of what she calls “say data” — what devs are saying — versus the “do data” — what’s happening in reality. And then alongside this combination of qualitative and quantitive data she is also running qualitative interviews looking to map out the developer journey, from ideation to production.
It helps the TV 2 DevEx team understand where they can integrate feedback and optimize the experience. In these smaller sessions, they not only ask about the steps but, importantly, how engineers feel at each step, Jeppesen said, like “What are the pains and gains here? And they’ll tell you about frustrations or things that are really good.”
Eventually, they get to the point where they can visualize the curves on an “as-is journey” versus the “to-be journey,” where observability signals and other incremental improvements can be applied to the lows. Then her team asked, “Did that raise your journey? Did that actually improve things for you?” she continued. “Then again you have a measurable way of actually improving the developer experience in the developer journey.”
These fixes don’t need to be full-fledged ideas, but rather her team will release a simple prototype or even a mock-up to show the developers. And the devs feel able to do the same thing, conceiving solutions to their own problems.
In the circumstances of the CLI, three engineers had excitedly drawn their idea across whiteboards, but it just wasn’t translating for their Product Owner. Then they created a GIF explaining how they’d like their own platform abstraction. She showed it to some other developers and they quickly had a shared understanding of how that would likely solve a big developer bottleneck, so they built something out to validate it.
“Don’t spend too much time before you actually go and validate,” Dahl Jeppesen emphasized, “but just try something, and consider the developers as users.”
They aren’t just measuring the technical developer experience, but the effect of people and processes too. She gave The New Stack the example that, “If we measure flow state and we figure out the biggest problem is they’re stuck in meetings or the agile framework that they are using isn’t work for them.”
Unsurprisingly, one impediment to the developer flow state that came up was too many meetings. Instead of building something, she had a non-technical but still very effective action item of advising agile coaches and leadership of this, as they could best influence the meeting culture.
Dahl Jeppesen also offers four steps to prioritizing developer experience:
- Know your users. Embrace the water cooler, start a free-to-join weekly huddle, and be curious.
- Learn from your company’s UX team. “DevEx and UX are kind of the same thing,” she argues, advocating for the UX methodologies of Double Diamond — discover, define, develop and deliver — and consider valuable, usable, feasible and viable in the Discovery Mode.
- Ensure a user-centric mindset in your team. That means developing a specific and shared vocabulary between your product team and your internal developer customers. She also recommends training them up on user-centric methods and employing hypothesis-driven development.
- Measure the value and experience of your users. This is the hardest step, Jeppesen emphasized, but you really can’t improve what you can’t measure. And if devs have metrics to measure the success of their applications, so should DevEx teams. Start small, she suggested, like with the adoption rate of your platform or tools, and make sure to make both qualitative and quantitative measurements.
