Meta Measures Developer Productivity via Software Supply Chains
What if developer productivity is not defined by how many lines of code you write but by your ability to affect other code, people and processes in the software supply chain?
So many developer productivity metrics obsess over speed or quantity. But new research from Meta found that neither of these things matters as much as you might think.
A team of machine learning researchers at Meta took a bet that the most common measurement of raw developer output — the number of changes a developer makes — is a “simplistic” and “misleading” measure of productivity. They contended that not all lines of code are created equal. In fact, the team concluded that focusing on lines of code abstracts away the complexity and diversity of any given task.
In a research paper, “Modeling the Centrality of Developer Output with Software Supply Chains,” this Meta team put forth a graph-based way to develop a more complex, holistic way to measure developer productivity across the software supply chain.
The researchers’ conclusion: Productivity is unlocked when you pair the right person with the right task.
Measuring Developer Productivity
You can’t improve what you can’t measure. And when you add up the cost of engineering salaries, there’s been a growing interest in measuring developer productivity — but it’s hard.
“We know that productivity is actually a cornerstone in software engineering, yet it’s something that we have struggled with measuring or quantifying,” Rui Abreu, research software engineer at Meta and one of the authors of this research, said at the Developer Productivity Engineering Summit, in London, this past June.
Rui Abreu, of Meta, shared how researchers at the company discovered that developers’ centrality to important tasks, and how complex those tasks are, influence dev productivity. (Photo by Jennifer Riggins)
“Even the SPACE framework, it will give some guidelines, but they don’t tell you exactly what to measure or how to measure,” he said, reflecting on his lifelong research into developers’ frustration.
“There are many units of activity; there’s whole lines of code, changes, the number of issues — these metrics are not exchangeable. We don’t know the relationship between one another.”
The common developer productivity metric of measuring lines of code, he continued, doesn’t account for differences across codebases or the complexity of projects. And, as we are learning with the rush of generative AI adoption, more code does not mean better.
Compounding this challenge, Meta could fill a stadium, not a boardroom, with its developers — who all bring different levels of experience.
“You cannot expect that the lines of code produced by a young developer is the same as a more senior developer because of the different responsibilities and even different effort in the company,” Abreu said.
The team was looking for measurements that view software development as the network or a supply chain it is.
Mapping People and Tasks
The research team at Meta looked to leverage graph modeling to help create more productive software supply chains, by factoring in not just software, but people and tasks, too.
Other factors can be added to these graph charts, including documentation.
This project looked to:
- Construct software supply chains from over the last decade, factoring in a diversity of programming languages, project sizes and application types.
- Create a set of models for developer output, replicating historical output and then enhancing it with these supply chain factors.
- Understand how the centrality of a developer’s work affects output.
Instead of lines of code, the research team looked to graph a social map of the tasks of Meta’s roughly 40,000 engineers around two domains:
- Complexity of task
- Centrality and importance to the development process
“If we map our development environments as a software supply chain, what we’ll have is a huge network that exists as an example of a network or itself a very small subset of the network,” Abreu said, describing the methodology. “Once you have this graph, the key thing that we’re after is to look at the nodes that are the most important.”
Examples of a node include:
- A file that a particular team changes frequently
- Dependencies between method calls
- Code changes over time
- Authorship
- Team organization, like engineer to manager
- Functions
- Diffs
Then, when the graph identifies the engineers deemed to be important, they can be automatically assigned the most important work.
Meta applied several algorithms to establish centrality:
- Software supply chain networks of people, processes and components.
- Google’s PageRank centrality, which has pages becoming less popular over time, unless heavily interlinked or networked.
- Katz centrality, also known as alpha centrality, which measures the degree of influence a node within a social network has in relation to other first-degree nodes.
The team then expanded its graph indicators to better predict output, which they defined as accepted changes that get deployed. These further predictors included:
- Tenure in years at the company because, Abreu noted, it was too hard to measure the tenure of a project.
- Whether someone has other employees reporting to them.
- Prior teams.
- Frequency of changing teams.
- Number of prior managers.
- Community size.
- ReadMe files or anything touched by developers.
From there, the project tried to relate impact, importance and centrality across the different nodes of the software supply chain.
Complexity and Centrality
This project confirmed previous research that developers new to a project do easier and less central tasks, which may or may not result in more lines of code created. On the other hand, more senior developers tend to write fewer lines of code because they are usually solving harder problems that are more central to the core infrastructure, the value of an organization or the customer.
The research process, Abreu said, was akin to Google giving more importance to more relevant websites. These results were consistent whether they followed the timeline of pull requests, merge requests or customer notifications. As a developer becomes more experienced, productivity — by those less sophisticated measurements — tends to reach a plateau, after about 18 months on the same project.
“It’s not that this experienced dev is not doing anything,” Abreu said. It’s that the dev has become so central in a year and a half that they are doing other things than writing code. The number of changes they make might slow, but their centrality increases, which in turn increases their impact on the team’s and code’s future.
This research also replicated the results of three other research projects within Meta and the finding that engineers across teams will reach a plateau of complexity somewhere between six to 24 months on a project.
The Meta researchers found that as a developer becomes more experienced, productivity tends to reach a plateau after about 18 months on the same project. But that doesn’t mean the dev isn’t central to the project’s success.
“You are junior to the build baseline. You do a lot of things. You see a lot of output in terms of number of requests, but eventually you reach a plateau where you produce less and less pulls,” Abreu said. “This doesn’t mean that you are not being productive in your job. You are more central; you are more experienced.”
Typically, while the output goes down, the centrality to certain files around a given theme increases over time.
The longer you are on a project, he continued, the more likely you are to spend more time in planning meetings and other strategic activities not measurable by pull requests.
This graph-powered research is mostly applied to developer productivity at the team level.
Abreu was careful to emphasize that Meta isn’t de-anonymizing individuals, including not providing detailed information to managers who have less than three reports. “Even though we could,” he said, “we are not highlighting people so management directors can go back” and move an individual to only staging work or otherwise gaming the system in a way that could hurt someone’s career.
Following the trend of other developer experience efforts, this information is available to the individual developers, to help them not only understand their own productivity but to look for teams within the company that they can progress with.
What’s Next: Replicating the Study
Next, Abreu and his team hope to replicate this study across other enterprises and large open source software communities.
The team is looking to verify their unified concept of software supply chains, which characterizes for statistical modeling three main node groups:
- People
- Software artifacts
- Tasks
This report shows that, at least at Meta, the team has figured out how global measures of the software supply chain explain variations in developer output better than the more common, simplistic measurements.
All of this work was confirmed via a logistic regression model, which found that centrality was the biggest predictor of developer productivity of a project.
This research also explored the impact of moving teams, which happens a lot at companies that are Meta’s size. While developers could move around projects, the research found that there was a bigger impact on productivity when managers moved around too much. Still, developers who worked with managers who had more centrality and time within the company were more productive.
Abreu concluded, “At least sometimes the machines can actually measure these variations and the complexity of different tasks and different efforts that a software engineer needs to do.”
