Towards a Better Understanding of Developer Productivity
Measuring Developer Productivity is difficult, because in order to measure it, we must first be able to define what it is. Developers and Managers often have different views on developer productivity. Developers tend to define productivity in terms of their own activity whereas managers tend to focus on outcomes and performance of the team.
Developer Productivity can be defined simply by metrics that are easiest to operationalize (counting code commits, pull requests) or by the result of these activities – delivering a new product feature, or an enhancement to a current product feature. But, as is the case in most things, there’s a lot of nuance which needs to be considered in order to get to a more complete, useful picture of developer productivity. You get what you measure: for example, a team optimizing for resolution time for issues in a release may be saddled with longer fixed times in the future due to incurred technical debt.
DORA
The proliferation of DevOps-based workflows and fully automated build and deploy pipelines led a few researchers at Google and other places to create DevOps Research and Assessment (DORA) – a program which led to the creation of one of the most popular sets of metrics that is used by many teams to measure their performance. Through years of research, the DORA team initially identified four key metrics to measure the performance of a software development team:
Deployment Frequency: How often an organization successfully releases to production
Lead Time for Changes: The amount of time it takes a commit to get into production
Change Failure Rate: The percentage of deployments causing a failure in production
Time to Restore Service: How long it takes an organization to recover from a failure in production
The first two metrics are an indicator of the velocity of a team, and how efficient their workflows are. The latter two are an indicator of stability, and robustness of the team’s software development processes. Keeping track of these metrics and continuously iterating to improve them can lead to improved outcomes.
SPACE
However, these metrics tell only a part of the story, starting from the time a commit is made. To better understand developer productivity, further research at Microsoft and GitHub led to the publication of the SPACE framework. SPACE defines five dimensions to measure, for a more accurate picture of developer productivity:
Satisfaction and Well-being: How fulfilled developers feel with their work, team, tools, or culture; well-being is how healthy and happy they are, and how their work impacts it
Performance: The outcome of a system or process
Activity: A count of actions or outputs completed in the course of performing work
Communication and Collaboration: How people and teams communicate and work together
Efficiency and Flow: The ability to complete work or make progress on it with minimal interruptions or delays, whether individually or through a system
To use this framework effectively, the authors make several recommendations. The idea is to combine some operationalized metrics (e.g., number of PRs, story points) with perceptions of the team to capture an accurate picture of the productivity of the team. In most cases, the best way to capture the perceptions of the team is simply to ask them!
Including metrics from multiple dimensions and types of measurements result in a balanced view. This provides a truer picture of what is happening, and this more balanced view helps to reinforce smarter decisions and tradeoffs among team members.
Here’s an example of measures around code reviews, an activity that occurs multiple times within most development teams, and can be rounded out to capture a more complete picture of their effectiveness.
Challenges to Measuring Developer Productivity
Advances in tooling available to software developers have added a new dimension to the developer productivity discussion. The availability and proliferation of AI-assisted coding tools have led to an expectation of a boost in productivity, cutting down delivery times and setting expectations of a quick turnaround to feature requests.
This can lead to adverse outcomes – the quality of the code produced by code assist tools can be very uneven, and the time saved by using these tools can be spent in debugging and fixing issues. The expectation of a quick turnaround can also lead to cutting corners, and not following best practices.
Quality can sometimes be at odds with the metrics used to measure Developer Productivity. For example, the product team may consider a feature complete and push for a release to production based on a demo of a proof of concept. For the engineering team, the “working demo” was just that – and they would not consider it complete until all the edge cases were correctly handled, tests were written and so on.
Another interesting wrinkle is the thought and care put into security – access to resources in the cloud, and to connected systems leads to credentials being accessed constantly, which can lead to the credentials leaking. A recent example of a breach at Zapier included attackers getting access to customer information via unauthorized access to its code repositories, which included a copy of real customer data being used for testing. Some amount of friction is therefore necessary, and useful. This can be solved, however, by setting up secure credential distribution mechanisms, tooling to allow teams to generate test data, and documentation.
Developer Experience – A Paradigm Shift
However, there is a paradigm shift currently underway, where the focus is moving away from measuring Developer Productivity in a silo, to focusing on improving the Developer Experience (DevEx). DevEx is not just about individual developer satisfaction; it directly influences the quality, reliability, maintainability, and security of software systems.
Investments in DevEx have been found to accrue benefits at multiple levels: individually, team and organization. For individual developers, positive outcomes include improved job performance, creativity, and learning. For teams, improved code quality and technical debt and at the organization level, positive outcomes include improved retention, innovation, profitability, and the organization’s ability to achieve its goals.
A Human-Centered and Multifaceted Approach is the Way Forward
As researchers at Google found, software engineering is a creative endeavor and measuring the productivity of humans at this is inherently difficult. They concluded that measuring productivity needs to be thought about in a holistic and multifaceted way.
Frameworks such as SPACE can help us understand tradeoffs, and help measure productivity using more than one metric. In the long term, focusing on improving the Developer Experience will lead to happier and more productive developers, stronger teams, and more successful organizations.
References:
Developer productivity with Dr. Nicole Forsgren (the creator of DORA)
The SPACE of Developer Productivity – ACM Queue
Getting What You Measure – ACM Queue
Quantifying the impact of developer experience | Microsoft Azure Blog
Navigating the SPACE between productivity and developer happiness | Microsoft Azure Blog
How Developers and Managers Define and Trade Productivity for Quality