Pat Kua is a seasoned technology leader. He has more than 20 years of experience in the software industry and has worked in different roles during his career. Pat is a known author and speaker, and currently maintains a newsletter focused on tech leadership. You may already have read a few of his articles, as we shared in our Engineering Management Newsletter.
Having it in mind, I believe his opinion on Engineering Metrics is interesting for our audience. I reached out to Pat, who readily accepted to be interviewed by email. Thanks, Pat!
I sent his 4 questions that consider some of his works. Our team decided not to edit the answers, and you read them as he wrote below.
You write a lot about your experiences as a Tech Lead. Recently, you shared an interesting thread on Twitter saying engineering leaders feel like they should measure something. Still, you defend that before getting into it, they should answer the question, “What do you care most about?”
In other words, you’re saying that your goals should drive your metrics, not the other way around. Is that so? Could you elaborate a bit more on that?
Yes, that’s correct. Goals should always drive metrics and not the other way around. One main reason engineering leaders should care about this is because setting an ambitious goal is a key leadership responsibility. A great goal is one that people working on it empathise with, and one that will motivate and hit that intrinsic motivation. I’m key to see a metric that meets both of these criteria.
Software teams should aim to write good quality code, but defining what that looks like cannot be reduced to a single measure.
DORA / Four DevOps Metrics seem to be turning into a standard in the industry. Having the previous answer in mind, what are your thoughts on the matter? Do they fully represent engineering outcomes? Are they enough for measuring engineering outcomes?
I love the work the DORA team published, and I agree they are fast becoming industry standards. They are great measures, but like the previous answer, they are not substitutes for goals and I don’t think they fully represent and not enough to measure engineering outcomes. The DORA team emphasise this in their work too, that engineering is part of a wider system and thus engineering outcomes are not completely independent.
Let’s look at one of the DORA metrics as an example – Deployment Frequency. I’ve helped many organisations to increase their pace, helping them move from cadences such as twice yearly or quarterly to every two weeks, or from every two weeks to weekly or daily, and then trying to improve this. An organisation with a high Deployment Frequency has a great engineering capability. It doesn’t mean the organisation uses this capability well. In some of the agile and lean communities, you often hear the question, “Are you building the wrong things faster?” A high Deployment Frequency is meaningless if the engineering outcomes don’t add value, which might come from how work is prioritised or what features are planned. Engineering leaders often influence, but rarely control these decisions, usually in the hands of business or product folk.
You published an article in Martin Fowler’s blog talking about the use of metrics. In the section “Guidelines for a more appropriate use of metrics,” you list 4 actions: 1) Explicitly link metrics to goals; 2) Favor tracking trends over absolute numbers; 3) Use shorter tracking periods; 4) Change metrics when they stop driving change
After almost 10 years since the publication, would you change anything? What have you learned during this period that is not there?
I originally wrote this article as a chapter for the second ThoughtWorks’ Anthology book. Since books are harder to update than a web page, I wanted to write something that would remain relevant for a long time, so I appreciate this question. The original article was published in 2013, but I’m pretty sure I wrote it back in 2011!
Re-reading the article I don’t think there is much that I would change. The principles hold true today, and I still hear of teams who fall for the same traps. If there was anything I might add (although at the cost of potentially going stale) is an example around OKRs. OKRs are a popular planning tool today that, when used well, create strong alignment. Unfortunately many use OKRs poorly that create a lot of dysfunction, because they forget to apply these principles. As an example, many people use a metric as a replacement for a good objective (e.g. “Deliver 50% more story points in 6 months”). This objective is terrible because it doesn’t describe the intent and, like most metrics, easily manipulated. A well written OKR would be clear about the outcome (e.g. “Improve productivity by reducing distractions and rework”) and then pick two key results as indicators (not replacements) to track progress.
Many companies still don’t make good use of metrics. They either don’t measure anything at all, or they measure meaningless metrics. What would be your advice for engineering leaders that want to make the most out of metrics? Where to start from, apart from reading the guidelines you wrote?
Other than the guidance provided engineering leaders need to sit down and answer a set of really tough questions, such as:
- What’s most important to you and your team right now?
- What might be holding us back?
- What aren’t we doing that have the most impact?
- What impact do I care about right now?
These questions are the starting point to pick a priority, a single goal. Once you have the answers to these questions, you can continue onwards to think about what you would measure based on what you care about. Just like how good software is built, start small and iterate 🙂