Revamping SAFe's Program Level PI Metrics Part 2/6: Business Impact

“Managers shape networks’ behavior by emphasizing indicators that they believe will ultimately lead to long term profitability” – Philip Anderson, Seven Levers for Guiding the Evolving Enterprise

Introduction

In Part 1 of this series, we introduced the Agile Release Train (ART) PI metrics dashboard and gave an overview of the 4 quadrants of measurement. This post explores the first and arguably most important quadrant – Business Impact.

As you may have guessed from the rather short list, there can be no useful generic set of metrics. They must be context driven for each ART based on both the mission of the ART and the organisational strategic themes that gave birth to it. As Mark Schwartz put it so elegantly in The Art of Business Value, “Business value is a hypothesis held by the organization’s leadership as to what will best accomplish the organization’s ultimate goals or desired outcomes”.

Definitions

Rationale

Fitness Function

When reading The Everything Store: Jeff Bezos and the Age of Amazon, I was particularly taken by the concept of the fitness function. Each team was required to propose ".. A linear equation that it could use to measure its own impact without ambiguity. … A group writing software code for the fulfillment centers might home in on decreasing the cost of shipping each type of product and reducing the time that elapsed between a customer's making a purchase and the item leaving the FC in a truck". Amazon has since moved to more discrete measures rather than equations (I suspect in large part due to the bottlenecks caused by Bezos' insistence on personally signing off each team's fitness function equation), but I believe the “fitness function mindset” has great merit in identifying the set of business impact metrics which best measure the performance of an ART.

To illustrate based on three ART's I work with:

• An ART at an organisation which ships parcels uses "First time delivery %". They implement numerous digital features enabling pre-communication with customers to avoid delivery vans arriving at empty houses. Moving this a percentage point has easily quantifiable bottom-line ROI impacts.
• An ART focused on Payment Assurance at an organisation which leverages "Delivery Partners" to execute field installation and service work. Claims for payment submitted by these partners are complex and require payment within tight SLA's. A fitness function based on Payment lead time and cost savings based on successful claim disputes would again easily be mapped to quantifiable ROI.
• A telco ART focused on self-service diagnostics for customers. The fitness function in this case would reference “reduced quantity of fault-related calls to call centers” (due to the customer having self-diagnosed and used the tool to make their own service booking if required), “reduced quantity of no-fault-found truck rolls” (due to the tool having aided the customer in identifying ‘user error’), “increased first call resolution rates for truck rolls” (due to the detailed diagnostic information available to service technicians).

Considerations when selecting fitness function components
Obviously, the foremost consideration is identifying a number of components from which one can model a monetary impact. However, I believe two other factors should be considered in the identification process:

• Impact on the Customer
• Ensuring a mix of both Leading and Lagging Measures

Net Promoter Score (NPS) is rapidly becoming the default customer loyalty metric, and whilst Reichheld argues in The Ultimate Question 2.0 that mature NPS implementations gain the ability to quantify the value of a movement in a specific NPS demographic I have yet to actually work with an organization that has reached this maturity. However, most have access to reasonably granular NPS metrics. The trick is identifying the NPS segments impacted by the customer interactions associated with the ART’s mission and incorporating those measures.

When it comes to identifying useful leading metrics, there can be no better inspiration than the Innovation Accounting concepts explained by Eric Ries in The Lean Startup. In some cases (particularly Digital), it can also be as simple as taking the popular Pirate Metrics for inspiration. For many trains with digital products, I also believe abandonment rate is an extremely valuable metric given that an abandoned transaction tends to equate to either a lost sale or added load on a call center.

Program Predictability

This is the standard proxy result measure defined in SAFe. It is a great way of ensuring focus on objectives whilst leaving space for Product Owners and Managers to make trade-off calls during PI execution. In short, I paraphrase it as "a measure of how closely the outcomes from PI execution live up to the expectations established with Business Owners at PI planning and how clear those expectations were".

But wait, there's more!

A good train will use far more granular results metrics than those listed above. Each feature should come with specific success measures that teams, product owners and managers should be using to steer their strategy and tactics (fuel for another post), but I am seeking here a PI level snapshot that can be utilized consistently at portfolio levels to understand the success or otherwise of investment strategy.

A closing note on the Fitness Function

I believe the fitness function definition should be identified and agreed at the ART Launch Workshop. Well-launched ARTs will have all the key Business Owners present at this workshop, and I strongly believe that agreement on how the business impact of the ART will be measured is a critical component of mission alignment.

Series Context

• Part 1 – Introduction and Overview
• Part 2 – Business Impact Metrics (You are here)
• Part 3 – Culture Metrics
• Part 4 – Quality Metrics
• Part 5 – Speed Metrics 
• Part 6 – Conclusion and Implementation

“The gold standard of metrics: Actionable, Accessible and Auditable ... For a report to be considered actionable it must demonstrate clear cause and effect … Make the reports as simple as possible, so everyone understands them … We must ensure that the data is credible” – Eric Ries, The Lean Startup