The Power of Measurement

Peter Drucker, widely regarded as the father of modern management, coined the phrase “What gets measured gets managed” in 1954. However, this well-known saying is actually part of a more comprehensive and thought-provoking statement: “What gets measured gets managed — even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.”

Drucker’s insight highlights a crucial challenge in business analytics. While gathering data is essential, the real skill lies in identifying and prioritizing the right metrics to drive a business forward. By focusing on impactful metrics, you can ensure that what you measure and manage truly matters.

Although this blog primarily addresses product analytics in tech companies, the principles apply broadly to all businesses and types of analytics. The insights shared here stem from my experiences as a data professional across various company sizes and product types, including a startup (Digivizer), a scale-up (Immutable), and a tech giant (Facebook).

Prioritizing Metrics: A Strategic Approach

Aligning Metrics with Product Lifecycle Stages

Importantly, a company’s most critical metrics evolve over time. Consider Uber, for instance. Despite being unprofitable for nearly 15 years, it’s hailed as one of the most successful businesses in recent history. Why? Because Uber strategically prioritized rapid growth over immediate profitability in its early years.

Initially, Uber focused intensely on metrics like user growth and retention to establish market dominance. Only after becoming the leading ride-sharing company did Uber shift its focus towards profitability and financial sustainability. This strategy exemplifies how companies should align their metrics with the stages of the product lifecycle.

The Four Stages of the Product Lifecycle

To effectively prioritize metrics, you should consider the four stages of the product lifecycle:

1. 1. Introduction

1. 1. Growth

1. 1. Maturity

1. 1. Decline

Each stage presents unique challenges, and the metrics you focus on should help address these pressing issues. While specific tactical problems may vary, they generally stem from the following high-level questions:

1. 1. Introduction: Have we achieved product-market fit?

1. 1. Growth: How can we scale effectively?

1. 1. Maturity: How do we ensure profitability?

1. 1. Decline: How can we maintain user interest and slow the decline?

 

Let’s delve into each stage and explore the most relevant metrics and analyses.

Stage 1: Introduction – Achieving Product-Market Fit

The Crucial First Step

The Introduction stage marks the beginning of the product lifecycle and focuses primarily on achieving product-market fit. During this phase, product owners must determine whether their offering meets a genuine market need and resonates with the target audience. Understanding product-market fit involves assessing whether early adopters not only use the product but also find significant value in it. Establishing confidence in product-market fit lays the foundation for future growth and scalability.

Key Metrics for Product-Market Fit

Three essential metrics can provide clarity on whether you’ve achieved product-market fit. In order of importance, these are:

1. 1. Retention: Do users find value in the product?

       

      • • Example metrics: D30 Retention, Cohort Retention Curves

1. 1. Active Users: How many users does the product have? Is this number increasing?

       

      • • Example metrics: Daily Active Users (DAU), Monthly Active Users (MAU), Growth Accounting

1. 1. Stickiness: Is the product engaging and used frequently?

       

      • • Example metrics: DAU/MAU, Activity Frequency Histogram (sometimes called L28 Histogram)

Interpreting the Metrics

When used together, these three metrics can quantitatively measure product-market fit or point to the most critical product issues. You’ll likely encounter one of five potential scenarios:

1. 1. No long-term retention and low user growth (worst case):

       

      • • This scenario indicates no product-market fit.
      • • Users aren’t returning to use the product, and there’s a small market.
      • • This situation requires significant changes in the product and/or the target market.

1. 1. No long-term retention but high user growth:

       

      • • This represents the “leaky bucket” problem.
      • • Users are being acquired and using the product briefly, but all eventually churn.
      • • Focus here should be on fixing retention and slowing down growth.

1. 1. Long-term retention but low user growth:

       

      • • In this case, focus on adjusting the acquisition funnel to improve user growth.
      • • Alternatively, if the market proves to be small, consider pivoting to a larger market.

1. 1. Long-term retention, high user growth, but low stickiness:

       

      • • This indicates a utility product that users find valuable but use infrequently.
      • • Examples include tax preparation apps, travel websites, and event ticketing sites.
      • • Focus should be on exploring new features to make the product more engaging.

1. 1. Long-term retention, high user growth, and high stickiness (ideal state):

       

      • • Users are returning to the product, using it frequently, and user numbers are growing.
      • • This scenario demonstrates strong product-market fit.

Once an organization gains confidence in product-market fit, attention can shift to growth. This approach prevents wasteful spending on user acquisition only to have to pivot the product or market, or watch the majority of users churn.

Stage 2: Growth – Scaling Effectively

From Promising to Dominant

The Growth stage presents an opportunity for a product to evolve from promising to dominant. A prime example of effective scaling is Facebook’s famous “8 friends in 10 days” rule. Through funnel analysis and experimentation, Facebook discovered that new users who connected with at least 8 friends within their first 10 days were far more likely to remain active on the platform. This insight led to focused efforts on optimizing user onboarding and encouraging friend connections, significantly boosting user retention and stickiness.

During this stage, the key question becomes: How do we scale effectively while maintaining product quality and user satisfaction?

Broadening Analytics Focus

In the Growth stage, analytics should expand to include three main types:

1. 1. User Journey Analysis: How can we optimize the user experience?

       

      • • Example metrics: Conversion Rate, Time to Convert, Funnels

1. 1. Experimentation: How can we determine whether a change will positively improve key metrics?

       

      • • Example methods: A/B Testing, Multivariate Testing

1. 1. ‘Aha’ Analysis: What moment causes a step-change in a user’s retention and stickiness?

       

      • • Example metrics: A combination of user journey analysis, experimentation, and product-market fit metrics

Implementing User Journey Analysis

When implementing user journey analysis, remember that less is often more. While it may be tempting to instrument every page and button in a product, this approach can be burdensome for engineering to implement and difficult to maintain. Instead, start with just a beginning and end event — these two events will allow you to calculate a conversion rate and a time to convert. Expand beyond two events only to include critical steps in a user journey. Ensure that events capture user segments such as device, operating system, and location.

Building Experimentation Capabilities

Experimentation is a skill that requires practice. Start building this capability early in a product and company’s lifecycle, as it’s more challenging to implement than a set of metrics. Develop this skill by involving product, engineering, and data teams in experiment design. Experimentation isn’t only crucial in the Growth stage but should remain a fundamental part of analytics throughout the rest of the product lifecycle.

Uncovering the ‘Aha’ Moment

‘Aha’ Analysis helps identify pivotal moments that can turbocharge growth. These are the key interactions where users realize the product’s value, leading to loyalty and stickiness. Facebook’s “8 friends in 10 days” was their users’ ‘aha’ moment. This analysis requires analysts to explore a variety of potential characteristics and can be challenging to identify and distill down to a simple ‘aha’ moment. Be sure to use a hypothesis-driven approach to avoid getting lost in a sea of data.

Stage 3: Maturity – Ensuring Profitability

Shifting Focus to Long-Term Sustainability

In the Maturity stage, the focus shifts from rapid growth to optimizing for profitability and long-term sustainability. This phase involves refining the product, maximizing efficiency, and ensuring the business remains competitive. Companies like Apple, Netflix, and Amazon have successfully navigated this stage by honing in on cost management, increasing user revenue, and exploring new revenue streams.

Key Metrics for Profitability

During this stage, the focus shifts primarily to monetization metrics:

• • How can we be profitable while maintaining a high-quality product and satisfied customer base?

• • Example metrics: Customer Acquisition Cost (CAC), Customer Lifetime Value (LTV), LTV:CAC Ratio, Monthly Recurring Revenue (MRR)

Strategies for Profitability

Monetization metrics have clear objectives in terms of trying to increase revenue and decrease costs. Marketing and Go-To-Market teams often own CAC reduction, while product teams typically own LTV and MRR improvement. Strategies can range from optimizing advertising spend and reducing time to close sales deals to cross-selling and bundling products for existing users.

As a general guideline, a LTV:CAC ratio of 3:1 to 4:1 is often used as a target for B2B software companies, while B2C targets are closer to 2.5:1. However, these targets can vary based on industry and business model.

Stage 4: Decline – Maintaining Interest and Slowing the Decline

Navigating the Inevitable Decline

As Jeff Bezos famously said, “Your margin is my opportunity.” As products mature, profitability inevitably declines. Competitors identify your opportunity and increase competition, existing users migrate to substitutes and new technologies, and markets become saturated, offering little growth. In this phase, maintaining the existing user base becomes paramount.

Key Analyses for the Decline Stage

In Stage 4, a broad set of useful metrics can be adopted. Some key types of analysis include:

1. 1. Churn Prediction Modeling: Can we identify users likely to churn and intervene?

       

      • • Example models: Logistic Regression, Tree Models, Neural Networks

1. 1. Power User Analysis: What can we learn from the most engaged users?

       

      • • Example metrics: Stickiness, Feature Usage, Transaction Volume

1. 1. Root Cause Analysis: What are the root cause drivers of key metrics?

       

      • • Example analysis: Quarterly Business Reviews, Issue Driver Trees

Leveraging Predictive Models

By creating churn prediction models and analyzing feature importance, you can identify characteristics of users who are likely to churn and deploy intervention measures. Given that new user growth has slowed, retaining existing users becomes critical. This analysis may also help resurrect previously churned users.

Learning from Power Users

Power user analysis seeks to understand the most engaged users and their characteristics. These users are the highest priority to retain and have the product-usage behavior that would ideally be shared across all users. Look for users who are active every day, who spend long periods in the product, who use the most features, and who spend the most. Deploy measures, such as loyalty programs, to retain these users and identify pathways to increase the number of power users.

Digging Deep with Root Cause Analysis

Root cause analysis is essential for delving into specific problem areas within a mature product. Given the complexity and scale of products at this lifecycle stage, having the capability to conduct bespoke deep dives into issues is vital. This type of analysis helps uncover the underlying drivers of key metrics, provides confidence in product changes that are costly to implement, and can help untangle the interdependent measures across the product ecosystem.

Conclusion

“Focus is about saying no.” — Steve Jobs. Product analytics is a bottomless pit of potential metrics, dimensions and visualizations. To effectively use product analytics, companies must prioritize metrics down to a few focus areas at any one time. These metrics can be supported by a range of other measures, but must have the following:

• • Teams aligned on which metrics should be prioritized

• • Teams who deeply understand the definition of key metrics

• • Metrics that are tied to a key product question

• • A tangible action which can be taken to improve the metric

This can be achieved by prioritizing the right metrics at each product lifecycle stage — Introduction, Growth, Maturity, and Decline. From achieving product-market fit to scaling effectively, optimizing for profitability, and maintaining user interest, each phase demands a clear focus on the most relevant problems to solve.

Remember, it’s not about measuring everything; it’s about measuring what matters. In the words of Steve Jobs, let’s say no to the noise and yes to what truly drives our products forward.

Addendum

I avoided listing too many specific metrics in the sections above and only provided some example metrics for each product lifecycle stage. Instead I focused on the over-arching themes to focus analytics against. But, if you are looking for the long list of options, there are some good resources linked below.

• • Product Metrics: The Ultimate Guide

• • The SaaS Metrics That Matter

• • Product Analytics Metrics: Comprehensive Guide for Growth

• • 15 Important Product Metrics You Should Be Tracking

• • Product Analytics Metrics: What To Track & How To Optimize Them?

• • Powerful tips to enhance your Prompts in the AI World