How To Get 10x The Value From Your Product Analysis

9 min readDec 29, 2020

Imagine this:

Does this sound familiar? Maybe not? Does the below reality sound more familiar?

You’re on a product team developing a mobile application ─ you’re responsible for ensuring product quality, informing roadmapping, and rollouts of new features. You’ve heard that users aren’t using a feature in the way that you had hoped. You first re-engage with the experience first hand on your iPhone, and reach out to a few users to form a fresh hypothesis. You engage with an analyst on your team to look at what the data says: it says something either completely unexpected, or doesn’t have the level of granularity or correct angle to be meaningful. You re-engage with the analyst, shedding more light on the situation and where you need the analysis to take a different angle, pledging to include this detail upfront next time rather than assume a level of understanding. Repeat the process 2–3 more times. You now have a reasonable understanding of the issue (albeit with some gray areas), as well as reasonable confidence in the scope of the issue and size of the opportunity.

The point here is that most tech companies now have a breadth of data at their hands and, no matter the business-intelligence or product-analytics tools at one’s disposal, one of the greatest costs of turning this data into meaningful insights is at the point of translating hypotheses and questions into meaningful analysis to inform product strategy & execution.

When one person with intimate knowledge of the business owns both the analysis itself and the analysis’ ability to inform product strategy & execution, both are enhanced. This article is focused on arguing why the product operations role should be expected to perform the product analyst role.

The Product Ops Role (overview)
The Product Analyst Role (overview)
Why Product Ops Should Be Expected To Perform as a Product Analyst
Counter-argument: Why Product Ops Shouldn’t Be Expected To Perform as a Product Analyst
What a Successful Process Looks Like when Product Operations acts as a Product Analyst

The Product Ops Role

Product operations is still a relatively new role, but one that many expect to grow in popularity and relevance in the near future. Although product ops roles may manifest themselves differently on different teams within the same company or across companies, the purpose of the role is typically to enable product teams to achieve better outcomes. This article by Blake Bassett does a great job of laying out what the role is and the key tasks and competencies associated. I would summarize to say that product operations typically focuses on …

Ensuring the highest product quality, through bug triaging, dogfooding, product audits, and data analysis.
Acting as the driver of change management, by ensuring ops strategy is informed of and aligned to product strategy, by driving and leading roll-outs including developing strike-teams where appropriate, and by regularly communicating updates and feature expectations.
Informing product roadmapping/strategy, by accumulating user pain points, generating hypothesis across product teams and features, and by facilitating feedback amongst core stakeholders within and across product teams

Product ops is typically expected to be familiar enough with data to perform a certain, relatively low, level of analysis to estimate scope/impact but is not typically expected to answer more in-depth questions. For example, product ops should be able to answer questions such as “How many users will this feature impact, and what is a rough expected outcome on that cohort to our KPIs and to other metrics we care about?” Typically, product ops is not expected to answer more complex questions such as “What effect has this feature had in isolation to our users, controlling for other concurrent changes to their experience? How has their engagement changed over time, amongst different cohorts?”

To help with deeper analysis, enter Product Analysts….

The Product Analyst Role

Product Analysts are typically very comfortable and experienced working with data to uncover insights. I would summarize to say that product analysts…

Translate questions and hypotheses into structured analysis.
Own data analysis & visualization, and communicate the outcomes/insights to aid strategic decisions.
Measure the impact & efficiency of product launches & business processes.

Product Analysts are expected to be able to clearly communicate their findings while having a certain level of understanding of the business — but they are not expected to be an expert in the business including a pulse on its rapid changes. As a result, first and second pass analyses can be often performed at an angle that is insufficient to drive product outcomes, resulting in rework, or worse, in meaningless outcomes.

Why Product Ops Should Be Expected To Perform as a Product Analyst

Simply put, an analysis’ ability to affect outcomes is only realized when it informs and enhances business insights — and it is primarily the business insights, not the analysis, that is the main driver of outcomes. Great business insights on their own go far — great business insights mixed with great supporting analysis goes further — but analysis on its own falls flat without business insights.

a) There are diminishing returns that time has on the quality of analysis, and diminishing returns that the quality of an analysis has on its ability to affect outcomes. For example, spending 50% more time on an analysis may improve its quality by 20%, but improving the quality of analysis by 20% may affect outcomes by just 5% since analysis mostly serves to enhance business insights. See below where a, b, and c on the left hand side graph correspond to d, e, and f on the right hand side graph. Whether or not this tradeoff is worth it will depend on the circumstances: Is this a new or mature product? Is the cost of development small, or large? Many times, it is worth it to invest more time on deep, quality analysis, but I’d argue that most companies spend too much time on data analysis and not enough time close to the business to generate meaningful insights.

b) The translation from hypotheses to analysis costs time and diminishes quality, reducing the ability to generate a meaningful analysis and thereby the ability of the analysis to affect outcomes. Analysis is performed in response to asking questions and in turn helps to generate better, more informed questions along with better, more informed strategy — see the circle graph below (“How Analysis Informs Strategy & Execution”). But there are costly handoffs made whenever multiple parties are involved: work must be summarized, learnings must be translated, and detail is lost — think of the quality of an image degrading each time it is compressed. Beyond degrading quality, these trade offs also take time, often having to revisit and revise analysis to better fit the original hypothesis/motivation.

However, when the individual performing the analysis is an expert in the business and is the one responsible for asking the questions and informing strategy & execution, there are no handoffs to be made: analysis, questions, and hypotheses flow freely, more quickly and at a greater quality as there is less revision to be done and as an in-depth knowledge of the business drives a more nuanced analysis. Further, when the individual responsible for driving the strategy & execution has a firm understanding of the source data and its limitations, we see a more nuanced approach to strategy & execution.

Taken as a whole, when one individual owns the hypothesis, the questions, and the corresponding analysis, we save time and have a greater impact on product outcomes.

Counter-argument: Why Product Ops Shouldn’t Be Expected To Perform as a Product Analyst

Acknowledging all of the reasons above, there are valid reasons for why one could argue for the separation of responsibility between product analyst and product operations.

This is asking too much of product operations. Time spent doing analysis isn’t time that can be spent communicating with internal and external customers, gathering business insights, or focusing on change management. Further, analysis often requires hours of undisturbed time: something that can be difficult to do when in a product operations role to say the least.
The wrong incentives will be at play. In separate roles, a product analyst is an impartial party dedicated to an unbiased assessment of a hypothesis. Going unchecked, product operations performing their own analysis may be disincentivized from attempting to disprove their own hypotheses for the sake of expediency or proving themselves right.
Customer feedback may become muted. Occasionally, word from customers/the business is at odds with what one sees in data — customers may say they hate a new experience, but data may show more engagement. Investing too much time in the data may limit one’s ability to listen to the customer, which is more often true than the data is.
It may be difficult to hire an individual with strong analytical abilities and exceptional communication skills. Today, product operations hires tend to favor skills such as stakeholder management and communication over analytical abilities, while product analysts tend to be hired for analytical ability and strong communication when speaking to those findings but not necessarily when speaking to other subjects.

While this all may be true, and while I’ve worked with many strong product operations specialists who were reasonably weak analytically, I still believe that you can get 10x the value from your product analytics when it is owned by product operations.

What a Successful Process Looks Like when Product Operations Acts as a Product Analyst

Overall, typical product operations responsibilities are more support oriented and responsive while typical product analyst responsibilities are more deliberate and require more dedicated working time.

Carve out time to dedicate to analysis. Mark off a 3–4 hour calendar time block more than a week ahead of time and put yourself on do-not-disturb. Either allow yourself to be interrupted only for emergencies (e.g. outages) or identify someone who can take over that responsibility for you should the need arise. If you’re in the office, place a red post-it note on the top of your monitor to indicate that you’re not to be disturbed. If there are emails or slack messages that you receive, they can wait — 90%+ of “urgent” questions can wait a few hours without harm.
Be fanatical about the 80/20 principle (Pareto principle). As I demonstrated above, the reason that product operations can be trusted to perform product analysis is because there are significantly diminishing returns to time spent on analysis and that analyses have on improving outcomes. Have the courage to say “no” to follow-up questions that are raised by managers but aren’t relevant and won’t change outcomes.
Offload maintenance and reporting to a non-tech business analyst when possible. Building and maintaining detailed reporting and models can be cumbersome. After creation, ask a non-tech business analyst to maintain your dashboarding as the underlying data tables change. At Uber, our data tables change all the time and the maintenance required adds up — but you don’t need intimate understanding of the motivation behind dashboards to update them as minor adjustments to data tables are made.

Overall, I’d love to see more product operations specialists hired who are expected to perform their own analyses in order to improve the outcomes of product teams. That is, after all, the purpose of product operations to begin with.