Data vs Intuition

A question from this newsletter’s mailbag (yes you can email me) recently piqued my interest - a reader asked whether their answer to my “data vs intuition” WMT interview question passed the sniff test. I’m copy/pasting the original question and rationale below:

“walk me through a product trade-off where the data in your hands vs the intuition in your head didn’t jive, and how you made sense of it”

from this I can learn:

• what level of trade-offs you have made

• if you use data and to what degree

• do you over-rely on data to guide

• how you build / refine product instinct

• if you follow curiosity or index on delivery

The reader’s answer was sometimes you see a data point that seems too good to be true, and talking to customers can provide the context to gauge whether it is or isn’t. I ask this question in phone screens pretty regularly, and it’s pretty common to hear an example where the internal metrics dashboard and the external customer voice are the source of the tension. One of the reasons I like this question is I’m always looking to learn new ways to blend quantitative and qualitative insights, which I think is the real magic of product development. But this reader’s response led me to another realization - one way to connect quant and qual insights is to think of them as 2 ends of a shared view, zoomed-out at one end and zoomed-in on the other end. For example, in the situation the reader shared, the data insight was a zoomed-in view, but the customer conversation layered on the zoomed-out view.

Here are some concrete examples:

• interviewing a set of users with a common profile can help you shortlist 1 shared problem to scope and solve for them (zoomed-out / qualitative), but running an A/B test with different UX variations can help identify the optimal solution path to take (zoomed-in / quantitative)

• looking at a category’s TAM can determine whether your company should invest in the space (zoomed-out / quantitative), but talking to analysts who cover the domain can shape the differentiated solution you build (zoomed-in / qualitative)

I know “blend qual and quant when making product decisions” is not a new takeaway, but being aware of whether you’re zoomed out or zoomed in and knowing what altitude to bridge things is useful. To go back to the original WMT question, I think the best examples of data vs intuition conundrums are when the zoomed-out and zoomed-in views don’t reconcile:

• we built a thing that we heard a legitimate need for, but now no one is using it

• we optimized a painful user workflow, but the user survey reactions were meh

• we knew the adoption curve of our product follows a particular pattern, but that turned out not to be true when we entered a new geographic region

• the intended usage of our product was X, but then users kept using it for Y

As always, I’d also love to hear from readers on their attempts to reconcile data vs intuition - please chime in via comments👇. And if you enjoyed this post, please consider subscribing.

---

further reading / references

• I’m a fan of open-ended, multi-layered, tangent-spawning interview questions that can fill up the allotted time - I call them “walk me throughs…” (WMTs)

• I’ve highlighted top-down directives as a cause of metrics malfunction before, but in this context I’d rephrase as a leadership team with a zoomed out view that doesn’t jive with the product team’s zoomed in view is a reason you can end up with a south star metric (aka a bastardized north star metric)

• my recent tongue-in-cheek post about product onions (based on a Twitter thread where the exec team was pushing revenue as a product team goal) is an example of the above zoomed-out / zoomed-in dissonance that can arise

---

childish drawing / interpretation