Andon Cords: When to Cancel a Product Launch
I recently polled subscribers to see what topics they’d like to hear about next, and “screwing up product launches” was unexpectedly the top choice. Below is a recap of on the most confounding releases I’ve ever been involved in (please try not to laugh out loud). Also, I want to preemptively share a counter-example of one of my well-executed launches, which incorporates many of the lessons learned during this episode.
One of the joys of working at Amazon (remember my intense first week) was the hundreds of little tools available to every employee to explore service interfaces (the result of an architectural mandate). Every dimension of the commerce lifecycle (catalog, inventory, pricing, orders, fulfillment, returns) had very powerful, internal admin GUIs accessible by anyone to play around with. I would sometimes just jump from wiki to wiki, perusing the tl;dr of each teams’ toolchain. Certain tool names stuck with you because they were tongue-in-cheek jokes, memorable acronyms, or just unfamiliar terms. One such utility that stuck with me was the Andon Cord.
Basically, if there was an issue with an item listed on Amazon.com, anyone could “pull” the Andon Cord (a principle borrowed from Toyota’s manufacturing process). The result was the immediate halting of the item’s sale, because something (quality, price, availability, review) was off. I really, really wanted to pull the Andon Cord someday!
Fast forward to the actual launch debacle. A peer of mine had been working on incorporating the Kindle and Audible customer experiences together for 1+ year; this was the culmination of Amazon’s acquisition of the audiobook platform and involved unifying the purchase of digital content on either the web or device store, turning on the ability to play audiobooks on the latest generation Kindle device, and providing an in-country specific catalog based on regional publisher agreements. The launch had been pushed several times due to various delays (UX debate, press timing, title selection) and now was planned to overlap with my colleague’s long-awaited world tour. I had been on the Kindle team a few months and was very eager to actually manage a launch (I’d shadowed a couple of major events and thought the best way to learn was to shepherd a release, even if it wasn’t a product that I’d been intimately involved with). So, we worked out a setup where he put together the launch plan and I covered while he was on vacation.
What could go wrong? Apparently, everything.
I got involved only a couple of weeks before launch, and while I understood the MVP experience and release plan, I didn’t have the tribal knowledge that comes with having dealt with all manner of product development issues from inception on. The core flow was pretty simple: buy an audiobook and make sure it plays. The key dimensions that mattered were as follows:
quality of audiobook download / playback on-device
accuracy of titles available for users in different locales
consistency between web store vs. device store purchase
The night before the go-live we ran through all scenarios with our QA team and felt good. The day of the launch we had multiple offices (Kindle in Seattle, Audible in New Jersey) join a video conference and walk through the release checklist. We had a deployment moratorium in effect to ensure there wouldn’t be any complicating variables during the rollout. Everyone had a test script (book titles, user logins, etc) and multiple devices (WiFi, 3G) handy. We started flipping switches, and within minutes would be ready to cut the press release…
Setback #1
As soon as one end-to-end test worked (buy an audiobook, download it, play successfully) we all loosened up. We started joking around and going off script - buying random titles, toggling WiFi on and off, trying to play books in non-English languages, etc. One person played an audiobook and the sound came out all garbled.
Everyone stopped.
It’s your device. It’s a corrupt download. It’s an encoding issue.
Did you download over 3G or WiFi? Do you have the right software build? Try it again!
Other people started having the same issue. Attempts to reproduce it were intermittent. The same folks on the same device with the same title wouldn’t be able to recreate it (they would delete and re-buy the title and everything would be fine).
The clock was ticking, and we couldn’t suss out the underlying issue, so we paused the launch.
Within hours, we figured out that new (incorrectly configured) hosts had been added to the cluster of servers that served audiobook content. One of the (new) engineers on the team heard about the launch and thought it would be good to add additional capacity for load. This person didn’t know the proper host config procedure or, more importantly, that a moratorium was in effect. Silly (but well-intentioned), and not the end of the world, so no reason to not re-try the launch that same afternoon.
Setback #2
There was relief as we started our second attempt. The first try hadn’t worked, but we knew exactly what had gone awry, and were confident because it felt like the type of issue we’d laugh about once everything was said and done.
We quickly breezed through the buy/download/play scenario, and started going through the portion of the test script where we had spoofed logins for users from different countries, just to ensure that the list of titles available for purchase was accurate. One of the folks in the room (who maintained content publisher relationships for the Kindle team but not the Audible team) noticed that the # of titles available when he set his country to Mexico was the same as his usual login (in the US). How could the exact same # of audiobooks be available for sale in both Mexico and the US?
We quickly started trying random countries (I picked Turks and Caicos in a panic) and lo and behold many of them had the same selection count as the US. We were definitely not correctly enforcing in-country content restrictions, which was both a customer promise and legal policy blunder.
To further complicate things, it was unclear whether this warranted hitting the brakes on the press release. The top 10 countries in terms of audiobook buyers were good to go. Given this was a joint venture, the Kindle and Audible teams couldn’t agree on whether we pause the launch / sort out the issue OR go ahead with comms / triage it later. Execs were looped in, egos were bruised, and the launch was halted. For the second time in a day.
It took into the wee hours of the night, but we were able to root cause the issue. We had a full audiobook catalog that included every title. We had denylists per country from publishers, that were updated periodically and enforced dynamically. So any time you were viewing the store, based on the provided (or derived) country of residence, the view would be filtered accurately. Apparently there was a bug where we hadn’t properly ingested the denylist for a bunch of countries outside of the top 10. But our test plan only ever ran through the top 10, so it had been months of staleness that no one had noticed. It was just luck that someone happened to try that scenario during the go-live. The sense was now developing that rigor was lacking, and there was a strong push to go back to the drawing board. I pushed for, and got, “one last chance” the next day.
Setback #3
There was real tension in the room. I said something ridiculous to kick things off like “third time’s the charm”. It wasn’t.
We were testing various buying scenarios, and again, through serendipity, someone noticed that a particular title was priced differently than it had been the day before during launch testing.
Several people spoke up and highlighted that was a known issue. When the Amazon and Audible content stores had been integrated, reconciling item data (e.g. title description, audiobook price) was a matter of keeping 2 different catalog systems in sync. Whereas Amazon had reduced pricing change propagation latency over a decade plus (thanks to features like deal of the day), Audible had never focused on that (their differentiators were the listening experience and content selection). Someone made the argument that price increases / drops were part of commerce - if it swung lower for the customer, good timing, and if it swung higher, support would refund the difference for anyone perturbed enough to make the claim.
I didn’t have the benefit of having been in the trenches with this group for months, but I had read the press release around Amazon’s purchase of Audible. The same audiobooks were also sold on Apple’s iTunes store and played on iOS devices (there was an ongoing industry debate on purpose-built reading devices like the Kindle vs multi-purpose consumption devices like the iPad).
I was very uneasy at this point. I was trying to ship a product that I hadn’t masterminded, and the ridiculousness of the situation was dawning on me. I could see the figurative Andon Cord dangling, begging me to pull it. My CXBR radar was on high alert. I asked a question that I already knew the answer to.
“Do we at least make sure that we’re never selling the same content on iTunes for cheaper?”
“Hmm. I don’t think we ever thought through that angle.”
I did a quick test. As feared. And then I pulled the Andon Cord.
It took months to re-architect catalog systems to ensure pricing changes did not put us in an awkward situation from a PR perspective. Audiobooks cost what they cost, and if there is ever a deal, it’s definitely available on Amazon properties, while cascading to other storefronts is secondary. It’s the same idea as a retail store honoring their online prices, but not worrying about what their products are going for on eBay. It was obvious in hindsight what the ideal customer experience should be, but it was a journey to figure it out, and not something that could have happened during a war room operating on fumes.
My teammate came back from his trip, horrified to hear the story of the triple flop, but eager to overcome the perception now plaguing the product. I didn’t get a fourth chance, but he got his first, and it was flawless. In the end, he shipped a much better product, and I like to think that it was always his to do so.
So what did I learn, and what can you take away?
retrospective learnings need to be operationalized and future-proofed - Amazon had perfected all manner of procedures around launch windows (e.g. deployment moratoriums) but a recently acquired company (Audible) had not been looped in
to PM a product requires honing instinct, which can’t be manufactured if you skip steps - being present for the entire lifecycle results in intuition (e.g. P0/P1) that’s handy when turbulence hits, but landing another pilot’s plane is unfamiliar
all the good (and bad) habits of a cross-functional team come across at ship time - any muscles built around communication pay off in speed of decision-making, any hangups around collaboration get exacerbated by the moment’s intensity
pulling the Andon Cord is hard, because we need time to acknowledge mistakes - part of me was willing to compromise, part of me just wanted to be done with it, and part of me wanted a 4th shot at it after re-jiggering the plan “my way”
failure is learning, but more importantly, the safety to fail multiple times is key - I still can’t believe my management chain gave me 3 chances in 24 hours, and even after that fiasco I went on to manage several key products without oversight
Would love to hear from readers (use the comments below) on their product launch disasters…
further reading / references
again, read this First Round Review article which proves I learned from this mess
Andon Cords: a quick primer, a detailed history, and Amazon’s implementation
childish drawing / interpretation
