Big Data: Principles and Examples Vol. 2

In this volume, we discuss Product Recommendations.
Product Recommendations

Product recommendations span far more than just films. We have all seen these recommendations hundreds of times while shopping online. You are looking at one product, and five or ten others are recommended, either as alternatives to consider, or perhaps accessories, based on what shoppers who are in some way like you have done in the past.

Many frame the product recommendation problem as one of prediction. How can I predict which product you are likely to buy next, and recommend it to you? That’s actually a lot better than what contestants were asked to predict for the Netflix Prize. Unfortunately, prediction is not really the problem at all. If I am 100% accurate in my prediction, and you buy exactly what I predicted you would, I haven’t changed anything. I haven’t generated any incremental value for the retailer, or created much immediate value for the shopper, other than perhaps saving them a bit of time.

Instead, the problem is a subtly different one. I have to predict which products I can offer you that will result in sales that would not have otherwise occurred. Those are the sales that drive incremental revenue and value. This brings up two key issues.

First, I have to decide exactly what I want to measure. For example, is it just sales this session, or do I want to measure things like my ability to influence the shopper to return to my site more often in the future, and generate incremental sales that way? This is Principle 1.

Second, to measure the difference I made, I need a control group. I have to design and run a real scientific experiment if I want to understand what is truly incremental. This is Principle 2. Without the experiment, I cannot possibly say what people bought as the result of my recommendations vs. what people would have bought anyway.

I would never do that in a drug trial. I would never give an experimental drug to 100 patients with a disease and declare it was effective simply because 90 of them survived. I need a control group to get either a placebo or the best-known existing treatment. If only 20 of the control group survive the disease, my new drug is fantastic. But if 95 of the control group survive, my drug is at best ineffective and at worst counterproductive. It is a sad state of the Big Data world that there are still an awful lot of conclusions being drawn and publicized without any real experimental design or control groups. Principle 2 is all about trying to change that.

Of course, not every experiment needs to run as long or take as much time to set up as a clinical trial of a new drug. With the right infrastructure that embeds repeatable experimental best practices, many organizations can run tens if not hundreds of experiments simultaneously. Perhaps the best at this are large online businesses like Amazon, Google, and Bing. All three have made very deliberate platform investments over the years to ensure that almost anyone who can change anything about how the web site looks or works can also experiment to determine the effects of their change in a very controlled way.

While an experimental platform is immensely useful, the absolute most valuable asset required is an organizational culture that values experimentation, that recognizes that many experiments will fail, but that they will fail quickly and be turned off, and that the value driven from the successful experiments will be vast relative to the short-term cost of the failed ones. Once this cultural change takes place, it has been my experience that people very rapidly find ways to do large numbers of quick experiments that in aggregate drive real value. In brick-and- mortar retail, for example, it is now standard practice to test new concepts in a handful of stores for no more than 2 or 3 months, then decide how to improve them before rolling them out to hundreds of stores. A large chain can easily have 10 or more in-store experiments running at any given time.