Group Product Manager
Laura heads up product for consumer web, international, and LivingSocial at Groupon. She has a bachelor’s in mathematics from the University of Chicago and a master’s in computer science with specialization in machine learning from Georgia Tech. She has more than 10 years of experience in ecommerce product management at four Chicago tech companies, from early stage startups to publicly traded global companies. She is passionate about using analytics and machine learning to create a delightful customer experience.
At Groupon, we have a very data-driven philosophy of product management. In this blog post, I’ll talk through how we approach product ownership in a data-driven way, from financial forecasting to roadmapping to feature development to experimentation.
Financial Forecasting and Roadmap Creation
For every candidate feature, we calculate the projected financial upside according to the following formula:
feature_revenue_forecast = expected_lift x platform_factor x success_probability x platform_revenue
- feature_revenue_forecast is what we are trying to calculate (the expected revenue from the feature)
- expected_lift is the increase in conversions we expect from users in the treatment group vs. users in the control group. The vast majority of experiments fall between -1.0% and +1.0% lift.
- platform_factor is what percent of all users of the platform (whether iOS, android, mobile web, or desktop web) are part of the experiment. For example, for a test on the checkout page on mobile web, the platform factor will be 100%—100% of users who place an order on mobile web visit the checkout page during their journey. For a test on the getaways deal page, the platform factor will be much smaller, so the overall financial impact of the test will be smaller.
- success_probability is a haircut we apply to take into account that not all experiments will succeed. In fact, only about 30% of our experiments are successful. The success_probability for a given feature could be greater than or lower than 30%, depending on how confident we are that the experiment will succeed. For some experiments that are primarily for strategic reasons, such as maps improvements, we will use a success_probability of 90% or 100%.
- platform_revenue is the total revenue generated by the platform. For example, the platform_revenue for iOS is the total revenue from orders placed via the iOS app.
With this formula, we have a consistent and data-driven way to estimate the upside from each proposed initiative. Then, once each experiment concludes, we compare our estimates to the actual results, and over time we refine our estimations.
We use these estimated upside figures to create product roadmaps. In order to prioritize initiatives and determine the cutlist, we need to introduce another data point—the engineering effort required. Then, we use the following formula to calculate the ROI of each feature:
ROI = feature_revenue_forecast / level_of_effort
We then stack rank features according to their ROI.
ROI is an input into the creation of the product roadmap and the determination of the cutlist, but it is not the only input. I always like to ensure that there is a healthy amount of time spent on engineering excellence (site stability, paying down technical debt, reducing latency, library upgrades, increased test coverage, improved tooling). I also like to ensure that we have a customer focus. Many of our features come directly from customer feedback via focus groups, quantitative surveys, and app store feedback; the Wishlist feature was one of these. I also reserve a healthy amount of time for strategic initiatives that may not provide lift in the short term but that set us up for success in terms of the Groupon 2020 vision.
At Groupon we are lucky to have vast amounts of data that we can use to deliver a delightful product to our customers. We have worked with one million merchants to date; we have pumped more than $18 billion into local businesses; we have more than 1 billion Groupons sold; our app has been downloaded 171 million times, and we have saved customers more than $28 billion.
The Groupon platform handles tens of billions of user actions per month, and for machine learning algorithms that drive core product features our platform needs to make decisions (such as which deal to show the user next) in fractions of a second.
Developing product features that take advantage of these vast amounts of data in a performant way is an interesting challenge.
We use machine learning algorithms in a variety of ways to develop products here at Groupon:
- Supply intelligence – There are millions of merchants we could call at any time to get onto our platform; how do we pick the best ones?
- Fraud prevention – Fighting the bad guys in realtime.
- Discovery and personalization – Selecting which deals to show a given user in her mobile app deal feed.
- Image recognition – Identifying the best user-generated images with neural networks.
- Logistics – Getting ahead of the order rush by sending extra inventory to the right warehouse in advance of high demand.
- Customer support – AI-based chatbots to respond to and resolve customer issues instantaneously.
To make developing data-driven products faster, we built a generic, extensible machine learning platform at Groupon called Flux. Flux is the “Rosetta Stone” between data scientists and engineers.
To make the process for productionalizing machine learnings more robust, Groupon has an ETL management platform called Quantum Engineered Data (QED). QED reads from any source, and includes built-in data cleaning, error correction, and anomaly detection. Clean data is preserved and made available as a “feature catalog.” QED handles failures smartly, supporting falling back to yesterday’s model when appropriate. QED is able to plug into any source of truth—including streams, warehouse tables, and JSON endpoints.
QED gives us a lot more confidence in the robustness of our models. In general, subtle changes to a single data field can seriously impact model performance, and nuances in the data set could look fine to tests but fail in the real world.
Interested in joining Product?
This blog post would be incomplete without a brief discussion of Groupon’s monitoring tools. We have a healthy suite of realtime alerts on product and engineering KPIs. We use splunk for logging and wavefront for graphing. Each service is staffed with a 24/7 on-call schedule, with escalation handled by pagerduty.
Additionally, each product area and business are has an Amazon-style Weekly Business Review, where we look at metric trends longitudinally, identify areas of change or concern, and begin deep dives where appropriate.
There are 100 teams at Groupon that run experiments. At any given time, around 200 experiments are being run simultaneously on the Groupon platform.
Groupon has a dedicated team called Optimize that built a bespoke tech platform for running product experiments with mathematical rigor. The experimentation platform is called Finch Express. Finch Express is built with Ruby on Rails, Node.js, Ember.js, Python, R, and Hadoop/Hive. The team has filed three patents for its innovations on product experimentation.
Essentially, Finch Express uses a technique called Group Sequential Analysis, first developed by Abraham Wald in 1945. Group Sequential Analysis has been used extensively in high-risk clinical trials, such as heart valve studies, where it’s possible that one treatment is actually harming the patients. Ethically, we would want to stop a harmful clinical trial immediately—but statistically, checking the experiment results mid-run or “peeking” will vastly increase your rate of false positives and invalidate your statistical results.
Group Sequential Analysis provides a controlled, statistically rigorous way to “peek” at experiment results at set points during the experiment run. This allows Groupon to end an experiment early if it is losing money, and to roll out an experiment early if it is deemed an early winner (capturing more upside).
Finch Express does all of this automatically. Product managers create the experiment in Finch Express, add a description and screenshots (to save the details for future product managers to reference), and launch the experiment at 50/50. Finch Express does the heavy lifting of dynamically determining the appropriate lift sensitivity for the experiment (based on traffic and conversion rate), performing the Group Sequential Analysis calculations, deeming the experiment a “success,” “failure,” or “flat” (most experiments end flat), and even automatically rolling out or rolling back the experiment based on its results. Then, Finch Express reports on the financial results of the experiment. The experimentation platform prevents product managers from statistical no-nos, such as peeking, unbalanced bucketing, and concluding the experiment too early. As a result, our experimentation processes have a high degree of statistical rigor.
On average, Group Sequential Analysis allows us to conclude experiments an average of 57.53% earlier compared to simply running them to a single final checkpoint. This reduces the cost of failed experiments, hastens upside capture of successful experiments, and allows for much faster iteration and innovation.
To date, Groupon product managers have run a total of 2,500 experiments, thanks in large part to the proprietary and patent-pending experimentation platform.
Thanks for staying with me until the end! I hope this gives you an idea for how we use big data at global scale here at Groupon to create our product roadmaps, to innovate with new products and features, to monitor product performance, and to evaluate the impact of new initiatives. If you’re interested in learning more about data-driven product management opportunities at Groupon, have a look at our open roles.