Data-Driven Product Management at Groupon

Data-Driven Product Management at Groupon

Data-Driven Product Management at Groupon

Laura Hamilton
Group Product Manager
May 22, 2018

Introduction

At Groupon, we have a very data-driven philosophy of product management. In this blog post, I’ll talk through how we approach product ownership in a data-driven way, from financial forecasting to roadmapping to feature development to experimentation.

Hippo

https://www.howgoogleworks.net/

Financial Forecasting and Roadmap Creation

For every candidate feature, we calculate the projected financial upside according to the following formula:

feature_revenue_forecast =
expected_lift x platform_factor x success_probability x platform_revenue

Where:

  • feature_revenue_forecast is what we are trying to calculate (the expected revenue from the feature)
  • expected_lift is the increase in conversions we expect from users in the treatment group vs. users in the control group. The vast majority of experiments fall between -1.0% and +1.0% lift.
  • platform_factor is what percent of all users of the platform (whether iOS, android, mobile web, or desktop web) are part of the experiment. For example, for a test on the checkout page on mobile web, the platform factor will be 100%—100% of users who place an order on mobile web visit the checkout page during their journey. For a test on the getaways deal page, the platform factor will be much smaller, so the overall financial impact of the test will be smaller.
  • success_probability is a haircut we apply to take into account that not all experiments will succeed. In fact, only about 30% of our experiments are successful. The success_probability for a given feature could be greater than or lower than 30%, depending on how confident we are that the experiment will succeed. For some experiments that are primarily for strategic reasons, such as maps improvements, we will use a success_probability of 90% or 100%.
  • platform_revenue is the total revenue generated by the platform. For example, the platform_revenue for iOS is the total revenue from orders placed via the iOS app.

With this formula, we have a consistent and data-driven way to estimate the upside from each proposed initiative. Then, once each experiment concludes, we compare our estimates to the actual results, and over time we refine our estimations.

We use these estimated upside figures to create product roadmaps. In order to prioritize initiatives and determine the cutlist, we need to introduce another data point—the engineering effort required. Then, we use the following formula to calculate the ROI of each feature:

ROI = feature_revenue_forecast / level_of_effort

We then stack rank features according to their ROI.

ROI is an input into the creation of the product roadmap and the determination of the cutlist, but it is not the only input. I always like to ensure that there is a healthy amount of time spent on engineering excellence (site stability, paying down technical debt, reducing latency, library upgrades, increased test coverage, improved tooling). I also like to ensure that we have a customer focus. Many of our features come directly from customer feedback via focus groups, quantitative surveys, and app store feedback; the Wishlist feature was one of these. I also reserve a healthy amount of time for strategic initiatives that may not provide lift in the short term but that set us up for success in terms of the Groupon 2020 vision.

Dilbert

Image source: Dilbert.com

Data-Driven Features

At Groupon we are lucky to have vast amounts of data that we can use to deliver a delightful product to our customers. We have worked with one million merchants to date; we have pumped more than $18 billion into local businesses; we have more than 1 billion Groupons sold; our app has been downloaded 171 million times, and we have saved customers more than $28 billion.

The Groupon platform handles tens of billions of user actions per month, and for machine learning algorithms that drive core product features our platform needs to make decisions (such as which deal to show the user next) in fractions of a second.

Developing product features that take advantage of these vast amounts of data in a performant way is an interesting challenge.

We use machine learning algorithms in a variety of ways to develop products here at Groupon:

  • Supply intelligence – There are millions of merchants we could call at any time to get onto our platform; how do we pick the best ones?
  • Fraud prevention – Fighting the bad guys in realtime.
  • Discovery and personalization – Selecting which deals to show a given user in her mobile app deal feed.
  • Image recognition – Identifying the best user-generated images with neural networks.
  • Logistics – Getting ahead of the order rush by sending extra inventory to the right warehouse in advance of high demand.
  • Customer support – AI-based chatbots to respond to and resolve customer issues instantaneously.

Groupon Mobile App

To make developing data-driven products faster, we built a generic, extensible machine learning platform at Groupon called Flux. Flux is the “Rosetta Stone” between data scientists and engineers.

Flux capaciter

Image: Wikimedia Commons

Data scientists work primarily in R. Flux models are written in Java and Clojure for stability and speed. Python is the glue that connects R and Java. It all runs on Groupon’s large Hadoop cluster.

To make the process for productionalizing machine learnings more robust, Groupon has an ETL management platform called Quantum Engineered Data (QED). QED reads from any source, and includes built-in data cleaning, error correction, and anomaly detection. Clean data is preserved and made available as a “feature catalog.” QED handles failures smartly, supporting falling back to yesterday’s model when appropriate. QED is able to plug into any source of truth—including streams, warehouse tables, and JSON endpoints.

Smart people doing interesting work

QED gives us a lot more confidence in the robustness of our models. In general, subtle changes to a single data field can seriously impact model performance, and nuances in the data set could look fine to tests but fail in the real world.

machine learning - xkcd

Image credit: XKCD

Monitoring

This blog post would be incomplete without a brief discussion of Groupon’s monitoring tools. We have a healthy suite of realtime alerts on product and engineering KPIs. We use splunk for logging and wavefront for graphing. Each service is staffed with a 24/7 on-call schedule, with escalation handled by pagerduty.

Additionally, each product area and business has an Amazon-style Weekly Business Review, where we look at metric trends longitudinally, identify areas of change or concern, and begin deep dives where appropriate.

The data warehouse uses Teradata and Apache Hive.

Experimentation

There are 100 teams at Groupon that run experiments. At any given time, around 200 experiments are being run simultaneously on the Groupon platform.

Groupon has a dedicated team called Optimize that built a bespoke tech platform for running product experiments with mathematical rigor. The experimentation platform is called Finch Express. Finch Express is built with Ruby on Rails, Node.js, Ember.js, Python, R, and Hadoop/Hive. The team has filed three patents for its innovations on product experimentation.

Essentially, Finch Express uses a technique called Group Sequential Analysis, first developed by Abraham Wald in 1945. Group Sequential Analysis has been used extensively in high-risk clinical trials, such as heart valve studies, where it’s possible that one treatment is actually harming the patients. Ethically, we would want to stop a harmful clinical trial immediately—but statistically, checking the experiment results mid-run or “peeking” will vastly increase your rate of false positives and invalidate your statistical results.

Group Sequential Analysis provides a controlled, statistically rigorous way to “peek” at experiment results at set points during the experiment run. This allows Groupon to end an experiment early if it is losing money, and to roll out an experiment early if it is deemed an early winner (capturing more upside).

Finch Express does all of this automatically. Product managers create the experiment in Finch Express, add a description and screenshots (to save the details for future product managers to reference), and launch the experiment at 50/50. Finch Express does the heavy lifting of dynamically determining the appropriate lift sensitivity for the experiment (based on traffic and conversion rate), performing the Group Sequential Analysis calculations, deeming the experiment a “success,” “failure,” or “flat” (most experiments end flat), and even automatically rolling out or rolling back the experiment based on its results. Then, Finch Express reports on the financial results of the experiment. The experimentation platform prevents product managers from statistical no-nos, such as peeking, unbalanced bucketing, and concluding the experiment too early. As a result, our experimentation processes have a high degree of statistical rigor.

On average, Group Sequential Analysis allows us to conclude experiments an average of 57.53% earlier compared to simply running them to a single final checkpoint. This reduces the cost of failed experiments, hastens upside capture of successful experiments, and allows for much faster iteration and innovation.

To date, Groupon product managers have run a total of 2,500 experiments, thanks in large part to the proprietary and patent-pending experimentation platform.

correlation - xkcd

Image credit: XKCD

Conclusion

Thanks for staying with me until the end! I hope this gives you an idea for how we use big data at global scale here at Groupon to create our product roadmaps, to innovate with new products and features, to monitor product performance, and to evaluate the impact of new initiatives. If you’re interested in learning more about data-driven product management opportunities at Groupon, have a look at our open roles.

Laura heads up product for consumer web, international, and LivingSocial at Groupon. She has a bachelor’s in mathematics from the University of Chicago and a master’s in computer science with specialization in machine learning from Georgia Tech. She has more than 10 years of experience in ecommerce product management at four Chicago tech companies, from early stage startups to publicly traded global companies. She is passionate about using analytics and machine learning to create a delightful customer experience.

Laura Hamilton

Group Product Manager

Modularization of Android Apps

Modularization of Android Apps

Modularization in Android Apps

The mobile team organized a Meetup yesterday in Palo Alto out of the new large Spontaneous Combustion conference room. We had about 30 engineers from the area attend plus a great turnout from our team. Eric Fararro introduced Groupon engineering as a whole, followed by a technical talk about application modularization / instant app preparation given by Aolei Zhang and Erik Kamp. We fielded questions about this topic after the talk and had a handful of engineers interested in joining the Groupon mobile team as a result!

Special thanks to Stephane Nicolas for mentoring us through the talks, Daniel Molinero and Cody Henthorne for all the great feedback and pointers, and Lupe Leon for all the help organizational-wise.

Architecture Patterns for Backends beyond SOA

Architecture Patterns for Backends beyond SOA

Architecture Patterns for Backends beyond SOA

Javier Cano, Senior Software Engineer
Sergey Burkov, Senior Java Developer
December 13, 2017

In the Merchant Experience team specifically, and in Groupon in general, we have to deal with the challenge of scale and performance that our global business imposes. We make heavy use of SOA and microservices in our platform, though that is usually not enough. The solutions that we need make us explore and try different architectural patterns that move beyond what a SOA approach can provide. In this short talk we’ll explore some of these alternatives architectures, which problems they solve and how they integrate in microservices platform.

You can see lots more video of Grouponers and their smart friends on our YouTube channel.

Messaging at (Groupon) scale

Messaging at (Groupon) scale

Messaging at (Groupon) Scale

Nikita Berdikov
Senior Software Engineer
December 13, 2017

Every company is using messaging one way or another. So do we at Groupon. Messaging platform allows distributed heterogeneous services communicate with each other in asynchronous publish-subscribe fashion. Let’s talk about problems it helps to solve and problems it creates (especially from the owners of messaging infrastructure point of view). In addition we will go through tools we have built around messaging for better monitoring, maintenance and issues.

You can see lots more videos from Grouponers and other smart people on out YouTube channel.

Groupies Hold Pink Ribbon Breakfast to Fight Breast Cancer

Groupies Hold Pink Ribbon Breakfast to Fight Breast Cancer

VOLUNTEERING

Groupies Hold Pink Ribbon Breakfast to Fight Breast Cancer

14 November 2017

Enjoying pink baked goods (for a cause)

From the Sydney Office

We all have a special woman in our lives, so let’s support them and each other at the Pink Ribbon Breakfast and Bake-Off Competition!

For winning the Global Volunteer-A-Thon, Groupon ANZ was provided the opportunity to not only celebrate but also give back. Together, we enjoied a cupcake, shared a breakfast and heard some important details around Breast Cancer.

Each Groupon team was invited to bring their best baked goods to the table and go head to head in the Ultimate Groupon Bake Off to find out just who is the Bake Leader once and for all!

It was a great event, organized by our staff, with personal stories shared regarding loved ones who had impacted by breast cancer.

Everyone loves a good “camera timer” joke.

From the Sydney Office

And Breakfast Tea in the Melbourne Office

A Note from the Melbourne Office

Just wanted to share our little Pink Ribbon Morning Tea we had this morning in Melbourne. Thanks for everyone who was involved!

Ryan and Craig outdid themselves in the baking department and there were plenty of tasty treats for a good cause to go round!

We’ve opened the morning tea up to the rest of The Hub office space and hope to raise quite a bit of money by the end of the day! I’ll loop back in with the final count tomorrow 🙂

Great to have Ryan and Steve T in the office as well (and our little celebrity Liam the Cavoodle!) – and what a morning to welcome Daniel to the Melbourne Office as well! Go Tea!!!

We Raised a Massive $1173.30!

Thank you to each and everyone of you who baked (or bought) in some items of food. Thank you to the Taskforce, Women@work (Tina, Amanda, Melissa, Naomi, Ryan, Ang, Kiera, Val & Cara) who organized it all, to Steve who judged the bake off & a massive Thank you to everyone who generously donated!

Geekfest: Web.js (Full Stack Javascript)

Geekfest: Web.js (Full Stack Javascript)

Web.js (Full Stack Javascript)

Jaime Garcia Diaz
Software Engineer
November 14, 2017

1
2
3
4
5
6
7
Javascript is one of the most popular programming languages.
It's flexibility has impacted the way the web is being built.
Let'
s build a full-stack application with Javascript.
We'll touch on integrating with Docker, Mongo, Nextjs, Graphql, React and MaterialUI.
Recommended for anyone interested on Javascript and how it can be used on different web architecture tiers.
Demo:
https://github.com/jgarciadiaz/demo-events

See all Geekfest videos from Groupon and our friends.

Kannada Rajyotsava Celebrations in Bangalore

Kannada Rajyotsava Celebrations in Bangalore

Kannada Rajyotsava

Kannada Rajyotsava Celebrations at Groupon’s Bangalore Office

Karnataka Rajyotsava is celebrated on 1 November of every year. This was the day in 1956 when all the Kannada language-speaking regions of South India were merged to form the state of Karnataka.

Data Driven Chicago, Full Video Recap

Data Driven Chicago, Full Video Recap

ENGINEERING & MACHINE LEARNING

Data Driven Chicago has its second data industry event with Echo Global Logistics, Reverb, Sprout Social and Trunk Club

7 November 2017

Complete recording of Data Driven Chicago, November 2, 2017

Recorded at Groupon HQ in Chicago

If you missed it, two of Groupon’s own were joined by folks from Echo Global Logistics, Reverb, Sprout Social and Trunk Club to talk about the current challenges and innovations in the data-driven product  and machine-learning space. This event featured presentations by:

  • Ilhan Kolko (Echo)
  • Andrew Lisy and Laura Hamilton (Groupon)
  • Tyler Hanson (Reverb)
  • Mary Feigenbutz and Greg Reda (Sprout Social)
  • Laurie Skelly and Elizabeth Cleveland (Trunk Club)
  • Moderated by Alli Diedrick (Built In Chicago)

Data Driven Chicago (Second Edition)

Data Driven Chicago (Second Edition)

Data Driven Chicago

Ilhan Kolko (Echo)
Andrew Lisy and Laura Hamilton (Groupon)
Tyler Hanson (Reverb)
Mary Feigenbutz and Greg Reda (Sprout Social)
Laurie Skelly and Elizabeth Cleveland (Trunk Club)
Moderated by Alli Diedrick (Built In Chicago)
November 2, 2017

Showcasing some of the great data-driven and machine-learning talent here in Chicago. Brought to your by Groupon, hosted by Built In Chicago.

See more videos from Groupon and our friends.

Groupon’s project managers are ‘the air traffic controllers of the engineering department’

Groupon’s project managers are ‘the air traffic controllers of the engineering department’

PROJECT MANAGEMENT & ENGINEERING

Groupon’s project managers are ‘the air traffic controllers of the engineering department’

30 October 2017

The Groupon Project Management Team in Chicago

Photograph by Chris Murphy

It was 4 a.m. when confirmation finally came in from the engineers: Groupon had successfully completed the online integration of LivingSocial, which the company acquired in 2016.

Groupon’s project management team had been toiling for months, working with engineers and key stakeholders on this large-scale project. With about 50 other staffers from different departments, they waited and watched into the wee hours, some via conference, until the “cutover” was finally confirmed. This means that users could now access the integrated content — and two weeks ahead of schedule.

That’s when an engineer came in with a bottle of Cristal and a “cutover” cannoli cake.

“Besides my wedding cake, that was the best cake I’ve ever had,” said Karen Hyatt, technical project manager. “There was something about being bleary-eyed at 4 a.m., eating cutover cake with the team you worked so hard with over six months that was really just special.”

Read the complete article at Built In Chicago.

Leadership Habits

Leadership Habits

About the Leader Habits

These seven Leader Habits were finalized in 2017 by combining the data we collected from ~600 managers (representing a true sample of our global and cross-functional leadership population) and defining what we see as priorities based on Groupon’s long-term strategy.

Remove roadblocks and make connections for others; know when to roll up your sleeves and be a doer
At Groupon, we have a laser focus on execution. Great leaders not only supervise from above but also roll up their sleeves and tackle the work head on. As a leader, your job is to clear out distractions and provide the right resources so your team can hyper-focus on reaching goals.

Build a strategy and vision by getting the team to debate and commit; influence the right people to bring others with you
A culture of high performance starts with setting an effective strategy and vision. We expect our leaders to challenge themselves and their teams to figure out what really matters, make tough choices about priorities, and deliver on a strategy that inspires their team to row together in the same direction.

Be positive, calm, and flex with change; learn as you go and evolve with the business to make things better
Change is a constant reality and we need to continuously evolve to do what is best for our customers and business. It’s not enough to simply adjust to change – leaders must model resilience and continuous learning so that Groupon can strive above and beyond the competition.

Prioritize goals for impact; distill complex issues into crisp messages; take a practical approach to messy problems
This Habit is all about bringing focus and practicality to what we do and how we do it and getting rid of unnecessary complexity. As a leader, you should be identifying things that drag down our ability to move fast and proactively work to improve them.

Drop the ego; be open to feedback and willing to change your direction; let the best idea win
A “my way or the highway” approach won’t get you far. Instead, being authentic and real is what earns trust and helps build relationships with others. Be open about your strengths and own your weaknesses.

Trust your gut but know the facts and figures cold; be more than two questions deep; test smartly, learn quickly
We expect our leaders to understand the nitty-gritty of what their employees are doing on an operational level. If not, we create the risk of getting off track and moving in the wrong direction. Leaders have to make tough decisions on a regular basis. And, to make good decisions, it is essential to have the right data, details, and numbers at your fingertips.

Lead by example; listen, encourage, and deliver the tough messages; give direct, frequent feedback and develop others
Coaching is one of the most critical skills for a manager at any level. A great leader will increase not only quantity but also quality of conversations with peers and direct reports. This will build trust and increase productivity and retention of the employees on your team.

Groupon is a massive data-driven experiment — this team helps run it

Groupon is a massive data-driven experiment — this team helps run it

DATA SCIENCE & ENGINEERING

29 August 2017

Groupon is a massive data-driven experiment — this team helps run it

Groupon has tweaked and tested every corner of its e-commerce platform to find out precisely what makes customers click. Its platform is one of the world’s most optimized online destinations, but Groupon is still running daily experiments to add new features that increase business — and get rid of features that don’t.

Comprised of both data scientists and engineers, the Optimize team has built a tech platform for running those experiments with scientific rigor. We spoke with three team members about their efforts to reshape how Groupon thinks about data.

Read the complete article at BuiltInChicago