Data-Driven Product Management at Groupon

Data-Driven Product Management at Groupon

PRODUCT MANAGEMENT

May 2018

Data-Driven Product Management at Groupon

Laura Hamilton

Group Product Manager

Laura heads up product for consumer web, international, and LivingSocial at Groupon. She has a bachelor’s in mathematics from the University of Chicago and a master’s in computer science with specialization in machine learning from Georgia Tech. She has more than 10 years of experience in ecommerce product management at four Chicago tech companies, from early stage startups to publicly traded global companies. She is passionate about using analytics and machine learning to create a delightful customer experience.

At Groupon, we have a very data-driven philosophy of product management. In this blog post, I’ll talk through how we approach product ownership in a data-driven way, from financial forecasting to roadmapping to feature development to experimentation.

Financial Forecasting and Roadmap Creation

For every candidate feature, we calculate the projected financial upside according to the following formula:

feature_revenue_forecast = expected_lift x platform_factor x success_probability x platform_revenue

Where:

  • feature_revenue_forecast is what we are trying to calculate (the expected revenue from the feature)
  • expected_lift is the increase in conversions we expect from users in the treatment group vs. users in the control group. The vast majority of experiments fall between -1.0% and +1.0% lift.
  • platform_factor is what percent of all users of the platform (whether iOS, android, mobile web, or desktop web) are part of the experiment. For example, for a test on the checkout page on mobile web, the platform factor will be 100%—100% of users who place an order on mobile web visit the checkout page during their journey. For a test on the getaways deal page, the platform factor will be much smaller, so the overall financial impact of the test will be smaller.
  • success_probability is a haircut we apply to take into account that not all experiments will succeed. In fact, only about 30% of our experiments are successful. The success_probability for a given feature could be greater than or lower than 30%, depending on how confident we are that the experiment will succeed. For some experiments that are primarily for strategic reasons, such as maps improvements, we will use a success_probability of 90% or 100%.
  • platform_revenue is the total revenue generated by the platform. For example, the platform_revenue for iOS is the total revenue from orders placed via the iOS app.

With this formula, we have a consistent and data-driven way to estimate the upside from each proposed initiative. Then, once each experiment concludes, we compare our estimates to the actual results, and over time we refine our estimations.

We use these estimated upside figures to create product roadmaps. In order to prioritize initiatives and determine the cutlist, we need to introduce another data point—the engineering effort required. Then, we use the following formula to calculate the ROI of each feature:

ROI = feature_revenue_forecast / level_of_effort

We then stack rank features according to their ROI.

ROI is an input into the creation of the product roadmap and the determination of the cutlist, but it is not the only input. I always like to ensure that there is a healthy amount of time spent on engineering excellence (site stability, paying down technical debt, reducing latency, library upgrades, increased test coverage, improved tooling). I also like to ensure that we have a customer focus. Many of our features come directly from customer feedback via focus groups, quantitative surveys, and app store feedback; the Wishlist feature was one of these. I also reserve a healthy amount of time for strategic initiatives that may not provide lift in the short term but that set us up for success in terms of the Groupon 2020 vision.

Data-Driven Features

At Groupon we are lucky to have vast amounts of data that we can use to deliver a delightful product to our customers. We have worked with one million merchants to date; we have pumped more than $18 billion into local businesses; we have more than 1 billion Groupons sold; our app has been downloaded 171 million times, and we have saved customers more than $28 billion.

The Groupon platform handles tens of billions of user actions per month, and for machine learning algorithms that drive core product features our platform needs to make decisions (such as which deal to show the user next) in fractions of a second.

Developing product features that take advantage of these vast amounts of data in a performant way is an interesting challenge.

We use machine learning algorithms in a variety of ways to develop products here at Groupon:

  • Supply intelligence – There are millions of merchants we could call at any time to get onto our platform; how do we pick the best ones? 
  • Fraud prevention – Fighting the bad guys in realtime.
  • Discovery and personalization – Selecting which deals to show a given user in her mobile app deal feed.
  • Image recognition – Identifying the best user-generated images with neural networks.
  • Logistics – Getting ahead of the order rush by sending extra inventory to the right warehouse in advance of high demand.
  • Customer support – AI-based chatbots to respond to and resolve customer issues instantaneously.

To make developing data-driven products faster, we built a generic, extensible machine learning platform at Groupon called Flux. Flux is the “Rosetta Stone” between data scientists and engineers.

Data scientists work primarily in R. Flux models are written in Java and Clojure for stability and speed. Python is the glue that connects R and Java. It all runs on Groupon’s large Hadoop cluster.

To make the process for productionalizing machine learnings more robust, Groupon has an ETL management platform called Quantum Engineered Data (QED). QED reads from any source, and includes built-in data cleaning, error correction, and anomaly detection. Clean data is preserved and made available as a “feature catalog.” QED handles failures smartly, supporting falling back to yesterday’s model when appropriate. QED is able to plug into any source of truth—including streams, warehouse tables, and JSON endpoints.

QED gives us a lot more confidence in the robustness of our models. In general, subtle changes to a single data field can seriously impact model performance, and nuances in the data set could look fine to tests but fail in the real world. 

 

Interested in joining Product?

Monitoring

This blog post would be incomplete without a brief discussion of Groupon’s monitoring tools. We have a healthy suite of realtime alerts on product and engineering KPIs. We use splunk for logging and wavefront for graphing. Each service is staffed with a 24/7 on-call schedule, with escalation handled by pagerduty.

Additionally, each product area and business are has an Amazon-style Weekly Business Review, where we look at metric trends longitudinally, identify areas of change or concern, and begin deep dives where appropriate.

The data warehouse uses Teradata and Apache Hive.

Experimentation

There are 100 teams at Groupon that run experiments. At any given time, around 200 experiments are being run simultaneously on the Groupon platform.

Groupon has a dedicated team called Optimize that built a bespoke tech platform for running product experiments with mathematical rigor. The experimentation platform is called Finch Express. Finch Express is built with Ruby on Rails, Node.js, Ember.js, Python, R, and Hadoop/Hive. The team has filed three patents for its innovations on product experimentation.

Essentially, Finch Express uses a technique called Group Sequential Analysis, first developed by Abraham Wald in 1945. Group Sequential Analysis has been used extensively in high-risk clinical trials, such as heart valve studies, where it’s possible that one treatment is actually harming the patients. Ethically, we would want to stop a harmful clinical trial immediately—but statistically, checking the experiment results mid-run or “peeking” will vastly increase your rate of false positives and invalidate your statistical results.

Group Sequential Analysis provides a controlled, statistically rigorous way to “peek” at experiment results at set points during the experiment run. This allows Groupon to end an experiment early if it is losing money, and to roll out an experiment early if it is deemed an early winner (capturing more upside).

Finch Express does all of this automatically. Product managers create the experiment in Finch Express, add a description and screenshots (to save the details for future product managers to reference), and launch the experiment at 50/50. Finch Express does the heavy lifting of dynamically determining the appropriate lift sensitivity for the experiment (based on traffic and conversion rate), performing the Group Sequential Analysis calculations, deeming the experiment a “success,” “failure,” or “flat” (most experiments end flat), and even automatically rolling out or rolling back the experiment based on its results. Then, Finch Express reports on the financial results of the experiment. The experimentation platform prevents product managers from statistical no-nos, such as peeking, unbalanced bucketing, and concluding the experiment too early. As a result, our experimentation processes have a high degree of statistical rigor.

On average, Group Sequential Analysis allows us to conclude experiments an average of 57.53% earlier compared to simply running them to a single final checkpoint. This reduces the cost of failed experiments, hastens upside capture of successful experiments, and allows for much faster iteration and innovation.

To date, Groupon product managers have run a total of 2,500 experiments, thanks in large part to the proprietary and patent-pending experimentation platform.

Thanks for staying with me until the end! I hope this gives you an idea for how we use big data at global scale here at Groupon to create our product roadmaps, to innovate with new products and features, to monitor product performance, and to evaluate the impact of new initiatives. If you’re interested in learning more about data-driven product management opportunities at Groupon, have a look at our open roles.

Meet Gabi, our Senior Director of International Marketing Ops

Meet Gabi, our Senior Director of International Marketing Ops

GABI STAFFORD

Senior Director International Marketing Ops
Dublin
Joined: 2014

Our Director of International Marketing Ops shares her love of Groupon’s open, diverse culture and how working here has lifted her game.

_____________________________

Tell us about your Groupon story so far.

I have been at Groupon for three and a half years and I’m on my 4th job and my 5th boss! One thing you can say about Groupon is that it is never, ever boring. I love the fast-paced, results-based environment. I have learned more than I could ever have imagined at my time here and working at Groupon has truly lifted my game. When I started in 2014 I was Head of International Email and now three and half years later I am Senior Director of International Marketing Operations, a role with responsibility for all online international marketing channels. It’s been an exciting and wild ride.

 

What’s the secret to succeeding at Groupon? What type of person tends to excel here?

Success at Groupon is based on results: you need to be passionate, results-focused and data-driven to excel here. Creativity and an entrepreneurial bent are also definitely beneficial. You should like being constantly challenged and thrive on problem solving. All of that being said, respect, integrity and inclusion are core Groupon values so you should also enjoy working with lots of truly individual and awesome people.

 

What is Groupon like as a workplace for women? Does it feel like a diverse, inclusive space?

My personal experience of Groupon is definitely that it is diverse and inclusive. As a mother, the flexibility and family first focus that Groupon offers is something that I haven’t experienced in other companies. My daughters are particularly fond of the soft drinks and chocolate on tap when they come in to play table tennis or foosball. We are very lucky, particularly in International to have a significant number of female leaders. Our International President, CMO and HR Leader are all women. At Groupon the sky is literally the limit!

 

“There is real power in coming together as a group not only to support one another but also to harness our skills and talent to positively impact the representation of women in STEM, particularly in Ireland.”

How significant is it that we are launching Women@Groupon?

Women@Groupon is very close to my heart. I work with a lot of incredibly talented, passionate women and Women@Groupon is one way to celebrate and nurture that talent here in Dublin while also having a few laughs: another important part of life here! There is real power in coming together as a group not only to support one another but also to harness our skills and talent to positively impact the representation of women in STEM, particularly in Ireland. Dublin Women@Groupon is all about having an eye on the future.

 

What does your mother think you do?

I think she pictures me on the starship Enterprise constantly talking on my state of the art video phone. She loves the fact that I work globally for a well-recognised brand, even if she’s a bit vague on the details!

Kathleen, EMEA Facilities Manager

Kathleen, EMEA Facilities Manager

KATHLEEN McRORY

Facilities Manager, EMEA
Dublin
Joined: 2016

In her own words, Kathleen is responsible for “everything you see, touch, taste or break from above the floor to below the ceiling!” So as EMEA Facilities Manager her days are busy to say the least! Read on to find out more about her Groupon story and why she loves her job.

______________________

I started here officially in November 2016 but back in 2014 when Groupon was based in Harcourt Road (in Dublin’s city centre) we had been a client in Regus where I was working, so it feels a bit fated that I found my way back to the green green grass of my Groupon home! When I was interviewing for roles in 2016 and I told my parents one of them was for Groupon my Dad almost fainted with excitement “The Groupon!!! I LOVE “THE” GROUPON!!” I think it’s the first company I worked for that my parents were familiar with and already loved so they knew I’d be in safe hands! They aren’t sure what it is I do but they tell people I run something in “the” Groupon so I guess that’s close enough!

I think understanding people and being willing to put the work in is key to succeeding here. If you visit our office you can tell that everyone is working and owning that work. I look after 140 people day-to-day and they are all individual and unique. The secret to success here is getting to know that you can’t group all people into one neat bundle. You have to take the time to understand them and their needs. To listen, try to be patient and never take yourself too seriously! Groupon is not like anywhere I have ever worked. You are encouraged to be yourself, to be unafraid to shine, to express your opinion and to be involved in shaping our future. We make it a matter of course to think outside the box: we don’t just talk about it, we do it.

“Job satisfaction in any company can dip and rise. The best part of working here is that when there is a dip, there is a troop of fierce, strong and motivating men and women in your corner ready to lift you up and help you adjust your perspective.”

Groupon for Women

As a woman working in a male-driven area of the business I have been so lucky to have a boss who champions me and respects my opinion. As a woman working in Groupon as a whole I am surrounded by amazing females who I am constantly learning from. We are so lucky to have women in senior management who lead by example every day and female colleagues who lift each other up without a second thought. 

My work is ever-changing and endlessly challenging in all the best ways: from nursing kittens who were born under a floor in one of my meeting rooms to reimagining ways to make each office even more welcoming than before! Working in different countries, understanding cultures and dynamics to make each place that bit better.  I have spent my short time at Groupon laughing (mostly at myself…) rushing from place to place, feeding the masses, growing and learning at a rapid pace which keeps me on my toes and always taking the time to enjoy it. I have never had a dull day or moment. I have never gone home and thought “I didn’t do my best today.” Job satisfaction in any company can dip and rise. The best part of working here is that when there is a dip, there is a troop of fierce, strong and motivating men and women in your corner ready to lift you up and help you adjust your perspective.

 

Meet Rachel, HRBP in Dublin

Meet Rachel, HRBP in Dublin

RACHEL KEPPEL

HR Business Partner
Dublin
Joined: 2017

Resilience, pragmatism, resolve and an ability to work well with cats. Rachel talks about some of the surprising secrets to success at Groupon.

_______________________________

Tell us about your Groupon story so far.

I have been in Groupon 10 months now and work as HR Business Partner in the Dublin office primarily supporting the Tech, Marketing and Coupons teams in EMEA.  Most of my teams are located in Dublin however I am also responsible for some of our teams in Poland, Netherlands, Berlin, Italy, France, and London.   

 

What’s the secret to succeeding at Groupon? 

In my opinion the key to success in Groupon is resilience, pragmatism and resolve.  Groupon moves fast so it is important that you can keep up with the pace while driving your agenda. On the flip side you need to have a focus on making things fun.  We believe in making our lives less boring which means we all have a responsibility to build a great place to work. You need to have a sense of humor, and a commitment to excitement.              

 

What is Groupon like for women? Does it feel like a diverse, inclusive space to work?

I truly feel that Groupon is a great place to work for women.  Achieving a high number of women in leadership positions is a challenge that many firms face.  However within the Groupon office in Dublin, and indeed the wider business, we are very fortunate to have a strong female leadership team.  These inspirational leaders not only pave the way for our female employees but also instil a culture of inclusion within the entire office. We celebrate difference and know that having a diverse workforce changes the way we work and leads to greater success. As an employer Groupon has a lot to offer women.  

“We celebrate difference and know that having a diverse workforce changes the way we work and leads to greater success.”

How significant is it that we are launching Women@Groupon?

This group will not only offer a fantastic resource to empower our staff but also provide an avenue to encourage more women to consider a career in Tech, specifically at Groupon. By making employee stories and profiles such as mine available to external audiences, people can get an authentic sense of what Groupon has to offer as an employer.  We have also committed to making connections with local schools and show kids what a future in Tech has to offer.

These initiatives are owned by our employees and we all see the significance and are energised to make a difference.  We want to add value not only to the Groupon community but the community as a whole.

 

What excites you about your work? What’s on the horizon that keeps you engaged and interested?

I am excited to learn more about the business, ensure that our people strategy supports our success and ultimately continue to make Groupon a great and fun place to work.

What’s the strangest thing you’ve seen in the office?

This one is more unique than strange … Within my first week of Groupon it was discovered that a cat had given birth in one of our meeting rooms.  Our lovely office Manager Kathleen stepped up and hand-reared the cats for several weeks. When they were ready to be adopted they were given a loving home with one of our Managers, Seamus!   

 

What does your mother think you do?

My mother thinks I interview people all day even though I haven’t worked in recruitment for at 5 years.     

 

Meet Sarah from our EMEA Recruitment Team

Meet Sarah from our EMEA Recruitment Team

SARAH O’NEILL

 

Recruiter for Netherlands & Belgium
Based in Dublin
Joined: 2015

 

Entrepreneurial spirit and a killer aim when throwing custard pies: some of the skills that help Sarah succeed at Groupon.

Tell us about your Groupon story so far.

I joined Groupon 2.5 years ago as a Talent Sourcer for the Corporate side of the business. My role quickly progressed to a full cycle Recruiter giving me the chance to build strong relationships with other TA team members as well as hiring managers and work on loads of different types of roles for our business across EMEA. Basically, whatever required an extra pair of hands or was of urgency, I was happy to jump on board with!

In the past year, I moved to the Netherlands and Belgium markets (though I’m based in Dublin) where I lead all recruiting as their dedicated Recruiter and Talent Partner. It’s a challenging role at times considering the language and location barriers but it works well and my hiring team are a blast to work with!

What’s the secret to succeeding at Groupon? 

I would say things like Tenacity, Resilience and Grit are qualities that help people excel at Groupon and someone with an entrepreneurial spirit tends to do well here. I’d also say, don’t take yourself too seriously!

What is Groupon for women? Does it feel like a diverse, inclusive workspace?

In my experience Groupon is a great employer for women: there is a genuine, human side to the Managers and Leadership Team who clearly understand that sometimes life gets in the way of the standard 9-6 working hours. Groupon’s International Business is led by many strong women and they help shape our inclusive culture. During my time here I have met and worked with many women who have shaped me professionally and who I will call upon as mentors for the rest of my career – you know who you are!

 

There is a genuine, human side to the Managers and Leadership Team who clearly understand that sometimes life gets in the way of the standard 9-6 working hours.

 

Last year Sarah joined the team in Warsaw for a couple of months to assist on Customer Service hiring.

How significant is it that we are launching Women@Groupon?

The launch of Women@Groupon is hugely significant for our current and future employees: to have our male and female colleagues advocate for women in the workplace, for parity and for empowerment is truly inspiring to say the least. Makes it hard to believe that only 100 years ago we couldn’t even vote, how times have changed!

 

What’s on the horizon at work that keeps you engaged and interested?

I’m driven by relationships and people, I get really excited about building teams of quality people and seeing them flourish and I also love seeing my Hiring Managers get really excited about their new starters! Groupon is always moving, changing and adapting which keeps me engaged and excited and pushes me to constantly strive for improvement. My hiring managers and stakeholders in NL/BE are like another team that I truly feel connected to: I have their best interests at heart and want the best for their teams, so that keeps me on my toes too.

 

What’s the strangest thing you’ve seen in the office?

I think I have become accustomed to the weird and wonderful things that happen in the big green “World of Groupon” but a stand-out memory would be from when I first joined and took part in a charity contest – right in the middle of the canteen – where we basically ‘pie’d’ a group of people, one of which was my manager at the time. I came from a pretty corporate background so that was crazy to me, it was for charity though so all in the name of a good cause!

 

In a little over 2 years Groupon has as brought me to work in Poland, London, Dublin and Amsterdam and now with Belgium in the mix too, the exposure to international recruitment and the opportunities it presents are endless: the sky is the limit and the people are amazing!

 

Meet Artur: kickboxing and fantasy novels and hitchhiking… all in a day for our CS agent in Katowice!

Meet Artur: kickboxing and fantasy novels and hitchhiking… all in a day for our CS agent in Katowice!

ARTUR

CUSTOMER SERVICE
KATOWICE
Joined: 2017

You are one of the very first people who started within the Katowice bunch. How has your time at Groupon been so far?
It has been great! While working in Groupon, I’ve met interesting people from all over the word, e.g. India or US. Time spent with them was one of the best experiences since I started my job.

Tell us a little bit about yourself!
I’m 20 year old IB (International Baccalaureate) student from Poland. I like reading fantasy, watching movies, training kick-boxing… my greatest dream is to hitchhike through all continents!

And what made you decide to apply for a job @Groupon?
Lack of dress code (slippers are allowed!), lots of benefits and a brand new office in the heart of Katowice.

Is there anything that surprised you @Groupon?
Absolutely brilliant chillroom!

 

Groupon in Katowice? Nice people, cool office and great benefits

 

What is your job about?
My job is about giving customers answers to their questions, concerns… basically helping them and often making their life a bit easier 😉

What is the funniest conversation you’ve had with a customer so far?
So, there was a customer, who besides asking me about Groupon business, also asked questions about me, my dreams, age, homeland, if I want to travel… we can always expect interesting conversations here!

It’s more than 100 people in the Katowice office now. What do you think about how Groupon is already affecting the local job market?
I think Groupon made a great opportunity for a lot of people to gain experience and to allow them to work & study at the same time.

Tell us, what is your superpower?
I can distract other colleagues from their tasks. Works every time!

What do you like the most about @Groupon so far?
Lots of breaks during which I can compete with friends in Just Dance 😀

Do you have any advice for people thinking about joining Groupon?
Just give it a go and see for yourself!

What about some holiday?
Going to hitchhike through Northern Poland soon!

 

Meet Kamil – Customer Service impresario and budding film director

Meet Kamil – Customer Service impresario and budding film director

Kamil Zabiegała

Customer Service Representative
Katowice
Joined: 2017

Kamil joined our Customer Service team in Katowice in 2017 and has been soaking up the Groupon culture ever since. We never knew he was such a movie buff either!

You were one of the first Customer Service agents to join our Katowice team, so how has your time been so far?
It’s amazing, I have to say I love the people, I love the challenges, love the mood and the way we work. Not mentioning the office space. It definitely is not boring, there is always someone around the corner to talk with, it’s electrifying.

Tell us something about yourself we don’t yet know!
I’m just a guy like any other – head full of ideas and creativity. I love the cinema – the director Christopher Nolan in particular – and my dream is to be a movie director someday too. Beside that, my hobbies are computer graphics, photography and creative writing.

So what made you decide to apply for a job at Groupon?
Corn flakes and fresh fruit for breakfast of course. But seriously – working in a big company is a really engaging experience, chatting and talking to people is a cool thing, and when I saw some photos of Groupon offices on the web, it amazed me how great it must be to work with so many people. Oh, and I love Katowice 😀

I love the people and the atmosphere in the office: even if we’re working on something serious, we’re always able to smile.

Is there anything that surprised you about Groupon?
Certainly the positive mood and generally high energy levels of people around here.

So what do you do here?
Help people! It sounds simple but it’s often more complicated. We use chat and email to respond to our customer’s queries whenever they have problems enjoying their purchase on Groupon website.

What do you like the most about @Groupon so far?
People and the fun we have in the office, even if we are working on something serious, we can do it in a cool way! I’m totally buying it!

Do you have any advice for people thinking about joining Groupon?
If you like making friends and facing challenges, don’t wait until tomorrow, just apply!

Meet Cillian – secondary school student, rugby fan and future marketing & tech wiz!

Meet Cillian – secondary school student, rugby fan and future marketing & tech wiz!

Cillian Moore

Transition Year Student
Balbriggan, Dublin, Ireland
16 Years Old

This week in our Dublin office we had the pleasure of Cillian’s company when he took some time to stop by for work experience and learn all about marketing for his secondary school (high school) tuition. He attends Ardgillan Community College in Balbriggan, North Dublin and, just like us, he loves getting involved in the community. Oh and he’s a major rugby fan and plays for his local team too!

My experience this week has shown me the wide range of technology and engineering jobs that are available nowadays, so that’s what I’d like to do someday.

Tell us what you learned about Groupon this week?

“It was interesting seeing all of the different people involved in marketing at Groupon. I never knew the impact of advertising and promotion in a company, so it was a real eye-opener for me to see all the types of functions needed to make a company like Groupon run. I also loved how the teams could see impact of their work at the end of the week.”

What do you hope to study at university?

“I’d like to get my Masters, maybe in Computer Science. My experience this week has shown me the wide range of technology and engineering jobs that are available, so that’s what I’d like to do someday.”

MOST importantly – did you enjoy yourself here this week?

“YES! The people are very friendly and the atmosphere is great. Everyone I worked with from the design team to the affiliates and marketing and recruiting teams was very helpful and happy to answer my questions and provide advice on my future career.  I moved around various teams during the week and the people were so welcoming and made me feel right at home. The atmosphere in here is relaxed and fun, everyone helps each other out and there is a creative vibe, which is really nice to be around”

We absolutely enjoyed having Cillian’s around – and being so enthusiastic about technology and engineering, we’ll be keeping a close eye on him for future roles here!

Hope to see you again soon Cillian.

#TeamGroupon