Experimentation
Culture Awards
Globally awarding your growth in experimentation culture
Sharing inspirational stories by the nominees & winnersThe 2024 live broadcast recording is now available
THE 2024 AWARD NOMINEES
2024 Rising Star Award nominees (jury voting)
2024 Team Award nominees (jury voting)
2024 Organization-wide Award nominees (jury voting)
2024 Community Award nominees (community voting)
The case submit deadline for the 2024 experimentation culture awards was March 31st 2024. We received 55 case submissions.The 2024 jury went through all the jury cases and rated each of them on a 1 to 5 scale (1= should not be nominated, 5 = should be nominated). Jury members were not allowed to vote on cases by friends, family, colleagues or clients. The top 5 in each category (based on average score) became the finalists for the awards.Cases that did not end up in the top 5, but scored a 3.5 on average without a single jury member scoring below 3 have received an honorable mention (finalists also need this score).All the jury finalists were interviewed through a video call. Based on these interviews, the jury selected a winner in each category.The judges did not vote for the community cases. The results are derived from the voting by the full #EXPCA community.Looking for the 2023 nominees & winners? Find them here!
RISING STAR NOMINEES
For organizations (teams) that grew from almost no experimentation, to scaling up experimentation quickly.
Apoteket (SE) - For the Product division: "Cultivating confidence: the journey to experimentation success".
IU (DE) - For the CRO team: "Empowering growth: the experimentation journey at IU".
Rising Star award Winner: NS (NL) - For the CRO center of excellence: "From almost nothing to experimentation by 9 product teams on web and app".
Serko (NZ) - For the Product delivery team: "From zero to hero, experimentation culture now at the heart of the growth journey".
Telenet (BE) - For the OptiMice team: "Unleashing the powerful beast of experimentation".
Click on the organization name to read the case description
See the 2023 rising star nominees & winner
May 23rd 2024 recording
TEAM NOMINEES
For organizations (teams) that grew from some people applying some CRO / Experimentation to a more solid set-up, and a strong team.
Team award Winner: A1 Telekom Austria (AT) - For the Experimentation team: "From obligation to empowerment: the birth of experimentation culture".
Comcast Cable (US) - For Digital experience & platforms: "Driving evidence-based decision making one test at a time".
Enchanting Travels (US) - For the Web team: "From zero to data dynamo: our journey to precision testing and streamlined processes".
Pon Automotive (NL) - For the Digital marketing automation team: "Super Tuesday".
TomTom GO Navigation App (NL) - For the D2C team: "The Battle of Transformation: From Web to App Testing".
Honorable mentions for:
Butlin's (UK) - For the CRO team: "Nurturing a test & learn culture".
Intuit Quickbooks (US) - For the Online ecosystem team: "Ecosystem experimentation process evolution".
Click on the organization name to read the case description
See the 2023 team award nominees & winner
May 23rd 2024 live broadcast
ORGANIZATION-WIDE NOMINEES
For organizations that grew from a strong experimentation team to experimentation more broadly adopted throughout the organization.
ANWB (NL) - For solving scaling issues with their integrated serverside experimentation platform.
Carwow (UK) - For their experimentation culture transformation and cutting test alignment meetings in half.
Nerdwallet (US) - For enabling true democratization: any employee can launch an experiment.
Organization-wide award Winner: Oda (NO) - For cultivating an experimentation culture to full bloom.
Sunweb Group (NL) - For propelling experimentation efforts within the organization: from processes, output to outcomes.
Honorable mentions for:
Boats Group (US) - Teams no longer test in isolation.
NatWest Retail (UK) - For reframing and evolving experimentation.
Click on the organization name to read the case description
See the 2023 organization-wide award nominees & winner
May 23rd 2024 live broadcast
COMMUNITY AWARD
The Community Award 2024, sponsored by Convert.com, is for an individual, a team, or an organization who really helps the broader experimentation community move forward.
Our community was able to submit cases (without self-nomination). After the 2024 live broadcast, all attendees could vote on the community award cases of which the case owners had accepted their nomination.
Community award Winner: Test & Learn Community - For offering a home to experimenters.
Honorable mentions for:
Conversion Ideas - For sharing valuable and affordable CRO courses.
Experimentation Jobs - For growing experimentation careers.
May 23rd 2024 live broadcast
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
THE 2023 AWARD NOMINEES
The case submit deadline for the 2023 experimentation culture awards was April 25th 2023. We received 69 case submissions.The 2023 jury went through all the cases and rated each of them on a 1 to 5 scale (1= should not be nominated, 5 = should be nominated). Jury members were not allowed to vote on cases by friends, family, colleagues or clients. The top 5 in each category (based on average score) became the finalists for the awards.Cases that did not end up in the top 5, but scored a 3.5 on average without a single jury member scoring below 3 have received an honorable mention (finalists also need this score).All the finalists will be interviewed through a video call. Based on these interviews, the jury will select a winner in each category.After we stopped with the individual category (experimentation is a team sport, was our reasoning, and we created a separate community award category), we added the Rising Star Award as a full category this year. The community award has no nominee overview; we will announce the winner in the live show.Looking for the 2022 nominees & winners? Find them here!
RISING STAR NOMINEES
For organizations (teams) that grew from almost no experimentation, to scaling up experimentation quickly.
Winner of the 2023 Rising Star award: Bupa Australia (AU) - For the Tiger Squad: "Proving the value of business experimentation through one product squad."
EDC Retail (NL) - For the CRO Team: "Lighting the experimentation fire in the boardroom".
Moneywise (CA) - For the Experiment Lab: "Finding leverage with an ambassador program structure for experimentation".
Gelato (NO) - For the Data Team: "Turning the tide to experimentation - challenging the status quo".
Gympass (BR) - For the Center of Excellence: "From zero to two in 6 months: scaling experimentation across marketing channels".
Honorable mention for:
Perlego (UK) - For the CRO Strategists: "How Perlego are trusting their growth to experimentation".
TEAM NOMINEES
For organizations (teams) that grew from some people applying some CRO / Experimentation to a more solid set-up, and a strong team.
DPG Media (NL) - For the Circulation Team: "From a CRO silo to experimentation in product teams".
ACLU (US) - For the Digital, Tech, and Analytics Team: "Scaling Experimentation at ACLU".
Heineken (NL) - For the E-business Team: "Embedding digital growth with their experimentation ecosystem".
Winner of the 2023 Team award: Tele2 Sweden (SE) - For the Experimentation Core Teams: "From a disengaged team to a fully focused machine".
ANWB (NL) - For the Web Analytics & CRO Team: "Facilitate central development of optimization through a center of excellence".
Honorable mentions for:
PVH (US) - For the CXO Team: "Breaking silos through journey-based optimization".
Smartphonehoesjes (NL) - For the CXO Team: "From CRO silo to CXO team".
ORGANIZATION-WIDE NOMINEES
For organizations that grew from a strong experimentation team to experimentation more broadly adopted throughout the organization.
Beter Bed (NL) - For how their experimentation program evolved from focusing on winners to gaining comprehensive insights.
Groover (FR) - For how they defined a testing framework to ensure alignment without compromising velocity.
Winner of the 2023 Organization-wide award: Drukwerkdeal (NL) - For how their CEO became their biggest fan.
Zillow (US) - For building the bridge between experimentation and business impact.
RTL Netherlands (NL) - For going from pushing experimentation in limited teams towards getting pulls from everywhere.
COMMUNITY AWARD
For an individual, a team, or an organization who really helps the broader experimentation community move forward. Note that the case owner can not submit these cases; they can only be nominated by someone else.
Winner of the 2023 Community Award:
Lucia van den Brink & Daphne Tideman (NL) - For starting the Female Speakers in Growth and Experimentation board and the Women in Experimentation community.
Honorable mention for:
Bhavik Patel (UK) - For running the CRAP Community with the Conversion Rate optimization, Analytics and Product meetups.
June 15th 2023 recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on June 15th, 6pm UTC
THE 2022 NOMINEES & WINNERS
The case submit deadline for the 2022 experimentation culture awards was May 21st 2022. We received 54 case submissions.The 2022 jury went through all the cases and rated each of them on a 1 to 5 scale (1= should not win, 5 = should win). Jury members were not allowed to vote on cases by friends, family, colleagues or clients. The top 5 in each category (based on average score) became the finalists for the awards.Cases that did not end up in the top 5, but scored a 3.5 on average without a single jury member scoring below 3 have received an honorable mention (finalists also need this score).All the finalists were interviewed through a video call. Based on these interviews the jury selected a winner in each category and also selected the winners for the community award and the rising star award.Looking for the 2021 nominees & winners? Find them here!
INDIVIDUAL NOMINEES
Amy Hartwig (US) - For growing experimentation at Microsoft DevDiv.
WINNER of the 2022 Community AWARD is: Tom van den Berg (NL) - For CRO weekly: a curated newsletter to help increase the quality of experimentation throughout the market.
Ana Catarina Cizilio (BR) - For empowering A/B-testing in Brazil.
WINNER of the 2022 Individual AWARD is: Bjarn Brunenberg (NL) - For experiment program spitting fire at TomTom.
Ellie Hughes (UK) - For becoming an experimentation evangelist at AND Digital
Honorable mention for:
Marianne Stjernvall (SE) - For sharing in-house lessons of building an experimentation culture.
TEAM NOMINEES
WINNER of the 2022 Team AWARD is: Vista (US) - For the Experimentation Hub: "Building a culture of experimentation in public".
WINNER of the 2022 Rising Star AWARD is: SMG (CH) - For the CRO Team: "Inform, inspire, involve, iterate. Revive & expand a culture of experimentation".
Videoland (NL) - For the Experimentation Team: "From 5.5 experiments per month to a center of excellence".
Accor (FR) - For the Experimentation Department: "Making experimentation great(ly known/used)".
ING (NL) - For the Global Conversion Centre of Excellence and Pricing team: "Connecting to experiment: the virtual conversion workshops lead to 77% more tests in 1 year".
Honorable mentions for:
DPG Media (NL) - For the CRO-team: "From CRO-team to centre of excellence".
Thinkific (CA) - For the Growth team: "Reducing barriers to experimentation".
ORGANIZATION-WIDE NOMINEES
Marks & Spencer (UK) - For culture, velocity & quality.
TL (CH) - For how do you energize the culture of experimentation in a 125 year old company?
WINNER of the 2022 Organization-wide AWARD is: ASOS (UK) - For embracing an experiment-oriented mindset.
Vattenfall Netherlands (NL) - For adopting data-driven customer experience optimization.
Beter Bed (NL) - For development accelerates experimentation.
July 7th 2022 live broadcast
This live broadcast had 120+ minutes of keynotes, interviews and winners
The live broadcast started on July 7th, 6pm UTC
THE 2021 NOMINEES & WINNERS
The nomination deadline for the 2021 experimentation culture awards was May 28th 2021. We received 62 case submissions.The 2021 jury went through all the cases and rated each of them on a 1 to 5 scale. They were not allowed to vote on cases by friends, family, colleagues or clients. The top 5 in each category (based on average score) became the finalists for the awards.All those finalists were interviewed. Based on these interviews the jury selected a winner in each category:- 2021 individual nominees & winner
- 2021 team nominees & winner
- 2021 organization-wide nominees & winnerIn 2020 we had 6 out of 15 finalists (case owners) being female, in 2021 this went down to 4 out of 15 finalists. We are concerned about this. We also believe we need to work harder on the full diversity perspective of our selected finalists / cases.This is why we are reconsidering the way we collect cases and judge these cases to have a more diverse representation of our industry as finalists of our 2022 awards.Looking for the 2020 nominees and winners? Find them here!
INDIVIDUAL NOMINEES
WINNER of the 2021 Individual AWARD is: Aleksander Fabijan (SI) - For it Takes a Flywheel to Fly: Leading Microsoft to share lessons in experimentation culture.
Alexandre Suon-Perhirin (FR) - For driving the adoption of A/B testing as a decision-making tool for strategic investments.
Peter Ernst (US) - For building a culture of experimentation in healthcare: find better ways to connect patients during COVID.
WINNER of the 2021 Community AWARD is: Rommil Santiago] (CA) - For launching Experiment Nation: a home for experimenters around the world, from beginner to pro.
Ruben de Boer (NL) - For building a worldwide CRO community with free coaching for charity and low barrier Udemy courses.
TEAM NOMINEES
WINNER of the 2021 Team AWARD is: bol.com (NL) - For Team Experimentation and the journey to make their colleagues smarter.
TOYOTA (FR) - For the CRO team. An eye on ROI: Executives and CRO experts nurture experimentation together.
ANWB (NL) - For the Web analytics & CRO-team and the CRO-training, onboarding and documentation of the CRO process.
iTech Media (UK) - For the Experimentation Team. Advocating their operational framework and increasing the velocity of experiments.
fonQ (NL) - For the CX Team. Rocket start CRO: from 0 to 30 A/B-tests in 6 months.
ORGANIZATION-WIDE NOMINEES
Rabobank (NL) - For the growth from ad-hoc to structural experimentation and optimization.
BAM - Bamboo Clothing (UK) - For the growth and journey to experimentation informing smarter decisions.
Convoy (US) - For enabling experimentation at scale in a B2B Marketplace.
WINNER of the 2021 Rising Star AWARD is: The Royal Mint] (UK) - For embedding a culture of experimentation at a company with over 1,100 years of history.
WINNER of the 2021 Organization-wide AWARD is: ING (NL) - For moving towards continuous experimentation.
July 8th 2021 live broadcast
This live broadcast will have 150 minutes of keynotes, interviews and winners
THE 2020 NOMINEES & WINNERS
The nomination deadline for the 2020 experimentation culture awards was on July 31st 2020. A total of 65 cases were submitted through our website.The 2020 jury went through all the cases and rated every single one of them on a 1 to 5 scale. The top 5 in each category became the final nominees for the awards.All those finalists have been interviewed. Based on these interviews the jury selected a winner in each category.Looking for the 2021 nominees? You can find them here.
INDIVIDUAL WINNER & NOMINEES
WINNER of the 2020 Individual AWARD is: Guido X Jansen (NL) - For evangelizing CRO in general and CRO culture in particular with the CRO Cafe Podcast.
Gabriela Denison (UK) - For building trust in fast, pain-free experimentation at ASOS.com.
Raoul Warren Doraiswamy (AU) - For building the CovidCRAP community. Supporting businesses affected by CoronaVirus for free.
Desiree van der Horst (NL) - For taking away the fear of sharing failures at Fingerspitz.
Stefan Thomke (US) - For writing "Experimentation Works", the book on experimentation for business leaders.
TEAM WINNER & NOMINEES
WINNER of the 2020 Team AWARD is: VodafoneZiggo (NL) - For the CRO Center of Excellence. Creating a culture of experimentation.
Taxfix (DE) - For the Experimentation taskforce. The path towards a strong experimentation culture.
Hallmark Cards Benelux (NL) - For the CRO team. Started from the bottom, now they are 100 tests further.
Microsoft (US) - For the Experimentation Platform Team. A/B Testing and Covid-19: Data-Driven decisions in times of uncertainty.
SAP (US) - For the Optimization Lab. Building a data-driven culture through widespread testing.
ORGANIZATION-WIDE WINNER & NOMINEES
WINNER of the 2020 Organization-wide AWARD is: Farfetch (PT) - For the circle path from insights to learnings. Change a company experimentation mindset in a data-driven approach.
GlaxoSmithKline (US) - For building a culture and practice of experimentation.
MongoDB (US) - For going from 0 to 100 within 6 months. Implementing and accelerating a research and testing program.
AllClear Travel Insurance (UK) - For the democratization of experimentation.
RS Components (UK) - For full stack experimentation transformation. The 'binding agent' between Product & Engineering.
LIFETIME ACHIEVEMENT AWARD
The 2020 jury has decided to hand out an INDIVIDUAL Experimentation Culture lifetime achievement AWARD
The WINNER is: Ronny Kohavi (USA) - For all the research and publications that helped so many experimenters to understand how to conduct trustworthy digital experiments.
LIFETIME ACHIEVEMENT AWARD
The 2020 jury has decided to hand out an ORGANIZATION-WIDE Experimentation Culture lifetime achievement AWARD
The WINNER is: Booking.com (NL) - For being a big example organization for experimentation, known and quoted by many decision makers throughout the world.
September 24th 2020 broadcast
The 2020 liveshow has 180 minutes of inspirational talks and interviews
COMMUNITY AWARD
HONORABLE MENTION:
Coversion Ideas
Conversion Ideas, founded by Ruben de Boer:
Making courses affordable.Submitter 1: "The Justin Timberlake of CRO""Ruben's goal is to give everyone a chance to learn CRO and move the whole community. He works so hard for this, in addition to his full-time job at Online Dialogue.With 4 Udemy courses that sell for ~13 euros (with Udemy keeping most of it) or for free and a free CRO Airtable course (that Airtable itself recommends to CRO specialists), he has over 15,000 students worldwide.Last year, over 5,000 new students joined, and 764000 minutes of videos were watched. He updated all courses with the latest theories and tools. Even better, everyone who lacks the budget and/or has no job can get free access to all courses.1500+ subscribers to his free CRO Tips newsletter receive two CRO/Experimentation tips every week and a free book. Ruben added free guides last year (The CRO Interview Prep Guide and Airtable automation guide), and more are coming.Furthermore, last year, he created a free Personal Branding & Online Presence course, a free CRO internship database (that failed unfortunately), partnered with vendors to give his students exclusive extended free trials to practice with the theories, appeared in numerous podcasts, shared his insights in 150+ LinkedIn posts (currently over 7600 followers) and answers questions daily to help people.Ruben deserves this award because you can see the pleasure on his face when he is helping others. He has and endless motivation to share his knowledge with everybody who needs it."Submitter 2: "My highest recommendation""I've had the pleasure of learning from and getting to know Ruben over the last several years. From delivering incredibly valuable (and highly affordable) Udemy courses to sharing insightful, practical content on LinkedIn, Ruben has clearly shown that he is a selfless contributor to the CRO/Experimentation community. He is also an A++ person!!"
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
COMMUNITY AWARD
HONORABLE MENTION:
Experimentation Jobs
Experimentation Jobs,
founded by Kevin Anderson and Tom van den Berg:
Growing experimentation careers."Experimentation Jobs is helping people to find their first or next role in the CRO/experimentation space. The selection of jobs is spot on. Tom and Kevin have been doing this for two years and have more than 1000 jobs on their platform.This initiative is helping experimenters and companies to find the right person for the job.Also the career interviews with people from the space helps people to find a path into the CRO/experimentation world or simply get inspired to look for another opportunity."
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
COMMUNITY AWARD WINNER:
Test & Learn Community
Test & Learn Community, founded by Kelly Wortham:
Offering a home to experimenters.Submitter 1: "Florence Nightingale of Experimentation":"Kelly has managed to create such a great forum to share, and learn from many experts. It can be quite daunting in the world of experimentation to ask questions which you may deem as fairly basic, or "I should know this" scenarios. Kelly created the "ask anything channel" which provides a safe place without fear of toxicity. Kelly (and the members in the forum) always take time out to help and guide other members, the sense of community is a very lovely thing. Great forum, great and very knowledgeable people, that are always kind and considerate in their responses. Democratization for all, without the fear of ridicule!"Submitter 2: "Test & Learn Community - A Place for Everyone""The TLC is about community, inclusion, and shared learning - the success of which requires approaching knowledge with a generosity of spirit. Ultimately community is about being of service to others. This is what the TLC is for and is why we are such strong supporters of it and Kelly.The TLC has grown over the last 10+ years into the nonprofit it is today with more than 1.400 members - but it has never left it's purpose of being a place where people can connect - whether in person or virtually - to teach, learn, and converse.Kelly as the Founder and President of the Board of the TLC is the driving force in creating the platform for this vast network of support for the community. Since 2012 she has been at the forefront of demanding a welcoming, and encouraging atmosphere for sharing and learning. The one thing that has always remained true to the original intention of the community - the TLC was designed as a place for people to teach, learn, and converse. And it remains so today. No sales or solicitation of members is ever allowed and all questions are welcome and encouraged. The environment is welcoming, a place where no one should ever feel worried that they'll get bullied for asking a ""dumb question"". That's not what the TLC is. Kelly ensures the space is safe and is adamant about the no bullying or derogatory language agreement, no matter what one’s expert level may be.The value of the TLC to ALL members of our community is why we have nominated Kelly Wortham"Submitter 3: "Community Leader""Kelly not only strives on creating a space for education about experimentation, she creates a community of like minded individuals where we can expand our education and further the understanding of others processes on experimentation. There’s not really any other community that can provide a friendly environment to talk stats and learn without getting egotistical."Submitter 4: "TLC is home for experimenters""Kelly has invested in the TLC consistently for over a decade and has made it THE destination for experimentation professionals. The last year has seen an explosion in the growth of the community (maybe 1,500 now?) as well expansion in the community offerings including a newsletter and a few events beyond the typical monthly panels. She has managed to make it a place where experimentation friends hangout, exchange banter, share interesting stuff, recruit, seek work, talk nerdy and welcome new faces. She actively promotes a safe environment where people can learn, where diverse voices can be heard, and yet no one is walking on egg shells. For me its an indispensable resource where I have access to some of the greatest minds in the field. Kelly orchestrates all of this pretty selflessly IMO. She deserves some widespread recognition of all these efforts. I know she's on this judge panel, but allow her to recuse herself just this once!"
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
ORGANIZATION-WIDE NOMINEE:
ANWB - Netherlands
Solving scaling issues by implementing an integrated serverside experimentation platform."After building CRO processes and culture and improving maturity, we started to run into scaling issues:• Our webanalytics & CRO team is responsible for experimentation and needed in every step of the process. We grew to a team of >25 people, but we can’t keep on growing forever.• There’s an increased demand top-down for measuring impact and making decisions data driven.• We got more and more disturbed by the client-side implementation of our experimentation platform.We combined this into one solution: a server-side experimentation platform, integrated with cms.• The server side implementation removes flickering. Beside the performance improvement, this is required in order to be able to run experiments on components like main navigation.• The integration with cms makes it easy to build experiments and thus makes experimentation accessible to a large group of (potential) users.• We moved ownership for building and monitoring experiment from webanalists to content marketers.• It’s a lot less error-sensitive.• We shortened the process in the building-phase.We use this project to boost experimentation by communicating about it in central communication points, through presentations in different chapters (Content, Product Owners, UX, Front-end) and giving several workshops.The coming year we’ll be working on automating processes. The new experimentation platform is the first step in our plan to scale from 120 to 850 experiments per year, our goal for 2026."
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
ORGANIZATION-WIDE NOMINEE:
Carwow - United Kingdom
Experimentation culture transformation and cutting test alignment meetings in half."Carwow has gone through an experimentation culture transformation! We went from a company feeling that AB tests slow down product innovation, to seeing how making data driven decisions ultimately means we can ship the right things faster.The analytics team decided that we need to do more testing, as a large portion of our time was spent on pre/post analysis for changes released without testing, which also resulted in low confidence decision making. To ensure we built lasting change, we invested in incentives and time saving processes that led our stakeholders to testing rather than shaming them for releasing without measurement in mind.• First, we launched an experimentation platform to make reading tests easy and fun, we hosted talks on experimentation, and loudly celebrated experimentation wins.• We also helped stakeholders understand in hindsight where testing could have been helpful, and even went back and tested some of those changes!• Finally, we introduced, iterated on, and agreed to a practical decision framework and tool called “When To Test”. This framework was approved by leadership, thereby saving time and meetings to decide what gets tested and what doesn’t.We now have less than half as many hours spent discussing testing plans! Slowly, our teams have become more comfortable and at ease with testing and have a clear understanding the of the value of moving with velocity (in the direction of a better product) and not just the speed of releasing changes."
May 23rd 2024 recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
ORGANIZATION-WIDE NOMINEE:
Nerdwallet - United States
Enabling true democratization: any employee can launch an experiment."This organization started at 6 experiments per quarter company wide and then developed a strong experimentation culture through a staged approach to achieve a 1900% increase in velocity and hundreds of thousands in net incremental revenue per day.The addition of a specialised team that centralised experiments for paid performance marketing channels resulted in 5X velocity and large growth of marketing spend to soak up the extra ROAS generated from conversion rate increases.With that success, the effectiveness of experimentation was respected and the specialist team was graduated to a centre of excellence team which toured their best practices to the rest of the company even though corporate restructuring discouraged centralisation.But by continuing to provide superior and persistent spearheading, (and the ability for any employee in the company to launch an experiment) the team has increased experimentation coverage to 95% of their entire population of consumers and 125 experiments per quarter (as of Q1).Initially only a web team effort, we have now expanded to apps, marketing, ML models, etc. while maintaining quality control, strategic influence and best practices by building them as the default configurations of the mechanisms for experimentation."
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
ORGANIZATION-WIDE WINNER:
Oda - Norway
Cultivating an experimentation culture to full bloom."Over the past 12 months, Oda has seen a significant increase in cultural maturity surrounding experimentation and the growth of related practices. We went from a novice stage to intermediate with some parts of our experimentation program/culture stretching out on the advanced stage. Today, all new product development goes via gradual rollouts and A/B tests, with everyone in Product & Tech equipped to interpret test results.This transformation began with the adoption of an experimentation platform in early 2022, which proved to be cost-effective and user-friendly. Within 6 months, the incremental cost of running a new test approached 0. We further increased the robustness of the platform by enabling teams to run A/A tests and have recently implemented pre-experiment bias detection mechanisms, indicating a deeper level of maturity.By placing emphasis on documentation, templates, and guides, we enhanced teams' autonomy in testing, while bi-weekly community sessions facilitated knowledge sharing. The introduction of risk profiles increased the frequency of low-risk tests, and introducing guardrail metrics in assessing success of tests helped align experimentation with Oda's business strategy.Lastly, on organization, our hybrid setup for A/B testing, consisting of a Centre of Excellence alongside autonomous cross-functional teams, has empowered Product Managers to prioritize evidence-guided experimentation as a means of learning and optimizing our product."
May 23rd 2024 live recording
This broadcast has 120+ minutes of keynotes, interviews and winners
The live broadcast started on May 23rd, 6 pm UTC
ORGANIZATION-WIDE NOMINEE:
Sunweb Group - Netherlands
Propelling experimentation efforts within the organization:
from processes, output to outcomes."In November 2023, Sunweb restructured, forming Team Conversion with customer-facing Product Owners, Product/UX designers, and CRO specialists, reshaping our approach to experimentation.Within three months, we surpassed 2023 and 2022's experimentation velocity (currently +60), affirming our target of +200 experiments this year with a +25% win rate and +70% conclusive rate.Conclusiveness, a key metric, propels further iterations despite outcomes. Additionally, we aim for a 20% CVR uplift EOY, tracked through revised revenue models considering effect over time and factors like false discovery rate.To meet our targets, we host bi-weekly hypotheses workshops using qualitative and quantitative analyses ("Product boost"), engage in bi-weekly user research, and hold monthly deep dives with our customer care team.Each product team communicates with at least one customer weekly, using opportunity solution trees. To maintain velocity, we've implemented a dual-track experiment pipeline—product teams handle development experiments, while the CRO guild conducts low or no-code experiments.Experimentation is now intrinsic to Sunweb's success, supported by transparency and accountability through automation in our improved Airtable setup, facilitating company-wide progress sharing.With dedication, collaboration, and a strong belief in experime