Experimentation
Culture Awards

Broadcasted live on September 24th 2020
A virtual event with 6 keynotes, 15 nominees & 3 awards

See the detailed time schedule of the September 24th show


THE 2020 NOMINEES & WINNERS

The deadline for nominating yourself or someone else for the 2020 experimentation culture awards was on July 31st 2020. A total of 65 cases were submitted through our website.

The 2020 jury went through all the (non-duplicate) cases and rated every single one of them on a 1 to 5 scale. The top 5 in each category became the final nominees for the awards.

All finalists have been interviewed. Based on these interviews the jury selected a winner in each category. The winners were announced in the live show of September 24th 2020.

The 2020 liveshow has 180 minutes of inspirational talks and interviews

INDIVIDUAL WINNER & NOMINEES

Experimentation Culture Awards 2020 Individual award nominations
  • WINNER of the 2020 Individual AWARD is: Guido X Jansen (NL) - For evangelizing CRO in general and CRO culture in particular with the CRO Cafe Podcast.

  • Gabriela Denison (UK) - For building trust in fast, pain-free experimentation at ASOS.com.

  • Raoul Warren Doraiswamy (AU) - For building the CovidCRAP community. Supporting businesses affected by CoronaVirus for free.

  • Desiree van der Horst (NL) - For taking away the fear of sharing failures at Fingerspitz.

  • Stefan Thomke (US) - For writing "Experimentation Works", the book on experimentation for business leaders.

Click on the nominee to read more about the submitted case

TEAM WINNER & NOMINEES

Experimentation Culture Awards 2020 Team award nominations
  • WINNER of the 2020 Team AWARD is: VodafoneZiggo (NL) - For the CRO Center of Excellence. Creating a culture of experimentation.

  • Taxfix (DE) - For the Experimentation taskforce. The path towards a strong experimentation culture.

  • Hallmark Cards Benelux (NL) - For the CRO team. Started from the bottom, now they are 100 tests further.

  • Microsoft (US) - For the Experimentation Platform Team. A/B Testing and Covid-19: Data-Driven decisions in times of uncertainty.

  • SAP (US) - For the Optimization Lab. Building a data-driven culture through widespread testing.

Click on the nominee to read more about the submitted case

ORGANIZATION-WIDE WINNER & NOMINEES

Organization-wide award nominations
  • WINNER of the the 2020 Organization-wide AWARD is: Farfetch (PT) - For the circle path from insights to learnings. Change a company experimentation mindset in a data-driven approach.

  • GlaxoSmithKline (US) - For building a culture and practice of experimentation.

  • MongoDB (US) - For going from 0 to 100 within 6 months. Implementing and accelerating a research and testing program.

  • AllClear Travel Insurance (UK) - For the democratization of experimentation.

  • RS Components (UK) - For full stack experimentation transformation. The 'binding agent' between Product & Engineering.

Click on the nominee to read more about the submitted case

LIFETIME ACHIEVEMENT AWARD

The 2020 jury has decided to hand out an INDIVIDUAL Experimentation Culture lifetime achievement AWARD

  • The WINNER is: Ronny Kohavi (USA) - For all the research and publications that helped so many experimenters to understand how to conduct trustworthy digital experiments.

LIFETIME ACHIEVEMENT AWARD

The 2020 jury has decided to hand out an ORGANIZATION-WIDE Experimentation Culture lifetime achievement AWARD

  • The WINNER is: Booking.com (NL) - For being a big example organization for experimentation, known and quoted by many decision makers throughout the world.

September 24th, live broadcast

The 2020 liveshow has 180 minutes of inspirational talks and interviews

MongoDB Experimentation Culture Award nomination

ORGANIZATION-WIDE NOMINEE
RS Components

RS Components

Full Stack Experimentation Transformation - The 'binding agent' between Product & Engineering

I would like to highlight the incredible strides we've made in our organisation to build an experimentation culture that puts the customer at the heart of everything we do.

Enabled by our newly found experimentation Centre of Excellence 'Optimise Prime', over the last year we've thrown away our old practices and beliefs around experimentation to usher in a new wave of product & engineering led experimentation that is truly democratised and owned by all.

We've democratised our practices and processes, enabling the entire organisation to share in ideation and build products and solutions for our customers under no assumptions - delivered with an engineering first approach with full stack experimentation , backed by a robust business case and sizeable investment from the organisation, RS Components is well on its way to becoming a glowing success story in the experimentation arena. We still have a long way to go, but the future is so exciting for us!

The 2020 liveshow has 180 minutes of inspirational talks and interviews


MongoDB Experimentation Culture Award nomination

ORGANIZATION-WIDE NOMINEE
AllClear Travel Insurance

AllClear

Democratization of Experimentation

In 2018, AllClear was doing very few experiments. As it was growing by 54% YoY, experimentation became an important factor to accelerate growth and to make customer-centric decisions.

Starting its experimentation journey, AllClear invested in a new experimentation platform and consolidated all the test data in the new tool to foster faster decision making.

Aiming to democratize experimentation beyond the Experimentation Team, everyone has access to the tool, knows how to operate it and has access to the customer insights generated by everyone else.

Regular meetings and workshops with teams are organized to discuss tests insights and explore new opportunities. In addition, everyone is encouraged to contribute into the roadmap and share new experimentation ideas.

This structure works as an infinite loop of collaboration, continuous learning and idea sharing between the teams.

Nowadays, experimentation has become an integral part of business decision making and every new feature is tested.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


MongoDB Experimentation Culture Award nomination

ORGANIZATION-WIDE NOMINEE
MongoDB

MongoDB

From 0 to 100 within 6 months. Implementing and accelerating a research and testing program at MongoDB

Experimentation at Mongo was non-existent one year ago and until that time MongoDB was very much product lead, and quite successful.

But there was a strong culture and desire to apply scientific-thinking, hypothesis testing, as an underpinning to growth efforts as the company looked to expand beyond core channels and customers.

We instituted a test program framework, scaling the testing program from 0 to a current 100 test/yr pace. We worked (and are currently expanding) across the web, email, and product teams. This effectively installed a center of excellence model for hypothesis testing across the organization.

Key cultural wins were building project management guardrails for ingestion of ideas (submission, characterization, and prioritization) and communication of results (score carding, newsletters). We’re currently planning workshops for process/mindset internal training.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


GlaxoSmithKline Experimentation Culture Award nomination

ORGANIZATION-WIDE NOMINEE
GlaxoSmithKline

GlaxoSmithKline

Building a culture and practice of experimentation

GSK’s US Pharma brands were always open to testing and experimentation, but it wasn't until this year that the company invested in creating a centralized Experimentation Program.

They’ve been hard at work creating a centralized repository of ideas, rolling out templates to support the end to end process, and hosting a variety of interactive educational sessions with the theme “literacy tied to action.”

It would have been easy, and potentially justifiable, to put some of these efforts on hold when the pandemic hit to redirect resources elsewhere to cope with new constraints, but the team doubled-down.

In fact, the team is hosting the first ever Digital Measurement & Experimentation conference internally to inspire and educate on how to unlock value through hypothesis-driven measurement and experimentation in modern marketing.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Farfetch Experimentation Culture Award nomination

ORGANIZATION-WIDE NOMINEE & 2020 AWARD WINNER: Farfetch

Farfetch

The circle path from insights to learnings - change a company experimentation mindset in a data-driven approach

Farfetch is the global technology platform for luxury fashion, connecting creators, curators, and consumers.

2 years ago we identified the need to step-up our current experimentation practices. We started by assessing where we were and we used the Experimentation Growth Model to help guide and focus our objectives.

Since then, we have built a company-wide enablement Test & Learn Center of Excellence, onboarded new business areas and empowered all teams to use our single framework for experimentation.

This includes; our in-house experimentation platform (FABS) integrated with Optimizely; a centralized learnings hub and hypothesis one-pager; biweekly hypothesis peer-review sessions; monthly company-wide test & learn sessions sharing the latest learnings; our own results analysis algorithm (initially based on Stats Engine paper) which now supports us with continuous metrics, cycle metrics, business impact estimation/analysis and more.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Microsoft Experimentation Culture Award nomination

TEAM NOMINEE
Microsoft

Microsoft

A/B Testing and Covid-19: Data-Driven Decisions in Times of Uncertainty

Covid-19 caused many question marks at experimentation teams throughout the world. What to do now? Continue testing or not, are the learnings valid or not and so on.

The Microsoft Exp. team opened up their insights and views on this as a thought leading example for other experimentation teams. This helped a lot of experimentation teams to show their managers that they should not stop experimentation because of Covid-19:

https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/a-b-testing-and-covid-19-data-driven-decisions-in-times-of-uncertainty/

The 2020 liveshow has 180 minutes of inspirational talks and interviews


SAP Experimentation Culture Award nomination

TEAM NOMINEE
SAP

SAP

Building a data-driven culture through widespread testing

Amidst the pandemic, a small core of 3 full timers have propelled SAP Optimization Lab to new new heights. Our proudest accomplishment has been the data-driven culture we have helped proliferate across an organization of 1,200+ marketers around the world.

Ingredients to success:

Organizational penetration - 80+ experiments across 5 continents speaks to our global reach, but more important is the collaborative work with over 24 different teams that propelled those tests into the wild. Each test helps to pollinate “testing DNA” across the many stakeholders involved.

Our online test request form democratizes testing, yet also guides marketers towards “smart” test concepts - reinforcing the importance of sample size, goal and KPI considerations.

Our highly consumed "Insights Book" conveys our test learnings in a consistent, easy-to-consume format.

Our test database and visually-rich testing roadmap are open to all to view at any time, boosting program transparency and engagement.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Hallmark Experimentation Culture Award nomination

TEAM NOMINEE
Hallmark Cards Benelux

Hallmark

Started from the bottom, now we're 100 tests further

Our online department has been transformed in one year from an organization where changes were implemented based on subjective insights and gut feelings, to a data-driven organization where conversion is leading and we only implement changes based on validation of ideas. This transformation was initiated by a new CRO team.

The CRO team created this culture by:

1. Embracing every stakeholders feedback and use it to start a dialogue about experimentation, so they think along in gaining better test ideas;

2. Sharing learnings and results bi-weekly to the whole team;

3. Collaborating intensively with the development teams and their product owners;

4. Having complete endorsement of one management team member.

This all helped us to grow in one year from a culture where no tests were performed to one where we have performed over 100 tests. This led to new implementations and a lot of of new insights on our platform. Most importantly, we did not implement detrimental changes.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Taxfix Experimentation Culture Award nomination

TEAM NOMINEE
Taxfix

Taxfix

The path towards a strong experimentation culture

Since January 2020 our goal has been to improve conversion rates, but until then we had launched only one test. We realized we had to raise the level of our experimentation culture and make each member of the team confident to test. So we created an Experimentation Taskforce to create a process and train everyone on the importance of testing. In the following months, we reached important milestones:

√ We increased the number of experiments launched from just one in 2019, to two per week in 2020.

√ We increased the number of teams involved, from just one team last year to six teams now.

√ We have a structured process on how to test and had multiple workshops on why we should experiment. As a result, when we asked the teams to rate their level of confidence in testing, the grade for beginning of 2020 was 4.7, now it is 7.1.

√ Due to the success of our tests, even during the toughest weeks of the Covid-19 threat, we were always above target in terms of conversion rates.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


VodafoneZiggo Experimentation Culture Award nomination

TEAM NOMINEE & 2020 AWARD WINNER: VodafoneZiggo

VodafoneZiggo

CRO Center of Excellence - A culture of experimentation

Annemarie Klaassen and Farshad Soleymani (CRO Center of Excellence) have setup and expanded a companywide culture of experimentation.

In September we went through a reorganization in which CRO grew tremendously due to their efforts. We went from a handful to 15 CRO’ers helping out 26 squads within Digital and Marketing, collaborating with almost all other departments that now embrace our way of working.

They introduced a program management tool (Airtable) to facilitate a structured process for experimentation with experimentation databases, manuals for research and analyses, psychological strategies, prioritizing models etc.

They have facilitated multiple courses and coach to increase knowledge of CRO’ers, host a two weekly CoP, send out progress reports, setup a fail board, peer reviews and an extensive CRO Growth Scan.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Stefan Thomke Experimentation Culture Award nomination

INDIVIDUAL NOMINEE
Stefan Thomke

Stefan Thomke

Publishing an experimentation culture book for business leaders

Published in 2020, finally a book about experimentation written for business leaders.

Look at the Amazon recommendations: https://www.amazon.nl/Experimentation-Works-Surprising-Business-Experiments/dp/163369710X/ - this is a business book by a Harvard professor which is, and is going to be, read and followed-up by managers inside many organizations that need to move forward.

This book is supporting an experimentation culture embraced by upper management in many organizations.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Desiree van der Horst Experimentation Culture Award nomination

INDIVIDUAL NOMINEE
Desiree van der Horst

Desiree van der Horst

Taking away the fear of sharing failures

How to get everyone to share their failures and their successes? How we got from 7 lessons shared in 3 months to 96 lessons shared in the next 3 months.

Last year we had these magnetic boards on the wall, where marketers could post their fails and successes. Which seemed like a good idea, but nobody used it.

I asked my colleagues why they did not share their failures on that board:

1. I’m afraid that management thinks I’m stupid
2. I’m afraid my work is not good enough
3. I don’t want to stand out, because I don’t know what the consequences will be.

So our fears of what other people think of us, are holding us back.

When I asked about why they didn’t share successes, I got the following answers:

1. I’m not sure if my accomplishment is interesting enough for others
2. This result is probably not significant
3. It’s not my own success

Concluding...our own modesty keeps us hesitating if we’re good enough.

I also asked what they thought would be a good way to share these insights. What kind of platform would make them feel comfortable enough to start sharing? One answer was given countless times: it needs to be online. And it needed to be light and airy. Nothing heavy and serious. It shouldn’t cost any effort to share.

The solution: a new slack channel. We already use slack daily and it’s already incorporated in our work habits.

The main goal of this channel is that we can learn from each other's stupid mistakes, that we can have a laugh about it, show each other that we aren’t perfect. For seniors as well as trainees. We also have set the following rules:

1. This channel is to share successes and fails. To give out compliments or just nag a bit if you had a shitty day or troubling client. Be each other's support network.

2. Don’t publicly “shame” each other. If you have something to say about somebody's failure or success, send them a private message. Respect each other.

3. The channel is for marketers only. So no management, sales or other peeping toms. We’ve got each others backs. So never share an individual’s fail without their permission.

We started using this channel in the beginning of March. I asked our more senior marketers to step up a bit more and give examples of their own fails and successes. To lead with the right example. To make it easier for other (more junior) marketers to follow.

Back in the day, when we used the magnetic boards, maybe 9 fails were shared in 3 months. But using the slack channel, in 3 months time, 53 fails and 43 successes were shared!

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Gabriela Denison Experimentation Culture Award nomination

INDIVIDUAL NOMINEE
Gabriela Denison

Gabriela Denison

Building trust in fast, pain-free experimentation

We were not experimenting enough. When asked why it is the case, engineers and product owners often cite the lack of understanding on the math and process, and the frustration on how long it takes from having an experiment idea to actually learning something.

Gabriela, whose day job is to analyze and recommend upon experiment results, has since designed an internal Stats and Experiment Design 101 course. She also streamlined the experiment idea generation and planning process, and took the product ownership for the new automated dashboard that shows the metrics as the experiment runs, which began replacing manual analysis.

Her work has lead to weeks being shaved off from each experimentation cycle, and contributed to a 50% YoY increase in the number of experiments run. Engineers and product owners also report being more engaged in the process, and are now more confident to run future experiments by themselves, paving the way to scale experimentation at ASOS further.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Raoul Warren Doraiswamy Experimentation Culture Award nomination

INDIVIDUAL NOMINEE
Raoul Warren Doraiswamy

Raoul Warren Doraiswamy

Supporting businesses affected by CoronaVirus for free & building a community

When countries began going into lockdown, businesses moved online, many had limited experience selling their products online. In the same vein, CRO professionals found themselves being furloughed or laid off, looking for ways to help.

One man brought these communities together through a viral social media campaign, Raoul, when he created #CovidCRAP.

Raoul created a campaign which travelled around the world. He engaged with the CRO community first through LinkedIn, then with a website & slack group. His catchy (although, at times controversial) name & red lego caught the attention of CROs & businesses alike.

The main aim was to connect CROs with businesses who needed an hour’s worth of advice on how to improve their website.

Between March & April, he attracted 50-100 volunteers from all across the globe. With such vast coverage it made keeping track of the number of individual businesses helped difficult, but we know at least 100 businesses were helped for free within that timespan.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


Guido X Jansen Experimentation Culture Award nomination

INDIVIDUAL NOMINEE & 2020 AWARD WINNER: Guido X Jansen

Guido X Jansen

Evangelizing CRO in general and CRO culture in particular

Since 2019 Guido uses his weekly podcast to share the story of CRO practitioners in The Netherlands (2019) and beyond (since 2020) to address the biases we have in our own field, share cases and best practices and talk about implementing a culture of experimentation and validation.

Through this channel he is not only reaching CRO practitioners, but also more and more digital marketeers creating a more widespread understanding of what we as a CRO industry stand for and what organizations need to do for us to flourish.

The 2020 liveshow has 180 minutes of inspirational talks and interviews


SUBMIT YOUR AWARD CASE

Deadline for submission: July 31st, 2020 - 6.00pm UTC / GMT

You can submit your own case(s) through the form below (unless the deadline has passed of course). You are allowed to submit multiple case entries. If you would like to nominate someone else, you can use the same form.

After submitting your case you will receive an email with the confirmation of your entry. We will contact you in the second week of August if your case entry is selected as on of the final 5 nominees in a specific category.

Besides some contact info the signup form only asks for a max. 1000 chars. case description, which can be completed within 5 minutes. More info about the awards and nominee selection process can be found on our jury page.

Submit your case form

Our deadline to submit a case for our 2020 awards was July 31st, 6.00pm UTC / GMT. This means that our submit form is closed now. In the first week of August the jury has nominated the 5 finalists in each category and in September we announced the winners during our live broadcast.

If you were too late to submit something, don't worry - we will be back in 2021 with new awards.

Time left until the award case submission deadline:


    ABOUT THIS AWARDS EVENT

    The Experimentation Culture Awards were scheduled as part of the new Outperform Conference in Amsterdam, June 2020. This event got postponed to June 2021, due to the Covid-19 virus.

    This outbreak changed a lot, it caused a massive shift for many organizations. The health of employees and organizations are top priority now. Did it influence your experimentation culture?

    What have you, your team and your company done the past months to maintain a culture of experimentation, or to even grow it to a next level of maturity? This question is exactly the reason why we wanted to move forward with the Awards in 2020!

    On the deadline of July 31st 2020 we had received 65 amazing cases on evidence-based maturity growth in organizations. Stories that will inspire others in these strange times. Stories that inspire us all to keep on moving forward maintaining and improving a culture of experimentation.

    It's not about that one experiment, it's about improving the process, structure, trustworthiness, democratization and motivation for experimentation and data-driven decision making.

    Read on for our awards, the jury and the nominee selection process

    • Individual award: something 1 person has done in the past months to keep or improve the culture of experimentation in an organization or even broader.

    • Team award: something a team has done in the past months to keep or improve the culture of experimentation in your organization or even broader.

    • Organization-wide award: something an organization has done in the past months to keep or improve the culture of experimentation in your organization or even broader.

    THE AWARDS JURY OF 2020

    After the submission deadline of July 31st 2020, 6.00pm GTM, every jury member has scored each entry on a 5 star rating scale. The top 5 in each category has been be contacted and is now being interviewed about their award entry. Based upon that extra information the jury will select the winner among the 5 nominees in every category and announce the winners in the broadcast of September 24th, 6.00pm UTC / GMT - 9.30pm UTC / GMT.

    The live show is done, but you can still register your free event ticket to get access to the recording with 6 keynotes, 15 interviews, 3 awards and 2 bonus items. Truly inspirational content on experimentation culture.


    GET FREE ACCESS TO THE RECORDING OF THE LIVE SHOW

    Register yourself to be able to watch the recording of the live broadcast. The events has 6 keynotes, the 15 final nominees and presents the winners. You can check the full schedule here.

    The full recording is accessible through our live stream platform
    If you registered earlier: check your email for login information

    (emails are mainly send by info@experimentationcultureawards.com, but login info also comes from no-reply@editmysite.com - use "Reset password")

    VIRTUAL SUMMIT SCHEDULE

    September 24th 2020

    2020 keynote speakers

    All times in the schedule are UTC / GMT: Moscow +3hrs, Paris +2hrs, London +1hr, New York -4hrs, Austin -5hrs, San Jose -7hrs

    • 5.50pm - 6.00pm: Livestream starts with a 10 minute countdown - you can start to log in

    • 6.00pm - 6.15pm: Introduction of the 2020 Awards by jury chair & host of the this show Ton Wesseling

    • 6.15pm - 6.30pm: Keynote presentation by Stephen Pavlovich - Tactics that could help to create a culture of experimentation

    • 6.30pm - 6.45pm: Keynote presentation by Marianne Stjernvall - How to build an experimental organization

    • 6.45pm - 7.105pm: Individual award nominees case introductions & announcing the winner!

    • 7.05pm - 7.20pm: Keynote presentation by Hazjier Pourkhalkhali - Who optimizes the optimizers

    • 7.20pm - 7.35pm: Keynote presentation by Casandra Cambell - Stop just optimizing; start deliberate learning instead

    • 7.35pm -7.45pm: Individual lifetime achievement award given by the 2020 jury to Ronny Kohavi

    • 7.45pm - 8.10pm: Team award nominees case introductions & announcing the winner!

    • 8.10pm - 8.20pm: Keynote presentation by Kelly Wortham - Experimentation Culture From individual to industry

    • 8.20pm - 8.30pm: Keynote presentation by Lukas Vermeer - Experimentation Culture at Booking.com

    • 8.30pm - 8.40pm: Organization-wide lifetime achievement award given by the 2020 jury to Booking.com

    • 8.40pm - 9.02pm: Organization award nominees case introductions & announcing the winner!

    • 9.02pm - 9.05pm: Closing by Ton Wesseling

    All keynotes are presentations by the Award jury members.

    2020 keynote speakers

    All nominations will be introduced during the event by the case owners. Do not forget to register to get access to the show and/or the recording!


    CONTACT FORM

    Use the form below to ask us your question(s). Our goal is to get back to you within 2 business days.

    Emails are send through info@experimentationcultureawards.com

    You will be contacted by a representative of the owner of the Experimentation Culture Awards: Help je Klant Holding BV, registered at the Chamber of Commerce in the Netherlands under number 30251897.


    CONTACT FORM COMPLETED

    Thank you for asking your question. Our goal is to get back to you within 2 business days. We will get in touch with you through info@experimentationculture.com

    TICKET REGISTRATION DONE

    Our welcome email should arrive in your email box in 2 minutes. It's send through info@experimentationcultureawards.com.

    Keep an eye for our emails on event announcements and the event recording login link.