Editor’s note: Yixin is a speaker for ODSC East this April 23-25. Be sure to check out his talk, “Experimentation Platform at DoorDash,” there!

At DoorDash, we believe in learning from our marketplace of Consumers, Dashers, and Merchants and thus rely heavily on experimentation to make data-driven product and business decisions.

Here are a few examples of what we experiment about:

• Algorithms: We continually refine our real-time logistic fulfillment systems, navigating through constraints and tradeoffs, optimizing objective functions, and employing advanced techniques to enhance the precision of assignment decisions.
• ML models: Our teams conduct numerous experiments displaying varying store rankings to users, employing diverse ranking and recommendation algorithms to tailor user experiences.
• Infrastructure changes: Beyond traditional A/B testing in products and algorithms, we employ it to evaluate infrastructure changes, such as comparing the performance of Redis and CockroachDB.
• Products: Given the intricacies of implementing Dashers’ peak demand pay, we continuously enhance our mobilization system, strategically allocating incentives to preempt any potential supply-demand imbalances.
• Business Policy: The consumer retention marketing team strives to cultivate enduring connections with customers, starting from their initial interaction with DoorDash by delivering pertinent marketing content to encourage repeat engagements. Employing A/B tests, we consistently refine our policies, selecting optimal strategies from an extensive range of options.

To ensure the highest quality data from our experiments, we’ve crafted an in-house experimentation platform. At its core, our platform encompasses three key components: an experimentation configuration platform, an experimentation analysis platform, and a metrics platform.

Over the years, our analysis platform, Curie, has undergone a remarkable transformation. What began as a basic experimentation result interface (MVP) has matured into a comprehensive platform that seamlessly integrates with various data ecosystems within DoorDash. Curie now interfaces with key components such as the metrics platform, experiment configuration platform, forecasting platform, and ML systems, among others.

In my forthcoming presentation at ODSCI’ll delve into the distinctive challenges and opportunities encountered while developing the Experiment Analysis Platform. Here are some key highlights to anticipate:

• Experimentation Platform in Big Data

In this segment, I’ll explore how our platform leverages Dagster for orchestrating metrics and analysis jobs. I’ll delve into our data storage and retrieval strategies using datalake and snowflake, enabling exploratory analysis with Databricks notebooks, and integrating with our machine learning (forecasting) platform for automated decision-making.

Speed is key in experimentation at DoorDash. I’ll discuss various techniques we’ve developed, such as surrogate metrics, variance reductions, metrics transformations, and triggered analysis, all aimed at accelerating the experimentation process.

Interference poses a significant challenge in experiment design at DoorDash. I’ll illustrate this through scenarios like switchback experiments in three-sided marketplaces and budget-split experiments in ads bidding. Furthermore, I’ll share our innovative solutions to mitigate interference.

• Scalability and Operational Efficiency

As our platform scales, processing hundreds or thousands of experiments per day, each with billions of data points, becomes imperative. I’ll highlight optimizations like moments algorithms that we’ve implemented to ensure speedy analysis while optimizing cloud costs.

• Beyond A/B testing – Causal Inference, Adaptive Learning, etc.

Traditional A/B testing isn’t always feasible or optimal. I’ll discuss how we’ve expanded our approach to include causal inference and adaptive learning, leveraging bandit algorithms to tackle complex scenarios beyond the scope of A/B testing. This refined outline maintains clarity and succinctness while ensuring each topic is clearly defined and emphasized.

About the Author:

Yixin Tang is an engineer manager on the experimentation platform team at DoorDash. His team is responsible for building a world-class experimentation platform that the product teams and various business analytics functions use to make timely, insightful, and intelligent decisions that optimize our business and product. Yixin has 10 more years of experience contributing to various initiatives related to customer-facing products, machine learning, optimization, and data-driven large-scale systems, with a focus on experimentation.

Connect with him on LinkedIn to read more from him and learn about his journey.