Designing Scalable Multivariate Testing Frameworks for High-Traffic E-Commerce Platforms
DOI:
https://doi.org/10.14419/47mq5944Published
10-12-2025Keywords:
Multivariate Testing; E-commerce Optimization; Scalability; Experimentation Framework; Statistical AnalysisAbstract
Abstract: High-traffic e-commerce outlets use experimentation to perfect the user experience and, in turn, conversions and revenue. Multivariate testing, however, acts as an elegant solution, considering changes occurring in multiple variables and their interaction effects simultaneously, thus generating richer insights than in the scenario of an A/B test. Nonetheless, the implementation of a scalable multivariate testing framework on such large platforms harbors considerable architectural challenges, data infrastructure issues, and, most importantly, statistical and operational integration concerns. This paper covers complete principles, system design considerations, and statistical methods that guide the construction of engineering-grade privacy-compliant experimentation systems that support millions of concurrent users. By considering architecture approaches, data pipelines, performance improvements, and integration with personalization, inventory, and marketing systems, current best practices for performing experimentation as a fundamental operational capability are offered. Topics discussed in case studies offer evidence-based stories on successes and pitfalls on the canvas, emerging trends such as reinforcement learning and privacy-preserving analytics that will dictate the future of experimentation in e-commerce.
References
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. https://doi.org/10.1017/9781108653985.
Xu, Y., Chen, N., Fernandez, A., Sinno, O., & Bhasin, A. (2015). From infrastructure to culture: A/B testing challenges in large-scale social networks. In Proc. 21st ACM SIGKDD (pp. 2227–2236). https://doi.org/10.1145/2783258.2788602.
Petrović, G., & Ivanković, M. (2018). State of mutation testing at Google. In Proc. ICSE-SEIP (pp. 163–171). https://doi.org/10.1145/3183519.3183521.
Li, X., Makkie, M., Lin, B., Fazli, M. S., Davidson, I., Ye, J., & Quinn, S. (2016). Scalable fast rank-1 dictionary learning for fMRI big data analysis. In Proc. KDD (pp. 511–519). https://doi.org/10.1145/2939672.2939730.
Tang, D., Agarwal, A., O’Brien, D., & Meyer, M. (2010). Overlapping experiment infrastructure: More, better, faster experimentation. In Proc. KDD (pp. 17–26). https://doi.org/10.1145/1835804.1835810.
Bakshy, E., Eckles, D., & Bernstein, M. S. (2014). Designing and deploying online field experiments. In Proc. WWW (pp. 283–292). https://doi.org/10.1145/2566486.2567967.
Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving sensitivity of online controlled experiments using pre-experiment data. In Proc. WSDM (pp. 123–132). https://doi.org/10.1145/2433396.2433413.
Cobbe, J., Lee, M. S. A., & Singh, J. (2021). Reviewable automated decision-making: A framework for accountable algorithmic systems. In Proc. FAccT (pp. 598–609). https://doi.org/10.1145/3442188.3445921.
Bakshy, E., Messing, S., & Adamic, L. A. (2015). Exposure to ideologically diverse news on Facebook. Science, 348(6239), 1130–1132. https://doi.org/10.1126/science.aaa1160.
Deng, A., & Shi, X. (2016). Data-driven metric development for online controlled experiments. In Proc. KDD (pp. 77–86). https://doi.org/10.1145/2939672.2939700.
Antunes, B., Cordeiro, J., & Gomes, P. (2012). Context-based recommendation in software development. In Proc. RecSys (pp. 171–178). https://doi.org/10.1145/2365952.2365986.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. SIGKDD Ex-plorations, 11(1), 10–18. https://doi.org/10.1145/1656274.1656278.
Canfora, G., & Di Penta, M. (2006). Service-oriented architectures testing: A survey. In: International Summer School on Software Engineering (pp. 78–105). Springer. https://doi.org/10.1007/978-3-540-95888-8_4.
Veeraraghavan, K., Chen, P. M., Flinn, J., & Narayanasamy, S. (2011). Detecting and surviving data races using complementary schedules. In Proc. SOSP (pp. 369–384). https://doi.org/10.1145/2043556.2043590.
Martin, A., Anslow, C., & Johnson, D. (2017). Teaching agile methods: 10 years, 1000 release plans. In Proc. Agile Conference (pp. 151–166). Springer. https://doi.org/10.1007/978-3-319-57633-6_10.
Bakshy, E., Rosenn, I., Marlow, C., & Adamic, L. (2012). Social networks in information diffusion. In Proc. WWW (pp. 519–528). https://doi.org/10.1145/2187836.2187907.
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018). RNN for multivariate time series with missing values. Scientific Reports, 8, 6085. https://doi.org/10.1038/s41598-018-24271-9.
Junqué de Fortuny, E., Stankova, M., Moeyersoms, J., Minnaert, B., Provost, F., & Martens, D. (2014). Corporate residence fraud detection. In Proc. KDD (pp. 1650–1659). https://doi.org/10.1145/2623330.2623333.
Mertler, C. A., Vannatta, R. A., & LaVenia, K. N. (2021). Advanced and multivariate statistical methods. Routledge. https://doi.org/10.4324/9781003047223.
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web. DMKD, 18(1), 140–181. https://doi.org/10.1007/s10618-008-0114-1.
Kohavi, R., Henne, R. M., & Sommerfield, D. (2007). Practical guide to controlled experiments: Listen to your customers. In Proc. KDD (pp. 959–967). https://doi.org/10.1145/1281192.1281295.
Gui, H., Xu, Y., Bhasin, A., & Han, J. (2015). Network A/B testing: From sampling to estimation. In Proc. WWW (pp. 399–409). https://doi.org/10.1145/2736277.2741081.
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online controlled experiments at large scale. In Proc. KDD (pp. 1168–1176). https://doi.org/10.1145/2487575.2488217.
Ge, B., Li, X., Jiang, X., Sun, Y., & Liu, T. (2018). Dictionary learning for fMRI signal sampling. Frontiers in Neuroinformatics, 12, 17. https://doi.org/10.3389/fninf.2018.00017.
Wester, B., Devecsery, D., Chen, P. M., Flinn, J., & Narayanasamy, S. (2013). Parallelizing data race detection. In Proc. ASPLOS (pp. 27–38). https://doi.org/10.1145/2451116.2451120.
Petrović, G., Ivanković, M., Fraser, G., & Just, R. (2021). Practical mutation testing at scale: A view from Google. IEEE TSE, 48(10), 3900–3912. https://doi.org/10.1109/TSE.2021.3107634.
Kapur, N., Lytkin, N., Chen, B. C., Agarwal, D., & Perisic, I. (2016). Ranking universities via career outcomes. In Proc. KDD (pp. 137–144). https://doi.org/10.1145/2939672.2939701.
Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Five puzzling A/B outcomes explained. In Proc. KDD (pp. 786–794). https://doi.org/10.1145/2339530.2339653.
Burger, M., Sergeev, F., Londschien, M., Chopard, D., Yèche, H., Gerdes, E. C., … Faltys, M. (2024). Towards foundation models for critical time series. In AIM-FM Workshop @ NeurIPS 2024.
Kohavi, R., & Longbotham, R. (2023). Online controlled experiments and A/B tests. In Encyclopedia of Machine Learning and Data Science (pp. 1–13). Springer. https://doi.org/10.1007/978-1-4899-7502-7_891-2.