Abstract: Online experimentation has been widely used to evaluate an effect of a new feature of an online product on user engagement. One challenge is that an existence of outliers can often complicate the analysis of such experimental results. Thus, a procedure is provided herein to detect and remove outliers from experimental results. The procedure can use statistical tests based on parametric distributions of sample maximum or minimum. These tests can be performed using an inward testing procedure to identify multiple outliers. Finally, these filtered test results can be used to control delivery of a new feature of an online product.

Abstract: Online experimentation has been widely used to evaluate an effect of a new feature of an online product on user engagement. One challenge is that an existence of outliers can often complicate the analysis of such experimental results. Thus, a procedure is provided herein to detect and remove outliers from experimental results. The procedure can use statistical tests based on parametric distributions of sample maximum or minimum. These tests can be performed using an inward testing procedure to identify multiple outliers. Finally, these filtered test results can be used to control delivery of a new feature of an online product.

Abstract: The technologies described herein use a statistical test to determine whether differences between data sets of buckets in a bucket test, such as differences between averages of two buckets (e.g., differences between means of two buckets), are directionally larger than a predetermined or preset minimum threshold value. The statistical test may also provide an extension to specify the minimum threshold value as a percentage. Also, described herein are techniques for estimating different control variables of a bucket test, such as estimating minimum bucket size to provide sufficient statistical power with use of the minimum threshold value.

Abstract: Determine a plurality of first dwell durations for a plurality of first web pages, each first dwell duration indicating a time period a user has spent with a first web page. Access a plurality of first quality ratings for the first web pages, each first quality rating indicating a quality of a first web page as a part of a search result generated for a first search query. Access a predefined quality rating threshold. Correlate the first dwell durations and the first quality ratings. And, determine a dwell duration threshold, such that a second user spending a second dwell duration greater than or equal to the dwell duration threshold with a second web page indicates that the second user is satisfied with the second web page identified in a second search result generated by a search engine in response to a second search query requested by the second user.