Abstract: Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for imbalance detection in online experiments. In some implementations, a method includes obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices; generating a threshold for detecting imbalance using the alternative samplings of devices, a generated expected first set of devices, and a generated expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.

Abstract: The disclosed computer-implemented method includes accessing data that is to be used as part of a test implementation that has multiple potential outcomes. The method also includes determining that the test implementation is to be carried out using specified testing algorithms that test for at least one of the potential outcomes. The method next includes identifying portions of the accessed data that are to be used in the specified testing algorithms, and compressing the identified portions of the accessed data to remove portions of the accessed data that are unused in the specified testing algorithms. The method also includes executing the test implementation using the specified testing algorithms with the compressed accessed data. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: The disclosed computer-implemented method includes accessing data that is to be used as part of a test implementation that has multiple potential outcomes. The method also includes determining that the test implementation is to be carried out using specified testing algorithms that test for at least one of the potential outcomes. The method next includes identifying portions of the accessed data that are to be used in the specified testing algorithms, and compressing the identified portions of the accessed data to remove portions of the accessed data that are unused in the specified testing algorithms. The method also includes executing the test implementation using the specified testing algorithms with the compressed accessed data. Various other methods, systems, and computer-readable media are also disclosed.