Abstract: Embodiments of the present technology are directed to facilitating generation of experiment metric values, such as expected sample size and/or minimal detectable effect, for anytime valid confidence sequences (e.g., asymptotic confidence sequences). In one embodiment, a set of parameter values associated with an experiment using asymptotic confidence sequences are obtained. The set of parameter values include a minimal detectable effect and an uncertainty interval. Thereafter, an expected sample size for executing the experiment is determined based on the minimal detectable effect and the uncertainty interval. The expected sample size is provided for utilization in association with the experiment using asymptotic confidence sequences.

Abstract: A method, a system, and a computer program product for analyzing data collected during a randomized controlled experiment to determine an effect of variations of digital content. Determination of the effect includes execution of first and second testing sequences that prompt responses to first and second digital contents, respectively, from users. The testing sequences execute during a predetermined duration of time. Responses to the first and second testing sequences generate first and second test data, respectively. One or more confidence intervals for each first and second test data are generated at a randomly selected time during the predetermined duration of time. A testing metric indicating the effect of the second digital content over the first digital content is determined at the randomly selected time. The testing metric is determined at any time before expiration of the predetermined duration of time.

Abstract: Certain aspects and features of this disclosure relate to providing anytime-valid confidence sequences for multiple messaging treatments in an experiment. A process controls and/or corrects statistical error when multiple messaging treatments are being evaluated together. Messages can be stored, formatted, and transmitted from a communication server or other computing system. In one example, each test message from among multiple test messages is sent to an independent group of recipients over some period of time. An analytics application programmatically evaluates a metric related to message responses over time and determines a difference in the metric for each of several unique messages as compared to a baseline message. The analytics application also determines a confidence value and can display the changing confidence value in sequence over time along with the current difference, or lift, while maintaining the accuracy of the values.