Abstract: A method of executing an A/B testing service comprises deploying an instance of the A/B testing service between a client device and an A/B testing server and receiving by the A/B testing service from the A/B testing server a software development kit (SDK) key for a client device, the SDK key indicating an A/B testing environment to be associated with the instance. The method further comprises, in response to receiving the SDK key: storing data associated with the A/B testing environment in memory of the client device; providing a HyperText Transfer Protocol (HTTP) application programming interface (API) associated with the instance of the A/B testing service to one or more applications associated with the A/B testing environment; and executing, by a processing device, commands between the client device and the one or more applications on behalf of the A/B testing environment.

Abstract: A method includes providing, to a third-party content provider, code identifying a feature associated with digital content of the third-party content provider, wherein the identification comprises a feature variable placeholder associated with the feature. The method further includes receiving, by an experimentation platform, an indication of a first event type associated with the digital content, wherein the first event type identifies a first trackable action associated with the feature. The method further includes determining, by a processing device of the experimentation platform, that the first event type does not exist in a configuration associated with the feature; The method further includes, in response to the determining, generating the first event type in the configuration, generating and providing a graphical representation of the first event type for display in a user interface of the experimentation platform, and tracking the first event type on the experimentation platform.

Abstract: A method comprises receiving a first outbound request, from an internal user account of an internal platform, indicating a first action to be performed by a first third-party user account of a first third-party platform. In response to authenticating the first outbound request, the method further comprises sending an application programming interface (API) request to the first third-party platform to perform the first action on the first third-party platform on behalf of the internal user account. The method further comprises receiving a first inbound request, from the first third-party user account, indicating a second action to be performed on behalf of the internal user account on the internal platform. In response to authenticating the first inbound request, the method further comprises sending an internal request to the internal platform to perform the second action on the internal platform on behalf of the first third-party user account.

Abstract: A method of executing an A/B test includes, during execution of the A/B test, determining, by a processing device using a sequential frequentist test, that a sample ratio mismatch has occurred, wherein the sample ratio mismatch is determined before the A/B test ends the execution. The method further includes, in response to the determining, ending the execution of the A/B test before a previously scheduled end of the A/B test.

Abstract: A method includes receiving a training metric, the training metric indicating a parameter to be optimized by content recommendations. A machine learning algorithm maybe used to determine a plurality of different combinations of the recommendation algorithms in the experiment, and each of one or more of the plurality of combinations may be used to generate a content recommendation for one or more visitors. The statistical performance of each of the one or more combinations in optimizing the parameter (based on the content recommendations generated by those combinations) may be monitored and a higher percentage of visitors may be allocated to a combination that generates content recommendations that are the most effective at optimizing the parameter.

Abstract: A method of executing an A/B test includes configuring the A/B test to comprise a first plurality of users in a control group and a second plurality of users in a test group, wherein the first plurality of users and the second plurality of users are to be provided two different versions of a webpage. The method further includes, while the A/B test is executing, determining, by a processing device, that a sample ratio mismatch corresponding to the second plurality of users has occurred, wherein the sample ratio mismatch is determined before the A/B test ends executing. The method further includes, in response to the determining, ending the executing of the A/B test before a previously scheduled end of the A/B test.

Abstract: A method includes receiving an identification of a feature associated with digital content of a third-party content provider, wherein the identification comprises a feature variable placeholder associated with the feature. The method further includes receiving a configuration of a feature flag associated with the feature. The method further includes determining, by a processing device of an experimentation system, a plurality of feature variable values corresponding to the feature variable placeholder. The method further includes configuring, by the processing device, one or more rules on the experimentation system to determine: when and to whom the feature is to be deployed, based on the feature flag; and which of the plurality of feature variable values is to be deployed when and to whom.

Abstract: A method includes receiving, from a computing device, a request for content hosted by a content provider and determining one or more experiment variations for the requested content that the computing device is assigned to based, at least in part, on the request. The method further includes generating, by a processing device, a hash value of the one or more experiment variations and transmitting the hash value to the computing device and to a caching server.

Abstract: A method includes determining a plurality of epochs from a timeline of an experiment, wherein each of the plurality of epochs corresponds to a constant traffic allocation. The method further includes calculating, by a processing device, first and second mean values corresponding to a baseline of the experiment and a variation of the experiment for each epoch, respectively. The method further includes determining, for each epoch, a difference between the first mean value and the second mean value to generate a plurality of differences. The method further includes calculating, by the processing device, a weighted sum of the plurality of differences to generate a final difference in mean values.

Abstract: A method includes detecting a first plurality of user interactions with first dynamic content of a website. The method further includes detecting a second plurality of user interactions with second dynamic content of a website. The method further includes providing a graphical user interface (GUI) that illustrates the first and second pluralities of user interactions to a client device. The method further includes determining a correlation between the first and second plurality of user interactions and a conversion metric. The method further includes determining, by a processing device, that the first dynamic content has a higher conversion metric than the second dynamic content. The method further includes sending a first variation recommendation for the first dynamic content to the client device.

Abstract: A browsing history associated with a user of a client device is received from the client device. One or more keywords associated with a target audience for a variant of a web page are received from a content provider system. A score is determined for the user based on the browsing history for the user and the one or more keywords associated with the target audience. The user is identified as part of the target audience based on the score for the user satisfying a score threshold. The variant of the web page is caused to be presented at the client device associated with the user.

Abstract: A single-page application of a content provider is associated with a smart page established on an experiment system to allow a variation test to be performed on the single-page application. The smart page is associated with a trigger that determines when one or more corresponding conditions of the single-page application should be evaluated. Responsive to one or more of the conditions of the single-page application being satisfied, either an original version of the single-page web page or a variant of the single-page web page may be displayed.

Abstract: A method includes providing, to a third-party content provider, code identifying a feature associated with digital content of the third-party content provider, wherein the identification comprises a feature variable placeholder associated with the feature. The method further includes receiving, by an experimentation platform, an indication of a first event type associated with the digital content, wherein the first event type identifies a first trackable action associated with the feature. The method further includes determining, by a processing device of the experimentation platform, that the first event type does not exist in a configuration associated with the feature; The method further includes, in response to the determining, generating the first event type in the configuration, generating and providing a graphical representation of the first event type for display in a user interface of the experimentation platform, and tracking the first event type on the experimentation platform.

Abstract: A variation testing system environment for performing variation testing of web pages and applications is disclosed. Users requesting a view from a content provider are not randomly assigned to one of a plurality of variations of the view. Rather, a function is applied to each user's identifier in order to determine which variation of the view is provided to a client device of the user.

Abstract: A method of avoiding infinite loops of changes to a dynamic web page to display on a client device includes displaying the dynamic web page received from a content provider on the client device, wherein the dynamic web page comprises a modified element having an original modification. The method further includes receiving an indication that the modified element was updated to generate an updated modified element. The method further includes, in response to receiving the indication, determining, by a processing device, that reapplying the original modification to the updated modified element would result in an infinite loop on the dynamic web page. The method further includes, based on the determination, displaying the updated modified element on the dynamic without reapplying the original modification to the updated modified element.

Abstract: A content variation experiment system for performing variation testing of web pages is disclosed. A content provider receives requests for a web page undergoing an experiment. The content provider determines a variation from a plurality of variations of the web page to provide to the user. The content provider makes the determination without sending a network request to an experiment definition system used to define the experiment thereby reducing network latency.

Abstract: A method includes receiving an identification of a feature associated with digital content of a third-party content provider, wherein the identification comprises a feature variable placeholder associated with the feature. The method further includes receiving a configuration of a feature flag associated with the feature. The method further includes determining, by a processing device of an experimentation system, a plurality of feature variable values corresponding to the feature variable placeholder. The method further includes configuring, by the processing device, one or more rules on the experimentation system to determine: when and to whom the feature is to be deployed, based on the feature flag; and which of the plurality of feature variable values is to be deployed when and to whom.

Abstract: Webpages are optimizing through the use of experiments that compare the responses of viewers that are either presented with the original webpage or a variant thereof. One or more variants are first defined through the use of a browser-based editor application that initially examines the webpage for a snippet of code. The snippet can be added to the webpage, if missing, and the webpage returned to the editor application. Changes made to the webpage to define a variant are saved in variation code. When the webpage is later requested by multiple viewers, in each instance the viewer receives the webpage with the snippet, the snippet instructs the browser to download a file, and the instructions of the file determine whether the viewer will see the variant or the original webpage. Tracking viewer responses to the webpage and the variant allow a statistical basis for comparison to be developed.

Abstract: A variation testing system for performing variation testing of content is disclosed. The variation testing system allows an experimenter to continuously monitor the results of the variation test. In order to ensure that the experimenter is provided with valid results, the variation testing system may reset the variation test.

Abstract: Webpages are optimizing through the use of experiments that compare the responses of viewers that are either presented with the original webpage or a variant thereof. One or more variants are first defined through the use of a browser-based editor application that initially examines the webpage for a snippet of code. The snippet can be added to the webpage, if missing, and the webpage returned to the editor application. Changes made to the webpage to define a variant are saved in variation code. When the webpage is later requested by multiple viewers, in each instance the viewer receives the webpage with the snippet, the snippet instructs the browser to download a file, and the instructions of the file determine whether the viewer will see the variant or the original webpage. Tracking viewer responses to the webpage and the variant allow a statistical basis for comparison to be developed.