Abstract: Systems and methods for automatically recovering from software regression in a cloud computing environment. One example method includes determining, with an electronic processor, that a batch software update has been applied to the cloud computing environment. The method includes, in response to determining that a batch software update has been applied, transmitting a problem request to an event listener server. The method includes receiving, from the event listener server, a problem statement including a stack trace. The method includes determining, based on the stack trace, a software feature indicator. The method includes transmitting the software feature indicator to a root cause analyzer. The method includes receiving, from the root cause analyzer, a change list indicator and a relevancy score associated with the change list indicator. The method includes performing a mitigation action based on the change list indicator when the relevancy score exceeds a relevancy threshold.

Abstract: Systems and methods are described for verifying functionality of software. A set of code that is to be validated is identified. A first configuration is determined for the set of code that configures the code as a first build for validation. The first build is released for a first validation process. Prior to completion of validation of the first build, a second configuration is determined for the set of code that configures the code as a second build for validation. The second build is released for a second validation process prior to completion of validation of the first build. The first and second validation process are staged so that the first and second builds can be reverted independently of one another in the event of a validation issue. The first and second validation process are independently completed in the absence of a validation issue.

Abstract: Traditionally, when a feature is updated or a new feature is released, the feature undergoes internal testing and validation before external distribution. However, some features may receive proportionately less internal usage than customer usage reflected externally. Low internal usage of features can lead to weak telemetry data, which can allow code regressions (e.g., bugs) to go undetected until the features are released to customers. Accordingly, accelerated internal feature usage is provided to mirror external customer usage. Highly used features are dynamically identified and, any deficiencies in internal feature usage are identified. Tenant sites estimated to generate at least a portion of the deficiency in feature usage are identified. These sites may be migrated or replicated to internal validation rings to generate additional internal feature usage.

Abstract: A method of and system for selecting users for a rollout process of a feature is carried out by receiving an indication of the rollout process for the feature being rolled out, accessing a rollout plan, the rollout plan including a plurality of stages for the rollout process, and selecting users from a user population for each of the plurality of stages of the rollout process. Selecting the users from a user population includes examining a property to determine if a user in the user population is indicated as opted into being a late-stage receiver, and upon determining that the user is opted into being the late-stage receiver, excluding the user from the user population for one or more early stages of the rollout and including the user into the user population in one or more late stages of the rollout process.

Abstract: Disclosed herein is a system for automating the causality detection process when upgrades are deployed to different resources that provide a service. The resources can include physical and/or virtual resources (e.g., processing, storage, and/or networking resources) that are divided into different, geographically dispersed, resource units. To determine whether a root cause of a problem is associated with an upgrade event that has recently been deployed, a system is configured to use telemetry data to compute an upgrade-to-upgrade score that represents differences between two different upgrade events that are deployed to the same resource unit. The system is further configured to use telemetry data to compute an upgrade unit-to-unit score that represents differences between the same upgrade event being deployed to two different resource units. The scores can be used to output an alert, for an analyst, that signals whether a recently deployed upgrade event is the cause of a problem.

Abstract: A method of and system for selecting users for a rollout process of a feature is carried out by receiving an indication of the rollout process for the feature being rolled out, accessing a rollout plan, the rollout plan including a plurality of stages for the rollout process, and selecting users from a user population for each of the plurality of stages of the rollout process. Selecting the users from a user population includes examining a property to determine if a user in the user population is indicated as opted into being a late-stage receiver, and upon determining that the user is opted into being the late-stage receiver, excluding the user from the user population for one or more early stages of the rollout and including the user into the user population in one or more late stages of the rollout process.

Abstract: Systems and methods for automatically reducing regression for a software payload applied to a plurality of computing platforms by a software updater. One example method includes receiving a health request associated with the payload, and retrieving, from an escalation engine, a plurality of identifiers identifying a subset of the plurality of computing platforms that have completed deployment of the payload, and determining a plurality of ULS tags associated with the payload. The method includes querying an anomaly detector for failure data, including pre and post-deployment data, for the subset corresponding to the ULS tags, detecting a potential software regression associated with the payload by comparing the pre and post-deployment data, and querying a root cause analyzer based on the potential regression. The method includes receiving an identifier identifying a potential root cause for the potential regression, and transmitting an event based on the potential regression and the potential root cause.

Abstract: Systems and methods are described for verifying functionality of software. A set of code that is to be validated is identified. A first configuration is determined for the set of code that configures the code as a first build for validation. The first build is released for a first validation process. Prior to completion of validation of the first build, a second configuration is determined for the set of code that configures the code as a second build for validation. The second build is released for a second validation process prior to completion of validation of the first build. The first and second validation process are staged so that the first and second builds can be reverted independently of one another in the event of a validation issue. The first and second validation process are independently completed in the absence of a validation issue.

Abstract: Traditionally, when a feature is updated or a new feature is released, the feature undergoes internal testing and validation before external distribution. However, some features may receive proportionately less internal usage than customer usage reflected externally. Low internal usage of features can lead to weak telemetry data, which can allow code regressions (e.g., bugs) to go undetected until the features are released to customers. Accordingly, accelerated internal feature usage is provided to mirror external customer usage. Highly used features are dynamically identified and, any deficiencies in internal feature usage are identified. Tenant sites estimated to generate at least a portion of the deficiency in feature usage are identified. These sites may be migrated or replicated to internal validation rings to generate additional internal feature usage.

Abstract: Disclosed herein is a system for automating the causality detection process when upgrades are deployed to different resources that provide a service. The resources can include physical and/or virtual resources (e.g., processing, storage, and/or networking resources) that are divided into different, geographically dispersed, resource units. To determine whether a root cause of a problem is associated with an upgrade event that has recently been deployed, a system is configured to use telemetry data to compute an upgrade-to-upgrade score that represents differences between two different upgrade events that are deployed to the same resource unit. The system is further configured to use telemetry data to compute an upgrade unit-to-unit score that represents differences between the same upgrade event being deployed to two different resource units. The scores can be used to output an alert, for an analyst, that signals whether a recently deployed upgrade event is the cause of a problem.

Abstract: Systems and methods relate to determining whether a particular failure of a web page feature is related to a recently implemented modification, and, when applicable, automatically taking one or more actions to return the web page to a functioning state (revert the modification). One embodiment provides a system for web page recovery. The system includes at least one server including an electronic processor and memory. The memory stores instructions that, when executed by the electronic processor, cause the system to obtain a web page log, detect a failure of a feature of a web page related to the web page log, identify one or more change packages related to the web page, determine a set of possible root causes for the failure of the feature based on the one or more change packages, select one possible root cause, and revert the one possible root cause.

Abstract: Systems and methods for determining a severity level of a computing service. One system includes an electronic processor that is configured to receive telemetry data associated with one or more tenants of an online service providing services through a plurality of computing services and calculate, based on the telemetry data, a number of accesses of each of the plurality of computing services during a predetermined time period. The electronic processor is also configured to, for each of the plurality of computing services, assign a severity level to each computing service based on the number of accesses of each computing service during the predetermined time period. The electronic processor is further configured to, in response to detecting a failure of one of the plurality of computing services, initiate a response to the failure based on the severity level assigned to the one of the plurality of computing services.

Abstract: Systems and methods for determining a severity level of a computing service. One system includes an electronic processor that is configured to receive telemetry data associated with one or more tenants of an online service providing services through a plurality of computing services and calculate, based on the telemetry data, a number of accesses of each of the plurality of computing services during a predetermined time period. The electronic processor is also configured to, for each of the plurality of computing services, assign a severity level to each computing service based on the number of accesses of each computing service during the predetermined time period. The electronic processor is further configured to, in response to detecting a failure of one of the plurality of computing services, initiate a response to the failure based on the severity level assigned to the one of the plurality of computing services.

Abstract: Systems and methods for automatically reducing regression for a software payload applied to a plurality of computing platforms by a software updater. One example method includes receiving a health request associated with the payload, and retrieving, from an escalation engine, a plurality of identifiers identifying a subset of the plurality of computing platforms that have completed deployment of the payload, and determining a plurality of ULS tags associated with the payload. The method includes querying an anomaly detector for failure data, including pre and post-deployment data, for the subset corresponding to the ULS tags, detecting a potential software regression associated with the payload by comparing the pre and post-deployment data, and querying a root cause analyzer based on the potential regression. The method includes receiving an identifier identifying a potential root cause for the potential regression, and transmitting an event based on the potential regression and the potential root cause.

Abstract: Systems and methods for automatically recovering from software regression in a cloud computing environment. One example method includes determining, with an electronic processor, that a batch software update has been applied to the cloud computing environment. The method includes, in response to determining that a batch software update has been applied, transmitting a problem request to an event listener server. The method includes receiving, from the event listener server, a problem statement including a stack trace. The method includes determining, based on the stack trace, a software feature indicator. The method includes transmitting the software feature indicator to a root cause analyzer. The method includes receiving, from the root cause analyzer, a change list indicator and a relevancy score associated with the change list indicator. The method includes performing a mitigation action based on the change list indicator when the relevancy score exceeds a relevancy threshold.

Abstract: Systems and methods relate to determining whether a particular failure of a web page feature is related to a recently implemented modification, and, when applicable, automatically taking one or more actions to return the web page to a functioning state (revert the modification). One embodiment provides a system for web page recovery. The system includes at least one server including an electronic processor and memory. The memory stores instructions that, when executed by the electronic processor, cause the system to obtain a web page log, detect a failure of a feature of a web page related to the web page log, identify one or more change packages related to the web page, determine a set of possible root causes for the failure of the feature based on the one or more change packages, select one possible root cause, and revert the one possible root cause.