Abstract: The disclosed online system includes a number of processing blocks, including a feeding service that sends a data stream and a consuming service that receives a data stream, without sending back acknowledgements for the data (i.e., “nontransactional”). The system handles failure in the feeding service and/or the consuming service by adding reference points into the data stream and backing up a current sum that is maintained by the consuming service. Upon a failure of the consuming service, the system obtains the last reference point, restores the backup copy of the database, and starts reading the backup copy from that check point. To address a failure of the feeding service, the feeding service creates an identifier for each message in the data stream, and upon failure, the consuming service checks for messages that have the same identifier within a specified period of time, and discards any duplicate messages.

Abstract: The disclosed online system includes a number of processing blocks, including a feeding service that sends a data stream and a consuming service that receives a data stream, without sending back acknowledgements for the data (i.e., “nontransactional”). The system handles failure in the feeding service and/or the consuming service by adding reference points into the data stream and backing up a current sum that is maintained by the consuming service. Upon a failure of the consuming service, the system obtains the last reference point, restores the backup copy of the database, and starts reading the backup copy from that check point. To address a failure of the feeding service, the feeding service creates an identifier for each message in the data stream, and upon failure, the consuming service checks for messages that have the same identifier within a specified period of time, and discards any duplicate messages.

Abstract: In a distributed system a plurality of devices (including computing units, storage and communication units) are monitored by an automated repair service that uses sensors and performs one or more repair actions on computing devices that are found to fail according to repair policies. The repair actions include automated repair actions and non-automated repair actions. The health of the computing devices is recorded in the form of states along with the repair actions that were performed on the computing devices and the times at which the repair actions were performed, and events generated by both sensors and the devices themselves. After some period of the time, the history of states of each device, the events, and the repair actions performed on the computing devices are analyzed to determine the effectiveness of the repair actions.

Abstract: In a distributed system a plurality of devices (including computing units, storage and communication units) are monitored by an automated repair service that uses sensors and performs one or more repair actions on computing devices that are found to fail according to repair policies. The repair actions include automated repair actions and non-automated repair actions. The health of the computing devices is recorded in the form of states along with the repair actions that were performed on the computing devices and the times at which the repair actions were performed, and events generated by both sensors and the devices themselves. After some period of the time, the history of states of each device, the events, and the repair actions performed on the computing devices are analyzed to determine the effectiveness of the repair actions.

Abstract: Verification of Internet connectivity using multiple prior connection attempts to Internet destination(s). The Internet destinations may be destinations that have high reliability and that do not easily have intermediating systems that might deny a connection request. Such an Internet destination might be, for example, root Domain Name Server (DNS) servers. Connection attempt results are obtained by for at least some of the connection attempts, tracking which resulted in success and failure. Internet connectivity is then verified based on the collective results, rather than relying on any one single connection attempt. In one embodiment, the frequency of the connection attempts may depend on a current state of the Internet connection.

Abstract: Verification of Internet connectivity using multiple prior connection attempts to Internet destination(s). The Internet destinations may be destinations that have high reliability and that do not easily have intermediating systems that might deny a connection request. Such an Internet destination might be, for example, root Domain Name Server (DNS) servers. Connection attempt results are obtained by for at least some of the connection attempts, tracking which resulted in success and failure. Internet connectivity is then verified based on the collective results, rather than relying on any one single connection attempt. In one embodiment, the frequency of the connection attempts may depend on a current state of the Internet connection.