Abstract: The disclosure is directed to optimizing data writes in a distributed computing system, e.g., a social networking application. The system employs a write-ahead log (WAL) and a hybrid logical clock (HLC) together to improve data writes across data centers in the distributed computing system, including improving: (1) consensus monitoring and enforcement in replicating data across the data centers; (2) scalability of data writes, e.g., so that different data centers can together handle a sudden increase in data writes during specified periods and consistently maintain the time ordering of the data writes; (3) replication configuration of data centers so that client devices can specify a replication hierarchy per transaction, e.g., at the time of writing the data, rather than specifying at data center configuration time; and (4) a draining process to write transactions committed to the WAL to data storage systems in the data centers.

Abstract: The disclosure is directed to optimizing data writes in a distributed computing system, e.g., a social networking application. The system employs a write-ahead log (WAL) and a hybrid logical clock (HLC) together to improve data writes across data centers in the distributed computing system, including improving: (1) consensus monitoring and enforcement in replicating data across the data centers; (2) scalability of data writes, e.g., so that different data centers can together handle a sudden increase in data writes during specified periods and consistently maintain the time ordering of the data writes; (3) replication configuration of data centers so that client devices can specify a replication hierarchy per transaction, e.g., at the time of writing the data, rather than specifying at data center configuration time; and (4) a draining process to write transactions committed to the WAL to data storage systems in the data centers.

Abstract: The disclosure is directed to optimizing data writes in a distributed computing system, e.g., a social networking application. The system employs a write-ahead log (WAL) and a hybrid logical clock (HLC) together to improve data writes across data centers in the distributed computing system, including improving: (1) consensus monitoring and enforcement in replicating data across the data centers; (2) scalability of data writes, e.g., so that different data centers can together handle a sudden increase in data writes during specified periods and consistently maintain the time ordering of the data writes; (3) replication configuration of data centers so that client devices can specify a replication hierarchy per transaction, e.g., at the time of writing the data, rather than specifying at data center configuration time; and (4) a draining process to write transactions committed to the WAL to data storage systems in the data centers.

Abstract: Technology is disclosed for establishing a querying system and load balancing the multi-tiered querying system. A multi-tiered targeted query system can comprise three tiers: a web tier, an aggregator tier, and a shard tier. When load balancing of shards is performed, fan-out can occur, increasing latency. The disclosed technology performs load balancing while minimizing the amount fan-out increase. Selecting the databases to move can comprise determining which databases on that server are causing the most load, and determining if moving any of these databases will increase an expected amount of fan-out above an acceptable threshold value. Determining the expected amount of fan-out increase incurred by moving a database can be based on an analysis of a number of friend relationships between that database and other databases on the same or other servers.

Abstract: Technology is disclosed for a multi-tiered querying system to target queries to systems storing data relevant to the query. A multi-tiered targeted query system comprises at least three tiers: a web tier, an aggregator tier, and a shards tier. Servers at the web tier can be configured to service user data requests and pass them to servers at the aggregator tier. Servers at the aggregator tier can be configured to determine which selected shard servers have the requested information; formulate queries for the selected shard servers; send the queries to the selected shard servers; and aggregate results from the selected shard servers. Servers at the shard tier can be configured to store data, receive queries on that data, and return results for received queries.

Abstract: The disclosure is directed to optimizing data writes in a distributed computing system, e.g., a social networking application. The system employs a write-ahead log (WAL) and a hybrid logical clock (HLC) together to improve data writes across data centers in the distributed computing system, including improving: (1) consensus monitoring and enforcement in replicating data across the data centers; (2) scalability of data writes, e.g., so that different data centers can together handle a sudden increase in data writes during specified periods and consistently maintain the time ordering of the data writes; (3) replication configuration of data centers so that client devices can specify a replication hierarchy per transaction, e.g., at the time of writing the data, rather than specifying at data center configuration time; and (4) a draining process to write transactions committed to the WAL to data storage systems in the data centers.

Abstract: Embodiments are disclosed for improving cache or memory efficiency of a social network system. A method according to some embodiments includes steps of: receiving an instruction to improve cache or memory efficiency of social graph data of a social graph; generating based on the social graph a partitioning tree including multiple bottom-level buckets, the partitioning tree dividing the vertices of the social graph into the bottom-level buckets and ordering the bottom-level buckets such that a social network metric regarding the vertices is optimized; assigning user IDs to the vertices of the social network in a numerical sequence based on the ordering of the bottom-level buckets; storing the social graph data of the users in storage locations in an order according to the numeral sequence of the assigned user IDs of the vertices.

Abstract: Systems and methods are provided for adding new nodes to a computer networked system. The systems and methods may identify a first set of nodes in a networked computer system. The first set of nodes may be included in a first hash computation that clusters the first set of nodes into communities. An application shard space including a first space and a second space may be generated. The first set of nodes may be mapped to application shards in the first space based on the first hash computation. The application shards in the first space may be assigned to a first set of machines of the networked computer system. The second space may be maintained for mappings of nodes not included in the first hash computation to application shards in the second space.

Abstract: A technology for routing traffic from similar users to a same server cluster to improve data center efficiency is disclosed. When a traffic routing server receives a request from a user, the traffic routing server determines an identifier of a partition to which the user is assigned. The user and many other users with whom the user shares a social attribute are co-located in the same partition. The traffic routing server then computes a hash of the identifier using a hash function and locates a server cluster on a consistent hash ring using the computed hash. The traffic routing server then sends the request from the user to that server cluster. By consistently sending requests from users assigned to the same partition to the same server cluster, the technology improves cache hit rates and reduces data duplication across the server clusters, which in turn improves datacenter efficiency.

Abstract: Systems and methods are provided for adding new nodes to a computer networked system. The systems and methods may identify a first set of nodes in a networked computer system. The first set of nodes may be included in a first hash computation that clusters the first set of nodes into communities. An application shard space including a first space and a second space may be generated. The first set of nodes may be mapped to application shards in the first space based on the first hash computation. The application shards in the first space may be assigned to a first set of machines of the networked computer system. The second space may be maintained for mappings of nodes not included in the first hash computation to application shards in the second space.

Abstract: Technology is disclosed for establishing a querying system and load balancing the multi-tiered querying system. A multi-tiered targeted query system can comprise three tiers: a web tier, an aggregator tier, and a shard tier. When load balancing of shards is performed, fan-out can occur, increasing latency. The disclosed technology performs load balancing while minimizing the amount fan-out increase. Selecting the databases to move can comprise determining which databases on that server are causing the most load, and determining if moving any of these databases will increase an expected amount of fan-out above an acceptable threshold value. Determining the expected amount of fan-out increase incurred by moving a database can be based on an analysis of a number of friend relationships between that database and other databases on the same or other servers.

Abstract: Embodiments are disclosed for improving cache or memory efficiency of a social network system. A method according to some embodiments includes steps of: receiving an instruction to improve cache or memory efficiency of social graph data of a social graph; generating based on the social graph a partitioning tree including multiple bottom-level buckets, the partitioning tree dividing the vertices of the social graph into the bottom-level buckets and ordering the bottom-level buckets such that a social network metric regarding the vertices is optimized; assigning user IDs to the vertices of the social network in a numerical sequence based on the ordering of the bottom-level buckets; storing the social graph data of the users in storage locations in an order according to the numeral sequence of the assigned user IDs of the vertices.

Abstract: Technology is disclosed for establishing a querying system and load balancing the multi-tiered querying system. A multi-tiered targeted query system can comprise three tiers: a web tier, an aggregator tier, and a shard tier. When load balancing of shards is performed, fan-out can occur, increasing latency. The disclosed technology performs load balancing while minimizing the amount fan-out increase. Selecting the databases to move can comprise determining which databases on that server are causing the most load, and determining if moving any of these databases will increase an expected amount of fan-out above an acceptable threshold value. Determining the expected amount of fan-out increase incurred by moving a database can be based on an analysis of a number of friend relationships between that database and other databases on the same or other servers.

Abstract: A real-time index system is provided for maintaining the consistency of an index of a database. The real-time index system can receive data associated with an inconsistency detected in the index, compute an index key value from the received data, and obtain the index record and data record associated with the computed index key value. The real-time index system can further compare the data in the data record with the corresponding data in the index record to identify a mismatch in data between the data in the index record and the data in the data record. The real-time index system can maintain the consistency of the index by modifying a portion of data in the index record to be consistent with a corresponding portion of data in the data record.

Abstract: Technology is disclosed for establishing a querying system and load balancing the multi-tiered querying system. A multi-tiered targeted query system can comprise three tiers: a web tier, an aggregator tier, and a shard tier. When load balancing of shards is performed, fan-out can occur, increasing latency. The disclosed technology performs load balancing while minimizing the amount fan-out increase. Selecting the databases to move can comprise determining which databases on that server are causing the most load, and determining if moving any of these databases will increase an expected amount of fan-out above an acceptable threshold value. Determining the expected amount of fan-out increase incurred by moving a database can be based on an analysis of a number of friend relationships between that database and other databases on the same or other servers.

Abstract: A technology for routing traffic from similar users to a same server cluster to improve data center efficiency is disclosed. When a traffic routing server receives a request from a user, the traffic routing server determines an identifier of a partition to which the user is assigned. The user and many other users with whom the user shares a social attribute are co-located in the same partition. The traffic routing server then computes a hash of the identifier using a hash function and locates a server cluster on a consistent hash ring using the computed hash. The traffic routing server then sends the request from the user to that server cluster. By consistently sending requests from users assigned to the same partition to the same server cluster, the technology improves cache hit rates and reduces data duplication across the server clusters, which in turn improves datacenter efficiency.

Abstract: Technology is disclosed for a multi-tiered querying system to target queries to systems storing data relevant to the query. A multi-tiered targeted query system comprises at least three tiers: a web tier, an aggregator tier, and a shards tier. Servers at the web tier can be configured to service user data requests and pass them to servers at the aggregator tier. Servers at the aggregator tier can be configured to determine which selected shard servers have the requested information; formulate queries for the selected shard servers; send the queries to the selected shard servers; and aggregate results from the selected shard servers. Servers at the shard tier can be configured to store data, receive queries on that data, and return results for received queries.

Abstract: A real-time index system is provided for maintaining the consistency of an index of a database. The real-time index system can receive data associated with an inconsistency detected in the index, compute an index key value from the received data, and obtain the index record and data record associated with the computed index key value. The real-time index system can further compare the data in the data record with the corresponding data in the index record to identify a mismatch in data between the data in the index record and the data in the data record. The real-time index system can maintain the consistency of the index by modifying a portion of data in the index record to be consistent with a corresponding portion of data in the data record.