Abstract: A system, apparatus, and methods are provided for balancing partition distribution across nodes within a message broker cluster so as to balance the broker nodes' workloads. During operation, the system receives a stream of messages at the cluster, wherein the message stream is divided into topics, the topics are divided into partitions, and replicas for each partition are distributed among the nodes of the message brokering cluster. Upon detection of an imbalance in the nodes' workloads by a monitor (e.g., as indicated by uneven resource consumption), an analyzer considers various possible remedies (e.g., reassigning/demoting/promoting a replica), estimates their likely impacts on the workload, and determines whether they satisfy hard and/or soft goals of the system. The analyzer generates a plan that satisfies the hard goals and that may satisfy some or all soft goals, and passes it to an executor for implementation.

Abstract: The disclosed embodiments provide a system for facilitating interaction within a social network. During operation, the system identifies a first member of a social network with a first activity level that is lower than a threshold. Next, the system uses the first activity level to boost a score associated with recommending an interaction with the first member to a second member of the social network. The system then uses the boosted score to generate output for modulating subsequent interactions in the social network.

Abstract: A system, apparatus, and methods are provided for self-healing and balancing partition distribution across nodes within a message broker cluster. During operation, the system receives a stream of messages at the message brokering cluster, wherein the message stream is divided into partitions and replicas for each partition are distributed among a set of nodes within the message brokering cluster. Responsive to a change in the number of nodes within the message brokering cluster, the system (1) determines a set of replicas to be migrated within the message brokering cluster, (2) divides the set of replicas into multiple chunks, wherein each chunk includes one or more of the replicas to be migrated to a new node, and (3) migrates the set of replicas one chunk at a time, wherein replicas not corresponding to the single chunk do not begin migrating until all replicas within the single chunk finish migrating.

Abstract: The disclosed embodiments provide a system for facilitating interaction within a social network. During operation, the system obtains a set of features associated with two members of a social network, wherein the features comprise a member feature and an activity feature. Next, the system analyzes the features to predict an effect of a potential interaction between the two members on subsequent interactions between the two members in the social network. The system then uses the predicted effect to generate output for modulating the subsequent interactions in the social network.

Abstract: The disclosed embodiments provide a system that verifies correctness in a graph database. During operation, the system obtains a set of records from a source of truth for a graph database storing a graph, wherein the graph includes a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. Next, the system uses the records to automatically generate a set of test cases containing a set of queries of the graph database. The system then transmits the queries to the graph database and receives, from the graph database, a set of query results in response to the queries. Finally, the system performs a comparison of the query results and a set of expected results of the test cases to verify a correctness of the graph database.

Abstract: The disclosed embodiments provide a system that searches a graph database. During operation, the system obtains a log-based representation of a graph database storing a graph, wherein the graph comprises a set of nodes, a set of edges between pairs of nodes in the set of nodes, and a set of predicates. Next, the system obtains a first query containing a first search pattern for searching the log-based representation. The system then matches the first search pattern to one or more records in the log-based representation. Finally, the system outputs, as a first result of the first query, the one or more records in a subgraph of the graph.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of partitions containing a set of records, wherein the records include a set of values for a measure and a set of dimensions associated with the values. Next, the system reorganizes the records across the partitions by performing a distributed sort of the records by the measure. For each dimensional subset in the records, the system counts occurrences of the dimensional subset in each of the partitions and groups values of the counted occurrences by the dimensional subset so that the values reside in a single processing node. The system uses the values to identify one or more locations in the partitions for calculating a statistic for the dimensional subset and uses the location(s) to calculate the statistic. Finally, the system outputs the statistic in response to a query containing the dimensional subset.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a count of actions associated with a member of a social network, wherein the count comprises a set of action types and a number of actions associated with the member for each of the action types. Next, the system uses the count to calculate a set of scores for measuring the value of the social network to the member, wherein the set of scores is associated with researching and contacting people, building a network, keeping up with connections, staying informed and building knowledge, establishing and managing a reputation, and getting hired. The system then outputs the scores for use in characterizing and improving the value of the social network for the member.

Abstract: The disclosed embodiments provide a system for performing data management. During operation, the system obtains a first schema with a first syntax for describing a first data set and a second schema with a second syntax for describing a second data set. Next, the system converts the first schema into a first standardized form with a standardized syntax and the second schema into a second standardized form with the standardized syntax. The system then outputs the first and second standardized forms for use in accessing the first and second data sets.

Abstract: The disclosed embodiments provide a system for managing execution of a data flow. During operation, the system obtains a data dependency description for a data flow, wherein the data dependency description includes data sources to be consumed by the data flow, data targets to be produced by the data flow, and one or more data ranges associated with the data sources and the data targets. Next, the system uses the data dependency description to determine an availability of the data sources in an execution environment. After the availability of the data sources in the execution environment is confirmed, the system generates output for initiating execution of the data flow in the execution environment.

Abstract: A system, method, and apparatus are provided for repartitioning a topic of a publish-subscribe message system. The topic is originally configured with N partitions (N>1) hosted by multiple brokers for storing messages to be consumed by multiple consumers. The repartitioning process causes one or more collections of partitions to be created in addition to the original collection of N partitions. Afterward, when a new message is received for the topic and has an associated partition key, the key is processed (e.g., hashed) once to identify a target collection of partitions and is then reprocessed to identify a destination partition within the target collection. Consumers may be automatically subscribed to new partitions. For example, a consumer subscribed to the ith partition of the original N partitions may be subscribed to the ith partition of each additional collection of partitions.

Abstract: The disclosed embodiments provide a system for maximizing the propensity of content consumption according to content complexity and content consumption time. During operation, the system receives a request from a user to access a content feed. Based on a time of receipt, the system identifies a current seasonality that corresponds to reading habits of the user during a time period encompassing the time of receipt. The system then accesses a profile for the user to obtain a reading speed of the user and the reading habits of the user for the current seasonality. Next, the system: identifies a subset of content items; estimates a reading session length; and determines a complexity, with regard to the user, for each content item. The system then creates the content feed by ordering the subset of content items according to the reading session length and the complexity of each content item.

Abstract: A system, method, and apparatus are provided for performing set operations on variable-length data that has been stored and indexed to facilitate reverse reading. The set operation is performed on data records having any or all of multiple key/value pairs—such as a union of all data records having the key/value pairs, an intersection, or a difference. For each key/value pair, a tracking entity is initialized to identify a most recently stored record that has the key/value pair. Each record contains a key offset for each key that is common to the records, and the key offset stores an identity (e.g., offset to) the next most recently stored record that has the same value for the key. Each tracking entity is used to support traversal of the data records having the associated key/value pair, during which the set operation is performed to add qualifying records to a result.

Abstract: The disclosed embodiments provide a system for improving use of a social network. During operation, the system obtains a set of member features associated with a member of a social network and a set of attribute features associated with a set of member attributes. Next, the system analyzes the member features and the attribute features to predict a propensity of the member to accept recommendations of the member attributes as profile edits to a member profile of the member. The system then uses the predicted propensity to output a subset of the member attributes as recommended profile edits to the member.

Abstract: The disclosed embodiments provide a system for improving use of a social network. During operation, the system obtains user feedback associated with recommending a set of member attributes as profile edits to a set of members in a social network. Next, the system analyzes the user feedback to determine a set of acceptance rates of the member attributes. The system then uses the acceptance rates to update a taxonomy of the member attributes for use in improving recommendations of the member attributes to the members.

Abstract: A system, method, and apparatus are provided for limiting or preventing write amplification. Full documents (which may be stored as blobs) are stored as monolithic entities in that only the entire documents can be written or read. When one or more fields or other portions of a document are determined to be “hot,” those fields are stored in a separate structure, possibly adjacent to the document. When only these fields are updated, the document need not be rewritten. For read requests, both structures are read and merged to yield a true version of the document. Fields may be considered hot based on criteria such as frequency of access, ratio of the fields' size to the size of the document, type of document, the application associated with the document, etc.

Abstract: The disclosed embodiments provide a system for processing textual content. During operation, the system obtains a content item containing a set of text units. For each text unit in the set of text units, the system obtains a similarity score representing a similarity of the text unit to other text units in the content item and calculates a ranking score for the text unit from a combination comprising a text unit frequency for the text unit, the similarity score, and a position weight associated with a position of the text unit in the content item. The system then ranks the set of text units by the ranking score and uses the ranking to display a summary containing a subset of the text units in the content item.

Abstract: The disclosed embodiments provide a system for performing skill-based recommendation of events. During operation, the system obtains member attributes for a member of an online professional network. Next, the system matches the location of the member and one or more of the member attributes to event attributes of a set of events. The system then uses the member attributes and the event attributes to calculate a set of relevance scores representing a relevance of the events to the member. Finally, the system uses the set of relevance scores to output one or more of the events as recommendations to the member.

Abstract: The disclosed embodiments provide a system for filtering irrelevant actor updates from content feeds. During operation, the system identifies a set of candidate content items to display to a user. Then, for each content item in the set of candidate content items, the system obtains a content score for the content item, determines a second user associated with the content item, determines a propensity for the user to click on content associated with the second user, and applies a penalty to the content score when the propensity is less than a pre-determined threshold. Finally, the system orders the set of candidate content items according to the content scores, and forwards the ordered set of content items for display to the user.

Abstract: A system, method, and apparatus are provided for propagating a status among related events. A relationship component of the system maintains a graph of the related events and, upon assignment of a status to an event identifies one or more related events (e.g., the root event, other ancestor events, descendant events). A retention component of the system includes multiple nodes that maintain user records (e.g., sharded by user identifier) storing events associated with the members and that also maintain separate event status tables to identify statuses of selected events. For example, an event status table may store statuses of root events and/or other ancestor events from which a given subsequent event may inherit or derive a status. An event status may reflect a quality or characteristic of the event (e.g., spam, low quality, language of the event), a status of the associated user (e.g., fame, level of influence), etc.

Abstract: A system, method, and apparatus are provided for capturing a snapshot of variable-length data records that are indexed and sequentially stored in a manner that facilitates reverse reading. Each data record has a fixed number of keys, a key offset for each key that leads to another record with the same key value, and size metadata identifying a size of the data record (and possibly the key offsets). An index identifies, for each known value of each key, an index offset to a first entry (e.g., the most recently stored entry) that has the key value. Capturing a snapshot includes identifying a final record within the snapshot (e.g., based on time), copying the index, and pruning it as necessary to omit records not consistent with the snapshot (e.g., to omit data records stored after a final time corresponding to the snapshot).

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of overall scores representing predicted purchase behaviors of a set of customers with an educational technology product. Next, the system displays a graphical user interface (GUI) comprising a customer prioritization chart for the educational technology product. The system then displays representations of the overall scores in the customer prioritization chart. Finally, the system displays, in the GUI, the set of overall scores and a breakdown of the overall scores into a set of sub-scores that characterize different components of the overall scores.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of features for a customer of an educational technology product. Next, the system uses the set of features to calculate an overall score representing a predicted purchase behavior of the customer with the educational technology product. The system then uses multiple subsets of the features to calculate a set of sub-scores that characterize different components of the overall score. Finally, the system outputs the overall score and the sub-scores for use in managing sales activity with the customer.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a set of records from a set of inputs, with each record containing an entity key, a partition key, and one or more attribute-value pairs. Next, the system sorts and indexes the records by the entity key for each partitioned input. The system then processes a query of the records by matching entity key values to the sorted and indexed records for each input partition in the query. Next, the system merges the subset of records with the same entity key values into records, with each merged record containing an entity key field and a single field that includes a list of attribute-value pairs from the subset. Finally, the system outputs the merged records in response to the query.

Abstract: A system, method, apparatus and graphical user interface are provided for navigating a multi-component application or service—such as an application that executes on a mobile device and that includes multiple features, pages, content items, utilities, settings, etc. A shortcut bar provides rapid access to application components that correspond to multiple shortcuts, and the shortcut bar can be opened, by swiping it, to reveal a set of options associated with an open shortcut (e.g., the shortcut closest to where the user swiped on the bar). The options can be scrolled (e.g., vertically) or swiped in another direction (e.g., sideways) to access options associated with other shortcuts. The shortcut bar and the options provide set paths of navigation to desired content, without regard to the current context (e.g., the displayed content). Navigation paths to the content via the current context will vary, depending on the displayed content.

Abstract: A system, apparatus, and methods are provided that support the passing of oversized messages within a publish-subscribe messaging system. During operation, the system subscribes to a message stream brokered by a message brokering cluster that imposes a maximum message size. Responsive to receiving a given message of the message stream, the system (1) determines an offset to commit to the message brokering cluster, wherein the determined offset is based on whether one or more segments of one or more oversized messages are buffered, and (2) commits the determined offset to the message brokering cluster. Responsive to recovering from a crash, wherein the offset at which to resume the receipt of messages from the message stream is unknown, the system then retrieves the determined offset from the message brokering cluster and resumes the receipt of messages from the message brokering cluster based on the determined offset.

Abstract: The disclosed embodiments provide a system for detecting and managing inefficiency in external services. During operation, the system obtains a snapshot of a data stream transmitted over an external service from a computer system at a first time. Next, the system applies a compression technique to the snapshot to obtain a set of inefficiency metrics for a set of data elements in the snapshot. The system then outputs the set of inefficiency metrics with additional attributes of the data stream to improve identification of inefficiency in the data stream.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system uses a creation time of a first value of a metric from a nearline data store to obtain a second value of the metric from a source of truth. Next, the system calculates a difference between the first and second values. When the difference exceeds a threshold, the system uses the difference to correct a current value of the metric in the nearline data store.

Abstract: The disclosed embodiments provide a system for detecting and managing inefficiency in local storage. During operation, the system obtains a first snapshot of data in local storage of a computer system, wherein the first snapshot comprises a first set of data elements in the local storage at a first time. Next, the system applies a compression technique to the first snapshot to obtain a first set of inefficiency metrics for the first set of data elements. The system then outputs the first set of inefficiency metrics with additional attributes of the data to improve management of inefficiency in the data.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a global version of a statistical model and a user-specific version of the statistical model for a user. Next, the system applies the global version to member features of the user and job features of a set of jobs to generate a first ranking of the jobs for the user. The system then applies the user-specific version to the member features and the job features for a highest-ranked subset of jobs in the first ranking to generate a second ranking of the jobs for the user. Finally, the system outputs at least a portion of the second ranking as a set of job recommendations.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.

Abstract: The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

Abstract: A system is provided that ensures read-after-write consistency. During operation, the system receives, from a user, a write to a record having a primary key in a master key-value store, wherein the write specifies a secondary key for the record. The system then caches the secondary key and the primary key in a cache entry in a cache, wherein the cache entry is associated with the user. Next, the system applies the write to the master key-value store. Prior to propagation of the write from the master key-value store to a derived key-value store that maps secondary keys to primary keys, the system receives from a given user a query for the record, the query comprising the secondary key and not the primary key. Next the system translates the secondary key to the primary key by querying the cache when the given user is the user.

Abstract: A system, apparatus, and method are provided for implementing secondary profiles for members of an online application or service. Each member has a corresponding primary profile populated by the member, and a secondary profile populated with information from data sources other than the member. Each fact or entry in the secondary (or inferred) profile is accompanied by a confidence score reflecting confidence in the source of the fact, confidence that the fact is correctly associated with this member, and/or other factors. A given fact may be obtained or extracted from multiple sources, with each copy or version assigned a separate confidence score. In response to a request to identify members having a particular attribute, in addition to identifying members that have the attribute in their primary profiles, members having the attribute in their secondary profiles may be identified if the corresponding confidence scores are greater than a threshold.

Abstract: The disclosed embodiments provide a system for operating a switch fabric. During operation, the system identifies network traffic for transmission between two access switches in a switch fabric. Next, the system selects a subset of the network traffic for forwarding on an expedited fabric path comprising a physical link between the two access switches that isolated from other physical links in the switch fabric. Next, the system forwards the subset of the network traffic on the expedited fabric path.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a first set of rules for assigning a first set of sales professionals to a first set of accounts, wherein the first set of rules comprises a representative load rule, a matching rule, and a balancing rule. Next, the system applies an optimization technique to the first set of rules and a first set of parameters associated with the first set of sales professionals and the first set of accounts to produce a first set of assignments of the first set of sales professionals to the first set of accounts. The system then outputs the first set of assignments for using in managing sales activity of the first set of sales professionals.

Abstract: A system, apparatus, and method are provided for implementing secondary profiles for members of an online application or service. Each member has a corresponding primary profile populated by the member, and a secondary profile populated with information from data sources other than the member. Each fact or entry in the secondary (or inferred) profile is accompanied by a confidence score reflecting confidence in the source of the fact, confidence that the fact is correctly associated with this member, and/or other factors. A given fact may be obtained or extracted from multiple sources, with each copy or version assigned a separate confidence score. In response to a request to identify members having a particular attribute, in addition to identifying members that have the attribute in their primary profiles, members having the attribute in their secondary profiles may be identified if the corresponding confidence scores are greater than a threshold.

Abstract: A system, method, and apparatus are provided for indexing and sequentially storing variable-length data in a manner that facilitates reverse reading. Each entry stored in a log file, database, or other repository includes a data record having a fixed number of keys, a key offset corresponding to each key, and size metadata identifying a size of the data record (and possibly the key offsets). Each key offset is an offset to another entry (e.g., the matching key offset of the entry) whose data record features the same value for the corresponding key. An index identifies, for each given value of each key, an index offset to a first entry (e.g., the most recently stored entry) that has the given value for that key. Retrieving records matching a particular key value therefore simply involves following the corresponding index offset and then some number of key offsets.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a first configuration for processing a first set of content items from a first data source and a second configuration for processing a second set of content items from a second data source. For each content item in the first set of content items, the system uses mappings from the first configuration to transform original fields from the content item into required fields in a record representing the content item. Next, the system generates, from the required fields, a document key for the content item. The system also performs deduplication of multiple records with the document key and stores a single record with the document key. Finally, the system uses the second configuration to generate, from the second set of content items, a set of records independently of processing the first set of content items.

Abstract: A system, apparatus, and methods are provided for managing on-demand profiling of one or more instances of a software application executing on a plurality of machines within one or more data centers. During operation, the system executes the one or more instances of the software application on the plurality of machines. Next, the system publishes, to a command channel, a command message that comprises a profiling request, wherein the profiling request specifies a subset of the machines. The system then receives, via a data channel, one or more data messages from the subset of the machines, wherein the data messages comprise data gathered by the subset of the machines in response to receiving the command message. Next, the system then evaluates the performance of the software application by aggregating and processing the data messages. Responsive to detecting an anomaly in the performance, the system then executes one or more remedies.

Abstract: A system, method, and apparatus are provided for identifying clients or consumers of one or more controlled resources within a computing environment (e.g., a data center, a computing cluster). The resources may be controlled via the enforcement of quotas, wherein different clients have different quotas for the controlled resource(s). A client associated with a given call to a resource is identified using a tuple formed from identifiers of multiple services in the call stack, such as a first service (e.g., a front-end or user-facing service) and an immediate service (e.g., a final service in the stack prior to receipt of the call at a gateway, broker, or other entity that manages access to the controlled resources).

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a first transaction timestamp from a record that is replicated across a set of data sources. Next, the system includes at least a portion of the first transaction timestamp in a first key of a first mapping of the record to a first set of values for the record from the set of data sources. The system then audits the record by comparing the first set of values in the first mapping. Finally, the system outputs a result of the audited record based on the compared first set of values.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains input data containing a set of replicated records from a set of data sources. Next, the system generates, in a data store, a first mapping of a first key to a first set of values for a first replicated record in the set of replicated records. The system then audits the input data by comparing the first set of values in the first mapping. Finally, the system outputs a result of the audited input data based on the compared first set of values.

Abstract: A system, method, and apparatus are provided for locally enforcing quotas on resources within a computing environment (e.g., a data center, a computing cluster) that includes multiple brokers, gateway servers, or other entities that control access to the resource(s). Instead of using global access statistics provided by all gateways, each server individually allows or denies requested access to a resource based on whether the requesting client has already exceeded its quota (or would exceed its quota if the request is approved). A client associated with a given call to a resource may be identified using a tuple formed from identifiers of multiple services in the call stack, such as a first service (e.g., a front-end or user-facing service) and an immediate service (e.g., a final service in the stack prior to receipt of the call at a gateway, broker, or other entity that manages access to the controlled resources).

Abstract: A system, method, and apparatus are provided for visualizing quotas on resources within a computing environment (e.g., a data center, a computing cluster) that includes multiple brokers, gateway servers, or other entities that control access to the resource(s). Through a visualization interface, an operator of a quota system can set, change, and/or delete quotas on resources and/or sub-resources for any number of clients or consumers of the resources. The visualization interface may also enable access to a log of changes to quotas and/or graphs or other displays of statuses of any selected quotas for any selected clients and resources/sub-resources.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system generates a first distribution of a set of partitions comprising a graph database across a first set of storage nodes in a first cluster. Next, the system replicates the graph database by generating a second, different distribution of the set of partitions across a second set of storage nodes in a second cluster. The system then identifies one or more partitions storing data associated with the query and uses a set of mappings comprising the set of partitions, the first and second sets of storage nodes, and the first and second clusters to select one or more storage nodes containing the one or more partitions. Finally, the system transmits one or more portions of the query to the selected storage nodes.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system identifies a query type of a query of a graph database storing a graph, wherein the graph comprises nodes, edges between pairs of nodes, and predicates. Next, the system determines, based on the query type, a number of clusters on which the graph database is replicated for fanning out of the query. The system then selects a set of storage nodes in the determined number of clusters for processing of the query. Finally, the system generates a fan-out of the query to the selected storage nodes.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system uses one or more queries of a data set to aggregate one or more measurements of user engagement with a set of content items within a second-degree network of a member. Next, the system calculates one or more metrics from the one or more measurements. The system then ranks the set of content items by the one or more metrics. Finally, the system uses the ranking to present one or more of the content items in a content feed to the member.

Abstract: The disclosed embodiments provide a system for processing data. During operation, the system calculates one or more quantiles from a distribution of historic user engagement with content within a member segment of a social network. Next, the system performs a comparison of the one or more quantiles with a current measurement of user engagement with a content item within the member segment. When the comparison indicates that the current measurement of user engagement exceeds a threshold, the system presents the content item to one or more members in the member segment.

Abstract: The disclosed embodiments provide a system for facilitating interaction within a social network. During operation, the system obtains a set of attributes of a social network of a first member and a set of historic interactions in the social network. Next, the system analyzes the attributes and the historic interactions to predict an effect of a potential interaction between the first member and a second member of the social network on subsequent interactions in the social network. The system then uses the predicted effect to generate output for modulating the subsequent interactions in the social network.