Abstract: An encoder system trains a compression model that includes an autoencoder model and a frame extractor model. The encoding portion of the autoencoder is coupled to receive a set of target frames and a previous state tensor for the set of target frames and generate compressed code. The decoding portion of the autoencoder is coupled to receive the compressed code and the previous state tensor for the set of frames and generate a next state tensor for the set of target frames. The frame extractor model is coupled to receive the next state tensor and generate a set of reconstructed frames that correspond to the set of target frames by performing one or more operations on the state tensor. The state tensor for the set of frames includes information from frames of the video that can be used by the frame extractor to generate reconstructed frames.

Abstract: An online system receives a request from a target user to present a poll to other users of the online system. The poll includes a question and answer options associated with the question. The online system provides the poll for display to the other users and receives a selection of an answer option from each of the other users. Also, the online system receives comments associated with the poll from some of the other users. The online system qualifies the comments. For example, the online system generates a comment group for each of the answer options that includes comments from users who selected the answer option. As another example, the online system generates an indication for each comment that shows which answer options was selected by the user having provided the comment. The online system provides the comment groups and/or answer indications to the target user and the other users.

Abstract: Mechanisms are provided, in a hierarchical feedback aggregation (HFA) system implemented in one or more data processing systems, for collecting and presenting user feedback information for a composite offering. A backend engine of the HFA system, implemented in a first data processing system, registers a hierarchical feedback model for the composite offering. A frontend engine of the HFA system, implemented in a second data processing system, receives user feedback for an identified component of the composite offering. The backend engine of the HFA system generates an aggregate user feedback score for the identified component based on a combination of the user feedback for the identified component and aggregate user feedback scores for child components of the identified component in the hierarchical feedback model. The backend engine outputs a representation of the generated aggregate user feedback score for the component to a user.

Abstract: Methods and apparatus to generate consumer data are disclosed. An example method of selecting a sample of transaction data corresponding to a membership program includes defining a first type of member of the membership program; defining a second type of member of the membership program; calculating, via a processor, a target for the sample; selecting, via the processor, a first portion of the transaction data for the first type of member in accordance with the target; generating, via the processor, an updated target by recalculating the target with the first portion of the transaction data removed from consideration; and selecting, via the processor, a second portion of the transaction data for the second type of member in accordance with the updated target.

Abstract: A method comprises displaying visual representations of a plurality of product alternatives each including at least one attribute variant to a respondent, receiving from the respondent an indication of a preferred one of the plurality of product alternatives, transmitting a request to the respondent to identify at least one attribute variant of a non-preferred product alternative that is preferred by the respondent to the corresponding attribute variant of the preferred one of the plurality of product alternatives and receiving a response from the respondent identifying at least one attribute variant of a non-preferred product alternative that is preferred by the respondent to the corresponding attribute variant of the preferred one of the plurality of product alternatives.

Abstract: A device configured to obtain at least a portion of a phone call and to identify a voice signal associated with a person on the phone call. The device is further configured to generate metadata for the phone call and a transcript for the phone call. The device is further configured to input the transcript and the metadata into a machine learning model and to receive a call profile from the machine learning model. The call profile includes a first call classification for the phone call. The device is further configured to identify a call log associated with the phone call that includes a second call classification for the phone call. The device is further configured to determine that the first call classification does not match the second call classification, to generate a feedback report that identifies the first call classification, and to output the feedback report.

Abstract: An item recommendation system includes a candidate item database system, a communication module, and a processor. Candidate items to be recommended to a user are stored in the candidate item database system in association with the access items accessed by the user. The communication module acquires one or more access items accessed by the user recently. The processor acquires one or more candidate items and correlation indexes thereof from the candidate item database system based on the one or more access items, and generates an item recommendation list, wherein the item recommendation list comprises the one or more candidate items and recommendation weights thereof, and the recommendation weights are determined based on the correlation indexes.

Abstract: Systems, methods, architectures, and computer program products for linking multiple devices are disclosed. In an example for linking a mobile device with a desktop device, an identifier of a mobile device can be received from a desktop computer. The identifier can be used to send a link to the mobile device. When the link is accessed, a code and a channel are generated. The mobile device is connected to the channel and the code is provided to the mobile device. The code is entered at the desktop device and the desktop device is connected to the channel responsive to the code being validated, thereby linking the desktop and mobile devices.

Abstract: The present invention relates to system and method for evaluating reviewer's ability to provide feedback. The system receives feedback given by the reviewer that includes qualitative feedback and quantitative feedback. The system performs scoring of qualitative feedback to evaluate level of noise, suggestions, appreciation, specificity and duplicate comments in the qualitative feedback. Further, the system performs scoring of quantitative feedback that includes realistic score, softness score and critical nature score. Subsequently, the scores of qualitative feedback and quantitative feedback are aggregated to provide a rank to the reviewer for the reviewer's ability to provide feedback.

Abstract: In one embodiment, a method includes sending, to a first client system of a first user, a first trending module having references to a first set of trending topics, where one of the references does not include a live badge; generate a trending-topic interface corresponding to a first trending topic in the first set of trending topics, wherein the trending-topic interface comprises a live module containing at least one live video associated with the first trending topic; storing a reference to the live module with a news-event object; querying the news-event object to determine whether a live module is associated with the first trending topic; and sending, to a second client system for display, a second trending module having references to a second set of trending topic that includes the first trending topic, wherein a reference to the first trending topic includes a live badge.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, to present a video. One of the methods includes obtaining one or more unstructured documents. The method includes obtaining, by a computer system, a data model, the data model identifying a type of fact that can be determined from the one or more unstructured documents. The method includes determining, by the computer system, a channel to extract facts from the document based on the type of fact. The method includes distributing, by the computer system, the one or more unstructured documents to the channel. The method includes extracting, by the channel, facts from the one or more unstructured documents. The method also includes storing the facts in a data model.

Abstract: Disclosed herein are system, method, and computer program product embodiments for optimizing the performance of a launchpad that launches web-client applications. Maintaining requisite MIME files for a standardized web framework within a single iframe and launching subsequent applications within that iframe can improve the performance of the launchpad. The MIME files may persist within the single iframe across the suite of applications, and further downloading of files as the user launches further applications in the suite of applications may not be required.

Abstract: Techniques are described for genomically defining digital genes encoding data visualization elements and potential incremental changes to the elements as the basis for a genetic selection process for automated generating of data visualizations. In one aspect, a method includes receiving set of input data. The method further includes generating digital genes that genomically define data visualization elements based on the input data, and that define potential incremental changes to the data visualization elements. The method further includes executing a genetic selection process with respect to one or more fitness functions on populations of candidate data visualizations that are based on the genomically defined data visualization elements. The method further includes outputting final data visualization output generated by the genetic selection process.

Abstract: Techniques are described for genomically defining digital genes encoding data visualization elements and potential incremental changes to the elements as the basis for a genetic selection process for automated generating of data visualizations. In one aspect, a method includes receiving set of input data. The method further includes generating digital genes that genomically define data visualization elements based on the input data, and that define potential incremental changes to the data visualization elements. The method further includes executing a genetic selection process with respect to one or more fitness functions on populations of candidate data visualizations that are based on the genomically defined data visualization elements. The method further includes outputting final data visualization output generated by the genetic selection process.

Abstract: The foregoing are among the objects attained by the invention, which provides, in some aspects, digital data processing methods for generation of customized user interfaces that present links, images or other components representing items of interest to a user in an order that is prioritized as a function of (a) representations in a multidimensional factor space of characteristics of the respective items, and (b) representations in that same multidimensional space of characteristics of “context-indicative items”—e.g., items in which the user has previously shown an interest, as indicated by clicks or other interactions with those items respective components in the user interface.

Abstract: A system and methodology which can effectively provide decision makers with a better means of making decisions in a way that greatly improves the availability, reliability, and relevance of the information which they provide and use to make decisions. The system and methodology facilitates maximizing mutual utility in the context of a mutual decision between multiple users and groups of users identified generally as Parties and Counterparties and performs user specified actions based on meeting mutual threshold parameters. The system provides significant technical advantages over the prior art in that it uses helps Parties and Counterparties identify optimal arrangements and configurations with less errors, fewer computational cycles, less storage medium, and a smaller amount of time than would be possible using prior art systems.

Abstract: An interactive media management server that facilitates management of interactive media and delivery of such interactive media. It comprises a first interface to an interactive media distribution server communicatively coupled to a plurality of mobile devices, and a second interface to a service provider system or a content provider system. It comprises a manager module for managing creation, distribution and tracking of a plurality of interactive media. The manager module, via the second interface, manages the delivery of the plurality of interactive media from the service provider system or the content provider system, and the communication of user responses to the plurality of interactive media to the corresponding service provider system or content provider system.

Abstract: According to one or more embodiments of the present invention, a system includes a user device to participate in a survey, and a server coupled with the user device to facilitate dynamic survey adjustment by performing a method that includes creating a survey based at least in part on a set of survey questions. The method further includes inserting an attentiveness question at a random position in the survey, wherein a response to the attentiveness question is indicative of an attentiveness of a survey taker to the survey. The method further includes, in response to receiving the response to the attentiveness question from the survey taker, performing one or more dynamic survey adjustments. The dynamic survey adjustments includes adjusting a weightage associated with one or more responses provided by the survey taker.

Abstract: A computing system may include a software application configured to receive instructions to merge a source code update with a branch of a software product, provide instructions to merge the source code update into the branch, and select software tests to execute on the branch as updated based on a portion of the branch modified by the source code update and a predetermined checkpoint within a development cycle of the software product. The software application may also be configured to provide instructions to execute the software tests on the branch as updated and receive feedback data indicating results of the software tests. The software application may be additionally configured to determine that the feedback data indicates that the branch as updated passed at least a threshold number of the software tests, and, in response, provide instructions to merge the branch as updated with a production branch of the software product.

Abstract: Embodiments leverage an artificial intelligence engine to generate customer-specific user experiences based on real-time analysis of customer responses to recommend and/or experienced features. Some embodiments access a profile of an end user comprising at least one first characteristic associated with the end user and extract end user information from a database of an entity server; identify a first trend related to resource utilization and/or management implemented by the end user based on the extracted end user information; determine an opportunity based on the identified first trend; in response to determining the opportunity, transmit control signals configured to cause the graphical user interface of the device of the end user to display graphically at least one recommendation based on the determined opportunity to the end user; and receive an input selecting or declining at least one recommendation. These inputs may correlate to a modified interface experience.

Abstract: The innovation relates to systems and/or methodologies for facilitating dynamic sustainability searches. A sustainability search component executes a query, and returns one or more results that satisfy a set of search criteria and a set of sustainability factors. The search criteria can include keywords, such as product types, process types, and additional features related to the keywords, such as price, location, brand, and so forth. The sustainability factors are alternative cost measures for a given process, product, or plant element. The sustainability search component can analyze the returned results, and rank the results based on sustainability.

Abstract: A process determines, with a processor, a virtual indicium position in a physical environment at which a virtual indicium is to be projected. The processor determines the virtual indicium position based on a point of intersection of a vector and an object within the physical environment. The vector originates at a head position of a user and is directed through a point of intended gaze on a virtual object toward the virtual indicium. Further, the processor sends, with the processor to an actuator, an instruction to adjust a position of a projector in operable communication with the actuator so that the projector projects the virtual indicium toward the virtual indicium position.

Abstract: In an example, an anonymized set of confidential data data values of a first confidential data type is obtained. Then an anonymized set of confidential data data values of a second confidential data type is also obtained. A multiplier following a log-normal distribution is determined for the anonymized set of confidential data data values of the first confidential data type. Then smoothing is performed independently for the anonymized set of confidential data data values of the first confidential data type and the multiplier. Percentiles for the anonymized set of confidential data data values of the second confidential data type are then determined using the smoothed anonymized set of confidential data data values of the first confidential data type and the smoothed multiplier.

Abstract: A system and an operating method of a questionnaire sticker applying to various communication software programs are described. The system includes a server, a user platform, a user end and a sending end. The server includes a group database and a questionnaire sticker database. The group database and the questionnaire sticker database are respectively accessed by a registration interface and a sticker interface included in the user platform. The user end inputs the registration information through the registration interface, so as to create a user group or join a friend group. The sending end selects one of the questionnaire sticker models through the sticker interface to form a questionnaire sticker. The questionnaire sticker is sent to the group members through the corresponding communication software.

Abstract: An online survey platform and process for correlating the bid price offered by a supplier for the completion of an out-of-network survey hosted on the survey platform to one or more intrinsic value indicators of the survey. If the supplier's bid price is less than or equal to the survey budget price set by the buyer, then the routing platform of the present invention will route the supplier's respondent(s) to the survey.

Abstract: Methods are provided for software build outcome prediction. For example, one method may comprise obtaining feature data associated with a software build, the feature data including one or more of the following: first data identifying a user to perform a modification on a set of one or more software artifacts, second data identifying the set of one or more software artifacts, and third data identifying a set of one or more reviewers to review the modification. The method may also comprise determining first probability data associated with the software build being unsuccessful given the feature data and second probability data associated with the software build being successful given the feature data; and predicting a software build outcome associated with the software build based on the first probability data and the second probability data.

Abstract: An encoder system trains a compression model that includes an autoencoder model and a frame extractor model. The encoding portion of the autoencoder is coupled to receive a set of target frames and a previous state tensor for the set of target frames and generate compressed code. The decoding portion of the autoencoder is coupled to receive the compressed code and the previous state tensor for the set of frames and generate a next state tensor for the set of target frames. The frame extractor model is coupled to receive the next state tensor and generate a set of reconstructed frames that correspond to the set of target frames by performing one or more operations on the state tensor. The state tensor for the set of frames includes information from frames of the video that can be used by the frame extractor to generate reconstructed frames.

Abstract: Provided herein are systems, methods and computer readable media for assigning leads. An example for method for assigning leads comprises receiving a list of one or more merchants, wherein the one or more merchants comprise a sales value and an indication of an ability to provide a service associated with at least one demanded offer, accessing sales resource data comprising at least one or more sales resources and an associated capacity for contacting one or more merchants, assigning one or more sales resources an assignment probability number, wherein an assignment probability number represents a probability that a sales resource will be randomly assigned a next supply source in an assignment process, adjusting an assignment probability number based on the associated capacity, and assigning at least a portion of the one or more merchants according to at least the assignment probability number of the one or more sales resource and the sales value of the one or more merchants.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for receiving, processing, and presenting user reviews, or logging user experiences, with products and services, e.g., restaurant services. One aspect is a method providing user interface elements including a region having panels each for receiving ratings of a general aspect of the product or service, a region having panels each for receiving ratings of a specific item of the product or service, where each panel includes an analog user interface element to input a rating and presents an image corresponding to the rating. Another aspect is a method that includes providing for each panel an analog user interface element for a user to input a rating for a corresponding item of the product or service, and accepting as a completed review a review input by the user that does not include any text input by the user.

Abstract: The provided solution makes it possible to select one item from a set of items, a multicomponent score being associated with each item of the set of items. After having computed a value characterising a relation ranking between a first item of the set of items and a second item of the set of items, the computed value being computed as a function of a plurality of score components associated with the first and second item of the set of items, respectively, the first item or of the second item is selected based on the computed value.

Abstract: A change management system is configured to control communication between one or more engineering subsystems utilized in the design and/or manufacture of a component. The change management system may be configured to facilitate communication between one or more engineering subsystems that are fragmented and/or not configured to communicate with one another. Further, the change management system may be configured to act as a virtual user when interfacing with one or more of the engineering subsystems. Additionally, the change management system may be configured to check data from one or more of the subsystems for compliance with one or more control rules governing the progression of a manufacturing/design project.

Abstract: An multistep guided system for mobile devices that facilitates the creation and dissemination of multistep guided activities from a source computer/device to a plurality of other recipient mobile devices, wherein the multistep guided activities is disseminated to the recipient mobile devices in a form that is compatible with the capabilities of the respective recipient mobile devices. The audio guided system comprises the source computer/device, the plurality of other recipient mobile devices and a server.

Abstract: An interactive media management server that facilitates management of interactive media and delivery of such interactive media. It comprises a first interface to an interactive media distribution server communicatively coupled to a plurality of mobile devices, and a second interface to a service provider system or a content provider system. It comprises a manager module for managing creation, distribution and tracking of a plurality of interactive media. The manager module, via the second interface, manages the delivery of the plurality of interactive media from the service provider system or the content provider system, and the communication of user responses to the plurality of interactive media to the corresponding service provider system or content provider system.

Abstract: A customer database system may store historic customer satisfaction information. A CSR terminal may facilitate an interaction between a service representative and a first customer, and a survey platform may: (i) detect that the CSR terminal is currently interacting with the first customer and that the first customer has an entry in the customer database system, and (ii) automatically trigger an analytics action at the CSR terminal. A relational database may receive solicited customer feedback information along with a structured escalation level category entered via the analytics action. The solicited customer feedback information may be parsed into unstructured text, and a big data application platform may: (i) execute an algorithm to assign a sub-category to the first customer interaction based on the unstructured parsed text, and (ii) assign the solicited customer feedback to positive, neutral, and/or negative sentiment pools.

Abstract: A method includes determining a recommended aspect for content that includes at least one image. The recommended aspect is determined at least in part based on activity data that indicates aspects of other content authored by or interacted with by a plurality of authors in at least one social network, website, application software, or mobile application software (app). The method further includes generating the content including the at least one image according to the recommended aspect. The recommended aspect is an aspect of the at least one image.

Abstract: An article articulating two different points of view is provided to a reader, with each point of view associated with a respective hashtag. Social media streams associated with social media outlets are also collected and analyzed to identify occurrences of the hashtags in the social media streams, and to collect and analyze information relating to those occurrences. Scoring parameter values are assigned to each occurrence, with each scoring parameter value based on the related information, such as an identity of a notable social media user associated with the occurrence. A weighted score value is generated based on the scoring parameter values for each hashtag, and the weighted score value is presented to the reader of the article in real time. Additional visual information, such as trend lines, identities of notable social media users associated with certain occurrences, and other information may also be provided to the reader in real time.

Abstract: The present disclosure relates to a method and system for obtaining interactive user feedback in real-time by feedback obtaining system. The feedback obtaining system establishes connection between user device of user and server of service provider based on user location received from user device, receives static data of user from server and dynamic data of user from capturing device located at site of service provider, identify contextual information associated with user based on static data and dynamic data, provide one or more feedback queries for user from database based on contextual information, provide one or more sub-feedback queries for user based on response of user for one or more feedback queries and obtains user feedback based on response of user for one or more sub-feedback queries and one or more feedback queries and implicit feedback. The use of implicit feedback together with actual feedback gives effective feedback of users.

Abstract: Various embodiments relate generally to computer science, software, and computer program and platform architectures, including receiving input data configured to identify one or more items, the input data being converted from a first data format to a second data format, parsing the input data in the second data format to extract event data associated with the one or more items, evaluating the event data to determine shipment data configured to identify a location, an account, and a first option associated with the one or more items, comparing the input data in the second data format to a threshold to determine a statistical probability associated with the one or more items, calling a data service to retrieve other data associated with the one or more items, and generating a retrieval option associated with other items if the statistical probability is greater than the threshold and the other data indicates the device is within a numerical range of a location at which the item is stored.

Abstract: Embodiments of the present disclosure relate to digitally converting physical form documents to electronic surveys. For example, a survey system receives an image capture of a physical form document and analyzes the image capture to identify fields including questions within the physical form document. The survey system creates an electronic survey corresponding to the physical form document based on the contents of the fields (e.g., text, objects). Specifically, the survey system generates electronic survey questions for the fields based on the contents of the fields. After generating electronic survey questions for the fields in the image capture, the survey system provides the electronic survey for display to a respondent. Additionally, the survey system can complete an electronic version (e.g., an image) of the form document including the responses.

Abstract: In one embodiment, a method and computer program product of distributing advertisement for increasing click-thru-rate is provided. The method comprises establishing communication between at least one user computing device and at least one website through a network, and displaying an ad to a user through the website. The user is then requested to participate in at least one event by way of the ad. After the user clicks on the ad, the user then travels to an advertiser's website or the advertiser's website is displayed to the user. After which, the most current total of the results of the event are displayed to the user in real time.

Abstract: Various embodiments relate generally to computer science, software, and computer program and platform architectures, including receiving input data configured to identify one or more items, the input data being converted from a first data format to a second data format, parsing the input data in the second data format to extract event data associated with the one or more items, evaluating the event data to determine shipment data configured to identify a location, an account, and a first option associated with the one or more items, comparing the input data in the second data format to a threshold to determine a statistical probability associated with the one or more items, calling a data service to retrieve other data associated with the one or more items, and generating a retrieval option associated with other items if the statistical probability is greater than the threshold and the other data indicates the device is within a numerical range of a location at which the item is stored.

Abstract: The early adopters and potential influencers (EAPI) system, method and computer-readable medium provide a way to identify early adopters and potential influencers. The EAPI system obtains a list of purchases for customers of merchants and/or subscriptions from a transaction tracking system. The EAPI system creates a time-based transaction network, and using a scoring function, the EAPI system determines an early adopter and/or potential influencer ranking among customers in the network. The EAPI system may use one or more customer attributes to determine a customer's influence with respect to different dimensions.

Abstract: A particular organization may undertake a plurality of different privacy campaigns, processing activities, etc. that involve the collection and storage of personal data. The system may be configured to collect customer satisfaction data, for example: (1) as part of a data subject access request submission form; (2) when providing one or more results of a data subject access request to the data subject; or (3) at any other suitable time. The customer satisfaction data may be collected in the form of a suitable survey, free-form response questionnaire, etc. (e.g., thumbs up vs. thumbs down, etc.). Additionally, by integrating a feedback survey with the data subject access request process, the system may increase a number of consumers that provide responses to the feedback survey. In particular embodiments, the system is configured to require the requestor to respond to the feedback survey prior to submitting the data subject access request.

Abstract: A system, method, and computer-readable medium are disclosed for performing an automated sales training activity operation, comprising: performing an assessment operation on an individual sales maker, the assessment operation ensuring the individual sales maker takes only the training needed by the individual sales maker; generating a unique personalized training plan for the individual sales maker; delivering training to the individual sales maker based upon the unique personalized training plan; certifying the individual sales maker based upon completion of the training, the certifying indicating a knowledge and proficiency level of the individual sales maker, the certifying also indicating progress of the individual sales maker towards completion of the unique personalized training plan and, presenting information regarding the certifying via a sales training activity user interface.

Abstract: A method includes forwarding one or more packets from a first addressable computing element to at least a second addressable computing element based on a topology-aware address assigned to the second addressable computing element. The topology-aware address includes a set of identifiers representing one or more topology attributes associated with the second addressable computing element.

Abstract: Methods, computer program products, and systems are presented. The method computer program products, and systems can include, for instance: examining data of a plurality of traveling users, wherein at least some of the plurality of traveling users have associated mobile client computer devices, and determining, by machine logic, based on the examining that certain traveling users of the plurality of traveling users are traveling in a group; deciding, by machine logic, based on the determining that one or more action is to be performed, wherein the deciding is in dependence on a processing of historical data, the historical data specifying activities of the certain traveling users when the certain traveling users travel in a group; and providing one or more output for performance of the one or more action.

Abstract: The present invention provides systems and methods that provide knowledge about a reaction (e.g. interest) of a group (normally of people) with respect to a set of elements, wherein each group member can react in a different manner. Assuming that the group is accessible by electronic means (e.g. Internet) this system may be based on a group response algorithm to estimate group reaction minimizing workload of group members and minimizing the time needed. To minimize or reduce group members' workload and time, the algorithm uses statistic tools. The group response system estimates group reaction with previously set statistic reliability. The present invention can use too in some cases, secondary algorithms for specific cases.

Abstract: Embodiments of the present disclosure relate generally to distributing a survey created for a first distribution channel across multiple distribution channels. More specifically, one or more embodiments of the present disclosure relate to providing a user with the ability to create a survey for one distribution channel and automatically provide the survey across multiple distribution channels. Further, one or more embodiments of the present disclosure improves survey administration by collecting responses to the survey from across the various distribution channels, compiling the responses, and providing the overall results of the survey to the user.

Abstract: The disclosure includes use of a feature-aware propagation model to identify one or more features of a product and one or more person(s), or members of a social network, to target, or user, for marketing the product having the identified features. The one or more person(s) identified using the model may be the person(s), or member(s), of a social network determined to have a maximum capability, relative to other members of the social network, for influencing the members of the social network in adopting, e.g., purchasing, a product having the identified features. In addition, parameters of the model may be determined using information about the social network, user preferences, and the products and features of the products.

Abstract: A social networking system builds a quality controlled and desired population-representative pool of human raters to provide ratings on content items to improve a feed ranking model used for providing its users with more relevant content. The system identifies a pool of candidate human raters for providing ratings on a feed of content items. For each candidate human rater of the pool of candidate human raters, the system presents a feed of content items based on a feed ranking model, obtains ratings on the feed of content items, and determines a score representing the consistency of the obtained ratings, the representativeness of the pool of human raters, or the relevance of the content provided by the ranking model. The system uses the computed scores to modify the ranking model used to present content to its users for improving the relevance of the presented content.