Abstract: A computer-implemented method is provided for taking one or more actions relating to therapy, the method comprising: obtaining data comprising audio data relating to a therapy session between a therapist and one or more patients; extracting text data from the audio data to form a transcript; dividing the transcript into a plurality of utterances; using at least a first part of a deep learning model to assign a semantic representation to each of the plurality of utterances to produce a plurality of assigned utterances; compiling the plurality of assigned utterances to form a representation of the therapy session; using at least a second part of a deep learning model, and an input comprising the representation of the therapy session, to obtain an output predicting a characteristic of the therapist, and/or the therapy, and/or the one or more patient; and causing the system to take one or more actions relating to the therapy, wherein the one or more actions are selected based on the output meeting one or more pre

Abstract: A learning model is created using a computer which includes receiving input from a user. The learning model includes comparing the input to the knowledge area in a knowledge database to assess a level of proficiency on topics within the knowledge area. The learning model includes determining topics of knowledge within the knowledge area where a user currently meets a proficiency threshold for one or more topics. A work topic is identified within the knowledge area where the user does not meet the proficiency threshold. Study material is presented to the user for the work topic of the knowledge area using an interactive mechanism. Feedback is received regarding the study material for the work topic from the interactive mechanism from the user. The learning model includes evaluating the feedback from the user to determine a score which indicates when the user meets a proficiency threshold for the work topic.

Abstract: The present invention relates to a method for reinforcing a multiple-choice QA model based on adversarial learning techniques, wherein incorrect answers are further generated based on a data set used in the process of training the multiple-choice QA model to enrich data which are learnable by the multiple-choice QA model.

Abstract: An information processing apparatus that switches a normal mode and an energy-saving mode includes control circuitry, an operation unit, and a device. The control circuitry controls processing of the information processing apparatus. The operation unit includes a display. The device mounted on the control circuitry or the operation unit writes change information indicating change of an operation mode in response to switching between the normal mode and the energy-saving mode. The device has a limit to a number of times of writing. The control circuitry counts the number of times of writing of the change information into the device, predicts a life of the device on a basis of a count value of the number of times of writing, and diagnoses appropriateness of the count value used for prediction of the life of the device when an increase rate of the count value exceeds a predetermined increase rate.

Abstract: Disclosed herein are system and method for generative assessment item development, encoding & analysis, wherein the generative assessment item includes a question and a plurality of possible responses to the question, the plurality of possible responses comprising at least two responses that are not independent, and wherein the plurality of responses are non-dichotomous.

Abstract: Systems and methods for automated custom training of a scoring model are disclosed herein. The method include: receiving a plurality of responses received from a plurality of students in response to providing of a prompt; identifying an evaluation model relevant to the provided prompt, which evaluation model can be a machine learning model trained to output a score relevant to at least portions of a response; generating a training indicator that provides a graphical depiction of the degree to which the identified evaluation model is trained; determining a training status of the model; receiving at least one evaluation input when the model is identified as insufficiently trained; updating training of the evaluation model based on the at least one received evaluation input; and controlling the training indicator to reflect the degree to which the evaluation model is trained subsequent to the updating of the training of the evaluation model.

Abstract: Disclosed is a method of training the dementia identification model. More particularly, the method may include obtaining test result data through at least one of a first task related to Stroop test, a second task related to a computational power test, and a third task related to a memory test; and labeling the test result data with a score value to train the dementia identification model.

Abstract: A method comprising accessing a first data structure that is associated with a first product prepared by a student and that includes first process data associated with a process performed by the student in generating the first product, analyzing the first data structure to generate a first characterization score based on the first product and the first process data, accessing a second data structure that is associated with a second product prepared by the student and that includes second process data associated with a process performed by the student in generating the second product, analyzing the second data structure to generate a second characterization score based on the second product and the second process data, and calculating a skill level change metric based on the first characterization score and the second characterization score indicating a change in ability level of the student over a course of the scenario-based assessment.

Abstract: A system and method for predicting risk of violence for an individual (primarily school violence, but not limited to school violence) performs the following steps: (a) receiving responses to questions from an individual; (b) extracting by a computerized annotator words or phrases from the questions and responses; (c) assigning by the annotator extracted word(s) or phrase(s) to at least one of a plurality of pre-defined categories; and (d) automatically identifying and scoring words or phrases that could be classified into the pre-defined categories by a trained machine-learning engine to produce a score reflecting relative risk of violence by the individual. The pre-defined categories include: expression of violent acts or thoughts of the individual; expression of negative feelings, thoughts or acts of others; expression of negative feelings, thoughts or acts of the individual; expression of family discord or tragedies; and expression of protective factors.

Abstract: A method, computer system, and a computer program product for recommending additional content to an author generating authored content is provided. The present invention may include monitoring a travel location associated with the author. The present invention may include calculating a multi-sensory region based on the travel location and a maximum sense distance value. The present invention may include selecting at least one piece of additional content from a corpus of additional content based on the multi-sensory region. The present invention may include generating a model based on the at least one piece of additional content. The present invention may include selecting a relevant piece of additional content from the data model based on determining a topic associated with the relevant piece of additional content matches an authored topic associated with the authored content. The present invention may include presenting the selected relevant piece of additional content to the author.

Abstract: A method for creating personalized lesson recommendations for a user is provided. In some embodiments, the method includes assigning a skill profile to the user, where the skill profile includes a skill vector associated with one or more fine-grained skills of the user. The method also includes assigning a difficulty profile to a task to be practiced by the user, where the difficulty profile includes a task difficulty vector associated with one or more fine-grained skills of the task. Further, the method include prioritizing the task within a list of recommended tasks offered to the user based on a comparison between the skill profile of the user and the difficulty profile of the task.

Abstract: An image forming apparatus includes a reader, a wait part, and a processor. The reader reads an image from a first sheet. In the wait part, the first sheet is made to wait. The processor is configured to make the first sheet wait in the wait part after the reading performed by the reader, and, after the first sheet is made to wait in the wait part, output the first sheet and a second sheet continuously. The second sheet records first information determined on the basis of an evaluation result obtained from evaluation of the first sheet.

Abstract: A teaching assistance method and a teaching assistance system using said method, the teaching assistance method comprising implementing behaviour detection of students in classroom images by means of using a trained depth tensor column network model, thus providing higher image recognition precision and reducing the hardware requirements for algorithms, and being able to be used on an embedded device, reducing the usage costs of the teaching assistance method; in addition, a teaching assistance system using said teaching assistance method has the same advantages.

Abstract: Mechanisms are provided for modifying response of cognitive systems to user requests based on a determined knowledge level of the user and knowledge level of the concepts referenced in the cognitive system responses. The mechanisms receive a response to a user submitted request received from a client computing system and determine a first knowledge level of the user for a domain of the user submitted request. The mechanisms determine a second knowledge level associated with concepts referenced in the response and whether the first knowledge level is different from the second knowledge level. The mechanism, in response to the first knowledge level being different than the second knowledge level, modify content of the response to include concepts associated with the first knowledge level, to thereby generate a modified response. The modified response is then output to the client computing system associated with the user.

Abstract: A method of obtaining a response to a query inputted by a user, the method comprising: receiving a user inputted query; representing the user inputted query as a sequence of embedding vectors using a first model; encoding the sequence of embedding vectors to produce a context vector using a second model; retrieving responses with associated response vectors; scoring response vectors against the context vector, wherein the scoring is a measure of the similarity between the context vector and a response vector; and outputting the responses with the closest response vectors, wherein the first model is configured to segment a user inputted query into a sequence of units from a vocabulary of units and represent each unit in the sequence as an embedding vector, wherein at least one of the units in the vocabulary is an incomplete word, and wherein the first model comprises parameters that are stored using eight bits per parameter; and wherein the second model has been trained using corresponding queries and response

Abstract: Systems and methods for defining and using an optimal burst boundary threshold to assess the reliability of a manual/automatic writing score are presented. Keystroke data, including inter-key interval data, such as inter-word interval data, may be gathered from writings. Clustering analyses may be performed on the inter-key interval data to determine an optimal number of bursts for the writings. An optimal burst boundary may be determined from the optimal number of bursts. Other burst-related measures and statistics, including the average and maximum burst lengths, may be determined from the writings based on the optimal burst boundary threshold. A score may be received for each of the writings. A validation indication metric may be generated for each of the writings based on the received score and the optimal burst boundary threshold. The resulting measures and statistics may be used or applied in different ways and provide personalized feedback as learning analytics.

Abstract: Embodiments of the present invention are directed to a two-way data transfer and synchronization system. The data transfer and synchronization system includes a provider system and a customer system. The provider system includes a web service hosted by a web server connected to a computer network and a first database. The customer system includes a client application, connected to the computer network and a second database (e.g., SIS database). The provider system cannot read data from the second database. The client application is configured to send requests to the web service and to receive HTTP responses from the web service. The responses from the web service are executed by the client application so that changes made to the second database are also made to the first database, and vice versa.

Abstract: A combination of knowledge representation and retrieval, assessments and personalization for a tutoring platform that provides an automated phased word exposition method for vocabulary learning and enable learners to learn words (or any learning objective) with sufficient proficiency and without excessive repetition. The phased learning objective method implements a moving average model that models the learner across multiple dimensions (e.g. reading, listening, writing, speaking) such that the platform is confident of the learner's understanding, while using minimal assessment doses, and such that the understanding transfers across experiences. The method implemented provides for multiple learning and assessment activities for a learning objective and leverages the same learner model for each learning dimension. Each word of a word set passes through learning and assessment-only phases for statistically significant evidence of learning.

Abstract: Accordingly, embodiments of the present invention advantageously account for item discrimination in a single-parameter logistic model used for measuring a test-taker's ability and an item's difficulty. Accounting for item discrimination improves the reliability of a test without increasing the number of test items. To account for item discrimination in a single-parameter logistic model, this invention uses the correlation between item response (correct or incorrect) and total test score or other measure of test-taker ability to obtain a Bayesian estimate of the correct-response probability (between zero and one). This correlation is a measure of item discrimination. The numerator in the formula for this correlation contains the difference between the average test score of test-takers who got the item right and the average test score of test-takers who got the item wrong.

Abstract: An inference device disclosed herein includes an identifying unit, and an inferring unit. The identifying unit identifies a set of a predetermined phenomenon, an effect having a causal relation with the predetermined phenomenon, and a polarity of the effect, from concept information in which a phenomenon, an effect having a causal relation with the phenomenon, a polarity to be an indicator of advantages and disadvantages caused by the effect to a user. The inferring unit infers an effect caused when the predetermined phenomenon occurs, and advantages and disadvantages caused by the effect based on the set identified by the identifying unit.

Abstract: Systems and methods are provided for automatically scoring a response and statistically revaluating whether it can be considered as aberrant. In one embodiment, a constructed response is evaluated via a pre-screening stage and a post-hoc screening stage. The pre-screening stage attempts to determine whether the constructed response is aberrant based on a variety of aberration metrics and criteria. If the constructed response is deemed not to be aberrant, then the post-hoc screening stage attempts to predict a discrepancy between what score an automated scoring system would assigned and what score a human rater would assign to the response. If the discrepancy is sufficiently low, then the constructed response may be scored by an automated scoring engine. On the other hand, if the constructed response failed to pass either of the two stages, then a flag may be raised to indicate that additional human review may be needed.

Abstract: Methods, computer program products, and systems are presented. The methods include, for instance: obtaining sample essays, sample annotations corresponding to the sample essays, and a subject content for building a subject domain comprehension model and an essay annotation model, by use of one or more neural network. The nodes of the subject domain comprehension model and the essay annotation model are interconnected based on respective relevancies for automatically annotating student works according to a standard of review corresponding to submitting students.

Abstract: The present disclosure features systems and methods for analyzing student learning and calibrating the difficulty of questions on a test or examination. In one embodiment, a method for analyzing the learning of a student includes administering, by an assessment agent, a task to a student, the task comprising a question having an associated difficulty. The assessment agent receives a response to the question from the student and evaluates the response to generate an observable, the observable comprising information related to the response. A posterior determination of the student's ability is then calculated by incorporating the observable into an ability model associated with the student, and the posterior determination of ability may be compared with the difficulty of the question, a skill acquisition probability, or other measure. The student's response, or a plurality of responses from students within a cohort, may be used to determine the difficulty of each question.

Abstract: A machine-learning component of a conversational user-interface system retrieves records of human question-and-answer sessions from heterogenous sources. A sequence of image-processing operations converts each source into a set of grayscale images. Each image is segmented into disjoint sections as a function of the textures, edges, and contours that make up the image's content. Each section is tagged with a vector of parametric values that identify characteristics of the section's content from which may be inferred semantic meaning. A cognitive function intelligently analyzes the vectors to classify each section as containing a question, an answer to a question, non-textual media, or other types of content. Another cognitive function merges and organizes the sections into question-answer pairs and the vectors associated with each pair are stored in a corpus that is submitted to the self-learning user interface during a machine-learning training session.

Abstract: The systems, methods and associated devices electronically collect, report and generate normalized educational outcome summaries of multiple different educational inputs, including didactic, experiential and problem solving events and/or assessments.

Abstract: A method processes scanned floating forms by comparing topological structures of an empty form and a corresponding filled form. The topological structure of each form includes vertical and horizontal ordered sequences of text phrases in the form, which describe directional relationships among text phrases but not their distances. A partial structure alignment method is used in the comparison, where insertions of text in the topological structure of the filled form relative to that of the empty form is not penalized, but deletions and substitutions are penalized. Based on this comparison, some text phrases in the filled form are matched to the those in the empty form, and unmatched text in the filled form is deemed user-filled data. The method further associates user-filled data with fields of the form based on a settings file of the empty form.

Abstract: A computer implemented method for evaluating examination responses includes receiving, with an examination evaluation server, an examination response, tokenizing the examination response into one or more n-word segments, extracting, from the response, a set of construct-relevant features indicating one or more scoring metrics, generating a set of scoring rules, and generating a score estimate as a function of one or more scoring rules from the set of scoring rules and one or more of the scoring metrics.

Abstract: A computer implemented method for identifying and removing similar examination items in a digital examination includes obtaining, with an examination evaluation server, a digital examination with a plurality of examination items, tokenizing a first examination item of the plurality of examination items into a first set of n-grams, tokenizing a second examination item of the plurality of examination items into a second set of n-grams, generating a first item vector from the first set of n-grams, generating a second item vector from the second set of n-grams, generating multiple item congruence values for corresponding similarity dimensions by comparing the first item vector with the second item vector, and identifying that the first and second examination items are a potential enemy item pair if one or more of the item congruence values exceed corresponding threshold values.

Abstract: A computer implemented method of examination scoring includes receiving, with an examination evaluation server, a language corpus with a plurality of corpus words, deriving, for each corpus word, a corpus sematic vector within a multi-dimensional semantic vector space, receiving first and second examination responses, tokenizing each examination response into a plurality of response words, identifying a set of matching response words from the plurality of response words from each examination response, identifying a set of semantic vectors corresponding to the matching response words for each examination response, and comparing the semantic vectors from the first examination response with the semantic vectors from the second examination response to derive a response semantic similarity value for use as an input for automated scoring.

Abstract: The invention provides a method of generating an adaptive partial report for an observer with an apparatus comprising a display, a user interface, and a processor. The apparatus can be a computer system or an electronic device, for example. The method includes the processor characterizing an iconic memory decay function for the observer. The characterization includes determining a prior for a plurality of parameters. The method further includes the processor determining a first stimulus for a first trial based on the prior for the plurality of parameters, the display generating the stimulus for viewing by the observer, the user interface receiving input for the first trial and in response to the stimulus, the processor revising respective parameter values for the parameters based on the received input, and the processor determining a new stimulus for a next trial based on the revised parameter values.

Abstract: A method for test item selection is provided that includes a computer implemented test battery having at least two or more sections with a plurality of test items. An ability estimate is calculated from an earlier section(s) of the at least two or more sections and an initial item and subsequent items for a subsequent section are selected from the plurality of test items based upon the ability estimate(s) from the earlier section(s). Use of a more informative initial ability estimate in the item selection process can improve interim ability estimation accuracy and item selection efficiency while keeping item exposure and item usage rates at acceptable levels.

Abstract: A non-transitory computer-readable recording medium stores a scoring rule application target specification program that causes a computer to execute a process including: receiving specification of one or more areas in an image; and storing the one or more areas in a storage in a manner associated with a question number in order of being received.

Abstract: A computer-implemented method of calibrating an assessment model for assessing responses includes accessing a plurality of training responses with a processing system for training an assessment model. The processing system analyzes the plurality of training responses to derive values of multiple features of the training responses. The processing system trains the assessment model based on the values of the multiple features of the training responses and a portfolio score for each individual associated with the plurality of training responses utilized in the training. The portfolio score for each individual corresponds to a measure of proficiency based on multiple writing samples constructed by the individual. The processing system determines, based on said training, a weight for each of the multiple features. The processing system calibrates the assessment model to include the weights for at least some of the features such that the assessment model is configured to generate scores for responses.

Abstract: Mechanisms for automatically grading a large number of solutions provided by learners in response to an open response mathematical question. Each solution is mapped to a corresponding feature vector based on the mathematical expressions occurring in the solution. The feature vectors are clustered using a conventional clustering method, or alternatively, using a presently-disclosed Bayesian nonparametric clustering method. A representative solution is selected from each solution cluster. An instructor supplies a grade for each of the representative solutions. Grades for the remaining solutions are automatically generated based on their cluster membership and the instructor supplied grades. The Bayesian method may also automatically identify the location of an error in a given solution. The error location may be supplied to the learner as feedback. The error location may also be used to extract information from correct solutions. The extracted information may be supplied to a learner as a solution hint.

Abstract: This disclosure describes a system that delivers content (e.g. multimedia video), and analyzes and aggregates feedback of a user viewing the content. The user may interact with the multimedia video in the form of answering questions which are dynamically placed between video segments. The system may determine the response type of the user for a given question. The system may handle both positive and negative responses (e.g. correct and incorrect responses to questions). The system may help the user navigate the segments of the video.

Abstract: Methods, systems, and apparatus, including medium-encoded computer program products, for analyzing data include: receiving data including responses, and lack thereof, for items of a cognitive test including multiple item-recall trials; processing the data using a stochastic model of a cognitive process, in which a conditional probability distribution of future states of the cognitive process depend upon a present state; and encoding a result of the processing on a non-transitory computer-readable medium for use in an assessment related to cognition; wherein the processing using the stochastic model includes representing recall or recognition of an item in the multiple item-recall trials using distinct cognitive states; and wherein the processing using the stochastic model includes adjusting separate memory storage and retrieval parameters for each of the distinct cognitive states in the modeled cognitive process to account for position of the items in each respective trial of the multiple item-recall trials.

Abstract: Systems and methods are provided for automatically scoring essay responses to a prompt using a scoring model. A relevant word corpus and an irrelevant word corpus are accessed. A scoring model is generated by, for each of a plurality of words in the relevant word corpus, determining a topic signature score based on a number of appearances of that word in the relevant word corpus and a number of appearances of that word in the irrelevant word corpus. For each of a plurality of words in an essay response, a topic signature score is determined for that word. A score for the essay response is determined based on the identified topic signature scores.

Abstract: A method for managing and rendering one or more information nodes relative to a current focus. The method receives one or more principal topics from an analysis of the information nodes. The method determines a topic vector from a similarity of a first information node to each of the principal topics and a map from the topic vector to a storage location of the first information node. The method determines one or more current topics from a second information node that has been recently accessed by a user and a current focus vector from a similarity of the one or more current topics to the one or more principal topics. The method renders the current focus vector and the first information node according to a rendering algorithm that is applied to the current focus vector and the first information node.

Abstract: Selecting candidates from a set of candidates by receiving a set of input parameters about a single aspect of the candidates and applying multiple scoring methods to those input parameters to compute a candidate score via each scoring method. A subset of zero or more candidates is then selected from the set of candidates by applying one or more selection criteria to the set of scores. There is not necessarily a value for every input parameter of every candidate.

Abstract: Systems and methods are provided for measuring a user's English language proficiency. A constructed response generated by a user is received, the constructed response being based on a picture. The constructed response is processed to determine a first numerical measure indicative of a presence of one or more grammar errors in the constructed response. The constructed response is processed to determine a second numerical measure indicative of a degree to which the constructed response describes a subject matter of the picture. The constructed response is processed to determine a third numerical measure indicative of a degree of awkward word usage in the constructed response. A model is applied to the first, second, and third numerical measures to determine a score for the constructed response indicative of the user's English language proficiency. The model includes first, second, and third variables with associated first, second, and third weighting factors, respectively.

Abstract: Computer-implemented systems and methods are provided for automatically generating recitation items. For example, a computer performing the recitation item generation can receive one or more text sets that each includes one or more texts. The computer can determine a value for each text set using one or more metrics, such as a vocabulary difficulty metric, a syntactic complexity metric, a phoneme distribution metric, a phonetic difficulty metric, and a prosody distribution metric. Then the computer can select a final text set based on the value associated with each text set. The selected final text set can be used as the recitation items for a speaking assessment test.

Abstract: A method and survey server for analyzing social media content based on survey participation data related to a website. The survey server collects survey participation data comprising text and a metric. The survey participation data correspond to survey information received from users of a plurality of user devices in relation to a visit of the website. The survey server analyzes the text of the survey participation data, to generate a correlation between words or groups of words present in the text and the metric. Social media content comprising text and related to a visit of the website is also collected. The text of the social media content is analyzed, to identify the words or groups of words. The social media content comprising the words or groups of words is associated with the metric. The analysis of the text of the survey participation data may comprise performing a semantic categorization.

Abstract: A method/apparatus/system for educational intervention based on a response metric is disclosed. The notice is generated in response to the collection of user and question data, the sending of questions, the receipt of answers, the evaluation of the correctness of the answers, the generation of a response metric, the comparison of the response metric to a threshold, and the generation of the report or notice. The response metric can be reflect the scatter, randomness, and/or slope of student provided answer data, and can be a fractal dimension.

Abstract: A method, a system, and a computer program product for creating an assessment model are disclosed. Marked regions in an electronic document are identified. Thereafter, mark-ups within the marked regions are identified. Based on the identified marked regions and mark-ups the assessment model is created.

Abstract: An optimization engine builds a set of test forms, called a pallet for use in a qualification test setting. The pallet is generated using a optimization engine programmed with constraints and goals for each test form in the pallet and for the pallet in general. A test information floor is set at the cut point of the test which causes test items to be focused at the area of the test where the pass/fail decision is made. The test items may be separated into families by assigning test items selected for a test form in each test generation cycle to one of several families. Because each family has unique test items compared to other families, a person re-taking a qualification test from a particular pallet may be given a test form from another family, ensuring that no item from the first administration will appear on the subsequent test.

Abstract: A medical diagnosis support apparatus includes a training data obtaining unit that obtains training data, an inference means candidate creating unit that creates a plurality of inference means candidates based on the training data, an inference performance evaluation unit that evaluates the performance of the plurality of inference means candidates based on the training data, an information validity evaluation unit that evaluates the validity of information presented by each of the plurality of inference means candidates based on the training data, and an inference means selection unit that selects an inference means from the plurality of inference means candidates based on the performance of the plurality of inference means candidates and the validity of the information presented by each of the plurality of inference means candidates.

Abstract: Embodiments herein provide automated assessment of examination scripts, such as English for Speakers of Other Languages (ESOL) examination scripts, written in response to prompts eliciting free text answers. In an embodiment, the task may be defined as discriminative preference ranking. Further, a system employing such methodology may be trained and tested on a corpus of manually-graded scripts. Embodiments herein, unlike extant solutions, are relatively prompt-insensitive and resistant to subversion, even if the operating principles are known. Embodiments may also detect scripts which are linguistically good but non-responsive to prompts such as memorized responses.

Abstract: A semantic similarity evaluation method includes performing word vectorization processing separately on words in a first sentence and a word in a second sentence to obtain a first word vector and a second word vector; performing, in a preset word vector compression order, compression coding processing on the first word vector according to a first compression coding parameter to obtain a first statement vector; performing, in the preset word vector compression order, compression coding processing on the second word vector according to a second compression coding parameter to obtain a second statement vector; and determining a vector distance between the first statement vector and the second statement vector, and evaluating a semantic similarity between the first sentence and the second sentence according to the vector distance. The method is used to evaluate a semantic similarity.

Abstract: A method, a system, and a computer program product for creating and evaluating assessment sheets are disclosed. The method includes identifying one or more first location identifiers based on one or more markings provided in a scanned master assessment sheet. A machine-readable code (MRC) is generated based on the one or more first location identifiers. The MRC is then split into a plurality of portions and embedded at various locations on a scanned pre-assessment sheet. The scanned pre-assessment sheet with the embedded MRC is then printed to generate the assessment sheets. Thereafter, one or more second location identifiers are identified based on one or more markings provided by evaluatees in each of the assessment sheets provided to them. The one or more second location identifiers are then compared with the one or more first location identifiers. The result of evaluation is provided based on the comparison.

Abstract: A method and apparatus of communicating with multiple mobile station devices in a concerted effort is disclosed. According to one example method of operation the method may include identifying a number of mobile station devices as being present at the pre-defined venue location via a server and initiating an application on the server that establishes a feedback menu display. The method may also include transmitting the feedback menu display to each of the identified mobile station devices receiving feedback messages from a majority of the mobile station devices responsive to the feedback menu display questions, and processing the feedback messages and providing a result to the moderator device.