Abstract: Some implementations relate to generating a training and/or evaluation dataset with LLM prompts (e.g., derived from user queries) based on a prompt complexity. An input prompt, for example derived from a user query, is received. The input prompt is decomposed into a prompt tree comprising a plurality of nodes. The plurality of nodes comprise: a plurality of leaf nodes corresponding to simple sub-prompts of the input query; a plurality of branch nodes of sub-prompts each corresponding to multiple simple sub-prompts; and a root node corresponding to the input prompt. A prompt complexity is determined based on a path length of the prompt tree. The prompt complexity is compared to a threshold complexity. If the prompt complexity is above the threshold complexity, the input prompt is included in a set of training prompts and/or a set of evaluation prompts.

Abstract: Implementations relate to a method implemented by one or more processors, the method including: receiving natural language (NL) based input associated with a client device; generating, using a large language model (LLM) and based on processing the NL based input, LLM output; determining, based on the LLM output, a sequence of LLM responses, the sequence of LLM responses including at least one intermediate LLM response and a final LLM response. In some implementations, the method may further include causing the final LLM response to be rendered at the client device. In additional or alternative implementations, the method may further include storing, as an instance of training data for fine-tuning the LLM or an additional LLM, the NL based input along with the final LLM response.

Abstract: Implementations relate to improving instruction following capabilities of large language models (LLMs) using instruction decomposition, self-evaluation, and optionally progressive refinement. Processor(s) of a system can: obtain natural language (NL) based input, generate a plurality of candidate responses and evaluate the candidate responses based on instructions included in the NL based input, using an LLM, and progressively refine the candidate responses until it is determined that one or more termination criteria are satisfied. In some implementations, the NL based input can be received from a client device. In these implementations, a given candidate response that is progressively refined can be rendered for presentation at the client device and responsive to the NL base input. In additional or alternative implementations, the NL based input can be obtained from database(s). In these implementations, a given candidate response that is progressively refined can be utilized in fine-tuning of the LLM.

Abstract: Implementations described herein relate to using self-evaluation when utilizing a large language model (LLM) to generate a response to a natural language (NL) based input. The LLM can be used to process the NL based input to generate a plurality of responses, and to generate a critique of those responses by comparing the responses to a set of response evaluation criteria. One of the responses can then be selected based on the comparison with the set of response evaluation criteria which can vary from one NL based input to another. If the NL based input was obtained a user of a client device during an inference stage, then the selected response can be rendered for presentation to the user. If the NL based input was obtained during a training stage, then the selected response can be stored as a training instance and optionally in association with additional data.

Abstract: At least utilizing a custom corpus of documents to condition a large language model (LLM) when generating a response to a user query. In some implementations, a user query associated with a client device is received. An API query for an external application is generated by an LLM based on the user query. The external application has access to a custom corpus of documents comprising a plurality of documents. The external application is queried using the API query. Data representative of one or more documents in the custom corpus of documents is received from the external application in response to the API query. The LLM generates a response to the query that is conditioned on the data representing one or more of the documents in the custom corpus of documents received from the external application. The response to the user query is caused to be rendered on the client device.

Abstract: Techniques are described that provide a reading pane for reading one or more content portions of a work and a guide pane that displays a structured overview of additional information from the corpus that is focused by an analysis of information from the currently-visible content portion of the work. The contents of the currently-visible content portion displayed within the reading page are used to derive dynamic representations of the user's immediate interests. At the user's request, a structured persistent context-aware guide is computed and displayed. The persistent context-aware guide provides a framework for accessing other parts of the corpus having the most-related information. The elements in the persistent context-aware guide are other content portions or other organizations such as sections, chapters, or articles in the corpus. The persistent context-aware guide may also incorporate additional closely-related information elements from outside the corpus including advertisements.

Abstract: A system and method for content-based message distribution are provided. An incoming message with a recipient address and a tag address including at least one content tag associated with one or more users is received. The content tag within the tag address and a recipient associated with the recipient address are identified. The recipient is added to the content tag as one of the users. The incoming message is displayed to at least one of the users associated with the content tag.

Abstract: A system and method for triaging of information feeds is provided. A plurality of information feeds are received. At least one topic is identified from each information feed. At least one topic is presented to a user in topic facet including a plurality of identified topics. A selection of one of the plurality of topics is received from the user. The user interface is updated to display only the feeds that contain the selected topic.

Abstract: A system and method for generating an information stream summary using a display metric is provided. An information stream including a plurality of information stream items is received. A display metric is calculated for each of the plurality of information stream items. The information stream items are grouped into one or more summary objects. A size is assigned to each of the one or more summary objects and the one or more summary objects are displayed based on the assigned size.

Abstract: A system and method for identifying topics for short text communications is provided. Tokens are extracted from a short text communication. A query is generated using the extracted tokens. The query is applied to a set of documents. Those documents in the set that match the query are identified as search results. Salient terms associated with each of the search results are identified. A threshold is applied to the identified salient terms. The salient terms that satisfy the threshold are selected as topics for the short text communication.

Abstract: A system and method for recommending interesting content is provided. A content profile including a vector of words extracted from messages associated with a user is obtained. One or more content pointer profiles each including a vector of words associated with a content pointer located within one or more messages are obtained. The content profile is compared with each of the content pointer profiles. A relevance score is determined for that content pointer by calculating a similarity between the user words and the content pointer words. The content pointers are ranked based on the relevance score. A threshold is applied to select the content pointers and those content pointers that satisfy the threshold are selected as the content pointers most relevant to the user.

Abstract: A system and method for providing mixed-initiative curation of information within a shared repository is provided. Static content is retrieved from a shared storage associated with a shared information management client. Dynamic content including one or more information items satisfying a similarity threshold with the static content is identified as similar dynamic content. An interactive visualization is generated within the shared information management client from information extracted from the static content and the similar dynamic content. The interactive visualization of the information is linked with the static content and the similar dynamic content. A selection of the information of the similar dynamic content is received from within the visualization. The static content in the shared storage is updated with the similar dynamic content linked with the selected information.

Abstract: A system and method for content-based message distribution are provided. An incoming message with a recipient address and a tag address including at least one content tag associated with one or more users is received. The content tag within the tag address and a recipient associated with the recipient address are identified. The recipient is added to the content tag as one of the users. The incoming message is displayed to at least one of the users associated with the content tag.

Abstract: A system and method for content tagging and distribution through email are provided. A user-to-tag record including content tags, each associated with one or more users, is maintained. An incoming email message with a tag address is received. The tag address is processed to identify a content tag, which is applied to the user-to-tag record. Users associated with the identified content tag are determined. The incoming email message is provided to the associated users.

Abstract: A system and method for supporting targeted sharing and early curation of information is provided. A digital data item selection by a user within a personal information management client is identified. One or more documents in a shared information repository similar to the digital data item are recommended including selecting recommendation criteria. The recommendation criteria are applied to the digital data item and the one or more documents. The one or more documents satisfying the recommendation criteria are identified as the similar documents. The similar documents are displayed visually proximate to the digital data item in the personal information client. A selection of one of the similar documents is received and the selected similar document in the shared information repository is updated with the digital data item.

Abstract: A system and method for identifying users relevant to a topic of interest is provided. A query comprising one or more topics is executed against a corpus of messages. Voting users associated with the messages matching the query are identified. A set of candidate users comprising users connected to the voting users is generated. A relevancy score is computed for each candidate user. The candidate users are ranked by their respective relevancy score.

Abstract: A system and method for identifying users relevant to a topic of interest is provided. A query comprising one or more topics is executed against a corpus of messages. Voting users associated with the messages matching the query are identified. A set of candidate users comprising users connected to the voting users is generated. A relevancy score is computed for each candidate user. The candidate users are ranked by their respective relevancy score.

Abstract: Methods, apparatus and articles of manufacture therefor, are disclosed for annotating documents in a viewer. In one embodiment, the method includes: associating the document with a user; identifying one or more sub-document elements in the document; augmenting the document by making words within the document selectable; displaying the augmented document in the viewer; and responsive to each user input selecting a word in the document displayed in the viewer, further augmenting the document displayed in the viewer by inserting the selected word following the sub-document element from which the word was selected.

Abstract: A system and method for generating an information stream summary using a display metric is provided. An information stream including a plurality of information stream items is received. A display metric is calculated for each of the plurality of information stream items. The information stream items are grouped into one or more summary objects. A size is assigned to each of the one or more summary objects and the one or more summary objects are displayed based on the assigned size.

Abstract: A system and method for providing mixed-initiative curation of information within a shared repository is provided. Static content is retrieved from a shared storage associated with a shared information management client. Dynamic content including one or more information items satisfying a similarity threshold with the static content is identified as similar dynamic content. An interactive visualization is generated within the shared information management client from information extracted from the static content and the similar dynamic content. The interactive visualization of the information is linked with the static content and the similar dynamic content. A selection of the information of the similar dynamic content is received from within the visualization. The static content in the shared storage is updated with the similar dynamic content linked with the selected information.