Abstract: A command map GUI that illustrates command usage patterns of one or more entities. A server receives and stores command usage data from a plurality of users/clients, each user/client executing a software application having a set of commands. A current user/client displays, via a GUI, graphical representations of the command usage data received from the server. The server receives new command usage data from a user/client included in the one or more entities. In response, the server automatically updates command usage data for the one or more entities, pushes the updated command usage data to the current user/client, and causes the GUI to display graphical representations of the updated command usage data. Thus, the command usage data of the one or more entities is automatically updated and displayed in the GUI in real-time or near real-time as new commands are continually invoked by users in the one or more entities.

Abstract: A command map GUI that illustrates command usage patterns of a first entity and/or a comparison between the first entity and a second entity. A server receives and stores command usage data from a plurality of clients, each client executing a software application that enables a set of commands. A client displays the GUI based on command usage data received from the server. The GUI displays a circle chart comprising a plurality of segments representing various command categories, each segment including a wedge that represents the amount of usage of the corresponding command category. The GUI also displays a plurality of data points, each data point representing a command, wherein the distance from the center of the circle chart represents the amount of usage of the corresponding command. The GUI may display a certification and/or an unused command recommended for a user based on command usage patterns of the user.

Abstract: A command map GUI that illustrates command usage patterns of a first entity and/or a comparison between the first entity and a second entity. A server receives and stores command usage data from a plurality of clients, each client executing a software application that enables a set of commands. A client displays the GUI based on command usage data received from the server. The GUI displays a circle chart comprising a plurality of segments representing various command categories, each segment including a wedge that represents the amount of usage of the corresponding command category. The GUI also displays a plurality of data points, each data point representing a command, wherein the distance from the center of the circle chart represents the amount of usage of the corresponding command. The GUI may display a certification and/or an unused command recommended for a user based on command usage patterns of the user.

Abstract: Methods, systems, and apparatus, including medium-encoded computer program products, for searching through spatial recordings by constructing queries. One of the methods include constructing a query based on input received at a three-dimensional (3D) space displayed at a visual programming interface, wherein the input includes positioning of objects at 3D positions within the 3D space over time, wherein the input defines at least one spatial orientation between at least two objects from the objects in the 3D space; executing the query to search a database of 3D recordings to find at least one segment from at least one 3D recording that includes the at least two objects and matches the spatial orientation between the at least two objects, as defined in the input, at a specific point in time; and presenting the at least one segment from the at least one 3D recording via the visual programming interface.

Abstract: One embodiment of the present invention sets forth a technique for designing and generating a smart object. The technique includes receiving a first input indicating a smart object behavior of a smart object that includes a smart device embedded in a three-dimensional (3D) object; in response to the input, generating computer instructions for the smart device, wherein the computer instructions, when executed by the smart device, cause the smart object to implement the smart object behavior; and transmitting the computer instructions to the smart device.

Abstract: A command map GUI that illustrates command usage patterns of a first entity and/or a comparison between the first entity and a second entity. A server receives and stores command usage data from a plurality of clients, each client executing a software application that enables a set of commands. A client displays the GUI based on command usage data received from the server. The GUI displays a circle chart comprising a plurality of segments representing various command categories, each segment including a wedge that represents the amount of usage of the corresponding command category. The GUI also displays a plurality of data points, each data point representing a command, wherein the distance from the center of the circle chart represents the amount of usage of the corresponding command. The GUI may display a certification and/or an unused command recommended for a user based on command usage patterns of the user.

Abstract: A subtask assignment system for assisting groups of workers to complete a task more efficiently. The task may comprise a physical assembly task of an object. The physical assembly task comprises a plurality of subtasks that must each be completed to complete the physical assembly task. The subtask assignment system may include an assignment server connected to a plurality of user devices via a network, each user device being operated by a particular worker. The assignment server may execute an assignment engine that receives inputs describing the overall task and group of workers, generates a task model representing the task based on the inputs, populates a parameters table based on the inputs, and automatically determines subtask assignments for the group of workers based on the task model and the parameters table. The assignment server determines an optimal subtask to assign to each worker based on the parameters.

Abstract: Features described herein generally relate to providing spatiotemporal guidance within a 360-degree video. Particularly, while a 360-degree video is being displayed to a user, a current location of the view of the user within the 360-degree video is determined. Additionally, regions of interest may be identified within the 360-degree video, along with their location and a time in which the regions of interest are active within the 360-degree video. A visual guide is then overlaid onto the 360-degree video that indicates a current location of the view of the user, as well as locations of one or more regions of interest, as well as the time during which each region of interest is active (able to be viewed) within the 360-degree video. By viewing the visual guide, the user may be able to anticipate a location and timing of regions of interest within the 360-degree video and adjust their gaze to the correct location at the correct time to view such regions of interest.

Abstract: An Adaptive Tangible User Interface (ATUI) for use in an extended reality environment in which tangible interfaces are composed in real time based on identified affordances of existing objects in the physical environment and the input tasks of a user. An extended reality system can be provided with instructions executable by one or more processors to perform processing including: generating a representation of the real-world environment within a field of view of the user; identifying physical objects within the real-world environment; generating a set of possible performable gestures afforded by available object affordance factors; determining potential input tasks; composing performable gestures for the potential input tasks; and selecting a physical object for use as an adaptive tangible user interface. Techniques for designing a virtual user interface, overlaying the virtual user interface on a selected ATUI physical object, and maintaining alignment of the virtual user interface, are also provided.

Abstract: One embodiment of the invention disclosed herein provides a system that includes a mirror apparatus with a first surface to which a half-silvered mirror film is applied, where the mirror apparatus transmits a transmitted image from a second surface to the first surface. The system further includes a servo-controlled dimmer that adjusts a level of ambient light associated with the mirror apparatus. The system further includes a motion sensing device that tracks positions of a plurality of points associated with an object; wherein the object is situated on the half-silvered mirror film side of the mirror apparatus. The system further includes a computing device including a memory that stores instructions that, when executed by a processor included in the computing device, cause the processor to control the servo-controlled dimmer to adjust the ambient light such that both the transmitted image and a reflected image is visible on the first surface.

Abstract: Methods, systems, and apparatus, including medium-encoded computer program products, for collaboration and view sharing between users when performing editing operations over a shared document. A first portion of a shared document is displayed to a first user in a user interface of a first instance of a collaboration application of a first user. The displayed first portion comprises a first location of the first user within the shared document. In the user interface, an indication specifying a relative locational direction from the first location towards a second location of a second user within the shared document is provided. A second portion of the shared document is being displayed to the second user through a second instance of the collaboration application during a conference call between a set of users, where the displayed second portion includes the second location of the second user within the shared document.

Abstract: A hybrid workstation enables a virtual reality (VR) interface, a traditional (TD) interface, and transitions between the interfaces. The VR interface comprises three-dimensional (3D)-based software and hardware components. The TD interface comprises two-dimensional (2D)-based software and hardware components. The state of the hybrid workstation is defined by three parameters comprising interface (VR interface or TD interface), position (seated or standing), and movement (stationary or room-scale). The hybrid workstation detects a transition from a current state to a next state upon determining that any of the three parameters have changed. The hybrid workstation then determines a transition response based on the particular transition that is detected. The transition response comprises a set of operations that are performed on the VR interface and/or the TD interface that mitigate the disruption and inefficiency caused when the particular transition occurs.

Abstract: A finger device initiates actions on a computer system when placed in contact with a surface. The finger device includes instrumentation that captures images and gestures. When in contact with a surface, the finger device captures images of the surface and gestures made on the surface. The finger device also transmits the images and gesture data to the computer system. An application on the computer system matches the images received from the finger device to a representation of the surface, identifies an action associated with the surface representation and gesture, and executes the action. Instrumenting the finger instead of the surface, allows a user to configure virtually any surface to accept touch input.

Abstract: A computer-implemented method for generating a virtual collection of digital materials includes: generating a virtual three-dimensional (3D) design workspace; and, in response to a first operation that is associated with a first media file: automatically generating a virtual 3D representation of the first media file; and automatically incorporating the virtual 3D representation of the first media file into the virtual 3D design workspace.

Abstract: One embodiment of the present application sets forth a computer-implemented method for remotely monitoring performance of a task in real time, comprising receiving, in real time from a first student computing device, first student data that includes a first set of actions performed in a first student application instance executing on the first student device, comparing a first student action included in the first set of actions to a first target action, in response to comparing the first student action to the first target action, identifying a first issue indicator associated with the first student action, determining a first remedy associated with at least one of the first target action and the first issue indicator, and performing a first remediation action associated with the first remedy.

Abstract: In various embodiments, a pattern-based recommendation subsystem automatically recommends workflows for software-based tasks. In operation, the pattern-based recommendation subsystem computes an expected distribution of frequencies across command patterns based on different distributions of frequencies across the command patterns. The expected distribution of frequencies is associated with a target user, and each different distribution of frequencies is associated with a different user. The pattern-based recommendation subsystem then applies a set of commands associated with the target user to a trained machine-learning model to determine a target distribution of weights applied to a set of tasks. Subsequently, the pattern-based recommendation subsystem determines a training item based on the expected distribution of frequencies and the target distribution of weights. The pattern-based recommendation subsystem generates a recommendation that specifies the training item.

Abstract: Approaches for generating virtual representations of smart objects in a 3D visual programming interface. The interface displays a 3D virtual environment containing virtual objects that represents a real-world environment containing smart objects. The 3D virtual environment displays virtual objects in a manner that is spatially accurate relative to the physical objects in the real-world environment. For each virtual object representing a physical object, a logic node (port node) is displayed, the port node representing the set of functions associated with the physical object. The interface enables users to create, delete, or modify different types of logic nodes (representing functions) and create, delete, or modify links (representing data connections) between logic nodes within the 3D virtual environment. The authoring of the logic nodes and links produces an executable program.

Abstract: In various embodiments, a task-based recommendation subsystem automatically recommends workflows for software-based tasks based on a trained machine-learning model that maps different sets of commands to different distributions of weights applied to a set of tasks. In operation, the task-based recommendation subsystem applies a first set of commands associated with a target user to the trained machine-learning model to determine a target distribution of weights applied to the set of tasks. The task-based recommendation subsystem then performs processing operation(s) based on at least two different distributions of weights applied to the set of tasks and the target distribution to determine a training item. Subsequently, the task-based recommendation subsystem generates a recommendation that specifies the training item. Finally, the task-based recommendation subsystem transmits the recommendation to a user to assist the user in performing a particular task.

Abstract: An automated robot design pipeline facilitates the overall process of designing robots that perform various desired behaviors. The disclosed pipeline includes four stages. In the first stage, a generative engine samples a design space to generate a large number of robot designs. In the second stage, a metric engine generates behavioral metrics indicating a degree to which each robot design performs the desired behaviors. In the third stage, a mapping engine generates a behavior predictor that can predict the behavioral metrics for any given robot design. In the fourth stage, a design engine generates a graphical user interface (GUI) that guides the user in performing behavior-driven design of a robot. One advantage of the disclosed approach is that the user need not have specialized skills in either graphic design or programming to generate designs for robots that perform specific behaviors or express various emotions.

Abstract: In one embodiment of the present invention, a method for multiple device interaction includes detecting an orientation of a first device relative to a second device. The method also includes detecting a first gesture performed with either the first device or the second device, wherein the first gesture causes a first action that is based at least in part on the orientation of the first device relative to the second device.