Abstract: A novel interpretable and steerable deep sequence modeling technique is disclosed. The technique combines prototype learning and RNNs to achieve both interpretability and high accuracy. Experiments and case studies on different real-world sequence prediction/classification tasks demonstrate that the model is not only as accurate as other state-of-the-art machine learning techniques but also much more interpretable. In addition, a large-scale user study on Amazon Mechanical Turk demonstrates that for familiar domains like sentiment analysis on texts, the model is able to select high quality prototypes that are well aligned with human knowledge for prediction and interpretation. Furthermore, the model obtains better interpretability without a loss of performance by incorporating the feedback from a user study to update the prototypes, demonstrating the benefits of involving human-in-the-loop for interpretable machine learning.

Abstract: A system includes a display device, a memory configured to store a visual analysis application and image data including a plurality of images including detectable objects; and a processor, operatively connected to the memory and the display device. The processor is configured to execute the visual analysis application to learn generative factors from objects detected in the plurality of images, visualize the generative factors in a user interface provided to the display device, receive grouped combinations of the generative factors and values to apply to the generative factors to control object features, create generated objects by applying the values of the generative factors to the objects detected in the plurality of images, combine the generated objects into the original images to create generated images, and apply a discriminator to the generated images to reject unrealistic images.

Abstract: A multimodal visual time aware graphical display system provides point of interest inquiry controllable and dynamic graphical effects on visualization such that when time and traffic condition aware graphics is enabled. The system processes traffic information based on current and past traffic data and visualize the geographical boundaries.

Abstract: A computing system includes a processing system with at least one processing unit. The processing system is configured to receive a depth map with a first boundary of an object. The processing system is configured to receive a color image that corresponds to the depth map. The color image includes a second boundary of the object. The processing system is configured to extract depth edge points of the first boundary from the depth map. The processing system is configured to identify target depth edge points on the depth map. The target depth edge points correspond to color edge points of the second boundary of the object in the color image. In addition, the processing system is configured to snap the depth edge points to the target depth edge points such that the depth map is enhanced with an object boundary for the object.

Abstract: Motion windows are generated from a query activity sequence. For each of the motion windows in the query activity sequence, a corresponding motion window in the reference activity sequence is found. One or more difference calculations are performed between the motion windows of the query activity sequence and the corresponding motion windows in the reference activity sequence based on at least one criterion associated with physical meaning. Abnormality of the motion windows is determined based on the one or more difference calculations. A standardized evaluation result of the query activity sequence is output based on the detected abnormal motion windows in the query activity sequence.

Abstract: Using a global optimization, a cycle within a frame buffer including frames corresponding to one or more cycles of query activity sequences is detected. The detection includes creating a plurality of cycle segmentations by recursively iterating through the frame buffer to identify candidate cycles corresponding to cycles of a reference activity sequence until the frame buffer lacks sufficient frames to create additional cycles, computing segmentation errors for each of the plurality of cycle segmentations, and identifying the detected cycle as the one of the plurality of cycle segmentations having a lowest segmentation error. Cycle duration data for the detected cycle is generated. Frames belonging to the detected cycle are removed from the frame buffer. The cycle duration data is output.

Abstract: A method for generating a graphical display of a bipartite graph includes receiving bipartite graph data, generating, a first meta-node including at least two nodes in the first set of nodes in the bipartite graph data and a second meta-node including at least two nodes in a second set of nodes in the bipartite graph data based on the bipartite graph data using a minimum description length (MDL) optimization process to generate the first meta-node and the second meta-node. The method further includes generating a first graphical depiction of the first meta-node and the second meta-node, the graphical depiction including a single edge connecting the first meta-node and the second meta-node to provide a summarized display of the bipartite graph data.

Abstract: A method for motion estimation in an augmented reality (AR) system includes receiving inertial sensor data and image data during movement of the AR system generating a probability map based on the inertial sensor data and the image data, the probability map corresponding to one frame in the image data and including probability values indicating that each pixel in the one frame is in an inertial coordinate frame or a local coordinate frame with a convolutional neural network encoder/decoder, identifying visual observations of at least one landmark in the local coordinate frame based on the image data and the probability map, and generating an estimate of secondary motion in the local coordinate frame based on a first prior state in a hidden Markov model (HMM) corresponding to the local coordinate frame and the visual observations of the at least one landmark in the local coordinate frame.

Abstract: A system for determining a quantitative accuracy of a test movement relative to a reference movement includes a display output device, a memory, and a processor operatively connected to the display output device and the memory. The memory stores motion capture data and programming instructions. The processor executes the programming instructions to determine a quantitative accuracy of the test movement relative to the reference movement. A method, executable by the processor, for determining the quantitative accuracy includes receiving, with the processor, motion capture data that includes the reference movement and the test movement. The motion data is split into individual movements, and the test movement is aligned with the reference movement. The processor computes a quantitative accuracy of the test movement relative to the reference movement, and generates, with the display output device, a visualization representative of the test movement. The computed accuracy is encoded into the visualization.

Abstract: A navigational system includes a hybrid-dimensional visualization scheme with a multi-modal interaction flow to serve for digital mapping applications, such as in car infotainment systems and online map services. The hybrid-dimensional visualization uses an importance-driven or focus-and-context visualization approach to combine the display of different map elements, including 2D map, 2D building footprint, 3D map, weather visualization, realistic day-night lighting, and POI information, into a single map view. The combination of these elements is guided by intuitive user interactions using multiple modalities simultaneously, such that the map information is filtered to best respond to the user's request, and presented in a way that presents both the focus and the context in a map in an aesthetic manner. The system facilitates several use cases that are common to the users, including destination preview, destination search, and virtual map exploration.

Abstract: A novel map representation called Probabilistic Surfel Map (PSM) for dense visual SLAM is disclosed. The PSM maintains a globally consistent map with both photometric and geometric uncertainties encoded in order to address inconsistency and sensor noise. A key aspect of the visual SLAM method disclosed herein is the proper modeling and updating of the photometric and geometric uncertainties encoded in the PSM. Strategies for applying the PSM for improving both the front-end pose estimation and the back-end optimization are disclosed. Moreover, the PSM enables generation of a high quality dense point cloud with high accuracy.

Abstract: A method and device for determining a footprint of a 3D structure are disclosed. The method includes: receiving mesh data for the 3D structure that includes vertices and edges that form polygons of the 3D structure; determining a connection graph including candidate nodes and candidate lines by (i) identifying all edges having a vertex that is less than threshold height value, and (ii) mapping the identified edges and vertices thereof onto a 2D plane; determining an adjacency list that indicates, for each candidate node, which other candidate nodes are connected to the candidate node by a candidate line; and generating a footprint of the 3D structure based on the connection graph and the adjacency list, the footprint including vertices corresponding to a selection of the candidate nodes and including edges corresponding to a selection of the candidate lines.

Abstract: A method for displaying graphics of clouds in a three-dimensional (3D) virtual environment includes generating a filtered texture based on a threshold filter applied to a cloud texture where the filter threshold corresponds to cloud coverage information in weather data of a geographic region. The method further includes mapping the filtered texture to a geometric surface corresponding to a sky dome in the 3D virtual environment, coloring a plurality of texels in the mapped filtered texture on the geometric surface stored in the memory based on an isotropic single-scatter color model, and generating a graphical depiction of the 3D virtual environment including at least a portion of the geometric surface corresponding to the sky dome with clouds based on the plurality of texels of the filtered texture that are colored and mapped to the geometric surface.

Abstract: A visual SLAM system comprises a plurality of keyframes including a keyframe, a current keyframe, and a previous keyframe, a dual dense visual odometry configured to provide a pairwise transformation estimate between two of the plurality of keyframes, a frame generator configured to create keyframe graph, a loop constraint evaluator adds a constraint to the receiving keyframe graph, and a graph optimizer configured to produce a map with trajectory.

Abstract: A method for motion estimation in an augmented reality (AR) system includes receiving inertial sensor data and image data during movement of the AR system generating a probability map based on the inertial sensor data and the image data, the probability map corresponding to one frame in the image data and including probability values indicating that each pixel in the one frame is in an inertial coordinate frame or a local coordinate frame with a convolutional neural network encoder/decoder, identifying visual observations of at least one landmark in the local coordinate frame based on the image data and the probability map, and generating an estimate of secondary motion in the local coordinate frame based on a first prior state in a hidden Markov model (HMM) corresponding to the local coordinate frame and the visual observations of the at least one landmark in the local coordinate frame.

Abstract: A method for extracting a pattern from spatio-temporal (ST) data includes receiving ST data, storing the ST data as s multi-dimensional array in a memory, and extracting at least one pattern from the ST data. The extracting includes generating a model approximating at least a portion of the array, and generating a visualization of a loading vector of the approximation. The ST data includes records with multiple categories of information, one of which is spatial, and one of which is temporal. Each dimension corresponds to a respective one of the categories of information. Generating the model includes applying tensor decomposition to the array, and extracting the at least one loading vector of the approximation. The extracted loading vector is indicative of a pattern in the ST data.

Abstract: A system and method for providing a location based interactive informational display includes processing circuitry outputting on a display device a map of a region represented with a first level of detail and including a location focusing graphical indicia overlaid on a sub-region of the map, outputting on the display device a details frame that includes information of the sub-region with a second level of detail that is higher than the first level of detail, receiving user input for modifying the sub-region on which the location focusing graphical indicia is overlaid, and, responsive to the user input, modifying the display of the location focusing graphical indicia and modifying the details frame to include information at the second level of detail corresponding to the modified sub-region.

Abstract: A method for generating a graphical display of a bipartite graph includes receiving bipartite graph data, generating, a first meta-node including at least two nodes in the first set of nodes in the bipartite graph data and a second meta-node including at least two nodes in a second set of nodes in the bipartite graph data based on the bipartite graph data using a minimum description length (MDL) optimization process to generate the first meta-node and the second meta-node. The method further includes generating a first graphical depiction of the first meta-node and the second meta-node, the graphical depiction including a single edge connecting the first meta-node and the second meta-node to provide a summarized display of the bipartite graph data.

Abstract: A method for automated generation of curves for computer graphics includes receiving a first control point corresponding to a first location in a two-dimensional plane and a second control point corresponding to a second location in the two-dimensional plane, receiving at least one constraint for at least one of the first control point and the second control point, and generating a first set of parameter points to generate a first spline connecting the first control point to the second control point based on the at least one constraint using a multigrid solver, the first set of parameter points including a first predetermined number of parameter points, each parameter point in the first set of parameter points corresponding to a location in the two-dimensional plane and a one-dimensional curvature parameter.

Abstract: A method for generating graphics of a three-dimensional (3D) virtual environment includes: receiving, with a processor, a first camera position in the 3D virtual environment and a first viewing direction in the 3D virtual environment; receiving, with the processor, weather data including first precipitation information corresponding to a first geographic region corresponding to the first camera position in the 3D virtual environment; defining, with the processor, a bounding geometry at first position that is a first distance from the first camera position in the first viewing direction, the bounding geometry being dimensioned so as to cover a field of view from the first camera position in the first viewing direction; and rendering, with the processor, a 3D particle system in the 3D virtual environment depicting precipitation only within the bounding geometry, the 3D particle system having features depending on the first precipitation information.