Abstract: Embodiments of this application disclose a method for training an image caption model, the image caption model including an encoding convolutional neural network (CNN) and a decoding recurrent neural network (RNN). The method includes: obtaining an image eigenvector of an image sample by using the encoding CNN; decoding the image eigenvector by using the decoding RNN, to obtain a sentence used for describing the image sample; determining a matching degree between the sentence obtained through decoding and the image sample and a smoothness degree of the sentence obtained through decoding, respectively; and adjusting the decoding RNN according to the matching degree and the smoothness degree.

Abstract: A video query method includes obtaining a media feature of a query media and a static image feature corresponding to a candidate video. The query media includes the target object, and the candidate video includes the moving object. A video feature of the candidate video is determined according to the static image feature and motion time sequence information of the moving object in the candidate video. Whether the moving object in the candidate video is related to the target object in the query media can be determined according to the media feature and the video feature.

Abstract: A human-model collaborative annotation system for training human annotators includes a database that stores images previously annotated by an expert human annotator and/or a machine learning annotator, a display that displays images selected from the database, an annotation system that enables human annotators to annotate images presented on the display, and an annotation training system. The annotation training system selects an image sample from the database for annotation by a human annotator, receives one or more proposed annotations from the annotation system, compares the human annotator's one or more proposed annotations to previous annotations of the image sample by the expert human annotator or machine learning annotator, presents attention maps on the display to draw the human annotator's attention to any annotation errors identified by the comparing, and selects a next training image sample from the database based on any errors identified in the comparing step.

Abstract: A video query method includes obtaining a media feature of a query media and a static image feature corresponding to a candidate video. The query media includes the target object, and the candidate video includes the moving object. A video feature of the candidate video is determined according to the static image feature and motion time sequence information of the moving object in the candidate video. Whether the moving object in the candidate video is related to the target object in the query media can be determined according to the media feature and the video feature.

Abstract: Embodiments of this application disclose a method for training an image caption model, the image caption model including an encoding convolutional neural network (CNN) and a decoding recurrent neural network (RNN). The method includes: obtaining an image eigenvector of an image sample by using the encoding CNN; decoding the image eigenvector by using the decoding RNN, to obtain a sentence used for describing the image sample; determining a matching degree between the sentence obtained through decoding and the image sample and a smoothness degree of the sentence obtained through decoding, respectively; and adjusting the decoding RNN according to the matching degree and the smoothness degree.

Abstract: A method for automatically selecting thematically representative music is disclosed. A processor is used for using a theme-related keyword to search a keyword-indexed video repository to retrieve videos associated with the theme-related keyword; analyzing the retrieved videos to select videos with music; and extracting music tracks and features from the selected videos. The method further includes selecting representative music related to the theme from the extracted music tracks using the extracted features; and storing the selected representative music in a processor accessible memory.

Abstract: A method for producing a blended video sequence that combines a still image and a video image sequence comprising: designating a first face in the still image, designating a second face in the video image sequence; detecting a series of video frames in the video image sequence containing the second face; identifying a video frame in the detected series of video frames suitable for transitioning from the first face into the second face; using a data processor to automatically produce a transition image sequence where the first face transitions into the second face, and a first background transitions into a second background; and producing the blended video sequence by concatenating the transition image sequence, and a plurality of video frames from the video image sequence starting from the identified video frame.

Abstract: Disclosed are systems and methods for administering cognitive training to a subject in need thereof.

Abstract: A method for improving a stereo image including a left view image and a right view image, comprising: using a data processor to automatically analyze the stereo image to determine an original stereo quality score responsive to relative positions of corresponding points in the left view image and the right view image; specifying a set of one or more candidate modifications to the stereo image; determining revised stereo quality scores based on each of the candidate modifications to the stereo image; selecting a particular candidate modification that produces a revised stereo quality score which indicates a higher quality level than the original stereo quality score; forming an output stereo image corresponding to the selected particular candidate modification; and storing the output stereo image in a processor-accessible memory.

Abstract: To increase the timeliness, objectivity, and efficiency in evaluating surgical procedures such as those performed by ophthalmology residents' learning of cataract surgery, an automatic analysis system for surgeries such as cataract surgery is provided to assess performance, particularly in the capsulorrhexis step on the Kitaro simulator. Computer vision technologies are employed to measure performance of this critical step including duration, centrality, circularity, size, as well as motion stability during the capsulorrhexis procedure. Consequently, a grading mechanism is established based on either linear regression or non-linear classification via Support Vector Machine (SVM) of those computed measures. Comparisons of expert graders to the computer vision based approach have demonstrated the accuracy and consistency of the computerized technique.

Abstract: A method of producing a stereo image from a temporal sequence of digital images, comprising: receiving a temporal sequence of digital images; analyzing pairs of digital images to produce corresponding stereo suitability scores, wherein the stereo suitability score for a particular pair of images is determined responsive to the relative positions of corresponding features in the particular pair of digital image; selecting a pair of digital images including a first image and a second image based on the stereo suitability scores; using a processor to analyze the selected pair of digital images to produce a motion consistency map indicating regions of consistent motion, the motion consistency map having an array of pixels; producing a stereo image pair including a left view image and a right view image by combining the first image and the second image responsive to the motion consistency map; and storing the stereo image pair in a processor-accessible memory.

Abstract: A system and method for measuring distances related to a target object depicted in an image and the construction and delivery of supplemental window materials for fenestration. A digital image is obtained that contains a target object dimension and a reference object dimension in the same plane. The digital image may contain a target object dimension identified by an ancillary object and a reference object dimension in different planes. Fiducial patterns on the reference and optional ancillary objects are used that are recognized by an image analysis algorithm. Information regarding a target object and its immediate surroundings is provided to an automated or semi-automated measurement process, design and manufacturing system such that customized parts are provided to end users. The digital image contains a reference object having a reference dimension and calculating a constraint dimension from the digital image based on a reference dimension.

Abstract: A method for automatically selecting thematically representative music is disclosed. A processor is used for using a theme-related keyword to search a keyword-indexed video repository to retrieve videos associated with the theme-related keyword; analyzing the retrieved videos to select videos with music; and extracting music tracks and features from the selected videos. The method further includes selecting representative music related to the theme from the extracted music tracks using the extracted features; and storing the selected representative music in a processor accessible memory.

Abstract: To increase the timeliness, objectivity, and efficiency in evaluating surgical procedures such as those performed by ophthalmology residents' learning of cataract surgery, an automatic analysis system for surgeries such as cataract surgery is provided to assess performance, particularly in the capsulorrhexis step on the Kitaro simulator. Computer vision technologies are employed to measure performance of this critical step including duration, centrality, circularity, size, as well as motion stability during the capsulorrhexis procedure. Consequently, a grading mechanism is established based on either linear regression or non-linear classification via Support Vector Machine (SVM) of those computed measures. Comparisons of expert graders to the computer vision based approach have demonstrated the accuracy and consistency of the computerized technique.

Abstract: A Method and System For Recognizing and Assessing Surgical Procedures from a video or series of still images is described. Evaluation of surgical techniques of residents learning skills in areas such as cataract surgery is an important aspect of the learning process. The use of videos has become common in such evaluations, but is a time consuming manual process. The present invention increases the efficiency and speed of the surgical technique evaluation process by identifying and saving only information that is relevant to the evaluation process. Using image processing techniques of the present invention, an anatomic structure of a surgical procedure is located on a video, timing of predefined surgical stages is determined, and measurements are taken from frames of the predefined surgical stages to allow the performance of a surgeon to be assessed in an automated and efficient manner.

Abstract: A method of producing a stereo image from a digital video includes receiving a digital video including a plurality of digital images captured by an image capture device; and using a processor to produce stereo suitability scores for at least two digital images from the plurality of digital images. The method further includes selecting a stereo candidate image based on the stereo suitability scores; producing a stereo image from the selected stereo candidate image wherein the stereo image includes the stereo candidate image and an associated stereo companion image based on the plurality of digital images from the digital video; and storing the stereo image whereby the stereo image can be presented for viewing by a user.

Abstract: A method for automatically selecting thematically representative music is disclosed. A processor is used for using a theme-related keyword to search a keyword-indexed video repository to retrieve videos associated with the theme-related keyword; analyzing the retrieved videos to select videos with music; and extracting music tracks and features from the selected videos. The method further includes selecting representative music related to the theme from the extracted music tracks using the extracted features; and storing the selected representative music in a processor accessible memory.

Abstract: A method of computing at least one photogenic route from a starting location to a destination location, including; computing photogenic values for images in a large collection representing a geographic region that includes the starting location and the destination location; computing a photogenic index for each route segment based on computed photogenic values of images taken along the route segment; computing at least one photogenic route from the starting location to the destination location and presenting the route(s) to a user.

Abstract: A method for representing a video sequence including a time sequence of input video frames, the input video frames including some common scene content that is common to all of the input video frames and some dynamic scene content that changes between at least some of the input video frames. Affine transform are determined to align the common scene content in the input video frames. A common video frame including the common scene content is determined by forming a sparse combination of a first basis functions. A dynamic video frame is determined for each input video frame by forming a sparse combination of a second basis functions, wherein the dynamic video frames can be combined with the respective affine transforms and the common video frame to provide reconstructed video frames.

Abstract: A method for determining a scene boundary location dividing a first scene and a second scene in an input video sequence. The scene boundary location is determined responsive to a merit function value, which is a function of the candidate scene boundary location. The merit function value for a particular candidate scene boundary location is determined by representing the dynamic scene content for the input video frames before and after candidate scene boundary using sparse combinations of a set of basis functions, wherein the sparse combinations of the basis functions are determined by finding a sparse vector of weighting coefficients for each of the basis functions. The weighting coefficients determined for each of the input video frames are combined to determine the merit function value. The candidate scene boundary providing the smallest merit function value is designated to be the scene boundary location.