Abstract: In one aspect, lines in image data of an event are automatically found and repaired. For example, the event may be a sporting event which is played on a field, and the line segment is a field line on the field which may be obscured by a player, game ball or other object. The line segment is automatically detected in a mask image, and a portion of the line segment which is occluded by the object is automatically determined, and the object is automatically removed. The line segment can also be repaired. Optionally, a virtual viewpoint of the event is provided from the image, with the line repaired and the object removed. In another aspect, an object in an image of an event is automatically located by detecting blobs in the image which meet at least one specified criterion, such as size, aspect ratio, density or color profile.

Abstract: 3d textured objects are provided for virtual viewpoint animations. In one aspect, an image of an event is obtained from a camera and an object in the image is automatically detected. For example, the event may be a sports event and the object may be a stationary object which is detected based on a known location, color and shape. A 3d model of the object is combined with a textured 3d model of the event to depict a virtual viewpoint which differs from a viewpoint of the camera. The textured 3d model of the event has texture applied from an image of the event, while the 3d model of the object does not have such texture applied, in one approach. In another aspect, an object in the image such as a participant in a sporting event is represented in the virtual viewpoint by a textured 3d kinematics model.

Abstract: An object is detected in images of a live event by storing and indexing templates based on representations of the object from previous images. For example, the object may be a vehicle which repeatedly traverses a course. A first set of images of the live event is captured when the object is at different locations in the live event. A representation of the object in each image is obtained, such as by image recognition techniques, and a corresponding template is stored. When the object again traverses the course, for each location, the stored template which is indexed to the location can be retrieved for use in detecting the object in a current image. The object's current location may be obtained from GPS data from the object, for instance, or from camera sensor data, e.g., pan, tilt and zoom, which indicates a direction in which the camera is pointed.

Abstract: Camera registration and/or sensor data is updated during a live event by determining a difference between an estimated position of an object in an image and an actual position of the object in the image. The estimated position of the object in the image can be based on an estimated position of the object in the live event, e.g., based on GPS or other location data. This position is transformed to the image space using current camera registration and/or sensor data. The actual position of the object in the image can be determined by template matching which accounts for an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image. The updated camera registration/sensor data can be used in detecting an object in a subsequent image.

Abstract: An object is detected in images of a live event by storing and indexing camera registration-related data from previous images. For example, the object may be a vehicle which repeatedly traverses a course. A first set of images of the live event is captured when the object is at different locations in the live event. The camera registration-related data for each image is obtained and stored. When the object again traverses the course, for each location, the stored camera registration-related data which is indexed to the location can be retrieved for use in estimating a position of a representation of the object in a current image, such as by defining a search area in the image. An actual position of the object in the image is determined, in response to which the camera registration-related data may be updated, such as for use in a subsequent traversal of the course.

Abstract: A video broadcast of a live event is enhanced by providing graphics in the video in real time to depict the fluid flow around a moving object in the event and to provide other informative graphics regarding aerodynamic forces on the object. A detailed flow field around the object is calculated before the event, on an offline basis, for different speeds of the object and different locations of other nearby objects. The fluid flow data is represented by baseline data and modification factors or adjustments which are based on the speed of the object and the locations of the other objects. During the event, the modification factors are applied to the baseline data to determine fluid flow in real time, as the event is captured on video. In an example implementation, the objects are race cars which transmit their location and/or speed to a processing facility which provides the video.

Abstract: A representation of an object in an image of a live event is detected by matching potential representation of the object against multiple types of templates. For example, the templates can include monochrome data, chrominance and/or luminance data, pixel data of the object from an earlier image, e.g., as a video template, an edge and morphology based template, a model of the object, or a predetermined static texture which is based on an appearance of the object. A weighting function may also be used. In one possible approach, a first type of template is used in an initial search area, and a second type of template is used in a smaller region of the initial search area. Based on a position of the optimum representation of the object in the image, a graphic can be provided in the image, or sensor and/or registration data of a camera can be updated.

Abstract: A representation of an object in an image of a live event is obtained by determining a color profile of the object. The color profile may be determined from the image in real time and compared to stored color profiles to determine a best match. For example, the color profile of the representation of the object can be obtained by classifying color data of the representation of the object into different bins of a color space, in a histogram of color data. The stored color profiles may be indexed to object identifiers, object viewpoints, or object orientations. Color data which is common to different objects or to a background color may be excluded. Further, a template can be used as an additional aid in identifying the representation of the object. The template can include, e.g., a model of the object or pixel data of the object from a prior image.

Abstract: A representation of an object in a live event is detected in an image of the event. A location of the object in the live event is translated to an estimated location in the image based on camera sensor and/or registration data. A search area is determined around the estimated location in the image. A direction of motion of the object in the image is also determined. A representation of the object is identified in the search area by detecting edges of the object, e.g., perpendicular to the direction of motion and parallel to the direction of motion, performing morphological processing, and matching against a model or other template of the object. Based on the position of the representation of the object, the camera sensor and/or registration data can be updated, and a graphic can be located in the image substantially in real time.

Abstract: A method is disclosed for three-dimensional rendering of a live event such as an automobile race over a client device such as a personal computer. Moving and stationary objects at the event may be rendered using computer-generated graphics of the moving and stationary objects and real time positional data giving the position of the moving objects over time. The positional data may be streamed to the client device and displayed in real time or substantially real time.

Abstract: A system adds a graphical image of the strike zone to a video or other image of a baseball game. The system determines location of the strike zone and the ball in real space. The locations of the strike zone and the ball are depicted in the video. Based on knowing the locations of the strike zone and the ball, the system can determines whether the pitch was a strike or a ball.

Abstract: A video effect is created that provides an experience to a viewer of freezing time during an event that is the subject of a video presentation, investigating the event during that frozen moment in time, and (optionally) resuming the action of the event. During that frozen moment in time, the video can move around the scene of the event and/or zoom in (or out) to better highlight an aspect of the event. In one embodiment, there will be a transition from video captured by a broadcast camera (or another camera) to a high resolution still image, movement around the high resolution still image, and a transition from the high resolution still image back to video from the broadcast camera (or another camera).

Abstract: A telestrator system is disclosed that allows a broadcaster to annotate video during or after an event. For example, while televising a sporting event, an announcer (or other user) can use the present invention to draw over the video of the event to highlight one or more actions, features, etc. In one embodiment, when the announcer draws over the video, it appears that the announcer is drawing on the field or location of the event. Such an appearance can be performed by mapping the pixels location from the user's drawing to three dimensional locations at the event. Other embodiments include drawing on the video without obscuring persons and/or other specified objects, and/or smoothing the drawings in real time.

Abstract: A system is disclosed for using camera attitude sensors with a camera. A camera assembly includes a tripod base, a tripod head interface mounted on the tripod base, a tripod head mounted on the tripod head interface and a camera mounted on the tripod head. The tripod head enables the camera to pan and tilt. The system also includes a first optical encoder for detecting the amount that the camera has been panned and a second optical encoder for detecting the amount that the camera has been tilted. Two inclinometers are mounted on the tripod head interface to measure attitude of the tripod head. Two gyroscopes (“gyros”) are mounted on the camera assembly. Data from the encoders, gyros and inclinometers are packaged and sent to graphics production equipment to be used for enhancing video captured by the camera.

Abstract: A system is disclosed for blending two image that makes use of a color map which indicates colors in a foreground can be mixed with the background and how much of each source to mix. One embodiment of the invention restricts the use of the color map to only pixels in the foreground that correspond to a graphic (or effect) in the background. Another embodiment makes use of a gray scale matte which stores blending values for each pixel in the foreground.

Abstract: A system is disclosed for enhancing video by use of a virtual surface. One or more positions are identified in a first image. These one or more positions are transformed to one or more locations in relation to the virtual surface. In subsequent video images (e.g. fields, frames, or other units), the one or more locations in relation to said virtual surface are transformed to one or more positions in the subsequent video images. The subsequent video images are enhanced based on the one or more transformed positions.

Abstract: A system uses GPS receivers and other sensors to acquire data about one or more objects at an event. The data acquired by the GPS receivers and the sensors is used to determine various statistics about the objects and/or enhance a video presentation of the objects. In one embodiment, the acquired data is used to determine three dimensional positions of the objects, determine the positions of images of the objects in a video and enhance the video accordingly. One exemplar use of the present invention is with a system for tracking automobiles at a race. The system determines statistics about the automobiles and enhances a video presentation of the race.

Abstract: A track model is created for use with a GPS receiver. In one embodiment, the track model is a set of planar surfaces which approximate the contiguous surface on which navigation takes place. The GPS receiver searches for an appropriate planar surface associated with its approximate position. Having found the appropriate planar surface, the GPS receiver constrains its position using the planar surface associated with its approximate position. Using the track model improves the accuracy of the computed position at the time and improves the ambiguity estimation process so that positions with greatly improved accuracy are available sooner.

Abstract: A system is disclosed that uses GPS and additional data to determine the location of an object. Typically, GPS receivers need valid data from four satellites to accurately determine a three dimensional location. If a GPS receiver is receiving valid data from fewer than four satellites, then additional data is used to compensate for the shortage of satellites in view of the GPS receiver. Examples of additional data includes a representation of the surface that the object is traveling on, an accurate clock, an odometer, dead reckoning information, pseudolite information, and error correction information from a differential reference receiver. An exemplar use of the disclosed system is to concurrently track a set of one or more automobiles during a race. The determined locations of the automobile can be used to provide route information, to generate statistics and/or to edit video of one or more of the automobiles.

Abstract: A three-dimensional model is created to represent an environment to be captured on video. A camera is fitted with pan, tilt and/or zoom sensors. An operator selects a location in the environment. The three-dimensional model is used to determine the three-dimensional coordinates of the location selected by the operator. Information from the pan, tilt and/or zoom sensors is used to transform the three-dimensional coordinates to a two-dimensional position in the video from the camera. Using the two-dimensional position of the video, a graphic is properly added to the video such that the graphic appears to be at the selected location in the environment.