SYSTEM AND METHOD FOR OBJECT RECOGNITION AND TRACKING IN A VIDEO STREAM
The invention provides a system method for object detection and tracking in a video stream. Frames of the video stream are divided into regions of interest and a probability is calculated for each region of interest that the region contains at least a portion of an object to be tracked. The regions of interest in each frame are then classified based on the calculated probabilities. A region of interest (RI) frame is then constructed for each video frame that reports the classification of regions of interest in the video frame. Two or more RI frames are then compared in order to determine a motion of the object. The invention also provides a system executing the method of the invention, as well as a device comprising the system. The device may be for example, a portable computer, a mobile telephone, or an entertainment device.
This invention relates to methods and systems for object detection and tracking, and to devices containing such systems.
BACKGROUND OF THE INVENTIONThe following prior art publications are considered relevant for an understanding of the invention:
Digital Image Processing by Rafael C. Gonzalez, Richard E. Woods and Steven L. Eddins, Prentice Hall (2004), 10.4.2—Region Growing.
E. Deja, M. M. Deja, Dictionary of Distances, Elsevier (2006).
Mahalanobis, P C (1936). “On the generalised distance in statistics”. Proceedings of the National Institute of Sciences of India 2 (1): 49-55).
Itakura F., “Line spectrum representation of linear predictive coefficients of speech signals,” J. Acoust.Soc. Am., 57, 537(A), 1975.
James M. Abello, Panos M. Pardalos, and Mauricio G. C. Resende (editors) (2002). Handbook of Massive Data Sets. Springer.
E. R. Berlekamp, Algebraic Coding Theory, McGraw-Hill 1968.
Richard W. Hamming. Error Detecting and Error Correcting Codes, Bell System Technical Journal 26(2):147-160, 1950.
Dan Gusfield. Algorithms on strings, trees, and sequences: computer science and computational biology. Cambridge University Press, New York, N.Y., USA, 1997).
U.S. Pat. Nos. 5,767,842 and 6,650,318
Entering data into a data processing device is accomplished using a data input device such as a keyboard, mouse, or joystick. Although electronic devices are constantly being miniaturized, the size of the various associated data input devices cannot be substantially decreased since they must conform to the size of a user's hands. Methods for inputting data have therefore been devised in which the user's hands do not have to touch the device. U.S. Pat. No. 5,767,842 to Korth, and U.S. Pat. No. 6,650,318 to Amon for example, disclose an optical system in which a camera is used to monitor a user's hand and finger motions. A software application interprets these motions as operations on a physically non-existent computer keyboard or other input device. In these systems, the camera has a fixed position, so that the background of the images remains constant. This allows the software application to make use of information present in the constant background in order to detect the user's hands in each image. This system, therefore, cannot be used in a device that in use is moved because, in this case, the background of the images is not constant, so there is no reliable background information in the images. Devices that are moved in use include hand-held devices such as a personal digital assistant (PDA), a mobile telephone, a digital camera, and a mobile game machine.
SUMMARY OF THE INVENTIONIn its first aspect, the present invention provides a system for object detection and tracking in a video stream. The system of the invention is based on two separate logical hierarchies. The first hierarchy partitions the video stream into regions of interest which act as standalone motion sensors in the environment, independently responsible for calculating the likeliness of the tracked object being present in the region. The second hierarchy monitors the behavior of the set of regions over time and, based on patterns of likeliness, calculates the position and motion parameters of the tracked object.
The system of the invention comprises a memory storing frames of a video stream to be analyzed by the system. A processor fetches frames of the video stream stored in the memory. An object detection module classifies regions of interest in each frame according to the probability that the region of interest contains at least a portion of a predetermined object to be tracked. As explained below, object detection by the object detection module does not involve edge detection of the objection in the frames. An object tracking module receives as its input the classified frames output by the object detection module and, by comparing consecutive classified frames, determines the motion of the object. The system of the invention may be used to input operating system (OS) commands to the device instead of, or in addition to, any input devices associated with the device such as a keyboard, mouse or joystick. The system of the invention my be used in any type of data processing device such as a personal computer (PC), a portable computer such as a PDA, a laptop or a palm plot, a mobile telephone, a radio or other entertainment device, a vehicle, a digital camera, a mobile game machine, a computerized medical device and a smart house product.
Depending on the application, the processor may optionally include a pattern to recognition module that identifies patterns of motion of the tracked object from among a predetermined set of object motions. The system may further comprise an OS command execution module that stores a look-up table that provides, for each of one or more of the predetermined motion patterns, an associated OS command. When one of the predetermined object motions is identified, the OS command associated with the motion is executed by the system.
In its second aspect, the invention provides a data processing device comprising the system of the invention. The data processing device may be, for example, a personal computer (PC), a portable computer such as a PDA, a laptop, or a mobile telephone, a radio or other entertainment device, a vehicle, a digital camera or a mobile game machine. The device of the invention has a video camera and processor configured to carry out object detection and object tracking, as explained above. The object to be detected and tracked may be for example a hand or finger of a user or a hand held stylus or other predefined or specific device.
The device of the invention comprises a memory that stores a look-up table that provides, for each recognized motion an associated OS command. When a motion pattern is detected by the pattern identification module, the OS command associated with the motion is looked up in the look-up and the OS command associated with the motion is then executed. The OS command may be, for example, activate functions such as Speaker On/Off, Next/Previous track in the MP3/IPTV, control map views in the GPS application and to switch on voicemail service,
In accordance with this aspect of the invention, the frames of the video stream are partitioned into two or more regions of interest. For each region of interest, a statistical analysis of the pixels in the region of interest is performed. For example, the statistical analysis may comprise generating a histogram for each of one or more functions defined on the pixels of the region of interest. The function may be, for example, an intensity of any one of the colors red, green, or blue of the pixels, or any one of the hue, saturation or luminance of the pixels. The histograms may be histograms of a single variable or may be multivariable histograms, in which the frequency of n-tuples of pixel properties is tallied. The statistical analysis may also comprise calculating values of statistical parameters such as an average, mode, standard deviation, or variance of any one or more of the histograms. The results of the statistical analysis of region of interest are used to classify the region according to the probability that the region contains at least a portion of the object being detected. For each frame analyzed, a “region of interest (RI) frame” is generated which is a representation of the classifications of the regions of interest of the frame.
One or more pattern detection modules are used to detect specific motion patterns of the object from the RI frames. Each pattern detection module outputs a probability that the specific motion pattern detected by the pattern detection module occurred during the time window. The outputs of the one or more pattern recognition modules are input to a motion recognition module that determines a motion pattern most likely to have occurred. The determination of the motion detection module is based upon the probabilities input from the pattern recognition modules and may also take into account an external input, for example, an input from the operating system or the application being run.
Thus, in its first aspect, the invention provides a system for object detection and tracking in a video stream, comprising:
-
- (a) a processor comprising an object detection module and an object tracking module;
wherein the object detection module is configured to:
-
- (i) calculate, for each of one or more regions of interest in each of two or more frames in the video stream, a probability that the region of interest contains at least a portion of an object to be tracked; and
- (ii) classify the regions of interest in each of the two or more frames according to the calculated probabilities and generate a region of interest (RI) frame for each video frame, the RI frame reporting the classification of regions of interest;
and wherein the object tracking module is configured to:
(i) compare two RI frames generated by the object detection module and determine a motion of the object.
The object tracking module may comprise one or more pattern detection modules, each pattern detection module being configured to calculate a probability that a specific pattern of motion of the tracked object during a time window occurred during the time window. The object tracking module may further comprise a motion recognition module determining a motion pattern most likely to have occurred based upon the probabilities generated by the one or more pattern detection modules. The determination of the motion recognition module may involve taking into account an external signal.
The system of the invention may further comprise an operating system (OS) command execution module configured to execute an OS command associated with an identified pattern of motion.
In its second aspect, the invention provides a method for object detection and tracking in a video stream, comprising:
-
- calculating, for each of one or more regions of interest in each of two or more frames in the video stream, a probability that the region of interest contains at least a portion of an object to be tracked;
- (ii) classifying the regions of interest in each of the two or more frames according to the calculated probability and generating a region of interest (RI) frame for each video frame, the RI frame reporting the classification of regions of interest; and
- (i) comparing two or more RI frames generated by the object detection module and determine a motion of the object.
The probability that a region of interest contains at least a portion of the object to be tracked may be obtained in a method comprising:
-
- (a) for each of one or more regions of interest in each frame in the video stream, calculating a statistical analysis of the pixels in the region of interest;
- (b) calculating a discrete classification of the region of interest in a calculation involving the statistical analysis of the region of interest in one or more previous frames of the video stream.
The statistical analysis may comprise generating a histogram for each of one or more functions defined on pixels of the region of interest. One or more of the functions may be selected from the group comprising:
-
- (a) an intensity of any one of the colors red, green, or blue of the pixels; and
- (b) any one of a hue, saturation or luminance of the pixels.
The method of the invention may further compris calculating values of statistical parameters of one or more of the functions. One or more of the statistical parameters may be selected from the group comprising:
-
- (a) an average;
- (b) a mode;
- (c) a standard deviation; and
- (d) a variance.
The step of comparing two or more RI frames may comprise:
-
- (a) for each frame, and for each of the classified regions of interest in the frame, comparing the classification of the region of interest with the classification of the region of interest in a plurality of frames obtained in a time window containing the frame;
- (b) determining, on the basis of the comparison, whether or not the selected region of interest contains the object to be tracked;
- (c) reclassifying, on the basis of this determination, the region of interest, according to whether or not the region of interest contains the object to be tracked; and
- (d) calculating one or more tracking parameters of the object's motion based upon changes in the states of two or more of the regions during a time window.
The tracking parameters may be selected from the group comprising:
-
- (a) direction of movement of the object;
- (b) a speed of movement of the object;
- (c) an acceleration of the object;
- (d) a width of the object in pixels; and
- (e) a height of the object in pixels; and
- (f) location of the object in the frame
In another of its aspects, the invention provides a data processing device comprising a system of the invention. The data processing device may be selected from the group comprising:
-
- (a) a personal computer (PC);
- (b) a portable computer such as a PDA or a laptop.
- (c) a mobile telephone;
- (d) a radio;
- (e) an entertainment device;
- (f) Smart Home;
- (g) a vehicle;
- (h) a digital camera
- (i) kitchen appliance;
- (j) an media player or media system,
- (k) location based devices; and
- (l) a mobile game machine.
- (m) a pico projector or an embedded projector
- (n) medical display device.
- (o) an in-car/in-air Infotainment system.
The device of the invention may further comprise one or both of a video camera and a display screen.
One or more of the patterns of motion of the tracked object may be selected from the group comprising:
-
- (a) a width of the object in pixels increased during the time window;
- (b) the width of the object in pixels decreased during the time window;
- (c) the object moved closer to the camera;
- (d) the object moved away from the camera;
- (e) the object moved in a predetermined path;
- (f) the object rotated;
- (g) the object was stationary;
- (h) the object performed any type of motion;
- (i) the object performed a flicking motion;
- (j) the object accelerated;
- (k) the object decelerated; and the object moved and then stopped.
The processor may further comprise an operating system (OS) command execution module configured to execute an OS command of the device associated with an identified pattern of motion. The OS commands may be selected from the group comprising:
-
- (a) depressing a virtual key displayed on a display screen of the device;
- (b) moving a curser appearing on a display screen of the device to a new location on the screen;
- (c) rotating a selection carousel;
- (d) switching between desktops;
- (e) running on the central processor unit a predetermined software application;
- (f) turning off an application;
- (g) turning the speakers on or off;
- (h) turning volume up/down;
- (i) skipping to the next or previous track in a media player or between IPTV channels;
- (j) controlling a GPS application;
- (k) switching on voicemail service;
- (l) navigating in photo/music album gallery;
- (m) scrolling web-pages, emails, documents or maps;
- (n) controlling actions in mobile games; and
- (o) controlling interactive video or animated content.
It will also be understood that the system according to the invention may be a suitably programmed computer. Likewise, the invention contemplates a computer program being readable by a computer for executing the method of the invention. The invention further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the method of the invention.
In order to understand the invention and to see how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:
The system 40 may further comprise an OS command execution module 51. In this case, the memory 44 stores a look-up table that provides, for each of one or more of the predetermined motion patterns, an associated OS command. When one of the predetermined object motions is identified, the OS command associated with the motion is executed.
A user input device 48 may be used to input any relevant data into the system 40, such as an identification of the video stream 2, or the parameters to be analyzed by the processor 46, as explained below. The video stream 2, as well as the results of the processing can be displayed on a display device 50, such as a CRT screen, LCD, or printer.
In step 30, the region of interest that was just analyzed is classified. The classification of a region of interest is a discrete function that describes the probability of the presence of the tracked object in the region of interest. The classification of the region of interest is determined in a method involving the statistical analysis of the region of interest in the current frame and the statistical analysis of the region of interest in one or more previous frames of the video stream. In one embodiment, a distance function is applied to calculate the similarity of various parameters and statistical features in the selected region to parameters and statistical features presenting a tracked object passing in the region. Region and object parameters may include for example the existence of different shapes and contours and their frequencies, while statistical features may include for example the histograms of hue, luminance and saturation and the color pattern. The combined distance result is compared to the results of the region of interest in previous frames. For example, a distance in hue parameters may indicate that an object of the same color as the tracked object has entered the region. This may cause the region to be classified with higher probability of containing the tracked object. The distance function may be, for example, a Euclidean Distance (E. Deja, M. M. Deja, Dictionary of Distances, Elsevier (2006)), a Mahalanobis Distance (Mahalanobis, P C (1936). “On the generalised distance in statistics”. Proceedings of the National Institute of Sciences of India 2 (1): 49-55) a Itakura saito Distance (Itakura F., “Line spectrum representation of linear predictive coefficients of speech signals,” J. Acoust.Soc. Am., 57, 537(A), 1975), a Chebyshev Distance (James M. Abello, Panos M. Pardalos, and Mauricio G. C. Resende (editors) (2002). Handbook of Massive Data Sets. Springer.), a Lee Distance (E.R. Berlekamp, Algebraic Coding Theory, McGraw-Hill 1968), a Hamming Distance (Richard W. Hamming. Error Detecting and Error Correcting Codes, Bell System Technical Journal 26(2):147-160, 1950), or a Levenshtein Distance (Dan Gusfield. Algorithms on strings, trees, and sequences: computer science and computational biology. Cambridge University Press, New York, N.Y., USA, 1997). The classification of the selected region of interest is stored in the memory 44 (step 31).
In step 32 it is determined whether another region of interest of the frame is to be analyzed by the detection module 45. If yes, then the process returns to step 24 with the selection of another region of interest in the current frame. Otherwise, the process continues with step 34 where a “region of interest (RI) frame” is generated for the video frame, and the process terminates. The RI frame of the input video frame is a representation of the classifications of the regions of interest of the frame.
The tracking module 47 receives as its input the RI frames generated by the detection module 45 during a time window of the video stream. The tracking module 47 may operate simultaneously with the detection module 45, receiving classified frames as they are generated by the detection module 45. Alternatively, the tracking module 47 may operate sequentially with the detection module 45, receiving the classified frames only after all of the frames of the video stream have been classified.
The camera 76 views a conical or pyramidal volume of space 86 indicated by broken lines. The camera 76 may have a fixed position on the device 72, in which case the viewing space 86 is fixed relative to the device 72, or may be positionable on the device 72, in which case the viewing space 86 is selectable relative to the device 72. Images captured by the camera 76 are digitized by the camera 76 and input to the processor 46 (see also
The memory 44 stores a look-up table that provides, for each test an associated OS command. When a motion pattern is detected by the pattern identification module 49, the OS command associated with the motion is looked up in the look-up table stored in the memory 44, and then the OS command associated with the motion is executed by the OS execution module 51. The OS command may be, for example, depressing a virtual key displayed on the display screen, moving a curser appearing on the display screen to a new location on the screen, running on the processor 46 a software application stored in the memory 44, or turning off the device 72. The device may provide an indication that the OS command was executed. For example, an OS command equivalent to depressing a key on the virtual keyboard may be indicated by briefly showing the key depressed on a virtual keyboard on the screen 4, or by briefly changing the appearance of the key. Other possibilities for indicating that the OS command was executed include briefly enlarging or otherwise changing the appearance of a depressed key or of the cursor on the screen 4, displaying an icon on the screen 4, producing a sound, and vibrating the device.
Claims
1.-30. (canceled)
31. A system comprising:
- a processor configured to: partition a first video frame in a video stream into at least a first region; classify the first region based on an analysis of one or more pixels within the first region, wherein a classification of the first region reflects a probability of a presence of a tracked object within the first region; compare the classification of the first region with a classification of another region of another video frame in the video stream; and based on the comparison of the classification of the first region and the classification of another region of another video frame in the video stream, determine a motion of the tracked object.
32. The system of claim 31, wherein to partition the first video frame the processor is further configured to partition the first video frame based on an axis of expected motion of the tracked object.
33. The system of claim 31, wherein to partition the first video frame the processor is further configured to:
- analyze one or more video frames in the video stream;
- determine, based on an analysis of the one or more video frames, one or more high variance regions within the one or more video frames; and
- omit the one or more high variance regions from the first region.
34. The system of claim 31, wherein to classify the first region the processor is further configured to classify the first region based on an analysis of one or more regions of one or more other video frames that precede the first video frame in the video stream.
35. The system of claim 31, wherein to classify the first region the processor is further configured to classify the first region based on a distance between a histogram of the tracked object and a histogram of the first region.
36. The system of claim 31, wherein to determine a motion of the tracked object the processor is further configured to detect a motion pattern of the tracked object based on the classification of the first region and one or more classifications of one or more other regions of one or more other video frames in the video stream.
37. The system of claim 31, wherein to determine a motion of the tracked object the processor is further configured to:
- apply a pattern recognition test to the first region and one or more other regions of in the video stream; and
- compute a probability that a motion pattern associated with the pattern recognition test occurred during a time window that includes the first video frame and the one or more other video frames.
38. The system of claim 31, wherein to determine a motion of the tracked object the processor is further configured to detect a motion pattern of the tracked object based on the classification of the first region and one or more inputs from an operating system.
39. The system of claim 31, wherein to determine a motion of the tracked object the processor is further configured to detect a motion pattern of the tracked object based on the classification of the first region and one or more inputs from an application.
40. The system of claim 31, wherein the processor is further configured to execute a command associated with the determined motion of the tracked object.
41. A method comprising:
- partitioning a first video frame in a video stream into at least a first region;
- classifying the first region based on an analysis of one or more pixels within the first region, wherein a classification of the first region reflects a probability of a presence of a tracked object within the first region;
- comparing the classification of the first region with a classification of another region of another video frame in the video stream; and
- based on the comparison of the classification of the first region and the classification of another region of another video frame in the video stream, determining, by a processor, a motion of the tracked object.
42. The method of claim 41, wherein partitioning the first video frame further comprises partitioning the first video frame based on an axis of expected motion of the tracked object.
43. The method of claim 41, wherein partitioning the first video frame further comprises:
- analyzing one or more video frames in the video stream;
- determining, based on an analysis of the one or more video frames, one or more high variance regions within the one or more video frames; and
- omitting the one or more high variance regions from the first region.
44. The method of claim 41, wherein classifying the first region further comprises classifying the first region based on an analysis of one or more regions of one or more other video frames that precede the first video frame in the video stream.
45. The method of claim 41, wherein classifying the first region further comprises classifying the first region based on a distance between a histogram of the tracked object and a histogram of the first region.
46. The method of claim 41, wherein determining a motion of the tracked object further comprises detecting a motion pattern of the tracked object based on the classification of the first region and one or more classifications of one or more other regions of one or more other video frames in the video stream.
47. The method of claim 41, wherein determining a motion of the tracked object further comprises:
- applying a pattern recognition test to the first region and one or more other regions of in the video stream; and
- computing a probability that a motion pattern associated with the pattern recognition test occurred during a time window that includes the first video frame and the one or more other video frames.
48. The method of claim 41, wherein determining a motion of the tracked object further comprises detecting a motion pattern of the tracked object based on the classification of the first region and one or more inputs from an operating system.
49. The method of claim 41, wherein determining a motion of the tracked object further comprises detecting a motion pattern of the tracked object based on the classification of the first region and one or more inputs from an application.
50. A non-transitory computer readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to:
- partition a first video frame in a video stream into at least a first region;
- classify the first region based on an analysis of one or more pixels within the first region, wherein a classification of the first region reflects a probability of a presence of a tracked object within the first region;
- compare the classification of the first region with a classification of another region of another video frame in the video stream; and
- based on the comparison of the classification of the first region and the classification of another region of another video frame in the video stream, determine, by the processing device, a motion of the tracked object.
Type: Application
Filed: Aug 1, 2016
Publication Date: Nov 24, 2016
Inventors: Nadav Israel (Alfei Menashe), Itay Katz (Tel Aviv), Dudi Cohen (Beer Sheva), Amnon Shenfeld (Tel Aviv)
Application Number: 15/225,661