System for processing video and audio information
A software development kit prioritizes certain aspects of an acquired image and introduces the prioritized aspects to a main processor. Alternatively, a coprocessor, or the coprocessor and the development kit, manipulate(s) the acquired image and introduce(s) the manipulated image to the processor. The reprogramming of either one of the development kit and the coprocessor may be initiated by either one of them or by the processor and the programming may be provided by the main processor. A central station and a gate array may also be individually reprogrammable by the main processor which sets up, programs and controls an intelligent imaging platform in accordance with the individual reprogramming. A reprogramming of an audio acquisition stage may also be initiated by that stage and any of the other stages and the processor and may be provided by the processor. The audio information may be related to the acquired image.
This is a divisional application of 10/113,480 (attorney's file IQINV-58963) filed in the United States Patent and Trademark Office on Mar. 29, 2002.
BACKGROUND OF A PREFERRED EMBODIMENT OF THE INVENTIONThis invention relates to a system for and method of processing an acquired image. More particularly, the invention relates to a system for, and method of, processing an image in a wide variety of ways, not previously capable of being accomplished in the prior art, to provide results which are enhanced compared to what has been able to be achieved in the prior art.
BACKGROUND OF THE INVENTIONSystems are now in use for processing an acquired image. For example, systems are now in use for processing an acquired image to determine the entrance into, and the departure of individuals from, a defined area such as an enclosure. Systems are also in use for determining the identity of individuals and objects in an enclosure. Systems are further in use for tracking the movement and variations in the positioning of individuals in an enclosure. These are only a few examples of different types of processing and uses of acquired images.
As of now, different processing and uses of acquired images require different types of systems to be constructed. For example, the same system cannot be used to identify an individual in a crowd and to track the movement of the identified individual in the crowd and particularly the movement of the individual in a defined area such as an enclosure or from one defined area to another defined area. The same system cannot also be used to magnify a particular portion of an acquired image and process that magnified portion. Since different systems are required to perform different functions, costs to individuals or organizations have increased, available space has become limited and complexities in operation have become magnified.
BRIEF DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTIONA software development kit prioritizes certain aspects of an acquired image and introduces the prioritized aspects to a main processor. Alternatively, a coprocessor, or the coprocessor and the development kit, manipulate(s) the acquired image and introduce(s) the manipulated image to the processor. The reprogramming of either one of the development kit and the coprocessor may be initiated by either one of them and by the processor and the programming may be provided by the main processor.
A central station and a gate array may also be individually reprogrammable by the main processor which sets up, programs and controls a camera in accordance with the individual reprogrammings.
A reprogramming of an audio acquisition stage may also be initiated by that stage and any of the other stages and the processor and may be provided by the processor. The audio information may be related to the acquired image.
BRIEF DESCRIPTION OF THE DRAWINGSIn the drawings:
The hardware section in
The audio signals preferably pass through a bus 35 between the audio codec or acquirer 32 and the field reprogrammable gate array 34. However, the audio signals could pass through a bus between the audio codec or acquirer 32 and the coprocessor 36. The system is more flexible when the audio signals pass between the audio codec or acquirer 32 and the field reprogrammable gate array 34 than when the audio signals pass between the audio codec or acquirer 32 and the coprocessor 36. The ability of the signals from the audio acquirer 32 to pass to either the gate array 34 or the coprocessor 36 may be seen by the extension of the bus 35 to the audio/video interface for the hardware section 28a.
Signals pass between the hardware section 28a and the software section 24 through a bus 38. Signals also pass between a miscellaneous input/output stage 40 (considered as hardware) and the software section 24 through a bus 42. Signals also pass through the hardware section 28b and the software section 24 through a bus 44. The hardware section 28b includes a compact flash card interface 46, a PC card interface 48 and a PCI interface 50. The hardware section 28b provides information storage and includes a capacity for providing information storage expansion and other (non-storage) expansion.
The software section 24 includes a video manipulator 52, an audio manipulator 54, an event generator 56, an event responder 58, a platform user interface 60 and a kernel operating system 62. Each of these stages has an arrow 64 disposed in an oblique direction at the bottom right corner of the stage. The oblique arrow 64 indicates that the stage is capable of being reprogrammed. The reprogramming of any stage with the arrow 64 can be initiated by any stage whether the other stage has the arrow 64 to indicate its capability of being reprogrammed. For example, the reprogramming of any of the stages 34, 36 and 52-62 (even numbers only) can be self initiated and can be initiated by any of the other stages 34, 36 and 52-62 and by any other stages such as the stages 70, 72 and 74. Thus, each of the stages 34, 36 and 52-62 (even numbers only) is illustratively able to be reprogrammed. Thus, the stages 34, 36, 52-62 and 65 (even numbers only) receiving such communications have an enhanced flexibility in operation in comparison to the stages which do not receive such reprogramming. Each reprogrammable stage including the stages 34, 36 and 52-62 (even numbers only) can also initiate reprogramming of itself. The reprogramming of each reprogrammable stage including the stages 34, 36 and 52-62 (even numbers only) can be initiated by almost any stage in the system, except for the image acquirer 30, the audio acquirer or codec 32, the miscellaneous input/output 40 and the storage and expansion stage 28b.
A software development kit 65 is indicated by a cloud designated as “platform configuration” with an arrow 64 in the upper left corner. The output from the software development kit 65 is introduced to a main processor 66 to control the operation of the main processor. The software development kit may be considered to be within the main processor 66. The main processor 66 reprograms individual ones of the stages 34, 36 and 52-64 (even numbers only) and the software development kit 65 to control the image acquired by the stage 30 from the camera and the audio acquired by the stage 32 from the camera.
The field programmable gate array 34 provides reprogrammable arrays of gates to clarify and sharpen the video data acquired from the image acquisition stage 30 and introduces the clarified image to the coprocessor 36. The coprocessor 36 manipulates the audio and video data depending upon the results desired to be obtained from the system 10. For example, different manipulations may be provided by the coprocessor 36 when the image is targeted on a single person or a group of people or on an inanimate object. The miscellaneous input/output stage 40 provides such information as motion sensing to indicate to an alarm panel that the camera has observed and detected motion in a scene. The hardware section 40 can also indicate to the camera that some external device has detected motion and wishes to inform the camera that an event worth observing is taking place. In addition, the hardware section 40 may also indicate to a lens to change the size of the iris in the lens. It will be appreciated that the hardware section 40 may perform a considerable number of function other than motion detecting.
The video manipulate stage 52 may manipulate an image to clarify the image as by correcting for color or extracting facial features. This is especially important when faces are in the image and the faces are to be matched against a database identifying a particular face. A similar type of manipulation is provided by the stage 54 with respect to audio information such as when a person is speaking. The event generator 56 matches the image from the stage 52 against the images in the database. This is important when the images are faces. The event responder stage 58 provides a response depending on the matching or lack of matching of the acquired image from the stage 52 and the image in the database. Although the matching has been discussed with reference to faces, the matching can be with respect to any physical object or any perceived state independent of a physical object.
The event responder 58 acts upon the output from the event generator 56 in accordance with the processing which is provided to obtain the desired results. The platform user interface 60 provides a mechanism for taking the information that the camera intelligent imaging platform 18 sees and the platform 60 calculates that information and presents the calculated information to the user. It also allows for the user to adjust the setting of the camera. The platform configuration 65 allows the user of the system to write code for customizing the camera to provide the desired result. The kernel operating system 62 provides the basic operation of the camera. It is well known in the art.
Although the stages 52-62 (even numbers only) and 65 constitute software, they may be disposed in the hardware section 28c, since they control the operation of the main processing hardware 66. The main processing hardware 66 is sometimes referred to in this application as a “main processor”. The main processor 66 is connected by the bus 25 to communication stages or channels in the intelligent imaging platform 18. The intelligent imaging platform 18 includes a subset of communication channels 70 (ethernet), 72 (serial) and 74 (firewire) in the communications arrangement 20. The channel 70 receives information from an Ethernet source. The channel 72 receives serial information from an external source. The channel 74 receives high speed information from software known as Firewire and communicates this information to hardware. The channels 70, 72 and 74 are representative of the different types of information that may be acquired by the currently active communication channels in the intelligent imaging platform 18. The representative channels such as the channels 70, 72 and 74 also receive information from the main processor 66 and supply information to the main processor.
The intelligent imaging platform 18 in turn communicates through the communications network 76 to the central station 14. As shown in
The stage 30 acquires the image from the camera and introduces the acquired image to the field programmable gate array 34. The gate array 34 clarifies the image in accordance with the desired processing to be provided of the image and introduces the signals representing the clarified image to the coprocessor 36. The coprocessor 36 manipulates the clarified image dependent upon the desired result to be obtained from the system shown in
The signals from the coprocessor 36 are further manipulated by the stages 52 and 54. The video manipulator 52 further enhances the quality of the acquired image. For example, the video manipulator 52 may identify individual faces in a crowd and may extract facial features of an individual. The event generator 56 may match the facial features against a database to identify the individual on the basis of this matching against the database.
The system 10 shown in
The degradation of the signal resolution with increases in distance is particularly troublesome when analog signals are processed many of the camera systems of the prior art processed analog signals. In contrast, the system of this invention operates on a digital basis. Coupled with the disposition of the camera controls in the camera, the digital operation of the system of this invention enhances the sensitivity and the reliability of the operation of the system 10.
The system 10 also has other advantages. This results in part from the flexibility in the construction and operation of the system. For example, all of the stages 34, 36, 52-62 (even numbers only) and 65 are reprogrammable. Furthermore, each of the stages 34, 36, 52-62 and 65 can be reprogrammed on the basis of a decision from that stage or any of the other of these stages. This flexibility in reprogramming provides for an enhanced sensitivity and reliability in the adjustments that can be provided in the operation of the camera, thereby providing an enhanced performance of the camera.
The output of the analyzer 92 is stored or archived as at 96 in
The miscellaneous input/output stage 40 in
The video and audio signals then flow through a bus 124 to the coprocessor 36. The output from the coprocessor 36 is provided to a bus 126. These signals then pass through the gate array 34 to PCI bus 38. The signals on the buses 120 and 122 may also be by-passed, through the field reprogrammable gate array 34 to the PCI bus 38 without passing through the coprocessor 36. The signals on the PCI bus 38 pass through the main processor 66 and through a communications bus 129 to the communications network 76 in
The different blocks are defined and determined by the interfaces of applicants' assignee. These interfaces are as follows:
-
- 1. The video interface 31 between the image acquirer 30 and the field reprogrammable gate array 34;
- 2. A coprocessor interface 142 between the gate array 34 and the coprocessor 36;
- 3. The hardware interface 38 between the gate array 34 and the main processor 66;
- 4. A communications interface 146 between the main processor 66 and the central station 14; and
- 5. A software interface 148 between the main processor 66 and the software development kit 65.
The request to initiate reprogramming of a reprogrammable block can come from anywhere in the system with the exception of such stages as the image acquirer 30, the audio acquirer or codec 32 and the miscellaneous input/output stage 40. However, the reprogramming is provided by the main processor 66.
-
- (a) “Remote Color Video Monitor with Archive”,
- (b) “Face Print Generation and Upload to Server”,
- (c) “Gun Shot Detection and Server Notification,” and
- (d) “Subject Tracking with Realtime Video Monitor.”
The second column in
The third column in
Column 5 in
Column 10 of
Although this invention has been disclosed and illustrated with reference to particular embodiments, the principle involved are susceptible for use in numerous other embodiments which will be apparent to persons of ordinary skill in the art. The invention is, therefore, to be limited only as indicated by the scope of the claims.
Claims
1-73. (canceled)
74. In a combination for processing an image to provide a desired result,
- a preprocessor for concentrating upon particular characteristics of an event,
- an analyzer for analyzing the particular characteristics of the event to identify individual characteristics of the event,
- an event generator for making decisions involving the individual characteristics upon the identification of the individual characteristics of the event,
- an event responder for providing action with respect to the individual characteristics of the event in accordance with the decisions made by the event generator, and
- main processor for providing a controlled operation of a camera in processing the image in accordance with the responses provided by the event responder.
75. In a combination as set forth in claim 74,
- the preprocessor, the analyzer, the event generator and the event responder being reprogrammable to adjust the individual characteristics of the image provided by the camera.
76. In a combination as set forth in claim 74 wherein
- the preprocessor, the analyzer, the event generator and the event responder are communicative with any other one of the preprocessor, the analyzer, the event generator, the event responder and the main processor to provide for an adjustment in the operation of any one of the preprocessor, the analyzer, the event generator and the event responder with which the main processor is communicating.
77. In a combination as set forth in claim 74,
- a post-processor for modifying the signals representing the image based upon the operation of the analyzer, the event generator and the event responder and for introducing the modified signals to the main processor to facilitate the processing of the acquired image by the main processor.
78. In a combination as set forth in claim 77,
- a camera responsive to the operation of teh main processor for providing the desired signals.
79. In a combination as set forth in claim 79,
- each of the preprocessor, the analyzer, the event generator, the event responder and post-processor being reprogrammable and the reprogramming of each of the preprocessor, the analyzer, the event generator, the event responder and the post-processor being initiated by any one of the preprocessor, the analyzer, the event generator, the event responder and the post-processor and the main processor.
80. In a combination for processing an image,
- an image acquirer fro acquiring the image and for converting the acquired image to signals representing the image,
- an event generator responsive to the signals from the image acquirer for prioritizing individual ones of the signals in accordance with the results desired to be obtained in the processed image,
- an event responder responsive to the prioritized signals for making decisions based upon the prioritized signals from the event generator and upon the desired results and for providing signals representing these decisions, and
- a hardware processor for processing the signals representing the decisions from the event responder to obtain the desired results.
81. In a combination set forth in claim 80,
- each of the event generator and the event responder being reprogrammable by the hardware processor in accordance with the results desired to be obtained in the acquired image.
82. In a combination as set forth in claim 81,
- a video manipulator responsive to the signals from the image acquirer for modifying the signals representing the acquired image to clarify the image represented by the signals and for introducing the modified signals to the event generator,
- the video manipulator being reprogrammable by the hardware processor in accordance with the results desired to be obtained in the acquired image.
83. In a combination as set forth in claim 80,
- an audio acquirer for acquiring audio information related to the image and for converting the audio information to signals representing the audio information, and
- an audio manipulator responsive to the audio signals for modifying the audio signals to clarify the audio information represented by the signals,
- The audio manipulator being reprogrammable by the hardware processor in: accordance with the results desired to be obtained in the acquired image.
84. In a combination as set forth in claim 81,
- a camera,
- the image acquirer, the event generator, the event responder, the hardware processor and the video manipulator being disposed in the camera.
85. In a combination as set forth in claim 83,
- camera
- the image acquirer, the event generator, the event responder, the hardware processor, the audio acquirer and the audio manipulator being disposed in the camera.
86-111. (canceled)
112. In a combination as set forth in claim 83,
- a camera
- the image acquirer, the event generator, the event responder, the hardware processor the audio acquirer and the audio manipulator being disposed in the camera.
113. In a combination for processing an image to provide a desired result, a first stage for identifying individual characteristics of an event,
- an intelligent imaging platform for indicating the individual characteristics of the event,
- an event responder for changing the individual characteristics of the event,
- a post processor for providing changes in the image in the intelligent imaging platform in accordance with the changes indicated in the event responder.
114. In a combination as set forth in claim 112 wherein
- the first stage, the event responder, the intelligent imaging platform and the post processor are reprogrammable.
115. In a combination as set forth in claim 113 wherein
- each of the first stage, the event responder, the intelligent imaging platform and the post processor is reprogrammable by any of the others or by itself.
116. In the combination set forth in claim 114,
- a database for providing a desired image,
- an event generator for providing a match between the individual characteristics of the event and the desired image in the database, and
- the event responder providing a response depending upon the matching or lack of matching of the individual characteristics of the event from the first stage and the desired image in the database.
117. In combination for processing an image to produce a desired result,
- a video manipulator for enhancing particular characteristics of the image,
- an event generator for concentrating on at least an individual one of the particular characteristics of the image.
- an intelligent imaging platform for indicating the image with the concentration on at least the individual one of the particular characteristics in the image, and
- a main processor including the event generator, the intelligent imaging platform for providing an image concentrated on the at least individual one of the particular characteristics in the image.
118. In the combination recited in claim 116 wherein
- video manipulator and the event generator are in the intelligent imaging platform.
119. In the combination recited in claim 116, including
- a database for indicating the desired image, and
- the event generator being operative to match the image in the video manipulator with the desired image in the database, and
- an event responder for providing a response depending upon the matching or lack of matching of the image in the video manipulator with the desired image in the database.
120. In combination for processing an event to provide a desired result,
- a first stage for identifying individual characteristics of the event,
- an event responder for processing the individual characteristics of the event, and
- an intelligent imaging platform for indicating the event as processed by the event responder.
121. In a combination as set forth in claim 119 wherein
- the first stage, the event responder and the intelligent imaging platform are reprogrammable.
122. In a combination as set forth in claim 120 wherein
- the first stage, the event responder and the intelligent imaging platform are self reprogrammable.
123. In a combination as set forth in claim 119 wherein
- the image provided by the first stage is matched against images in a database and the event responder provides a response depending upon the matching or lack of matching of these images and wherein
- a platform user interface processes the response from the event responder for introduction to a user.
124. In a combination as set forth in claim 120 wherein
- the image provided by the first stage is matched against images in a database and the event responder provides a response depending upon the matching on lack of matching of these images and wherein
- a platform user interface processes the response from the event responder for introduction to a user.
125. In a combination as set forth in claim 123 wherein
- each of the first stage, the event responder, the intelligent imaging platform and the platform user interface is reprogrammable by anyone of the first stage, the event responder, the intelligent imaging platform and the platform user interface.
126. In combination for processing an image to provide a desired result to a user, including,
- a preprocessor for manipulating the image to clarify the image,
- an event generator for matching the manipulated image against images in a database,
- an event responder for processing the output from the event generator, and
- platform user interface for processing the information from the event generator and presenting the processed information to the user.
127. In a combination as set forth in claim 125 wherein
- each of the preprocessor, the event generator, the event responder and the platform user interface is self reprogrammable and is programmable by the other ones of the preprocessor, the event generator, the event responder and the platform user interface.
128. In a combination as set forth in claim 125, including,
- an intelligent imaging platform for displaying the image, and
- a software development kit responsive to the output of the event generator for customizing the output of the intelligent imaging platform.
Type: Application
Filed: Feb 3, 2006
Publication Date: Jul 13, 2006
Inventors: Gregory Bone (Santa Monica, CA), Jonathan Walton (Newport Beach, CA)
Application Number: 11/347,764
International Classification: H04N 5/232 (20060101);