Method and system for screening luggage items, cargo containers or persons

Abstract

A system for screening luggage items, which includes an image generation device suitable for generating an image signal associated with a luggage item where the image signal conveys information related to the contents of the luggage item. The system also includes an apparatus having an input for receiving the image signal and a processing unit. The processing unit processes the image signal in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item. The processing unit generates a detection signal in response to detection of the presence of at least one target object in the luggage item. An output module conveys information derived at least in part on the basis of the detection signal to a user of the system. In alternative embodiments, the system may also be used to screen cargo containers and persons.

Description

FIELD OF THE INVENTION

The present invention relates generally to security systems and, more particularly, to methods and systems for screening luggage or cargo containers to identify certain objects located therein and for screening persons to identify certain objects located thereon.

BACKGROUND

Security in airports, train stations, ports, office buildings and other public or private venues is becoming increasingly important in particular in light of recent violent events.

Typically, security-screening systems make use of devices generating penetrating radiation, such as x-ray devices, to scan individual pieces of luggage to generate an image conveying the contents of the luggage. The image is displayed on a screen and is examined by a human operator whose task it is to identify on the basis of the image potentially threatening objects located in the luggage.

A deficiency with current systems is that they are entirely reliant on the human operator to identify potentially threatening objects. However, the performance of the human operator greatly varies according to such factors as poor training and fatigue. As such, the identification of threatening objects is highly susceptible to human error. Furthermore, it will be appreciated that failure to identify a threatening object, such as a weapon for example, may have serious consequences, such as property damage, injuries and human deaths.

Another deficiency with current systems is that the labour costs associated with such systems are significant since human operators must view the images.

Consequently, there is a need in the industry for providing a method and system for use in screening luggage items, cargo containers or persons to identify certain objects that alleviate at least in part the deficiencies of the prior art.

SUMMARY OF THE INVENTION

In accordance with a broad aspect, the invention provides a system for screening a luggage item. The system comprises an image generation device, an apparatus for processing image information and an output unit. The image generation device is suitable for generating an image signal associated with a luggage item, the image signal conveying information related to the contents of the luggage item. The apparatus for processing image information is in communication with the image generation device and comprises an input for receiving the image signal associated with the luggage item and a processing unit. The processing unit processes the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item. The processing unit generates a detection signal in response to detection of the presence of at least one target object in the luggage item. The output module conveys information derived at least in part on the basis of the detection signal to a user of the system.

For the purpose of this specification, the expression “luggage item” is used to broadly describe luggage, suitcase, handbags, backpacks, briefcases, boxes, parcels or any other similar type of item suitable for containing objects therein.

In a specific example of implementation, the image generation device uses penetrating radiation or emitted radiation to generate the image associated with the luggage item. Examples include, but are not limited to, x-ray, gamma ray, computed tomography (CT scan), thermal imaging and millimeter wave. The image signal generated may also be of any suitable format such as for example, VGA, SVGA, XGA, JPEG, GIF, TIFF and bitmap amongst others.

In accordance with a specific example of implementation, the output module includes a display adapted for generating an output display image conveying information derived at least in part on the basis of the detection signal in visual format. Optionally, the output module is adapted for generating image data conveying the location of at least one of the at least one target object whose presence in the luggage item was detected. Optionally still, the output module is adapted for generating image data conveying the location of at least one of the at least one target object whose presence in the luggage item was detected in combination with the image associated with the luggage item. In an alternative example of implementation, the output module is adapted for conveying information derived at least in part on the basis of the detection signal in audio format.

In accordance with a specific example of implementation, the detection signal conveys position information related to at least one of the at least one target object whose presence in the luggage item was detected.

In accordance with a specific example of implementation, the processing unit is responsive to detection of the presence of at least one target object to generate log information elements conveying a presence of at least one of the at least one target object whose presence in the luggage item was detected and for storing the log information data elements on a computer readable storage medium. The log information may include a time stamp data element indicating timing information associated to the detection of the presence of at least one target object in the luggage item.

In accordance with a specific example of implementation, the processing unit is operative for applying a correlation operation between data derived from the image signal and the plurality of target images to detect the presence of the at least one target object in the luggage item. The correlation operation may be effected optically, by using an optical correlator, or digitally using a programmed digital computer or dedicated hardware. In an alternative example of implementation, the comparisons between the image signal associated with the luggage item and at least some images in the plurality of target images is effected any suitable image processing algorithm.

In a specific example of implementation, the apparatus further comprises a second input for receiving the plurality of target images associated with target objects. In a specific implementation, the plurality of target objects includes at least one weapon.

In accordance with another broad aspect, the invention provides a method for screening a luggage item. The method includes receiving an image signal associated with the luggage item, the image signal conveying information related to the contents of the luggage item. The method also includes processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item. In response to detection of the presence of at least one target object in the luggage item, a detection signal is generated, which detection signal is then released.

In accordance with another broad aspect, the invention provides and apparatus suitable for screening a luggage item in accordance with the above described method.

In accordance with another broad aspect, the invention provides a computer readable storage medium including a program element suitable for execution by a computing apparatus for screening a luggage item, the computing apparatus comprising a memory unit and a processor operatively connected to the memory unit. The program element when executing on the processor is operative for screening a luggage item in accordance with the above-described method.

In accordance with another broad aspect, the invention provides an apparatus suitable for screening a luggage item. The apparatus comprises means for receiving an image signal associated with the luggage item, means for processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item. The apparatus also comprises means for generating a detection signal in response to detection of the presence of at least one target object in the luggage item. The apparatus also comprises means for releasing the detection signal.

In accordance with yet another broad aspect, the invention provides an apparatus suitable for screening a cargo container. The apparatus comprises an input for receiving an image signal associated with the cargo container, a processing unit in communication with the input and an output. The image signal conveys information related to the contents of the cargo container. The processing unit is operative for processing the image signal associated with the cargo container in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the cargo container. In response to detection of the presence of at least one target object in the cargo container, the processing unit generates a detection signal, which is then released at the output.

For the purpose of this specification, the expression “cargo container” is used to broadly describe an enclosures for storing cargo such as would be used, for example, in a ship, train, truck or an other suitable of container.

In accordance with yet another broad aspect, the invention provides an apparatus suitable for screening a person. The apparatus comprises an input for receiving an image signal associated with the person, a processing unit in communication with the input and an output. The processing unit is operative for processing the image signal associated with the person in combination with a plurality of target images associated with target objects to detect a presence of at least one target object on the person. The processing unit generates a detection signal in response to detection of the presence of at least one target object in the person. The detection signal is then released at the output.

In accordance with another broad aspect, the invention provides an apparatus for detecting the presence of one or more prohibited objects in a container. The apparatus comprises an input for receive data conveying graphic information on contents of the container and an optical correlator for processing the graphic information to detect depiction of the one or more prohibited objects.

Other aspects and features of the present invention will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments of the invention in conjunction with the accompanying Figures.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of the embodiments of the present invention is provided herein below, by way of example only, with reference to the accompanying drawings, in which:

FIG. 1 is a high-level block diagram of a system for screening a luggage item in accordance with a specific example of implementation of the present invention;

FIG. 2 is a block diagram of an output module suitable for use in connection with the system depicted in FIG. 1 in accordance with a specific example of implementation of the present invention;

FIG. 3 is a block diagram of an apparatus for processing images suitable for use in connection with the system depicted in FIG. 1 in accordance with a specific example of implementation of the present invention;

FIGS. 4a and 4b depict specific examples of visual outputs conveying the presence of at least one target object in the luggage item in accordance with specific examples of implementation of the present invention;

FIG. 5 is a flow diagram depicting a process for detecting a presence of at least one target object in the luggage item in accordance with specific examples of implementation of the present invention;

FIG. 6 shows three images associated to a target object suitable for use in connection with the system depicted in FIG. 1, each image depicting the target object in a different orientation, in accordance with a specific example of implementation of the present invention;

FIG. 7 shows a mosaic image including a plurality of sub-images associated with a target object suitable for use in connection with the system depicted in FIG. 1, each sub-image depicting the target object in a different orientation and scale, in accordance with a specific example of implementation of the present invention;

FIG. 8 is a block diagram a luggage screening process using an optical correlator in accordance with a specific example of implementation of the present invention;

FIG. 9 is a block diagram depicting the functioning of an optical correlator in accordance with a specific example of implementation of the present invention;

FIGS. 10 and 11 depict Fourier transforms of the spatial domain image for various numbers;

FIG. 12 shows two images associated to a person suitable for use in a system for screening a person in accordance with a specific example of implementation of the present invention;

FIG. 13 is a block diagram of an apparatus suitable for implementing at least a portion of the modules depicted in connection with the apparatus for processing images shown in FIG. 3 in accordance with a specific example of implementation of the present invention.

In the drawings, the embodiments of the invention are illustrated by way of examples. It is to be expressly understood that the description and drawings are only for the purpose of illustration and are an aid for understanding. They are not intended to be a definition of the limits of the invention.

DETAILED DESCRIPTION

Shown in FIG. 1 is a system 100 for screening a luggage item in accordance with a specific example of implementation of the present invention. The system 100 includes an image generation device 102, an apparatus 106 in communication with the image generation device 102 and an output module 108.

The image generation device 102 generates an image signal associated with a luggage item 104. The image signal conveys information related to the contents of the luggage item 104. The apparatus 106 receives the image signal associated with the luggage item 104 and processes that image signal in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item 104. In a specific implementation, the plurality of target images are stored in a database of target images 110. In response to detection of the presence of at least one target object in the luggage item 104, the apparatus 106 generates a detection signal conveying the presence of the at least one target object in the luggage item 104. Examples of the manner in which the detection signal can be derived are described later on in the specification. The output module 108 conveys information derived at least in part on the basis of the detection signal to a user of the system.

Advantageously, the system 100 provides assistance to the human security personnel using the system to detect certain target objects and decreases the susceptibility of the screening process to human error.

Image Generation Device 102

In a specific example of implementation, the image generation device 102 uses penetrating radiation or emitted radiation to generate the image associated with the luggage item 104. Specific examples of such devices include, without being limited to, x-ray, gamma ray, computed tomography (CT scans), thermal imaging and millimeter wave devices. Such devices are known in the art and as such will not be described further here. In a non-limiting example of implementation, the image generation device 102 is a conventional x-ray machine adapted for generating a x-ray image of the luggage item 104.

The image signal generated by the image generation device 102 and associated with the luggage item 104 may be convey a two-dimensional (2-D) image or a three-dimensional (3-D) image and may be in any suitable format Possible formats include, without being limited to, VGA, SVGA, XGA, JPEG, GIF, TIFF and bitmap amongst others. Preferably, the image signal is a format that can be displayed on a display screen.

Database of Target Images 110

In a specific example of implementation, the database of target images 110 includes a plurality of entries associated to respective target objects that the system 100 is designed to detect.

In a non-limiting implementation, for each entry associated to a target object at least one image (hereinafter referred to as a “target image”) is provided in the database of target images 110. The format of the target images will depend upon the image processing algorithm implemented by the apparatus 106. More specifically, the format of the target images is such that a comparison operation can be performed by the processing unit between the target images and data derived from the image signal associated with the luggage item 104.

Optionally, for each entry associated to a target object, a set of images is provided in the database of target images 110. For example, images depicting the target object in various orientations may be provided. FIG. 6 of the drawings depicts an example of arbitrary 3D orientations of a target object.

Optionally still, for each entry associated to a target object, characteristics of the target object are provided. Such characteristics may include, without being limited to, the name of the target object, its associated threat level, the recommended handling procedure when such a target object is detected and any other suitable information. Optionally still, each entry in the database of target images 110 is also associated to a respective target object identifier data element. In a non-limiting example of implementation, the database of target images 110 includes at least one entry associated to a weapon.

The specific design and content of the database of target images 110 may vary from one implementation to the next without detracting from the spirit of the invention. The design of the database is not critical to the present invention and as such will not be described further here.

Although the database of target images 110 has been shown in FIG. 1 to be a component separate from the apparatus 106, it will be appreciated that in certain embodiment the database of target images 110 may be part of apparatus 106 and that such implementations do not detract from the spirit of the invention. In addition, it will also be appreciated that in certain implementations, the database of target images 110 is shared between multiple apparatuses 106.

Output Module 108

In a specific example of implementation, the output module 108 conveys information derived at least in part on the basis of the detection signal to a user of the system.

A specific example of implementation of the output module 108 is shown in FIG. 2 of the drawings. As depicted, the output module includes an output device 202 and an output controller unit 200.

The output controller unit 200 receives the detection signal conveying the presence of the at least one target object in the luggage item 104 from apparatus 106 (shown in FIG. 1). In a specific implementation, the detection signal conveys position information related to the certain target object in the luggage item 104. Optionally, the detection signal also conveys a target object identifier data element. The target object identifier data element is associated to an entry in the database of target images 110.

In a first specific example of implementation, the output controller unit 200 is adapted to cause a display unit to convey information related to the certain target object in the luggage item 104. In a non-limiting example of implementation, the output controller unit 200 generates image data conveying the location of the certain target object in the luggage item 104. Optionally, the output controller unit 200 also extracts characteristics of the target object from the database of target images 110 on the basis of the target object identifier data element and generates image data conveying the characteristics of the certain target object in the luggage item 104. In yet another non-limiting example of implementation, the output controller unit 200 generates image data conveying the location of the certain target object in the luggage item 104 in combination with the image signal associated with the luggage item 104 and generated by the image generation device 102 (shown in FIG. 1).

In a second specific example of implementation, the output controller unit 200 is adapted to cause an audio unit to convey information related to the certain target object in the luggage item 104. In a specific non-limiting example of implementation, the output controller unit 200 generates audio data conveying the presence of the certain target object in the luggage item 104 and optionally the location of the certain target object in the luggage item 104 and the characteristics of the target object.

The output controller unit 200 then releases a signal for causing the output device 202 to convey information to a user of the system

More specifically, the output device 202 may be any device suitable for conveying information to a user of the system 100 regarding the presence of a target object in the luggage item 104. The information may be conveyed in visual format, audio format or as a combination of visual and audio formats.

In a first specific example of implementation, the output device 202 includes a display screen adapted for displaying in visual format information related to the presence of a target object in the luggage item 104. In a second specific example of implementation, the output device 202 includes a printer adapted for displaying in printed format information related to the presence of a target object in the luggage item 104. FIGS. 4a and 4b show in simplified format example of information related to the presence of a target object in the luggage item 104 presented in visual format. More specifically, in FIG. 4a, the image associated with the luggage item 104 is displayed along with a visual indicator (e.g., arrow 404) identifying the location of a target object (e.g., gun 402) detected by the apparatus 106. Alternatively, in FIG. 4b, a text message is provided describing the target object detected by apparatus 106. It will be appreciated that the output may include additional information without detracting from the spirit of the invention and that the example illustrated in FIGS. 4a and 4b have been provided for the purpose of illustration only.

In a third specific example of implementation, the output device 202 includes an audio output unit adapted for releasing an audio signal conveying information related to the presence of a target object in the luggage item 104.

In a fourth specific example of implementation, the output device 202 includes a set of visual elements, such as lights or other suitable visual elements, adapted for conveying in visual format information related to the presence of a target object in the luggage item 104.

The person skilled in the art will readily appreciate, in light of the present specification, that other suitable types of output devices may be used here without detracting from the spirit of the invention.

Apparatus 106

The apparatus 106 will now be described in greater detail with reference to FIG. 3. As depicted, the apparatus 106 includes a first input 310, a second input 314, an output 312 and a processing unit, generally comprising a pre-processing module 300, an image comparison module 302 and a detection signal generator module 306.

The first input 310 is for receiving an image signal associated with a luggage item from the image generation device 102 (shown in FIG. 1).

The second input 314 is for receiving target images from the database of target images 110. It will be appreciated that in embodiments where the database of target images 110 is part of apparatus 106, the second input 314 may be omitted.

The output 312 is for releasing a detection signal conveying the presence of a target object in the luggage item 104 for transmittal to output module 108.

The processing unit of the apparatus 106 receives the image signal associated with the luggage item 104 from the first input 310 and processes that image signal in combination with a plurality of target images associated with target objects received at input 314 to detect a presence of at least one target object in the luggage item 104. In response to detection of the presence of at least one target object in the luggage item 104, the processing unit of the apparatus 106 generates and releases at output 312 a detection signal conveying the presence of the at least one target object in the luggage item 104.

The process implemented by the various functional elements of the processing unit of the apparatus 106 is depicted in FIG. 5 of the drawings. At step 500, the pre-processing module 300 receives an image signal associated with the luggage item 104 is received via the first input 310. At step 501, the pre-processing module 300 processes the image signal in order to enhance the image, remove extraneous information therefrom and remove noise artefacts in order to obtain more accurate comparison results. The complexity of the requisite level of pre-processing and the related tradeoffs between speed and accuracy depend on the application. Examples of pre-processing may include, without being limited to, brightness and contrast manipulation, histogram modification, noise removal and filtering amongst others. It will be appreciated that all or part of the functionality of the pre-processing module 300 may actually be external to the apparatus 106, e.g., it may be integrated as part of the image generation device 102 or as an external component. It will also be appreciated that the pre-processing module 300 (and hence step 501) may be omitted in certain embodiments of the present invention without detracting from the spirit of the invention. As part of step 501, the pre-processing module 300 releases a modified image signal for processing by the image comparison module 302.

At step 502, the image comparison module 302 verifies whether there remain any unprocessed target images in the database of target images 110. In the affirmative, the image comparison module 302 proceeds to step 503 where the next target image is accessed and the image comparison module 302 then proceeds to step 504. If at step 502 all target images in the database of target images 110 have been processed, the image comparison module 302 proceeds to step 508 and the process in completed.

At step 504, the image comparison module 302 compares the image signal associated with the luggage item 104 against the target image accessed at step 503 to determine whether a match exists. The comparison may be effected using any image processing algorithm suitable for comparing two images. Examples of algorithms that can be used to perform image processing and comparison include without being limited to:

A—Image Enhancement

• • Brightness and contrast manipulation
  • Histogram modification
  • Noise removal
  • Filtering

B—Image Segmentation

• • Thresholding
    • Binary or multilevel
    • Hysteresis based
    • Statistics/histogram analysis
  • Clustering
  • Region growing
  • Splitting and merging
  • Texture analysis
  • Watershed
  • Blob labeling

C—General Detection

• • Template matching
  • Matched filtering
  • Image registration
  • Image correlation
  • Hough transform

D—Edge Detection

• • Gradient
  • Laplacian

E—Morphological Image Processing

• • Binary
  • Grayscale

F—Frequency Analysis

• • Fourier Transform
  • Wavelets

G—Shape Analysis and Representations

• • Geometric attributes (e.g. perimeter, area, euler number, compactness)
  • Spatial moments (invariance)
  • Fourier descriptors
  • B-splines
  • Chain codes
  • Polygons
  • Quad tree decomposition

H—Feature Representation and Classification

• • Bayesian classifier
  • Principal component analysis
  • Binary tree
  • Graphs
  • Neural networks
  • Genetic algorithms
  • Markov random fields

The above algorithms are well known in the field of image processing and as such will not be described further here.

In a specific example of implementation, the image comparison module 302 includes an edge detector to perform part of the comparison at step 504. In another specific example of implementation, the comparison performed at step 504 includes effecting a correlation operation between data derived from the image signal and the target images. In a specific example of implementation, the correlation operation is performed by an optical correlator. A specific example of implementation of an optical correlator suitable for use in comparing two images will be described later on in the specification. In an alternative example of implementation, the correlation operation is performed by a digital correlator.

The image comparison module 302 then proceeds to step 506 where the result of the comparison effected at step 504 is processed to determine whether a match exists between the image signal associated with the luggage item 104 and the target image. In the absence of a match, the image comparison module 302 returns to step 502. In response to detection of a match, the image comparison module 302 triggers the detection signal generation module 306 to execute step 510. Then, the image comparison module 302 returns to step 502 to continue processing with respect to the next target image.

At step 510, the detection signal generation module 306 generates a detection signal conveying the presence of the target object in the luggage item 104, and the detection signal is released at output 312. The detection signal may simply convey the fact that a target object has been detected as present in the luggage item 104, without necessarily specifying the identity of the target object. Alternatively, the detection signal may convey the actual identity of the detected target object detected as being present in the luggage item 104. As previously indicated, the detection signal may include information related to the positioning of the target object within the luggage item 104 and optionally a target object identifier data element associated to the target object determined to be a potential match.

Specific Example of Image Comparison Module 302 Including an Optical Correlator

As mentioned above, in a specific implementation of the image comparison module 302, step 504, which involves a comparison between the image signal associated with the luggage item 104 and the target images from the database of target images 110, is performed using a correlation operation. The correlation operation multiplies together the Fourier transform of the image signal associated with the luggage item 104 with the Fourier transform of a target image. The result of correlation operation provides a measure of the degree of similarity between two images.

In a specific implementation, the image comparison module 302 includes an optical correlator unit for computing the correlation between the image signal associated with the luggage item 104 and a target image from the database of target images 110. Specific examples of implementation of the optical correlator include a joint transform correlator (JTC) and a focal plane correlator (SPC).

The optical correlator multiplies together the Fourier transform of the image signal associated with the luggage item 104 with that of a target image and records the result with a camera. An energy peak measured with that camera indicates a match between the image signal associated with the luggage item 104 and the target image.

Advantageously, the optical correlator performs the correlation operation physically through light-based computation, rather than by using software running on a silicon-based computer, which allows computations to be performed at a higher speed than is possible with a software implementation and thus provides for improved real-time performance.

It will be appreciated that the correlation computation may also be implemented using a digital correlator. The correlation operation is computationally intensive and, in certain implementations requiring real-time performance, the use of a digital correlator may not provide suitable performance. In such implementations, an optical correlator will be preferred.

As described above, the database of target images 110 includes a plurality of target images associated to objects which the system 100 is designed to detect. In a specific example of implementation using a correlation operation, the database of target images 110 includes data indicative of the Fourier transform of the target image. This data will herein be referred to as a template or filter. The template will be retrieved later when performing a verification or identification operation. Image processing and enhancement can be performed to obtain better matching performance depending on the environment and application. In non-limiting examples of implementation the Fourier transform of the target image is digitally pre-computed such as to improve the speed of the correlation operation when the system is in use.

In a non-limiting example of implementation, the generation of the reference template or filter is performed in a few steps. First, the background is removed from the target image. In other words the target image is extracted from the background and the background is replaced by a black background. The resulting image is then processed through a Fourier transform function. The result of this transform is a complex image. A phase only filter (POF) for example will only contain phase information (between zero and 2 pi) which is mapped to a 0 to 255 range values. These 256 values correspond in fact to the 256 levels of gray of an image. The person skilled in the art, in light of the present specification, will readily appreciate that various types of templates or filters can be generated. Many methods for generating Fourier filters are known in the art and a few such methods will be described later on in the specification.

As a variant, in order to reduce the amount of data needed to represent the whole range of 3D orientations that a single target object can take, a MACE (Minimum Average Correlation Energy) filter is used to generate the template or filter. Typically, the MACE file includes combining several different 2D projections of a given object and encoding them in a single MACE filter instead of having one 2D projection per filter. One of the benefits of using MACE filters is that the resulting database of target images 110 would take less space since it would include fewer items. Also, since the number of correlations needed to identify a single target object would be reduced, the total processing time would also be reduced.

Another way of reducing the processing time is to take advantage of the linearity property of the Fourier transform. By dividing the target image into several sub-images, a composite image can be formed, herein referred to as a mosaic. When a mosaic is displayed at the input of the correlator, the correlation is computed simultaneously on all the sub-images without incurring any substantial time penalty. A mosaic may contain several different target objects or several different orientations of the same target object or a combination of both. FIG. 7 of the drawings depicts a mosaic including a target object in various orientations and scales. The parallel processing capabilities a mosaic effectively increase the throughput of the correlator.

FIG. 8 depicts a high level representation of a luggage screening process using an optical correlator. As shown, an image 800 associated with a luggage item is provided as input to the correlator and undergoes an optical Fourier transformation 804. The result of the transformation is multiplied 802 by the (previously computed) Fourier transform of a target image 804. The result of the multiplication of the two Fourier transforms is then processed through another optical Fourier transform 822 and the resulting signal is captured by a camera at what is referred to as the correlation plane, which yields the correlation output. The correlation output is released for transmission to the detection signal generator 306 where it is analyzed. A peak in the correlation output indicates a match between the image 800 associated with the luggage item 104 and the target image.

In a non-limiting example of implementation of an optical correlator, the Fourier transform of the image 800 associated with the luggage item 104 is performed as follows: The image is displayed internally on a small Liquid Crystal Display (LCD). A light beam projects the image through a lens that performs the equivalent of a Fourier transform on the image. The Fourier transform of the image is then projected on a second LCD screen on which is displayed the template or filter associated to the target image. The two multiplied Fourier transforms are then processed through a second Fourier lens which forces the light beam to converge to a CCD at the correlation plane. The CCD output is then sent to a frame grabber in the computer.

The inner workings of the aforementioned non-limiting example optical correlator are illustrated in FIG. 9. On the left hand side appears a laser source 900 that generates a light beam used to project images across the correlator. The light beam is directed first through a small set of lenses 902 used to expand its diameter in order to illuminate the whole surface of the first LCD screen 904 to the left. The image 800 associated with the luggage item 104 is displayed on the first LCD screen 904 either through a direct camera interface or provided as a VGA image by the computer. The first LCD screen 904 is illuminated by the light beam and the image is propagated through the correlator. In the illustrated example, the image 800 captured by the camera is that of a gun on a conveyor belt.

The light beam modulated by the first image on the first LCD screen 904 is then propagated through the second set of lenses 906, referred to as a Fourier lens since it performs the equivalent of the Fourier transform mathematical operation. The inherent properties of light are used to physically perform the appropriate calculations. Specifically, the propagation of light in vacuum is a function which corresponds to the kernel of the Fourier transform operation, thus the propagation of light along the axis of a Fourier lens represents a sufficiently strong approximation of this natural phenomenon to assert that the light beam undergoes a Fourier transform. Otherwise stated, a lens has the inherent property of performing a Fourier transform on images observed at its front focal plane, provided that this image is displayed at its back focal plane. The Fourier transform, which can normally be rather computation-intensive when calculated by a digital computer, is performed in the optical correlator simply by the propagation of the light. The mathematics behind this optical realization is equivalent to the exact Fourier transform function and can be modeled with standard fast Fourier algorithms. For more information regarding Fourier transforms, the reader is invited to consider B.V.K. Vijaya Kumar, Marios Savvides, Krithika Venkataramani, and Chunyan Xie, “Spatial frequency domain image processing for biometric recognition”, Biometrics ICIP Conference 2002. The contents of this document are incorporated herein by reference.

After going through the Fourier lens 906, the signal is projected on the second LCD screen 908 on which is displayed the target template, i.e., Fourier transform of the target image 804. When the Fourier transform of the image 800 associated with the luggage item 104 goes through the second LCD screen 908 on which the target template is displayed, the light beam crosses a second Fourier lens 910 which, again, optically computes the equivalent of a Fourier transform multiplication. This operation corresponds to a correlation in the spatial domain. The target image displayed on the second LCD screen 908 in fact induces a phase variation on the incoming light beam. Each pixel can potentially induce a phase change whose magnitude is equivalent to its grey level. As such the Fourier transform displayed on the first LCD screen 904 is multiplied with the Fourier transform of the target image 804, which is equivalent to performing a correlation.

The second Fourier lens 910 finally concentrates the light beam on a small area camera or CCD 912 where the result of the correlation is measured, so to speak. The CCD 912 in fact measures energy peaks at on the correlation plane. The position of a correlation peak corresponds in fact to the location of the target object center in the image 800 associated with the luggage item 104.

Referring back to FIG. 8, the CCD 912 communicates the signal from the optical correlator to the detection signal generator module 306. In this specific implementation, the detection signal generator module 306 is a computing unit including a frame grabber and software. The software is adapted to processing the signal received from the correlator to detects energy peaks as gray level video signals varying between 0 and 255. A strong intensity peak on the correlation plane indicates a match between the image 800 associated with the luggage item 104 and the target image 804. The location of the energy peak also indicates the location of the center of the target image in the image 800 associated with the luggage item 104.

Fourier Transform and Spatial Frequencies

The Fourier transform as applied to images is now described in general terms. The Fourier transform is a mathematical tool used to convert the information present within an object's image into its frequency representation. In short, an image can be seen as a superposition of various spatial frequencies and the Fourier transform is a mathematical operation used to compute the intensity of each of these frequencies within the original image. The spatial frequencies represent the rate of variation of intensity in space. Consequently, a smooth or uniform pattern mainly contains low frequencies. Sharply contoured patterns, by contrast, exhibit a higher frequency content.

The Fourier transform of an image f(x,y) is given by:

F(u,v)=∫∫f(x,y)e^{−f2π(ux+vy)}dxdy  (1)

where u, v are the coordinates in the frequency domain. Thus, the Fourier transform is a global operator: changing a single frequency of the Fourier transform affects the whole object in the spatial domain.

A correlation operation can be mathematically described by:

$\begin{array}{cc}C\left(\varepsilon ,\xi \right)={\int }_{-\infty }^{\infty }{\int }_{-\infty }^{\infty }f\left(x,y\right)h^{*}\left(x-\varepsilon ,y-\xi \right)xy& \left(2\right)\end{array}$

where ε and ξ represent the pixel coordinates in the correlation plane, C(ε,ξ) stands for the correlation, x and y identify the pixel coordinates of the input image, f(x, y) is the original input image and h*(ε,ξ) is the complex conjugate of the correlation filter.

In the frequency domain the same expression takes a slightly different form:

C(ε,ξ)=ℑ⁻¹(F(u,v)H*(u,v))  (3)

where ℑ is the Fourier transform operator, u and v are the pixel coordinates in the Fourier plane, F(u,v) is the Fourier transform of the image acquired with the camera f(x,y) and H*(u,v) is the Fourier transform of the filter of the reference template. Thus, the correlation between an input image and a target template is equivalent, in mathematical terms, to the multiplication of their respective Fourier transform, provided that the complex conjugate of the filter is used. Consequently, the correlation can be defined in the spatial domain as the search for a given pattern (template), or in the frequency domain, as filtering operation with a specially designed matched filter.

Advantageously, the use of optics for computing a correlation operation allows the computation to be performed in a shorter time than by using a digital implementation of the correlation. It turns out that an optical lens properly positioned (i.e. input and output images are located on the lens's focal planes) automatically computes the Fourier transform of the input image. In order to speed up the computation of the correlation, the Fourier transform of the target image is computed beforehand and submitted to the correlator as a mask. The target template (or filter in short) is generated by computing the Fourier transform of the reference template. This type of filter is called a matched filter.

FIG. 10 depicts the Fourier transform of the spatial domain image of a ‘2’. It can be seen that most of the energy (bright areas) is contained in the central portion of the Fourier transform image which correspond to low spatial frequencies (the images are centred on the origin of the Fourier plane). The energy is somewhat more dispersed in the medium frequencies and is concentrated in orientations representative of the shape of the input image. Finally, little energy is contained in the upper frequencies. The right-hand-side image shows the phase content of the Fourier transform. The phase is coded from black (0°) to white (360°).

Generation of Filters from Target Images

Matched filters, as their name implies, are specifically adapted to respond to one image in particular: they are optimized to respond to an object with respect to its energy content. Generally, the contour of an object corresponds to its high frequency contents. This can be easily understood as the contours represent areas where the intensity varies rapidly (hence a high frequency as per the section about Fourier Transforms and Spatial Frequencies).

In order to emphasize the contour of an object, the matched filter can be divided by its module (the image is normalized), over the whole Fourier transform image. The resulting filter is called a Phase-Only Filter (POF) and is defined by:

$\begin{array}{cc}POF\left(u,v\right)=\frac{H^{*}\left(u,v\right)}{H^{*}\left(u,v\right)}& \left(4\right)\end{array}$

Because these filters are defined in the frequency domain, normalizing over the whole spectrum of frequency implies that each of the frequency components is considered with the same weight. In the spatial domain (e.g. usual real-world domain), this means that the emphasis is given to the contours (or edges) of the object. As such, the POF filter provides a higher degree of discrimination, sharper correlation peaks and higher energy efficiency.

The discrimination provided by the POF filter, however, has some disadvantages. It turns out that, although the optical correlator is somewhat insensitive to the size of the objects to be recognized, the images are expected to be properly sized, otherwise the features might not be registered properly. To understand this requirement, imagine a filter defined out of a given instance of a ‘2’. If that filter is applied to a second instance of a ‘2’ whose contour is slightly different, the correlation peak will be significantly reduced as a result of the great sensitivity of the filter to the original shape. A new type of filter, termed a composite filter, is introduced to overcome these limitations.

In accordance with specific implementations, filters can be designed by:

• • Appropriately choosing one specific instance (because it represents characteristics which are, on average, common to all symbols of a given class) of a symbol and calculating from that image the filter against which all instances of that class of symbols will be compared; or
  • Averaging many instances of a given to create a generic or ‘template’ image from which the filter is calculated. The computed filter is then called a composite filter since it incorporates the properties of many images (note that it is irrelevant whether the images are averaged before or after the Fourier transform operator is applied, provided that in the latter case, the additions are performed taking the Fourier domain phase into account).

The latter form procedure forms the basis for the generation of composite filters. Thus composite filters are composed of the response of individual POF filters to the same symbol. Mathematically, this can be expressed by:

h_comp(x,y)=α_ah_a(x,y)+α_bh_b(x,y)+K+α_xh_x(x,y)  (5)

The filter generated in this fashion is likely to be more robust to minor signature variations as the irrelevant high frequency features will be averaged out. In short, the net effect is an equalization of the response of the filter to the different instances of a given symbol.

Composite filters can also be used to reduce the response of the filter to the other classes of symbols. In equation (5) above, if the coefficient b, for example, is set to a negative value, then the filter response to a symbol of class b will be significantly reduced. In other words, the correlation peak will be high if h_a(x,y) is at the input image, and low if h_b(X,y) is present at the input.

However, certain considerations are problems associated with these techniques. For one, if different images are to be grouped, averaged or somehow weighted into a single composite image, a few rules are to be followed. Failure to do this may result in an overall loss of accuracy. Consequently, the images should be appropriately chosen: if the images are too similar, no net gain in accuracy will be observed. On the other hand, if the images are too dissimilar, important features might become blurred. In the latter case, more than one filter might become necessary to fully describe one symbol. From the above it becomes evident that trade-offs are to be made, and the design and number of filters fully describing all the instances of a symbol should be gauged. The following paragraphs will address these issues.

To complete the design of the filters, there remains to select the right coefficients a, b, etc. as well as their numbers. On this aspect, it should be noted that if the filter is composed of too many coefficients, the response will degrade as a result of too much averaging. This means that the multiple characters used to generate the filter will each modify the response of the global filter. If too many are used, the filter will provide an average response. Therefore, the target images used to design the composite filter should be carefully selected. To overcome this problem and to ensure a good uniformity of the filter to the different occurrences of the same in-class character while preserving the discrimination capabilities, the following technique was used.

First a filter is created out of the image of a symbol (say a ‘2’). The filter is generated in the usual way and its response is tested on many instances of ‘2’s. Ideally, the response would be uniform but it is not because of the small discrepancies between the images.

The image of the ‘2’ that presents the weakest response to the filter is linearly combined (equation 5), using the original image. Note that the multiplying coefficients are smaller than unity so that the filter resulting from the composite image is not modified too drastically within one iteration of the process.

The same procedure is repeated, and again the image with the weakest response is added to the image used to form the filter. This process is repeated until all the images respond to the filter to within a given margin, at which point the desired filter has been generated. In a test case, the threshold was set at 85%.

One could argue that the same effect could be achieved without resorting to iteration. In the next section, we will describe a method, that, in theory, can be used to achieve that operation without resorting to iteration.

Adding ‘multiple versions’ of a ‘2’ while creating the filter may induce the filter to respond to other symbols as well (e.g. crosstalk will appear) and this behaviour will intensify with the number of symbols used to compute the filter. To prevent this, a negative background can be added to the symbols used for the filter generation.

To understand the mechanics of this process, assume that there are n classes of symbols to be recognized. Let it be further assumed that one is trying to optimize the filter for the ‘2’s. The process starts with the aggregation into one single symbol of all the symbols used in the development of the remaining n−1 classes (that is all symbols, the ‘2’s excepted). This image is then subtracted from all the instances of the ‘2’s using the linear combination process shown in equation 5, thereby reducing the probabilities that the filter for the ‘2’s will exhibit a response to another class of symbol.

The effect of the addition of this background is depicted in FIG. 11, where as usual the original image is on the left, the modified image is in the middle and the phase contents of the Fourier transform corresponding to the middle image are shown at right. The effect of this procedure can be seen by the appearance of a shadow around the edges of the symbols (middle vs left images).

With this strategy, if another character other than a ‘2’ is fed to the filter, one of its constituting areas will coincide with the negatively biased region of the image used to generate the filter. This will tend to minimize the filter crosstalk response. For example if an ‘8’ is fed to the filter of the ‘2’s, its upper left and lower right vertical segments will coincide with the negatively biased part of the filter and reduce the total correlation value.

Orthogonalization

As mentioned above, there is still another way to compute the filters. Mathematically speaking, it is possible to generate a filter from the linear combination of many instances of a symbol, each of the coefficients of the linear combination being adjusted in such a way that the filter response is maximum for the target symbol while being negligible for symbols belonging to other classes (when such a condition is met the equations—or filters in this case—are said to be orthogonalized). While the goal is the same as with the iteration process just described, the method is different as it relies on simultaneously solving a set of linear equations.

To carry out this computation, the following set of equations is to be solved:

h_amax=α₁^aa+α₁^bb+K+α₁ⁿn

h_bmax=α₂^aa+α₂^bb+K+α₂ⁿn  (6)

M

h_nmax=α_n^aa+α_n^bb+K+α_nⁿn

with:

$\begin{array}{cc}{H}_{n_{\max }}=\frac{\left(h_{n_{\max }}\right)}{\left(h_{n_{\max }}\right)}& \left(7\right)\end{array}$

as before and where H_n(max) stands for the POF filter computed with the object h_n(max), □_i^k are the weighing factors of the linear combination and a . . . n, the original symbol images.

In order to be orthogonalized, the set of equation 6 should obey the following constraint:

$\begin{array}{cc}\left[\begin{array}{c}{h}_{a\left(\max \right)}\\ h_{b\left(\max \right)}\\ M\\ h_{n\left(\max \right)}\end{array}\right]\otimes \left[abKn\right]=\left[\begin{array}{cccc}1& 0& K& 0\\ 0& 1& K& 0\\ K& K& K& K\\ 0& 0& K& 1\end{array}\right]& \left(8\right)\end{array}$

where stands for the correlation operator. Note that equation (7) simply states that the normalized response of a filter to a symbol of its class should be unity while its response to a foreign symbol should be zero.

The max subscript in equations (6) through (8) is used to indicate that the calculations were performed with the coordinates origin (reference point) of each and every Fourier plane image H_n, centred on its pixel of maximum correlation.

The tests performed with orthogonalized filters did not show the level of performance expected. To understand this, let us recall that an ideal filter would present a unitary response in the presence of its corresponding symbol and a null response to the other symbols. However, the presence of sidelobes (wings of decreasing intensity that surround the peak) render that statement true at the location of the correlation peak and at that location only. Thus, nothing can be said about the response of the filter in the vicinity of the correlation peak (e.g. the pixels surrounding the very peak itself).

Consequently, in order to obtain a truly orthogonalized set of filters, the filters would ideally need to take into account the response of all the pixels of the correlation plane (peak plus sidelobes) to all the symbols. While this can be achieved, the filter generation would, in this case, become a complex and time-consuming operation.

Since the iterative approach provides similar result in a more efficient way, the orthogonalization approach was not followed any further.

In a specific example of implementation, the correlator's video and graphics input are compatible with standard computer graphics (VGA) and NTSC video signals. The maximum image area processed by the correlator is equal to 640×480 pixels, and is independent of the size of the image, as opposed to digital systems that require more processing time and power as images get larger.

Another example is the sharing of a single optical correlator by multiple image generation devices.

Second Embodiment

Cargo Container Screening

Although the above-described screening system was described in connection with screening of luggage items, the concepts described above can readily be applied to the screening of other enclosures.

For example, in an alternative embodiment, a system for screening cargo containers is provided. The system includes components similar to those described in connection with the system depicted in FIG. 1. In a specific example of implementation, the image generation device 102 is configured to scan a large object (i.e. the cargo container) and possibly to scan the large object along various axes to generate multiple images associated to the cargo container. The image or images associated with the cargo container convey information related to the contents of the cargo container. Any suitable method for generating images associated to containers may be used. Such scanning methods for large objects are known in the art and as such will not be described further here. Each image is then processed in accordance with the method described in the present specification to detect the presence of target objects in the cargo container.

Third Embodiment

Screening of Persons

Moreover, the concepts described above can readily be applied to the screening of people.

For example, in an alternative embodiment, a system for screening people is provided. The system includes components similar to those described in connection with the system depicted in FIG. 1. In a specific example of implementation, the image generation device 102 is configured to scan a person and possibly to scan the person along various axes to generate multiple images associated to the person. The image or images associated with the person conveys information related to the objects carried by the person. FIG. 12 depicts two images associated with a person suitable for use in connection with a specific implementation of the system. Each image is then processed in accordance with the method described in the present specification to detect the presence of target objects on the person.

Specific Physical Implementation

Certain portions of the image processing apparatus 106 can be implemented on a general purpose digital computer 1300, of the type depicted in FIG. 13, including a processing unit 1302 and a memory 1304 connected by a communication bus. The memory includes data 1308 and program instructions 1306. The processing unit 1302 is adapted to process the data 1308 and the program instructions 1306 in order to implement the functional blocks described in the specification and depicted in the drawings. The digital computer 1300 may also comprise an I/O interface 1310 for receiving or sending data elements to external devices.

Alternatively, the above-described image processing apparatus 106 can be implemented on a dedicated hardware platform where electrical/optical components implement the functional blocks described in the specification and depicted in the drawings. Specific implementations may be realized using ICs, ASICs, DSPs, FPGA, optical correlator, digital correlator or other suitable hardware platform.

Although the present invention has been described in considerable detail with reference to certain preferred embodiments thereof, variations and refinements are possible without departing from the spirit of the invention. Therefore, the scope of the invention should be limited only by the appended claims and their equivalents.

Claims

1. An apparatus suitable for screening a luggage item, said apparatus comprising:

a) an input for receiving an image signal associated with the luggage item, the image signal derived on the basis of penetrating radiation and conveying information related to the contents of the luggage item;

b) a processing unit in communication with said input, said processing unit being operative for: i. processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item; ii. generating a detection signal in response to detection of the presence of at least one target object in the luggage item;

c) an output for releasing the detection signal.

2. (canceled)

3. (canceled)

4. (canceled)

5. (canceled)

6. The apparatus of claim 1, wherein the detection signal enables identification of at least one target object whose presence in the luggage item was detected.

7. (canceled)

8. An apparatus as defined in claim 1, wherein the detection signal conveys position information related to at least one target object whose presence in the luggage item was detected.

9. An apparatus as defined in claim 1, wherein the detection signal conveys information describing at least one characteristic of at least one target object whose presence in the luggage item was detected.

10. An apparatus as defined in claim 1, wherein said detection signal is operative for causing a display unit to convey information related to at least one target object whose presence in the luggage item was detected.

11. An apparatus as defined in claim 1, wherein said processing unit is responsive to detection of the presence of at least one target object to:

a) generate log information elements conveying a presence of the at least one target object whose presence in the luggage item was detected;

b) store said log information data elements on a computer readable storage medium.

12. An apparatus as defined in claim 11, wherein said log information elements include a time stamp data element.

13. An apparatus as defined in claim 1, wherein said processing unit being operative for processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item comprises said processing unit being operative to effect a correlation operation between data derived from the image signal and the plurality of target images.

14. (canceled)

15. An apparatus as defined by claim 13, wherein said processing unit comprises a digital correlator for effecting the correlation operation.

16. (canceled)

17. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. (canceled)

22. A method for screening a luggage item, said method comprising:

a) receiving an image signal associated with the luggage item, the image signal derived on the basis of penetrating radiation and conveying information related to the contents of the luggage item;

b) processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item;

c) generating a detection signal in response to detection of the presence of at least one target object in the luggage item.

23. (canceled)

24. (canceled)

25. (canceled)

26. (canceled)

27. A method as defined in claim 22, wherein the detection signal enables identification of at least one target object whose presence in the luggage item was detected.

28. (canceled)

29. A method as defined in claim 22, wherein the detection signal conveys position information related to at least one target object whose presence in the luggage item was detected.

30. A method as defined in claim 22, wherein the detection signal conveys information describing at least one characteristic of at least one target object whose presence in the luggage item was detected.

31. A method as defined in claim 22, wherein said detection signal is operative for causing a display unit to convey information related to at least one target object whose presence in the luggage item was detected.

32. A method as defined in claim 22, wherein, in response to detection of the presence of at least one target object, said method comprises:

a) generating log information elements conveying a presence of the at least one target object whose presence in the luggage item was detected;

b) storing said log information data elements on a computer readable storage medium.

33. A method as defined in claim 32, wherein said log information elements include a time stamp data element.

34. A method as defined in claim 22, wherein processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item comprises effecting a correlation operation between data derived from the image signal and the plurality of target images.

35. (canceled)

36. A method as defined by claim 34, wherein said correlation operation is effected at least in part by a digital correlator.

37. (canceled)

38. (canceled)

39. (canceled)

40. (canceled)

41. (canceled)

42. (canceled)

43. A system for screening luggage items, said system comprising:

a) an image generation device suitable for generating an image signal associated with a luggage item using penetrating radiation, the image signal conveying information related to the contents of the luggage item;

b) an apparatus in communication with said image generation device, said apparatus comprising: i. an input for receiving the image signal associated with the luggage item; ii. a processing unit in communication with said input, said processing unit being operative for: (a) processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item; (b) generating a detection signal in response to detection of the presence of at least one target object in the luggage item;

c) an output module for conveying information derived at least in part on the basis of said detection signal to a user of the system.

44-63. (canceled)

64. A system as defined in claim 43, wherein said output module is adapted for generating image data conveying the location of at least one target object whose presence in the luggage item was detected.

65. A system as defined in claim 43, wherein said output module includes a display adapted for generating an output display image conveying information derived at least in part on the basis of said detection signal in visual format.

66. A system as defined in claim 43, wherein said output module includes a display adapted for generating an output display image conveying information derived at least in part on the basis of said detection signal in visual format in combination with the image associated with the luggage item.

67. A system as defined in claim 43, wherein said output module is adapted for conveying information derived at least in part on the basis of said detection signal in audio format.

68. A computer readable medium including a program element suitable for execution by a computing apparatus for screening a luggage item, said computing apparatus comprising a memory unit and a processor operatively connected to said memory unit, said program element when executing on said processor being operative for:

a) receiving an image signal associated with the luggage item, the image signal conveying information related to the contents of the luggage item;

b) processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item;

c) generating a detection signal in response to detection of the presence of at least one target object in the luggage item;

d) releasing the detection signal.

69-86. (canceled)

87. An apparatus suitable for screening a luggage item, said apparatus comprising:

a) means for receiving an image signal associated with the luggage item, the image signal conveying information related to the contents of the luggage item;

b) means for processing the image signal associated with the luggage item in combination with a plurality of target images associated with target objects to detect a presence of at least one target object in the luggage item;

c) means for generating a detection signal in response to detection of the presence of at least one target object in the luggage item;

d) means for releasing the detection signal.

88-96. (canceled)