Video imaging using multi-ping sonar

Abstract

A sonar system comprising a sonar transmitter, a very large array two dimensional sonar receiver, and a beamformer section transmits a series of sonar pings into an ensonified volume of fluid at a rate greater than 5 pings per second, receives sonar signals reflected and scattered from objects in the ensonified volume, and beamforms the reflected signals to provide a video presentation and/or to store the beamformed data for later use. The parameters controlling the sonar system are changed so that the beamformer section treats the data from the receiver section with more than one set of parameters. The stream of data is treated either in parallel or in series by different beamforming methods so that at least one beam from the beamformer has more than one value.

Description

RELATED PATENTS AND APPLICATIONS

The following US patents and US patent applications are related to the present application: U.S. Pat. No. 6,438,071 issued to Hansen, et al. on Aug. 20, 2002; U.S. Pat. No. 7,466,628 issued to Hansen on Dec. 16, 2008; U.S. Pat. No. 7,489,592 issued Feb. 10, 2009 to Hansen; U.S. Pat. No. 8,059,486 issued to Sloss on Nov. 15, 2011; U.S. Pat. No. 7,898,902 issued to Sloss on Mar. 1, 2011; U.S. Pat. No. 8,854,920 issued to Sloss on Oct. 7, 2014; and U.S. Pat. No. 9,019,795 issued to Sloss on Apr. 28, 2015; U.S. patent application Ser. Nos. 14/927,748 and 14/927,730 filed on Oct. 30, 2015, Ser. No. 15/978,386 filed on May 14, 2018, Ser. No. 15/908,395 filed on Feb. 28, 2018, Ser. No. 15/953,423 filed on Apr. 14, 2018, Ser. No. 16/693,684 filed Nov. 11, 2019, and 62/931,956 and 62932734 filed Nov. 7, 2019, Ser. No. 16/362,255 filed on Mar. 22, 2019, and 62/818,682 filed Mar. 14, 2019 and are also related to the present application. The above identified patents and patent applications are assigned to the assignee of the present invention and are incorporated herein by reference in their entirety including incorporated material.

FIELD OF THE INVENTION

The field of the invention is the field generating and receiving of sonar pulses and of visualization and/or use of data from sonar signals scattered from objects immersed in a fluid.

OBJECTS OF THE INVENTION

It is an object of the invention to improve visualization using sonar imaging. It is an object of the invention to measure and record the positions and orientations, and images of submerged objects. It is an object of the invention to improve resolution of sonar images. It is an object of the invention to present sonar video images at increased video rates. It is an object of the invention to rapidly change the sonar image resolution between at least 2 pings of a series of pings. It is the object of the invention to change rapidly change the direction of the field of view on sonar images between at least 2 pings of a series of pings.

SUMMARY OF THE INVENTION

A series of sonar pings are sent into an insonified volume of water and reflected or scattered from submerged object(s) in the insonified volume of water. One or more large sonar receiver arrays of sonar detectors are used to produce and analyze sonar data to produce 3 dimensional images of the submerged object(s) for each ping. One or more parameters controlling the sonar imaging system are changed between pings to change the series of images. The resulting changed images are combined together to produce an enhanced video presentation of the submerged objects at an enhanced video frame rate of at least 5 frames per second. More than one of the parameters used to control the sonar imaging system are used to produce different 3D images from the same ping in a time less than the time between two pings.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows a sketch of the layout where the method of the invention may be used.

FIGS. 2A, 2B and 2C show side elevation, plan view and end elevation views of the sonar transmitter of the invention.

FIG. 3 shows possible configurations of the sonar transmitter of the invention.

FIGS. 4A and 4B show the sonar transmitter of the invention sending out pings in a 50 degree included angle and a 25 degree included angle.

FIGS. 5A, 5B and 5C show plan view, side elevation, and end elevation views of the sonar receiver of the invention.

FIG. 6 shows a flow chart of the method of the invention.

DETAILED DESCRIPTION OF THE INVENTION

It has long been known that data presented in visual form is much better understood by humans than data presented in the form of tables, charts, text, etc. However, even data presented visually as bar graphs, line graphs, maps, or topographic maps requires experience and training to interpret them. Humans can, however, immediately recognize and understand patterns in visual images which would be difficult for even the best and fastest computers to pick out. Much effort has thus been spent in turning data into images.

In particular, images which are generated from data which are not related to light are often difficult to produce and often require skill to interpret. One such type of data is sonar data, wherein a sonar signal pulse is sent out from a sonar generator into a volume of sea water or fresh water of a lake or river, and reflected sound energy from objects in the insonified volume is measured by a sonar receiver.

The field of underwater sonar imaging is different from the fields of medical ultrasonic imaging and imaging of underground rock formations because there are far fewer sonar reflecting surfaces in the underwater insonified volume. Persons skilled in the medical and geological arts would not normally follow the art of sonar imaging of such sparse targets. FIG. 1 shows a sketch of the system of the invention. A vessel 10 carrying the apparatus 11 of the invention is on the surface 14 of a body of water which we will call a part of a sea. The water rests on a seabed 13. It is understood that any fluid that supports sound waves may be investigated by the methods of the present invention. The apparatus 11 generally comprises a sonar ping transmitter (or generator) and a sonar receiver, but the sonar transmitter and receiver may be separated for special operations. Various sections of the apparatus are each controlled by controllers which determine parameters required for optimum operation of the entire system. In the present specification, a parameter is a specific value to be used which can be changed rapidly between pings. The parameters may be grouped in sets and the set can be switched, either by hand or automatically according to a criterion. The decision to switch parameters may be made by an operator or made automatically based on information gained from prior pings sent out by sonar transmitter or by information gained from the current ping. FIG. 1 shows the sonar transmitter sending out pulses of sound waves 12 which propagate into the water in an approximately cone shaped beam. The pulses 12 strike objects in the water such as stones 15 on the seabed 13, an underwater vessel 17, a swimming diver 18, and a sea wall 16. The vessel 17 may either be manned or be a remotely operated vessel (ROV). The objects underwater that have a different density than the sea water reflect pulses 19 as a generally expanding waves back toward the apparatus 11.

The term “insonified volume” is known to one of skill in the art and is defined herein as being a volume of fluid through which sound waves are directed. In the present invention, the sonar signal pulse of sound waves is called and defined herein as a ping, which is sent out from one or more sonar ping generators or transmitters, each of which insonifies a roughly conical volume of fluid. A sonar ping generator is controlled by a ping generator controller according to set of ping generator parameters. Ping generator parameters comprise ping sonar frequency, ping sonar frequency variation during the ping pulse, ping rate, ping pulse length, ping power, ping energy, ping direction with respect a ping generator axis, and 2 ping angles which determine a field of view of the objects. A ping generator preferably has a fixed surface of material 22 which is part of a sphere, but may shaped differently. Preferred ping generators of the invention are sketched in FIGS. 2 through 4. FIG. 2A shows a ping generator cross section 20 with piezo electric elements 21 sandwiched between electrically conducting materials 22 and 23. Material 25 between the piezo electric elements is electrically insulating. The electrically conducting material 22 is preferably a solid sheet of material which is grounded and is in contact with the seawater. Material 22 is thin enough that ultrasonic pressure waves can easily pass through it, but thick enough that water does not leak through it and get into the interior of the ping generator. The other end of the piezoelectric material elements 21 is energized by applying an ultrasonic frequency voltage to electrical elements 24 which are separated electrically from each other and which energize groups of piezo electric elements 21 to vibrate with the same phase and frequency. Wires 25 are sketched to show the electrical connections to the different segments 24. The plan view of the transmitter shows the elements 24 in FIG. 2B shown segmented into 9 segments. FIG. 3 shows other preferred segmentation schemes useful in the method of the invention. FIG. 4A shows the beam pattern of the outgoing sonar waves if all the elements 21 are energized with the same phase and frequency electrical signal. FIG. 4B shows the beam pattern of the outgoing sonar waves if only the elements 21 in the center section of FIG. 2B are energized with the same phase and frequency electrical signal. For the relative size and curvature of the surfaces 25 of FIG. 4A, the full beam has a divergence of 50 degrees and the restricted beam shown in FIG. 4B has a divergence of 25 degrees. By energizing appropriate combinations of electrodes, the beam may be sent out up, down, left, or right.

Ping generators of the prior art could send out a series of pings with a constant ping frequency during the ping. Ping frequencies varying in time during the sent out ping are known in the prior art. Changing the ping frequency pattern, duration, power, directions, and other ping parameters rapidly and/or automatically between pings in a series has not heretofore been proposed. One method of the invention anticipates that the system itself uses the results from a prior ping can be analyzed automatically to determine the system parameters needed for the next ping, and can send the commands to the various system controllers in time to change the parameters for the next ping. When operating in a wide angle mode at a particular angle and range, for example, a new object anywhere in the field of view can signal the system controllers to send the next outgoing ping the direction of the object, decrease the field of view around the new object, increase the number of pings per second according to a criterion based on the distance to the object, set the ping power to optimize conditions for the range of the object, etc. Most preferably, the system can be set to automatically change any or all system parameters to optimize the system for either anticipated or in reaction to unanticipated changes in the environment.

In a particularly preferred embodiment, the controller system may be set to change the sent out frequency alternately between a higher and a lower frequency. The resulting images alternate between a higher resolution and smaller field of view for the higher frequency, and a lower resolution and a larger field of view for the lower frequency. The alternate images may then be stitched after the receiver stage to provide a video stream at half the frame rate of the system available with unchanged parameters, but with higher central resolution and wider field of view, or at the same frame rate by stitching neighboring images.

Intelligent steering of the high-resolution, focused field of view on to a specific target of interest would mean that this technology would not necessarily be limited only to short range applications. If only one of the four steered pings, for example, needs to be continuously updated to generate real-time images, then the range limit could be significantly extended. The intelligent focusing may be implemented in a mode whereby a low-frequency, low-resolution ping with a large field of view is used to locate the target of interest. The subsequent high-frequency, high-resolution ping may then be directed to look specifically at the region of interest without having to physically steer the sonar head.

In this particularly preferred embodiment, additional intelligent and predictive processing and inter-frame alignment may be used to account for and track motion and moving objects. The priority of frame processing may be adapted to allow focus and higher refresh rate of images including the primary target, for example with the field of view centered on a primary target, or moving objects requiring the images that represent a portion of the field of view containing moving object to be updated more frequently.

The sonar receiver of the invention is a large array of pressure measuring elements. The sonar receiver is controlled by a sonar receiver controller according to set of sonar receiver parameters. The array is preferably arranged as a planar array shown in FIG. 5 because it is simpler to construct, but may be shaped in any convenient form such as a concave or convex spherical form for different applications. The array has preferably 24 times 24 sonar detecting elements, or more preferably 48 times 48 elements, or even more preferably 64 time 64 detectors, or most preferably 128 times 128 elements. A square array of elements is preferred, but the array may be a rectangular array or a hexagonal array or any other convenient shape. The detector elements are generally constructed by sandwiching a piezo electric material between two electrically conducting materials as shown for the sonar transmitter, but with an electrical connection to each element in the array. When a reflected sonar ping reaches the sonar detecting element, the element is compressed and decompressed at the sonar ping frequency, and produces a nanovolt analog signal between the electrically conducting materials. The nanovolt signals are amplified and digitally sampled at a sonar receiver sampling rate controlled by the sonar receiver controller, and the resulting digital signal is compared to a signal related to sent out ping signals to measure the phase and amplitude of the incoming sonar signals for each receiver element. The amplification or gain for the incoming sonar signals is controlled by the sonar receiver controller. If the sonar ping frequency is changed rapidly between pings, the sampling rate may also be changed to reflect the changed ping frequency. The incoming sonar ping is divided into consecutive slices of time, where the slice time is related to the slice length by the speed of sound in the water. A slice time parameter is set by the sonar receiver controller. For example, pings arriving from more distant objects can have wider slices than pings reflections from closer objects. Each slice contains a number of sonar wavelengths as the pulse travels through the water. The sonar receiver preferably has sonar receiver parameters controlled by the sonar receiver controller to have, for example, programable phase delays between the detector elements digital sampling times may be varied to achieve the same result. The sonar receiver may have parameters controlled by the sonar receiver controller which can be set to change the amplification or gain of the nanovolt electrical signals during the incoming sonar ping reflected signals. Prior art time varying gain (TVG) systems have used preplanned amplification ramps to correct for attenuation in the water column. This gain is applied based on range (distance from transmitter), but the gain profile does not change from ping to ping. Generally, the attenuation of the ultrasonic waves is higher for higher ping frequencies. Prior art changed the amplification factor by a preplanned schedule to even out the signals between the received first slice and the last slice of a ping. Prior TVG did not allow for the increased absorption by soft mud on the seafloor, for example. Since mud absorbs sound waves, the reflected sound waves are less intense as soon as the reflected slice reaches the mud. The TVG is changed on the next ping to boost the signals that reflect or are scattered by the mud. In the same way, the TVG is changed to boost or reduce the gain for slices that more strongly reflect or are scattered by a hard, highly reflecting object like the sea wall shown in FIG. 1.

A phase and amplitude of the pressure wave coming into the sonar receiver is preferably assigned to each detector element for each incoming slice, and a phase map may be generated for each incoming slice. A phase map is like a topographical map showing lines of equal phase on the surface of the detector array.

FIG. 5C sketches a reflected ping 54 reflected by first object at a range of 20 detector widths from the detector. The first object is on a line starting from the center of the detector and perpendicular to the detector surface. The scattered ping is shown having a spherical surface to reflect a wave with origin at the surface of the first object. The phase map for this ping will be a series of circular regions centered in the center of the detector, all having the same phase, and moving outward from the center of the detector as the various slices of the ping are analyzed. Reflected ping 55 indicates a second object located further away from the detector than the first object, and at an angle of 5 degrees to the right of the center line. Reflected ping 56 shows a third object located yet further away from the detector, and at an angle of 10 degrees to the left of the center line. Pings 55 and 56 produce similar rings originating to the left and right of the detector, and expanding as slightly elliptical rings outwardly from their centers (which are not located on the detector for the angles shown).

Applying additional gain control can be incorporated with Phase Filtering.

Phase map and data cleanup and noise reduction may be done optionally in the sonar receiver or in a beamformer section. The phase map and/or the digital stream of data from the detector are passed to the beamformer section, where the data are analyzed to determine the ranges and characteristics of the objects in the insonified volume.

The range of the object is determined by the speed of sound in the water and the time between the outgoing ping and the reflected ping received at the receiver. The data are most preferably investigated by using a spherical coordinate system with origin in the center of the detector array, a range variable, and two angle variables defined with respect to the normal to the detector array surface. The beamformer section is controlled by a beamformer controller using a set of beamformer parameters. The space that the receiver considers is divided into a series of volume elements radiating from the detector array and called beams. The center of each volume element of a beam has the same two angular coordinate and each volume element may have the same thickness as a slice. The beam volume elements may also preferably have thickness proportional to their range from the detector, or any other characteristic parameters as chosen by a beamformer controller. The range resolution is given by the slice thickness.

The beamformer controller controls the volume of space “seen” by the detector array and used to collect data. For example, if the sonar transmitter sends out a narrow or a broad beam, or changes the direction of the sent out beam, the beamformer may also change the system to only look at the insonified volume. Thus, the system of the invention preferably changes two or more of the system parameters between the same pings to improve the results. Some of the parameters controlled by the beamformer controller are:

• • Field-of-view
  • Minimum and maximum beamformed ranges
  • Beam detection mode such as (First Above Threshold FAT or maximum amplitude (MAX) or many other modes as known in the art)
  • Range resolution
  • Minimum signal level included in image
  • Image dynamic range
  • Array weighting function (used to modify the beamforming profile)
  • Applying additional gain post beamforming (this can be incorporated with Thresholding).

The incoming digital data stream from each sonar detector of the receiver array has typically been multiplied by a TVG function. A triangular data function ensures that the edges of the slices have little intensity to reduce digital noise in the signal. The TVG signal is set to zero to remove data that is collected from too near too and to far away from the detector, and to increase or decrease the signal depending on the situation.

In the prior art, the data have been filtered according to a criterion, and just one volume element for each beam was selected to have a value. For example, if the data was treated to accept the first signal in a beam arriving at the detector having an amplitude above a defined threshold (FAT), the three dimension point cloud used to generate an image for the ping would be much different from a point cloud generated by picking a value generated by using the maximum signal (MAX). In the FAT case, the image would be, for example, of fish swimming through the insonified volume and the image in the MAX case would be the image of the sea bottom. In the prior art, only one range in each beam would show at most one value or point and all the other ranges of a single beam would be assigned a zero.

In the present invention, the data stream is analyzed by completing two or more beamformer processing procedures in the time between two pings, either in parallel or in series. In a video presentation, the prior art showed a time series of 3D images to introduce another, fourth dimension time into the presentation of data. By introducing values into more than one volume element per ping, we introduce a 5^th dimension to the presentation. We can “see” behind objects, for example and “through” objects and “around” objects to get much more information. We can use various data treatments to improve the video image stream. In the same way, other ways of analyzing the data stream can be used to accomplish provide cleaner images, higher resolution images, expanded range images, etc. These different images imaging tasks to can be used on only one ping. The different images may be combined into a single image in a video presentation, or in more than one video at the frame rate the same as the ping rate.

If we are surveying a seawall, we Beamform the data before the wall (sea bottom—oblique to beams (low backscatter) soft (low intensity signals returned)) differently from the harbour wall (orthogonal to beams (high back scatter) hard, high intensity. If we know where a seawall is from a chart, the beamformer can use GPS or camera data to work out what ranges are before the wall and what are after and change TVG in the middle of the returned ping.

If we know the sea depth we can specify two planes, SeaSurfacePlane and SeatBottomPlane only data between the planes will be processed and sent from the head to the top end.

A large amount of data generated per second by prior art sonar systems has traditionally been discarded because of data transmission and/or storage limits. The present invention allows a higher percentage of the original data generated to be stored for later analysis.

FIG. 6 shows a flowchart of the method of the invention. The start 60 of the process of sending out a ping is to set all system parameters for all system controllers. Either all parameters are the same as the last ping, or they have been changed automatically by signals from stages of the previous ping. Step 60 sends signal to step 61 to send commands to transmitter 62. Transmitter sends data to receiver controller 63 to set parameters for receiver 64 and start receiver 64. Receiver receives analogue signals, samples the voltages from each element, and transmits data to the beamformer controller which sends data and instructions to the Beamformer section.

The beamformer analyses data and decides whether the next ping should change settings, and if so sends signals to the appropriate controller to change the settings for the next ping. The beamformer analyses the data in step 67 and decides either on the basis of incoming ping data or on previous instructions whether to perform single or multiple types of analysis of the incoming ping data. For example, the beamformer could analyze the data using both the FAT and MAX analysis, and present both images either separately or combined, so that there will be some beams having more than one value per beam. The reduced data is sent from step 67 to step 68 which stores or sends raw data or image data for further processing into a video presentation at a rate greater than 5 frames per second.

Obviously, many modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that, within the scope of the appended claims, the invention may be practiced otherwise than as specifically described.

Claims

1. A method of recording a 3D sonar image, comprising;

a) transmitting a series of sonar pings into a first volume of water, the series of sonar pings transmitted from a sonar ping transmitting device at a rate at least 5 pings per second, wherein the sonar ping transmitting device is controlled by sonar ping transmitting parameters, and wherein each sonar ping transmitting parameter is chosen from predetermined list of sonar transmitting parameter settings;

b) receiving sonar signals reflected or scattered from objects in the first volume of water from each of the series of sonar pings, the received sonar signals received by a large two dimensional array sonar receiving device;

c) wherein the sonar receiving device is controlled by sonar receiving parameters, and;

d) beamforming the received sonar signals from each of the series of sonar pings with a sonar beamforming device to form a three dimensional (3D) sonar image of the objects reflecting or scattering the received sonar signals, wherein the sonar beamforming device is controlled by a set of sonar beamforming parameters, and wherein each sonar beamforming device parameter is chosen from predetermined list of sonar beamforming device parameter settings;

e) changing at least one beamforming parameter in the time between any two sonar pings of the series of sonar pings to produce at least two real time beamformed data sets for the same ping; and

f) combining the at least two real time beamformed data sets to produce a single video frame image in the time between two sonar pings.

2. The method of claim 1, where the combined beamformed data sets have more than one value for at least one beam.

3. The method of claim 2, where the two real time data sets are the FAT data set and the MAX data set.

4. The method of claim 2, where the two real time data sets have different range settings.

5. The method of claim 2, where the two real time data sets have different time varying gain (TVG) settings.

6. The method of claim 2, where the two real time data sets have different Field-of-View (FOV) settings.

7. A method of real time three dimensional (3D) sonar imaging, comprising:

a) insonifying a volume of fluid with a series of sonar pings, the sonar pings, wherein the series of sonar pings are produced at a rate greater than 5 pings a second;

b) receiving for each of the series of sonar pings sonar signals reflected from one or more objects in the volume of fluid, wherein the sonar signals are received with a large 2D array of sonar signal detectors;

c) beamforming the reflected sonar signals to provide a series of (3D) sonar images of the one or more objects; wherein the beamforming procedure is changed from ping to ping to produce a series of images having different field of view (FOV) for each of at least two consecutive pings.

8. The method of claim 7, further comprising;

d) stitching at least two consecutive images of the series of 3D images produced in step c) to make a composite 3D image having a wider field of view than any one of the series of (3D) sonar images of step c).

9. The method of claim 8, wherein four consecutive images of the series of 3D images having different fields of view are stitched together to produce a single image.

10. The method of claim 9, further comprising;

e) recording a video of the composite images of step d), wherein the video shows composite single images stitched from four consecutive images, the composite images shown sequentially at a rate greater than 5 images per second.

11. The method of claim 7, further comprising;

d) identifying an object of interest from at least one image of the series of 3D images of the one or more objects; and

e) changing the field of view of at least one succeeding ping is to provide further images of the object of interest.

12. The method of claim 11, wherein step e) changes the field of view so that the beamformed image of the object of interest is approximately in the center of the changed field of view in succeeding pings.

13. The method of claim 7, wherein step c) changing the beamforming procedure from ping to ping includes inserting a programmable set of delays in sonar signals received by each element of the large array of sonar signal detectors.

14. The method of claim 8, wherein a subset of the at least two consecutive images of the series of 3D images is updated continuously to generate real time images.

15. The method of claim 8, wherein intelligent processing is used to account for and/or track motion of moving objects.

16. The method of claim 8, wherein predictive processing is used to account for and/or track motion of moving objects.

17. The method of claim 8, wherein interframe alignment is used to account for and/or track motion of moving objects.

18. The method of claims 16-18, wherein the portion of the field of view containing the motion of the moving objects is updated more frequently than the remaining portions of the field of view.