Geographically watermarked imagery and methods
An image is digitally watermarked to convey position data about the subject of the image, such as latitude, longitude and elevation of a point depicted therein. Additional information, e.g., about camera attributes, camera angle, and 3D warp-characterizing polynomial coefficients, can also be included.
Latest Digimarc Corporation Patents:
This application is a continuation-in-part of application Ser. No. 09/800,093, filed Mar. 5, 2001, (published as U.S. 2002-0124171 A1), the disclosure of which is incorporated by reference.
FIELD OF THE INVENTIONThe present invention relates to digital watermarking, and more particularly relates to digital watermarking as applied in the geophysical sciences.
BACKGROUND AND SUMMARY OF THE INVENTIONImagery commonly has a geographical aspect, e.g., a photograph may depict a particular place. It is often important to be able to precisely geographically locate a point within a picture or other image, e.g., for purposes of natural resources extraction, mapping, etc.
In accordance with one embodiment of the invention, auxiliary data is steganographically embedded within an image to permit locations of points within the image to be determined.
This and other features and advantages of the present invention will be more readily apparent from the following detailed description, which proceeds with reference to the accompany drawings.
Due to various factors (e.g., camera lens artifacts, and the perspective from which the photo was taken, etc.), the rectangular photo typically does not depict a rectangular area of land. Instead, the area of land depicted may actually be trapezoidal, or of other shape.
Overlaid on the
Each point depicted in the image 10 has a unique position that may be expressed by latitude and longitude (and, if 3D accuracy is desired, elevation) coordinates. In accordance with an illustrative embodiment of the invention, such position data for a single location depicted in the image is determined. (Various techniques can be employed, e.g., reference to a pre-existing map or database, ground-truth measurements using GPS equipment, etc.) This location, and the pixel 16 corresponding thereto in the image, are termed the “arbitrary origin” in the discussion that follows. (For expository convenience, the arbitrary origin in this discussion is the upper-left-most pixel in the image, and the ground point corresponding thereto.)
The image 10 is digitally watermarked across its extent with a payload that includes the coordinates of the arbitrary origin (e.g., latitude/longitude/elevation). In addition, the watermark payload also includes a parameter (e.g., angle 18) identifying the orientation of a vector pointing from the arbitrary origin to a known direction (e.g., true north). The watermark payload can also include a scale datum, e.g., indicating that 100 pixels to the right (along the row) from the arbitrary origin corresponds to a distance—on land—of 250 yards.
As noted, a rectangular image generally does not depict a rectangular piece of land. Moreover, even if a photo is taken from directly overhead—using a lens that introduces no aberrations—there is the slight complication posed by the fact that longitudinal lines are not parallel, but meet at the poles. Accordingly, if high accuracy is desired, the watermark can additionally convey coefficients for one or more polynomials (e.g., one for each coordinate axis), which model the apparent warp of the photographic depiction along different axes. (In an exemplary arrangement 5 coefficients of 8 bits each are provided for the latitude and longitude polynomials, and 6 coefficients of 8 bits each are provided for the elevation polynomial.)
Thus, in an exemplary embodiment, the watermark payload may comprise the following (196 bits total):
Digital watermarking is not belabored in this specification because such technology is well understood by artisans in the field of steganography. Briefly, however, watermarking typically works by making subtle changes to the brightness of image pixels, conveying message payloads that can be detected by suitable detector software or hardware. The embedding process generally adjusts to the unique characteristics of the image, placing a stronger watermark signal in areas with rich detail and a weaker watermark signal in areas with little detail. Because the payload is carried by the image's pixels, it is file-format independent. The payload can survive most normal processing operations, such as compression, edits, file format transformations, copying, scanning and printing. Some watermarking techniques are also robust against rotation and scaling, e.g., through use of embedded calibration data, or auto-correlation techniques.
Any watermarking technique can be employed in the present invention, provided the requisite number of watermark payload bits can be embedded without introducing objectionable corruption into the image. Suitable watermarking technologies are disclosed, e.g., in patent publications U.S. Pat. No. 6,122,403 U.S. Pat. No. 6,044,182, and WO 99/45705, and in pending U.S. application Ser. No. 09/503,881 (now U.S. Pat. No. 6,614,914).
In a particular embodiment, the watermark payload is represented in a single 128×128 pixel patch 20, which is then tiled across the image (with local scaling to reduce visibility). Each patch comprises 16,384 pixels. In
To enhance robustness, the watermark payload may be processed, e.g., by BCH, Reed-Solomon, convolutional, or turbo coding, or the like, to provide error detecting/correcting capability. Such coding has the effect of transforming the 196 bit payload bits into, e.g., 320 bits (“raw bits”). Each of the 16,384 pixels in the patch is encoded with one of these raw bits, so that each such bit is represented about 50 times per patch. The pixels corresponding to a single raw bit are desirably distributed across the patch, so that severe corruption of a small area of the watermarked image does not irretrievably lose certain raw bits.
On the detection side, the image is processed to retrieve the 320 raw bits, and then the 196 payload bits are determined from the raw bits. From these payload bits, a user of the image knows the geographical coordinates of the point at the arbitrary origin and, through use of the other encoded parameters, can deduce the geographical location of any other point depicted in the image.
In other embodiments, more elaborate watermark encoding can be used. For example, instead of tiling the identical watermark patch over and over across the image, each patch can be slightly different, e.g., encoding the position of that tile within the array of tiles. In one arrangement the tile position data is a pair of numbers indicating tile-row/tile-column offsets from the tile containing the arbitrary origin. Referring to
In still other embodiments, information about the image perspective can be conveyed through a watermark. Various forms of representation are possible. In one, the image perspective data can comprise the compass angle at which the camera is pointing (θ), and the elevation angle between the arbitrary origin point and the camera (φ). The former may be represented, e.g., by 10 bits, the latter by 8. Additionally or alternatively, the perspective data can identify the lens or its attributes, so that optical distortion of the image can be characterized. In an index-based system, a six-bit code can be used to identify one of 64 different lenses.
With different types of imaging systems, different forms of perspective information may be appropriate. For example, in so-called “whisk broom” cameras (i.e., those that repeatedly acquire line scans from a moving viewpoint), the perspective information may additionally include the starting and ending positions (the latter may be expressed as an offset from the former, allowing some payload conservation).
In yet other embodiments, elevation data for different points in the image can be encoded through watermarks. In one such arrangement, elevation data is determined for points at 64-pixel gridded spacings across the image. These points are designated in
In one such arrangement, the elevation data for each starred point is watermark-encoded in a 64 pixel by 64 pixel subpatch 24 centered around the star. Again, one such sub-patch 24 is shown in
(In the
The elevation watermark, based on patches 24, may be simply overlaid on the main watermark, based on patches 20. Desirably, however, there is some coordination between the two watermarks, so as to avoid extreme changes in any pixel values (as may occur, e.g., if both watermarks try to change a pixel by a maximum amount permitted by the respective watermarking technique). In one such coordination arrangement, each pixel in the image is assigned to one of the two watermarks. For example, 50-80% of the pixels in the image may be assigned to the main watermark, and 50-20% may be assigned to the elevation watermark. The assignment may be done based on a regular array, or a stochastic assignment may be used. In some cases, it may be prudent to allocate extra pixels to carry the elevation payload where—as in the upper right—the sub-patch 24 extends beyond the boundary of the image, limiting the number of pixels to convey elevation data (e.g., for arbitrary origin point 16).
(This coordination technique has applicability beyond the present context, and is generally applicable to a variety of watermarking applications in which plural watermarks are used.)
By techniques such as the foregoing, an image can be provided with extensive photogrammetric information that travels with the image, notwithstanding distortion, cropping, format conversion, etc.
This data can be exploited in various ways. One utilizes a computer system on which the image is displayed, e.g., on a monitor or screen. An operator uses an input device, such as a mouse, light pen, graphics tablet, or the like, to designate a particular point in the displayed image. In response to selection of the point (by clicking or other known selection technique), the computer processes the embedded watermark information and displays to the operator the precise latitude, longitude and elevation of the selected point.
Using the elevation data, the computer system can also generate, and render, a 3D view of the depicted landscape, from an arbitrary viewing angle. Surfaces that are hidden in the original image may be extrapolated using known techniques, and presented in a different color or texture to indicate their synthetic basis.
In embodiments in which the camera perspective is known, the projections of latitude and longitude lines on the depicted terrain can be adjusted, e.g., in accordance with variations in elevation. If the camera perspective is such that it is viewing down a downwardly-inclined slope, for example, the latitude or longitude lines that traverse this slope can be virtually placed more closely spaced together than would be the case if the camera view were orthogonal to the slope.
The mathematical manipulations associated with such operation are somewhat complex, but well within the skills of those working in the photogrammetric and mapping arts. To determine elevation at an arbitrary point and to generate 3D models, for example, the elevations at the starred points 22 are provided to an algorithm that applies a bi-cubic spline-fitting model so as to estimate the elevation at any point on the image.
(The computer system can take various forms, but most include conventional computer components such as one or more CPUs, volatile storage (e.g., RAM), non-volatile storage (e.g., ROM, fixed and removable magnetic disks, fixed and removable optical disks), interfaces (e.g., WAN, LAN, USB, modem, serial), input/output devices (e.g., monitor, keyboard, mouse, tablet, joystick, light pen), etc. Associated with the computer system is various software, including operating system software and applications software—the latter being programmed to perform the data processing and presentation operations detailed above. Naturally, such programming can be stored on fixed or removable computer storage media. In some embodiments, parallel or distributed computer architectures may be employed, e.g., with different components of the computer system being located remote from each other.)
A variety of aerial mapping and associated image database techniques can be used in conjunction with the present invention. Representative systems are shown, e.g., in U.S. Pat. Nos. 5,608,405, 5,926,581, 5,974,423, 6,023,278, 6,177,943, 5,995,681, 5,550,937 and 6,150,972.
To provide a comprehensive disclosure without unduly lengthening this specification, applicant incorporates by reference the patents publications and applications cited above.
Having described and illustrated the principles of the invention with reference to illustrative embodiments, it will be recognized that the invention can be modified in arrangement and detail without departing from such principles.
For example, while the “arbitrary origin” was the pixel in the upper-left comer of the image in the illustrative example, this placement is not critical. The arbitrary origin can be moved to any location, with relative measurements being adjusted accordingly.
Likewise, while the detailed embodiment contemplated that the coordinates of the arbitrary origin are literally encoded as part of the digital watermark payload, in other embodiments this need not be the case. Instead, e.g., the watermark payload can be an arbitrary identifier that identifies an entry in a data structure (e.g., table or database) in which the coordinate data is stored. The same index-a-remote-store approach can be used with any of the other payload data.
Although latitude/longitude/elevation were used as exemplary coordinates, it will be recognized that other coordinate geometries can alternatively be employed.
The main watermark payload is described as including coordinate, orientation, scale, and polynomial correction data. Depending on the application, certain of this data may be omitted, and/or certain additional information may be included in the watermark payload. The payload length is exemplary. Some embodiments can employ a payload that is considerably shorter (e.g., by abbreviating the bits dedicated to each data and/or omitting certain data). Other embodiments may employ a payload that is longer.
While elevation may be expressed in height above sea level, this need not be the case. Height relative to any other measure can alternatively be employed.
The illustrative embodiments' encoding of plural data (e.g., coordinate data, and lens data) in a single watermark payload is not essential. In other embodiments, the different elements of embedded information can be conveyed through distinct watermarks, e.g., layered over each other, interspersed between each other, coordinated with each other in the manner of the elevation watermark, etc.
In many embodiments, lossless data compression techniques (e.g., Lempel-Ziv based) can be employed to reduce the number of payload bits that are encoded in a watermark.
Although described in the context of watermarking for mapping and photogrammetric purposes, the principles detailed herein find application in many other watermarking applications, not limited to the purposes particularly detailed.
In view of the many embodiments to which the principles of the invention may be applied, it should be recognized that the detailed embodiments are illustrative only and should not be taken as limiting the scope of the invention. Rather, I claim as my invention all such embodiments as fall within the scope and spirit of the following claims, and equivalents thereto.
Claims
1. A method comprising digitally watermarking an image with a payload that represents at least two position coordinates for a point or area depicted within the image, wherein the payload additionally represents at least one of the following:
- (a) the direction of a vector extending from said image to a known compass point;
- (b) a scale datum, relating a span of pixels in the image to a span of terrain depicted in the image;
- (c) a polynomial tending to characterize an apparent warp of the image.
2. The method of claim 1 in which the payload represents the direction of a vector extending from said image to a known compass point.
3. The method of claim 1 in which the payload represents a scale datum, relating a span of pixels in the image to a span of terrain depicted in the image.
4. The method of claim 1 in which the payload represents a polynomial tending to characterize an apparent warp of the image.
5. The method of claim 4 in which said position coordinates include latitude and longitude.
6. A method comprising digitally watermarking an image with a payload that includes first and second portions, said watermarking using a tiled approach, wherein uniformly-sized patches of the image are processed in accordance with the payload, wherein the first portion is unchanging across all of said tiles, but the second portion changes between tiles, so that position information about each tile can be determined therefrom.
7. A method comprising digitally watermarking different regions of an image with different watermark payload data, wherein a first region of the image is watermarked with payload data relating to an elevation of terrain depicted in said first region, and a second region of the image is watermarked with payload data relating to an elevation of terrain depicted in said second region.
8. The method of claim 7 wherein both of said first and second regions are watermarked with payload data that represents latitude and longitude of a point depicted within said image.
9. A method comprising:
- receiving an image; and
- encoding the image with a digital watermark, wherein the digital watermark comprises a plural-bit payload including first and second portions, wherein multiple instances of the digital watermark are encoded in the image with the first portion remaining unchanged in each instance, but the second portion changes between instances so that position information of corresponding image areas can be determined there from.
10. A method to determine geo-locations of two or more areas depicted in an image, wherein the image comprises a digital watermark including a plural-bit payload embedded therein, said method comprising:
- analyzing the image or data corresponding to the image to recover the plural-bit payload;
- with reference to at least the plural-bit payload, determining a geo-location of a first area depicted in the image; and
- determining a geo-location of a second area depicted in the image by a relative position of the second area to the first area or to the first geo-location.
4504910 | March 12, 1985 | Araki et al. |
4631678 | December 23, 1986 | Angermiiller |
5113445 | May 12, 1992 | Wang |
5214757 | May 25, 1993 | Mauney et al. |
5280537 | January 18, 1994 | Sugiyama |
5329108 | July 12, 1994 | Lamoure |
5385371 | January 31, 1995 | Izawa |
5499294 | March 12, 1996 | Friedman |
5502576 | March 26, 1996 | Ramsay et al. |
5664018 | September 2, 1997 | Leighton |
5812962 | September 22, 1998 | Kovac |
5825892 | October 20, 1998 | Braudaway et al. |
5861841 | January 19, 1999 | Gildea et al. |
5889868 | March 30, 1999 | Moskowitz et al. |
5889898 | March 30, 1999 | Koren et al. |
5901178 | May 4, 1999 | Lee et al. |
5919730 | July 6, 1999 | Gasper et al. |
5958051 | September 28, 1999 | Renaud et al. |
5964821 | October 12, 1999 | Brunts et al. |
5987136 | November 16, 1999 | Schipper et al. |
5990826 | November 23, 1999 | Mitchell |
6005936 | December 21, 1999 | Shimizu |
6031914 | February 29, 2000 | Tewfik et al. |
6122403 | September 19, 2000 | Rhoads |
6130741 | October 10, 2000 | Wen et al. |
6175639 | January 16, 2001 | Satoh et al. |
6181802 | January 30, 2001 | Todd |
6185312 | February 6, 2001 | Nakamura et al. |
6205249 | March 20, 2001 | Moskowitz |
6243480 | June 5, 2001 | Zhao et al. |
6246777 | June 12, 2001 | Agarwal et al. |
6249226 | June 19, 2001 | Harrison |
6263438 | July 17, 2001 | Walker et al. |
6282362 | August 28, 2001 | Murphy et al. |
6289453 | September 11, 2001 | Walker |
6301360 | October 9, 2001 | Bocionek et al. |
6310956 | October 30, 2001 | Morito et al. |
6311214 | October 30, 2001 | Rhoads |
6320829 | November 20, 2001 | Matsumoto et al. |
6324573 | November 27, 2001 | Rhoads |
6332149 | December 18, 2001 | Warmus et al. |
6332193 | December 18, 2001 | Glass et al. |
6341350 | January 22, 2002 | Miyahara et al. |
6343138 | January 29, 2002 | Rhoads |
6351439 | February 26, 2002 | Miwa et al. |
6389151 | May 14, 2002 | Carr et al. |
6401206 | June 4, 2002 | Khan et al. |
6408082 | June 18, 2002 | Rhoads et al. |
6408331 | June 18, 2002 | Rhoads |
6411725 | June 25, 2002 | Rhoads |
6418232 | July 9, 2002 | Nakano et al. |
6427020 | July 30, 2002 | Rhoads |
6448979 | September 10, 2002 | Schena |
6493514 | December 10, 2002 | Stocks et al. |
6496802 | December 17, 2002 | van Zoest |
6498984 | December 24, 2002 | Agnew et al. |
6504571 | January 7, 2003 | Narayanaswami et al. |
6505160 | January 7, 2003 | Levy et al. |
6512835 | January 28, 2003 | Numao |
6522770 | February 18, 2003 | Seder et al. |
6532541 | March 11, 2003 | Chang et al. |
6542927 | April 1, 2003 | Rhoads |
6556688 | April 29, 2003 | Ratnakar |
6614914 | September 2, 2003 | Rhoads et al. |
6625297 | September 23, 2003 | Bradley |
6636249 | October 21, 2003 | Rekimoto |
6650761 | November 18, 2003 | Rodriguez et al. |
20010001854 | May 24, 2001 | Schena et al. |
20010019611 | September 6, 2001 | Hilton |
20010022667 | September 20, 2001 | Yoda |
20010026377 | October 4, 2001 | Ikegami |
20010026616 | October 4, 2001 | Tanaka |
20010026629 | October 4, 2001 | Oki |
20010030759 | October 18, 2001 | Hayashi et al. |
20010031064 | October 18, 2001 | Domescu et al. |
20010033674 | October 25, 2001 | Chen et al. |
20010034835 | October 25, 2001 | Smith |
20010039546 | November 8, 2001 | Moore et al. |
20010046307 | November 29, 2001 | Wong |
20010051964 | December 13, 2001 | Warmus et al. |
20020001395 | January 3, 2002 | Davis et al. |
20020002679 | January 3, 2002 | Murakami et al. |
20020006212 | January 17, 2002 | Rhoads et al. |
20020009209 | January 24, 2002 | Inoue et al. |
20020044690 | April 18, 2002 | Burgess |
20020046178 | April 18, 2002 | Morito et al. |
20020057340 | May 16, 2002 | Fernandez |
20020059520 | May 16, 2002 | Murakami et al. |
20020065844 | May 30, 2002 | Robinson |
20020069370 | June 6, 2002 | Mack et al. |
20020075298 | June 20, 2002 | Schena et al. |
20020080396 | June 27, 2002 | Silverbrook et al. |
20020095586 | July 18, 2002 | Doyle et al. |
20020095601 | July 18, 2002 | Hind et al. |
20020106105 | August 8, 2002 | Pelly et al. |
20020122564 | September 5, 2002 | Rhoads et al. |
20020124171 | September 5, 2002 | Rhoads |
20020124173 | September 5, 2002 | Stone |
20020135600 | September 26, 2002 | Rhoads et al. |
20020136531 | September 26, 2002 | Harradine |
20020147910 | October 10, 2002 | Brundage et al. |
20020159765 | October 31, 2002 | Maruyama et al. |
20020168069 | November 14, 2002 | Tehranchi et al. |
20020191810 | December 19, 2002 | Fudge et al. |
20030011684 | January 16, 2003 | Narayanaswami |
20030012562 | January 16, 2003 | Lawandy et al. |
20030032033 | February 13, 2003 | Anglin et al. |
20030040326 | February 27, 2003 | Levy et al. |
20030048908 | March 13, 2003 | Hamilton |
20030053654 | March 20, 2003 | Patterson et al. |
20030063319 | April 3, 2003 | Umeda et al. |
20030069693 | April 10, 2003 | Snapp et al. |
20030074556 | April 17, 2003 | Chapman et al. |
20030083098 | May 1, 2003 | Yamazaki et al. |
20030090690 | May 15, 2003 | Katayama et al. |
20040162981 | August 19, 2004 | Wong |
20040201676 | October 14, 2004 | Needham |
20040221244 | November 4, 2004 | Baldino |
0 947 953 | November 1999 | EP |
0 953 938 | November 1999 | EP |
935 872 | November 2001 | EP |
1220152 | July 2002 | EP |
2371934 | August 2002 | GB |
2000 41144 | February 2000 | JP |
WO99/17537 | April 1999 | WO |
WO 01/05075 | January 2001 | WO |
WO01/24113 | April 2001 | WO |
WO01/39121 | May 2001 | WO |
WO01/76253 | October 2001 | WO |
WO02/03328 | January 2002 | WO |
WO02/33650 | April 2002 | WO |
- Notice of Allowance (dated Dec. 17, 2004), Appeal Brief (dated Sep. 20, 2004) and Office Action (dated May 7, 2004) from parent U.S. Appl. No. 09/800,093.
- Notice of Allowance (dated Dec. 17, 2004), Appeal Brief (dated Sep. 20, 2004) and Office Action (dated May 14, 2004) from assignee's U.S. Appl. No. 10/002,954.
- U.S. Appl. No. 09/503,881, filed Feb. 14, 2000, Rhoads et al.
- U.S. Appl. No. 09/571,422, filed May 15, 2000, Rhoads et al.
- U.S. Appl. No. 10/371,995, filed Feb. 20, 2003, Rhoads et al.
- Bartlett, “Digital Watermarking, The Unseen Advantage” Geo Informatics, Jun. 2001 (3 pages).
- Bender et al., “Techniques for Data Hiding,” SPIE vol. 2420, Jan. 1995, pp. 164-173.
- Caronni, “Assuring Ownership Rights for Digital Images,” Published in the Proceedings of ‘Reliable IT Systems,’ VIS '95, HH. Brüggemann and W. Gerhardt-Häckl (Ed.), Vieweg Publishing Company, Germany, 1995, Jun. 14, 1994, 10 pages.
- Carp, “Steven wonders of the imaging world”, International Contact, Oct./Nov. 2000, pp. 36/I-36/IV.
- Friedman, “The Trustworthy Digital Camera: Restoring Credibility to the Photographic Image,” IEEE Transactions on Consumer Electronics, vol. 39, No. 4, Nov., 1993, pp. 905-910.
- http://web.archive.org/web/20010305033241/http://www.kodak.com/country/US/en/corp/researchDevelopment/technologyFeatures/digitalWatermarking.shtml, “Invisible Watermarking”; archive date of Mar. 5, 2001 (4 pages, including Internet Archive Wayback Machine cover page).
- Koch et al., “Digital Copyright Labeling: Providing Evidence of Misuse and Tracking Unauthorized Distribution of Copyrighted Materials,” OASIS Magazine, Dec. 1995, 3 pages.
- Manjunath, “Image Processing in the Alexandria Digital Library Project,” Proc. IEEE Int. Form on Research and Tech. Advances in Digital Libraries—ADL '98, pp. 180-187.
- Seybold Seminars: Keynote: Digital Imaging Day, comments from panel including Daniel Carp, panel discussion occurring on Aug. 28, 2000 (8 pages).
- Yeung et al., “Digital Watermarks: Shedding Light on the Invisible,” Nov.-Dec. 1998, IEEE Micro vol. 18, No. 6, pp. 32-41.
- Zhao et al., “Embedding Robust Labels Into Images for Copyright Protection,” Proc. of the International Congress on Intellectual Property Rights for Specialized Information, Knowledge and New Technologies (Vienna, Austria) Aug. 21-25, 1995, 10 pages.
- Zhao, “Digital Watermark Mobile Agents,” Proc. of NISSC'99, Arlington, VA, Oct. 18-21, 1999, pp. 138-146.
Type: Grant
Filed: Nov 28, 2001
Date of Patent: Sep 27, 2005
Patent Publication Number: 20020135600
Assignee: Digimarc Corporation (Beaverton, OR)
Inventor: Geoffrey B. Rhoads (West Linn, OR)
Primary Examiner: Kanjibhai Patel
Assistant Examiner: Abolfazl Tabatabai
Attorney: Digimarc Corporation
Application Number: 09/997,400