METHODS AND SYSTEM FOR EFFICIENT PROCESSING OF GENERIC GEOMETRIC CORRECTION ENGINE
An apparatus and method for geometrically correcting a distorted input frame and generating an undistorted output frame. The apparatus includes an external memory block that stores the input frame, a counter block to compute output coordinates of the output frame for a region based on a block size of the region, a back mapping block to generate input coordinates corresponding to each of the output coordinates, a bounding module to compute input blocks corresponding to each of the input coordinates, a buffer module to fetch data corresponding to each of the input blocks, an interpolation module to interpolate data from the buffer module and a display module that receives the interpolated data for each of the regions and stitch an output image. The method includes determining the size of the output block based on a magnification data.
The present invention relates to geometric correction engines and more specifically to correcting distorted images captured by lenses and geometrically correcting to generate an undistorted image.
Description of Related ArtLens geometric distortion issues in a camera system are common. The distortion can be common optical distortions, such as barrel distortion, pin-cushion distortion, or fisheye distortion. A correction therefore is needed to correct these types of distortions in hardware in combination with software or firmware. Affine transformation support is added to support the image and video stabilization application, where multiple images of the same scene need to be aligned. Perspective warp is an extension to affine transform and offers additional capabilities. Perspective transformations can align two images that are captured from different camera viewpoints or locations. This can also be used to rectify the left and right images of a stereo camera to reduce the complexity of a disparity computation. Perspective transformations can also generate new viewpoints.
Most digital cameras suffer from some degree of non-linear geometric distortion. A spatial transformation is required to correct the distortion. In automotive and security applications, cameras use wide angle lenses, including fisheye lenses to provide 180+° field of view. To visually present the scene to the user in an easy-to-consume representation, these distortions need to be corrected.
Another technique is back mapping which gives coordinates of the distorted image as a function of coordinates of the undistorted output image. Correction involves back-mapping each output pixel to a location in the source distorted image, and thus the corrected image is fully populated. As the distorted pixel locations mostly fall onto fractional coordinates, correction involves interpolation among the nearest available pixels.
The output image is divided into output blocks containing pixel information of the output image, for example blocks (0213, 0214) contain information of parts of the mat in the output image. The output blocks have a fixed size with a constant width and a constant height. The width of the output block represents the number of pixels in the block arranged in a row. The height of the output block represents the number of lines in the block. In current back mapping system, each of the output blocks is back mapped into input blocks after a perspective transform and pixel interpolation. For example, output block (0214) is back mapped to input block (0204) representing a portion of the white colored border of the mat in the image (0202). Similarly, output block (0213) is back mapped to input sub blocks (0203, 0223, 0233) representing data corresponding to top right corner data from the center white square of the mat. With a fixed output block size, it is clear that certain output blocks require less information compared to other output blocks. In order to back map after perspective transform, output block (0214) required data from one input block (0204), while output block (0213) required data from 3 input sub blocks (0203, 0223, 0233). The input blocks required for an output block may be together or spread apart in the input image. The input fetch blocks are stored off chip generally in a DDR memory. When the input blocks are together, the DDR bus may accommodate fetching the required input blocks in one cycle. However, if the input blocks are spread apart, multiple cycles may be required to fetch the input block due to addressing constraints and bus width. Latency difference for fetching is different for different input blocks resulting in performance loss and poor utilization of bandwidth. Therefore, with a fixed output block size, some output blocks require less information to be fetched versus some others. The ratio of the size of the input block size to the size of the output blocks required for the output block may be defined as a magnification factor or scaling factor. The size of the output block size is determined by the largest magnification factor. In other words, the smaller the output block size the larger the amount of data required to be fetched into an internal memory. The internal memory is fixed in size and the data required to be fetched for a high magnification factor output block may require many cycles to fetch the data resulting in a high bandwidth use and loss of performance. Therefore, there is a need to program the size of the output block size based on the magnification factor so that the bandwidth and cycles for fetching the input block data into an internal memory is minimized resulting in an enhanced performance.
The present invention is an apparatus and method for geometrically correcting a distorted input frame and generating an undistorted output frame. The apparatus includes an external memory block that stores the input frame, a counter block to compute output coordinates of the output frame for a region based on a block size of the region, a back mapping block to generate input coordinates corresponding to each of the output coordinates, a bounding module to compute input blocks corresponding to each of the input coordinates, a buffer module to fetch and store data corresponding to each of the input blocks, an interpolation module to interpolate data from the buffer module, and a display module that receives the interpolated data for each of the regions and stitch an output image. The method includes determining the size of the output block based on magnification data and identifying redundant regions in the output image.
The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, wherein:
While this invention is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detailed preferred embodiment of the invention with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention and is not intended to limit the broad aspect of the invention to the embodiment illustrated.
The numerous innovative teachings of the present application will be described with particular reference to the presently preferred embodiment, wherein these innovative teachings are advantageously applied to the particular problems of a geometric correction engine. However, it should be understood that this embodiment is only one example of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed inventions. Moreover, some statements may apply to some inventive features but not to others.
While this invention has been particularly shown and described with reference to preferred embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.
It should be noted that the terms “output image” and “output frame” are interchangeably used herein after to indicate an image displayed on a screen similar to frame (0202).
It should be noted that the terms “input image” and “input frame” are interchangeably used herein after to indicate an image displayed on a screen similar to frame (0201).
It should be noted that the terms “output block size” and “size of output block” are interchangeably used herein after to represent the size of an output block in an output frame. The output block size is defined by an output block width and an output block height. The terms “output block width” or the “width of the output block” are interchangeably used herein after to denote number of pixels arranged in a line. Similarly, the terms “output block height” or the “height of the output block” are interchangeably used herein after to denote number of lines in the output block. For example, an output block size may be defined as 16×20 to indicate an output block width of 16 and an output block height of 20. In other words, the number of pixels in the output block is 16 arranged in a row and the number of lines in the block is 20.
It should be noted that the terms “region size” and “size of region” are interchangeably used herein after to represent the size of a region in an output frame. The region size is defined by a region width and a region height. The region may comprise one or more output blocks. For example region size may be defined as 64×40 to indicate a region width of 64 and region height of 40. The region may be divided into equal output blocks with an output block size of 16×20. The output blocks size 16×20 may be arranged in 4 columns and the number of rows in the column is 2.
Histogram of Scaling Data with Scaling Thresholds (0500)
According to a preferred exemplary embodiment, a spatial plot (0520) of the scaling data of the output blocks based on the scaling thresholds is plotted for the output frame. The thresholds identified in the histogram (0500) enables the output frame to be divided into regions based on the thresholds. As illustrated in the spatial plot (0520), the area (0504) represents area with minimum magnification factors and the area (0503) represents area with maximum magnification factors. According to a preferred exemplary embodiment, the output frame is divided into regions based on the scaling factors. According to a further exemplary embodiment, the output block size is determined by the scaling factor within the region. It should be noted that the scaling factor within a region may be substantially the same. In other instances the scaling factor within a region may vary and within 200% of each other. In other instances the scaling factor within a region may vary and within 20-40% of each other. The more the number of thresholds selected from the histogram the more the number of available block sizes and the more flexibility in dividing the output frame into regions with identical scaling data.
Exemplary Spatially Adaptive Slicing Embodiment (0600)It should be noted that the output frame (0600) divided into 9 regions may not be construed as a limitation. The output frame may be divided into any number of regions as possible given the hardware limitations. The frame may be divided into 3 horizontal slices (0611, 0612, and 0613) corresponding to 3 heights RH1, RH2 and RH3, and frame may be divided into up to 3 vertical slices (0601, 0602, and 0603) corresponding to 3 widths RW1, RW2 and RW3. A total of 9 regions (0621-0629) may be created for the frame. Each region may be programmed with independent output block size. The output frame may be programmed with bigger block size for region with less spatial variation (i.e. scaling factor) and smaller block size for region with high spatial variation. This may improve the band width as well as performance as portion of the image with bigger block size increases. For example, in surround view application, the block size may vary from 16×20 to 112×64 for right camera and 32×8 to 136×160 for a front camera. The output frame may be programmed individually for input images from each of the cameras. According to an exemplary embodiment greater than 40% saving in band width may be achieved. According to a more preferred exemplary embodiment a greater than 80% saving in band width may be achieved. According to a most preferred exemplary embodiment a greater than 500% saving in band width may be achieved
Each of the regions may be programmed with a different block size as depicted in the output frame (0600). Region (0624), which has the maximum scaling factor (0503), is divided into 16 blocks with a minimum block size, while region (0623) which has the minimum scaling factor (0504) is divided into 2 blocks with a maximum block size. Similarly, regions (0622) and (0621) are divided into 4 blocks, regions (0623), (0626), and (0629) are divided into 2 blocks each, region (0625) is divided into 6 blocks and region (0628) is divided into 9 blocks. As clearly illustrated the output block is spatially sliced into regions with varying block sizes. According to a preferred exemplary embodiment, the size of each of the regions in the output frame may be equal. According to another preferred exemplary embodiment, the size of each of the regions in the output frame may not be equal. The output frame may be composed or constructed one region at a time in a raster scan mode. For example, region 0621 may be processed first, followed by regions 0622, 0623, 0624, 0625, 0626, 0627, 0628, and 0629 in that order. Within a region, a raster scan may be followed to compose the region. For example, within region 0621, block 0631 is processed first followed by blocks 0641, 0651, and 0661 in that order. Similarly, within region 0625, block 0635 is processed first followed by blocks 0645, 0655, 0665, 0675, and 0685 in that order. Similarly, block 0633 is processed followed by block 0643. The processing of each of the block is further described below in the method flow chart of
When a frame is divided into sub-set of 3×3 regions, the following combinations of regions may be supported.
-
- 3 vertical slices RW1=x, RW2=y and RW3=z
- 2 vertical slices RW1=x, RW2=y and RW3=0, Last region width has to be zero
- 1 vertical slice RW1=x, RW2=0 and RW3=0, Last two regions width has to be zero
- 3 horizontal slices RH1=x, RH2=y and RH3=z
- 2 horizontal slices RH1=x, RH2=y and RH3=0 Last region height has to be zero
- 1 horizontal slice RH1=x, RH2=0 and RH3=0 Last two regions height has to be zero
- A 2×2 region partitioning can be done by programming RW3=RH3=0.
The spatial adaptive slicing apparatus (0703) reads input frame from master port (0707) via DDR or on-chip and performs perspective transform as well as correction of distortion (including fisheye lenses). The output of apparatus (0703) can be sent to external memory (DDR) or sent to other hardware blocks such as scalar block or noise filter block for further pre-processing via local shared memory (0705). Spatial adaptive slicing apparatus (0703) may be targeted to operate at 120 frames per second (FPS) @300 MHZ with 2M pixel frame. That performance requirement may translate to BW of approximately 1.5 GBs. In order to meet the bandwidth requirement of 1.5 GBs, a read master interface may be limited to maximum response latency of 200 cycles.
The scalar block reads data from shared memory (0704) and may generate up to 10 scaled outputs from 2 inputs with various scaling ratios ranging from 0.5 to 1. The output of scalar block to shared memory (SL2) can be further noise filtered using noise filter block or written to DDR.
The noise filter block reads data from memory (eg. DDR or on-chip) to shared memory (0704) and performs bilateral filtering to remove noise. The output of noise filter block can be sent to external memory (e.g. DDR) from shared memory (0704) or can be further re-sized using the scalar block.
The shared level 2 (0704) memory block may be used to exchange data across hardware blocks such as apparatus (0703), scalar block, and noise filter block as well as to DMA Engine (UDMA).
A HTS (Hardware Thread Scheduler) block may be used for IPC communication among various hardware blocks such as apparatus (0703), scalar block, and noise filter block as well as to DMA Engine (UDMA). The message manager (0702) may be implemented as HTS.
The configuration manager (0701) may be used to program the hardware with typical network parameters. The system (0700) may further comprise typical hardware inputs such as clock, reset, network, data, and debug signals.
Exemplary Adaptive Slicing Apparatus Embodiment (0800)As shown in
Given the coordinates of the undistorted output image such as image (0600), the corresponding coordinates of the distorted input image may be calculated by combining the output coordinates and the offsets from an offset table. Distorted pixels from the input frame are read from the frame buffer, and buffered for the bilinear interpolation. After the interpolation, corrected image may be written back and stored to the SL2 shared memory.
The counter block (0802) computes output coordinates (0803) of the output frame for a region based on a size of the output block of the region. For example, for region (0621) in
In order to carry out geometric correction efficiently in time and frame buffer traffic, the hardware processes the output frame in small output blocks. Software running on HLOS or RTOS may configure appropriate parameters then initiate spatially adaptive slicing function by writing to a control register in the hardware. The hardware may store and maintain a bank of registers used for storing control and data information. The hardware controls the sequencing through output blocks, DMA transfers, and computation to process an entire image autonomously. An interrupt, if enabled, is asserted at the completion of the processing and composition of the output image.
The hardware may also be stalled and controlled on a macro-block basis by an external controller. An intermediate interrupt may be provided by the apparatus (0800) to facilitate the stalling of the hardware. Stalling the apparatus (0800) may be achieved by deasserting a write request enable on the output write port.
When a camera is viewing a scene from two different positions or when multiple cameras are viewing the scene from different positions, a transformation between the two viewing angles is needed to align the images. Under specific conditions, the class of geometric transformations known as “homography”, or planar-perspective transformation, will capture the geometric relationship between the images accurately. Common applications of homography transforms are to align (or stitch) multiple frames of the same scene to compute a panoramic output image. A second application is the alignment of planar surfaces in the world. Finally, perspective transforms are also useful in computing depth maps from a stereo image pair. By rectifying the two views, the search to compute disparity between the two views is simplified to a 1-D search problem. The homography is defined by a 3×3 transformation matrix, as in
haff=a*hu+b*vu+c (1)
vaff=d*hu+e*vu+f (2)
z=g*hu+h*vu+1 (3)
hp=haff/z
vp=vaff/Z
The affine transform is a subset of the perspective transformation. By setting g=h=0, hp=haff and vp=vaff. The mapping from destination coordinate to the source coordinate is expressed as
haff=a*hu+b*vu+c
vaff=d*hu+e*vu+f
Where hu and vu are horizontal and vertical coordinates of the source coordinates and heff and veff are the horizontal and vertical coordinates of the destination coordinates, a, b, c, and fare correlation and transform factors.
In image alignment applications, the homography matrices may be computed by locating corresponding points in the two frames and estimating the matrix parameters to transform the set of points in one frame onto the corresponding points in the second frame. In the stereo rectification application, the matrix is determined (pre-computed) at the calibration step and remains fixed.
In YCbCr mode, the offset table (0808) defines a (□x, □y) vector for a regular grid of output points. The grid can be fully sampled or down sampled. A fully sampled grid will define an offset vector for every output pixel, defining where to fetch the input data to compute the output pixel. This is the most precise definition and can capture rapidly changing offset tables. The drawback is that it will require a large amount of memory bandwidth as the geometric correction engine will be reading offset values for every output pixel. Since most offset tables are not expected to change rapidly in a small spatial region, a subsampled offset table may be read. Offset tables can be subsampled by powers of two in both horizontal and vertical directions and the subsampling factor is set in a register. This mode conserves memory bandwidth by reducing the amount of data read to describe the offset vectors, but requires more hardware to interpolate the missing offset vectors.
x1=xo+Δx
y1=yo+Δt
The output coordinates (0803) (hu vu) from the counter block may be input to a perspective transform block (0804) and after transformation using any of the equations (1,2,3) aforementioned the transformed output coordinates (0805) (hp vp) may be input to a mesh mapping block (0806). If back mapping is enabled in a mux block (0815), the coordinates (0807) (hd,vd) calculated by the back mapping block is input to the buffer block (0809). If back mapping is not enabled in a mux block (0815), the coordinates (0805) (hp,vp) calculated by the perspective transform block (0804) is input to the buffer block (0809).
Once mesh block fetch is completed, final input co-ordinates are calculated by applying back mapping on previously calculated perspective warp corner pixel co-ordinates. Additional padding is applied on top of these back mapped corner co-ordinates based on the interpolation type.
It is possible that the bounding box calculated by hardware may not cover all the input data required to generate particular output block. In such cases, software may apply an additional PixelPad, the amount of padding in input block in all directions. For each output pixel in the output block with a size OBW×OBH, the input pixels required are indeed bounded by back mapping of the 4 corners plus/minus the padding. More precisely, the input block may be determined by the following equations:
IBX_start=min(truncate(distortx(corner1)), truncate(distortx(corner3)))−Hw_Pad−PixelPad
IBX_end=max(truncate(distortx(corner2)), truncate(distortx(corner4)))+Hw_Pad+PixelPad
IBY_start=min(truncate(distorty(corner1)), truncate(distorty(corner2)))−Hw_Pad−PixelPad
IBY_end=maxn(truncate(distorty(corner3)), truncate(distorty(corner4)))+Hw_Pad+PixelPad
where corner1, corner2, corner3, and corner4 are upper-left, upper-right, lower-left, and lower-right corners of the OBW x OBH output block, and distortx(.), distorty(.) are X and Y coordinates of the corners after perspective wrap and back mapping.
For a geometric distortion correction, PixelPad is zeroed out to accommodate neighbor sets for all colors. Software may set the PixelPad such that information of the input blocks is not dropped. According to a preferred exemplary embodiment, the bounding module is further configured to add a buffer pixel pad to each of the input blocks increasing the size of the input blocks.
OBH and OBW may be chosen as reasonably large for efficient operation of adaptive slicing and geometric correction operation. According to a preferred exemplary embodiment, the region width ranges from 4 to 8094; and the region height ranges from 2 to 8094 According to a preferred exemplary embodiment the output block width ranges from 4 to 254; and the output block height ranges from 2 to 254. OBW is constrained to ensure efficient external memory write. Another constraint is that input block size, for each input block of the image, the allocated input buffer needs to accommodate without overflowing. If the parameters OBH, OBW are set too small, or PixelPad too large, performance may degrade and unintended and undesired external memory transfer may be happen.
An off-line utility program may be utilized to program and configure the apparatus (0800). Given offset table contents, processing parameters, and maximal input buffer size, the program computes an optimal set of OBW, OBH, and PixelPad to minimize input bandwidth. Another utility may be provided with a functional C model that computes the minimum PixelPad given a configuration and the processing block size with OBW (output block width) and OBH (output block height).
The hardware can be utilized to process a portion of the image, rather than the whole image. This allows an image to process through multiple software/hardware interactions to correct only a portion of the image to save time.
An intermediate interrupt, may also be provided on completion of each macroblock output write. This allows the geometric correction operation to be pipelined with other tasks. The apparatus (0800) output write stall after this event, waiting for a pulse on a start signal to begin writing the next macro block.
As the coordinates (0807) (hd,vd) calculated by the back mapping block (0806) are not integer values in most cases, either bi-cubic or bilinear interpolation is applied to the distorted pixels with the interpolation block (0810). Other interpolation techniques may also be applied. According to a preferred exemplary embodiment, interpolation data is interpolated with a bi-cubic or bilinear interpolation. Depending on register configuration, either bi-cubic or bilinear interpolation is used to interpolate the output Y pixels. In the case of bi-cubic interpolation, the distorted input pixel is interpolated from the 16 Y pixels in the 4×4 grid around the distorted input location, as shown in
Exemplary Geometric Correction of an Input Frame with a Spatially Adaptive Apparatus Method (1100).
As generally seen in the flow chart of
-
- 1) allocating a size for an output frame and dividing the output frame into blocks; each of the blocks having a block size; the block size having a width and a height; the width based on the pixels arranged in a line; the height based on the number of lines (1101);
- As described in
FIG. 2 , the size of the output frame may be allocated based on the frame to be displayed on a screen. Software or a utility may determine, allocate, and configure a hardware register with a frame size comprising a frame width and a frame height. For example, the output frame for a 4K display may be allotted to be 3840×2160. In another example a HD display of 1920×1080 or 2 Mega pixel may be allotted.
- As described in
- 2) capturing the input image with an optical device (1102);
- The captured image may be stored in an external memory with the individual fetch blocks.
- 3) mapping back each of the blocks in the output frame to correspond to blocks in the input image (1103);
- Software or a utility may determine and identify fetch blocks corresponding to each of the output blocks.
- 4) quantifying scaling data of each of the blocks in the output frame (1104); Scaling thresholds may be determined based on the histogram (0500). For example, with reference to histogram (0500), 3 scaling thresholds (0501, 0502, and 0505 are computed or in some cases chosen. Scaling threshold (0501) indicates a scaling factor of approximately 40 and the number of blocks with a scaling factor 40 is approximately 5. Similarly, scaling threshold (0505) indicates a scaling factor of approximately 5 and the number of blocks with a scaling factor 5 is more than 1200.
- 5) generating a histogram from the scaling data of the output frame (1105); a distribution of magnification factor (ratio of fetched block size compared to output block size) may be plotted on the x-axis versus the number of blocks. The histogram may be plotted by an offline utility or with software running with HLOS or RTOS after the magnification data is computed and stored for each of the output blocks in the output image
- 6) identifying scaling thresholds from the histogram (1106);
- 7) plotting a spatial domain plot of the output frame with the scaling data and the scaling thresholds (1107);
- a spatial plot (0520) of the scaling data of the output blocks based on the scaling thresholds may be plotted for the output frame.
- 8) dividing the output frame into regions based on the spatial domain plot and the scaling thresholds (1108);
- 9) computing and dividing each of the regions into output blocks based on the scaling thresholds and the scaling data within the region (1109); The output frame may be divided into any number of regions as possible given the hardware limitations. For example, the frame may be divided into up to 3 horizontal slices (0611, 0612, and 0613) and 3 vertical slices (0601, 0602, and 0603) for a total of 9 regions (0621-0629). Each region may be programmed with independent output block size. The output frame may be programmed with bigger block size for region with less spatial variation (i.e. scaling factor) and smaller block size for region with high spatial variation. This may improve the band width as well as performance as portion of the image with bigger block size increases. For example, in surround view application the block size may vary from 16×20 to 112×64 for right camera and 32×8 to 136×160 for a front camera.
- 10) programming the spatially adaptive slicing apparatus with a size for each of the regions and a size for each of the output blocks in each of the regions (1110); and
- For example an output block size may be defined as 16×20 to indicate an output block width of 16 and an output block height of 20. In other words, the number of pixels in the output block is 16 arranged in a row and the number of lines in the block is 20. The region size is defined by a region width and a region height. The region may comprise one or more output blocks. For example region size may be defined as 64×40 to indicate a region width of 64 and region height of 40. The region may be divided into equal output blocks with an output block size of 16×20. The output blocks size 16×20 may be arranged in 4 columns and the number of rows in the column is 2.
- 11) correcting the input frame geometrically for each of the regions across the output frame, composing and displaying the output frame on a graphical device (1111).
- 1) allocating a size for an output frame and dividing the output frame into blocks; each of the blocks having a block size; the block size having a width and a height; the width based on the pixels arranged in a line; the height based on the number of lines (1101);
Due to redundancy, some regions of the frame may not be used for stitching of final frame. By programming a register to disable the processing of the region, hardware skips the processing of particular region that is identified as redundant. A frame done signal may be sent on last pixel of last valid region. As an example shown in
A portion (1301) of an output image (1303) may be processed while skipping some regions, the following parameters may be programmed into an apparatus such as apparatus (0800).
-
- InitX X coordinate of upper-left corner of output frame
- InitY Y coordinate of upper-left corner of output frame
- FrameW Width of output compute frame
- FrameH Height of output compute frame
- FrameBase SDRAM address of upper-left corner of output frame (Y and Cb/Cr base may be needed in case of YCbCr420).
- FrameOfst SDRAM frame width (in bytes)
In some embodiments FrameW may be a multiple of OBW, FrameH may a multiple of OBH, and FrameBase a multiple of 64 (byte address).
As generally seen in the flow chart of
-
- 1. capturing an arbitrary shaped input image with a plurality of optical devices (1401);
- 2. processing each of the images captured with the plurality of optical devices (1402);
- 3. identifying redundant blocks and valid blocks in each of the regions (1403);
- 4. allocating an output frame with an output frame size and dividing the output frame into regions shaped as a rectangle (1404); Regions are decided after finding unused blocks for best mapping of unused blocks to skipped regions.
- 5. programming the apparatus and disabling processing for each of the redundant regions (1405); and
- 6. composing the output frame for each of the regions containing the valid blocks(1406).
As generally seen in the flow chart of
-
- 1. allocating an output frame with an output frame size and dividing the output frame into output blocks; each of the output blocks having an output block size; the output block size having a width and a height; the width based on the pixels arranged in a line; the height based on the number of lines (1601);
- 2. capturing an input image with an optical device (1602);
- 3. mapping back each of the output blocks in the output frame to input blocks in the input image (1603);
- 4. computing a size of the input blocks in the input image corresponding to each output blocks (1604);
- Input block computation may happens in the following sequence of processing. For each output block following steps are done in sequence.
- 1. Calculate input block.
- 2. Fetch input block (assuming the size fits in internal memory).
- 3. Processing of the pixels and write out.
- The aforementioned steps (1-3) are performed in sequence of each output block
- 5. for each of the output blocks, checking for size of the input blocks with the size of the internal memory (1605);
- 6. if the size of the input blocks is less than the size of the internal memory, programming an apparatus with input parameters, fetching the input blocks into an internal memory and processing the next output block in step (1605) until all the output blocks are processed and proceeding to step (1609); if not, proceeding to step (1605) (1606);
- For example, if the size of the input block is 640×20 and the size of the internal memory is 12800 bytes, then the input block is completely fetched in one fetch transaction or cycle or else step (1607) may be executed.
- 7. dividing the output block equally into sub blocks and checking if the size of the sub block is less than the size of the internal memory; if so, programming an apparatus with input parameters, fetching the input blocks into an internal memory, processing each of the sub blocks and processing the next output block in step (1605) until all the output blocks are processed and proceeding to step (1609); if not proceeding to step (1608) (1607);
- For example, if the size of the input block is 640×20 and the size of the internal memory is 6400 bytes, then the input block is divided into sub blocks. The size of the sub block is less than or equal to the size of the internal memory, the sub blocks are each fetched into the internal memory and processed sequentially to generate an output block.
- 8. dividing further until the size of divided sub blocks is less than the size of the internal memory, programming an apparatus with input parameters, fetching the input blocks into an internal memory, processing each of the divided sub blocks sequentially and processing the next output block in step (1605) until all the output blocks are processed (1608); and
- 9. composing the output frame for each of the blocks in the output frame (1609).
The present system may be broadly generalized as an apparatus comprising:
-
- (a) a counter block configured to compute output coordinates of the output frame for a region based on a size of the output block of the region;
- (b) a transform and back mapping block configured to generate input coordinates corresponding to each of the output coordinates;
- (c) a bounding module configured to compute input blocks corresponding to each of the input coordinates;
- (d) a buffer module configured to fetch data corresponding to each of the input blocks and store in an internal memory; and
- (e) an interpolation module configured interpolate data received from the buffer module; wherein
- the interpolated data for each of the regions is stitched, composed, and displayed on a display module.
This general system summary may be augmented by the various elements described herein to produce a wide variety of invention embodiments consistent with this overall design description.
Method SummaryThe present method may be broadly generalized as a method for geometrically correcting an input image with a spatially adaptive slicing apparatus and generating an output frame wherein:
the method comprising the steps of:
-
- a) allocating a size for an output frame and dividing the output frame into blocks; each of the blocks having a block size; the block size having a width and a height, the width based on the pixels arranged in a line; the height based on the number of lines;
- b) capturing the input image with an optical device;
- c) mapping back each of the blocks in the output frame to correspond to blocks in the input image;
- d) quantifying scaling data of each of the blocks in the output frame;
- e) generating a histogram from the scaling data of the output frame;
- f) identifying scaling thresholds from the histogram;
- g) plotting a spatial domain plot of the output frame with the scaling data and the scaling thresholds;
- h) dividing the output frame into regions based on the spatial domain plot and the scaling thresholds;
- i) computing and dividing each of the regions into output blocks based on the scaling thresholds and the scaling data within the region;
- j) programming the spatially adaptive slicing apparatus with a size for each of the regions and a size for each of the output blocks in each of the regions; and
- k) correcting the input frame geometrically for each of the regions across the output frame, composing and displaying the output frame on a graphical device.
This general method may be modified heavily depending on a number of factors, with rearrangement and/or addition/deletion of steps anticipated by the scope of the present invention. Integration of this and other preferred exemplary embodiment methods in conjunction with a variety of preferred exemplary embodiment systems described herein is anticipated by the overall scope of the present invention.
System/Method VariationsEmbodiments of the present invention anticipates wide variety of variations in the basic theme of construction. The examples presented previously do not represent the entire scope of possible usages. They are meant to cite a few of the almost limitless possibilities.
This basic system, method, and product-by-process may be augmented with a variety of ancillary embodiments, including but not limited to:
-
- An embodiment wherein the optical device is selected from a group consisting of: wide angle lens, fish eye lens, and automotive camera lens.
- An embodiment wherein the step of capturing the input image further comprises storing the input image in an external memory.
- An embodiment wherein the step of mapping back further comprises generating input coordinates corresponding to each of the blocks in the output frame; the input coordinates further comprising pixel information in the input frame.
- An embodiment wherein the size of the output frame configured with an output frame height and a output frame width; the output frame height ranges from 8 to 8094; and the output frame width ranges from 8 to 8094;
- An embodiment wherein the number of pixels in the line in step ranges from 8 to 8094;
- An embodiment wherein the number of lines in the line in step ranges from 8 to 8094;
- An embodiment wherein the step of mapping back further comprises identifying output corners for each of the blocks; performing a perspective transform on each of the corners; finding the input corners corresponding to the output corners using back mapping after the perspective transformation.
- An embodiment wherein the step of quantifying scaling data further comprises determining the number of input blocks required to be fetched for each of the blocks in the output frame.
- An embodiment wherein the step of identifying scaling thresholds is further determined by grouping output blocks with similar scaling data.
- An embodiment wherein the step of dividing the output frame into regions further comprises grouping output blocks with identical scaling data such that the size of each of the region is maximized.
- An embodiment wherein the step of programming the spatially adaptive slicing apparatus further comprises writing a size of each of the regions and a size of each of the blocks into a register in a register bank; the register bank maintained in an internal memory of the apparatus.
- An embodiment wherein the size of each of the regions is determined by a region width and a region height; the region width ranges from 8 to 8094; and the region height ranges from 8 to 8094.
- An embodiment wherein the size of each of the output blocks in each of the regions is determined by an output block width and a output block height; the output block width ranges from 4 to 254; and the output block height ranges from 2 to 254.
- An embodiment wherein the step of correcting the output frame further comprises correcting each of blocks in a region in a raster scan mode and moving to next region in a raster scan mode.
- An embodiment wherein the step of displaying the output frame further comprises fetching the input blocks for each of the output blocks into an internal memory without overflowing the internal memory.
One skilled in the art will recognize that other embodiments are possible based on combinations of elements taught within the above invention description.
Claims
1. A method for geometrically correcting an input image with a spatially adaptive slicing apparatus and generating an output frame, said method comprising the steps of:
- a) allocating a size for an output frame and dividing the output frame into blocks; each of the blocks having a block size; the block size having a width and a height; the width based on the pixels arranged in a line; the height based on the number of lines;
- b) capturing the input image with an optical device;
- c) mapping back each of the blocks in the output frame to correspond to blocks in the input image;
- d) quantifying scaling data of each of the blocks in the output frame;
- e) generating a histogram from the scaling data of the output frame;
- f) identifying scaling thresholds from the histogram;
- g) plotting a spatial domain plot of the output frame with the scaling data and the scaling thresholds;
- h) dividing the output frame into regions based on the spatial domain plot and the scaling thresholds;
- i) computing and dividing each of the regions into output blocks based on the scaling thresholds and the scaling data within the region;
- j) programming the spatially adaptive slicing apparatus with a size for each of the regions and a size for each of the output blocks in each of the regions; and
- k) correcting the input frame geometrically for each of the regions across the output frame, composing and displaying the output frame on a graphical device.
2. The method of claim 1 wherein the optical device is selected from a group consisting of: wide angle lens, fish eye lens, and automotive camera lens.
3. The method of claim 1 wherein the step (b) of capturing the input image further comprises storing the input image in an external memory.
4. The method of claim 1 wherein the step (c) of mapping back further comprises generating input coordinates corresponding to each of the blocks in the output frame; the input coordinates further comprising pixel information in the input frame.
5. The method of claim 1 wherein the size of the output frame configured with an output frame height and a output frame width; the output frame height ranges from 8 to 8094; and the output frame width ranges from 8 to 8094;
6. The method of claim 1 wherein the number of pixels in the line in step (a) ranges from 8 to 8094;
7. The method of claim 1 wherein the number of lines in the frame in step (a) ranges from 8 to 8094;
8. The method of claim 1 wherein the step of mapping back further comprises identifying output corners for each of the blocks; performing a perspective transform on each of the corners; with back mapping finding the input corners corresponding to the output corners after the perspective transformation.
9. The method of claim 1 wherein the step (d) of quantifying scaling data further comprises determining the number of sub input blocks required to be fetched for each of the blocks in the output frame.
10. The method of claim 1 wherein the step (f) of identifying scaling thresholds is further determined by grouping output blocks with similar scaling data.
11. The method of claim 1 wherein the step (h) of dividing the output frame into regions further comprises grouping output blocks with identical scaling data such that the size of each of the region is maximized.
12. The method of claim 1 wherein the step (h) of programming the spatially adaptive slicing apparatus further comprises writing a size of each of the regions and a size of each of the blocks into a register in a register bank; the register bank maintained in an internal memory of the apparatus.
13. The method of claim 1 wherein the size of each of the regions is determined by a region width and a region height; the region width ranges from 4 to 8094; and the region height ranges from 2 to 8094.
14. The method of claim 1 wherein the size of each of the output blocks in each of the regions is determined by an output block width and a output block height; the output block width ranges from 4 to 254; and the output block height ranges from 2 to 254.
15. The method of claim 1 wherein the step (k) of correcting the output frame further comprises correcting each of blocks in a region in a raster scan mode and moving to next region in a raster scan mode.
16. The method of claim 1 wherein the step (k) of displaying the output frame further comprises fetching the input blocks for each of the output blocks into an internal memory without overflowing the internal memory.
17. An apparatus for geometrically correcting an input frame and generating an output frame, said apparatus comprising:
- a) a counter block configured to compute output coordinates of the output frame for a region based on a size of the output block of the region;
- b) a transform and back mapping block configured to generate input coordinates corresponding to each of the output coordinates;
- c) a bounding module configured to compute input blocks corresponding to each of the input coordinates;
- d) a buffer module configured to fetch data corresponding to each of the input blocks and store in an internal memory; and
- e) an interpolation module configured interpolate data received from the buffer module; wherein the interpolated data for each of the regions is stitched, composed, and displayed on a display module.
18. The apparatus of claim 17 wherein an external memory block is configured to store the input frame; said external memory external to the apparatus.
19. The apparatus of claim 17 wherein the input frame is captured with an optical device; the optical device is selected from a group consisting of: wide angle lens, fish eye lens, and automotive camera lens.
20. The apparatus of claim 17 wherein the output frame is displayed on a display device; the display device selected from a group consisting of: automotive display, LED monitor, television screen, and LCD monitor.
21. The apparatus of claim 17 wherein the output coordinates comprise the four corners of a block in the output frame.
22. The apparatus of claim 17 wherein the input coordinates comprise the four corners of a block in the input frame after perspective transform of the output coordinates and back mapping.
23. The apparatus of claim 17 wherein input parameters to the apparatus comprise output frame width and output frame height, region width and region height; output block width and output block height within a region.
24. The apparatus of claim 17 wherein the region width ranges from 4 to 8094; and the region height ranges from 2 to 8094.
25. The apparatus of claim 17 wherein the output block width ranges from 4 to 254; and the output block height ranges from 2 to 254.
26. The apparatus of claim 17 wherein the bounding module is further configured to add a buffer pixel pad to each of the input blocks increasing the size of the input blocks.
27. The apparatus of claim 17 wherein the size of the internal memory ranges from 10 KB to 50 KB.
28. The apparatus of claim 17 wherein interpolation data is interpolated with a bi-cubic or bilinear interpolation.
29. The apparatus of claim 17 wherein the output frame is processed in a raster scan mode within each block in a region and within each region.
30. The apparatus of claim 17 wherein the interpolated data is further processed through modules configured to filter, reduce noise, scaled and formatted.
Type: Application
Filed: Sep 25, 2017
Publication Date: Mar 28, 2019
Inventors: Rajasekhar Reddy Allu (Plano, TX), Niraj Nandan (Plano, TX), Mihir Narendra Mody (Bangalore), Gang Hua (Katy, TX), Brian Okchon Chae (Johns Creek, GA), Shashank Dabral (Allen, TX), Hetul Sanghvi (Richardson, TX), Vikram VijayanBabu Appia (Dallas, TX), Sujith Shivalingappa (Bangalore)
Application Number: 15/714,862