LIGHT FIELD DISPLAY, ADJUSTED PIXEL RENDERING METHOD THEREFOR, AND VISION CORRECTION SYSTEM AND METHOD USING SAME
A method to automatically adjust user perception of an input image to be rendered on a digital display that has an array of light field shaping elements (LFSE), can include: digitally mapping the input image on a retinal plane of the user, and for each pixel digitally projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping; rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
The present disclosure relates to digital displays, and in particular, to a light field display, and adjusted pixel rendering method and computer-readable medium therefor, and vision correction system and method using same.
BACKGROUNDIndividuals routinely wear corrective lenses to accommodate for reduced vision acuity in consuming images and/or information rendered, for example, on digital displays provided, for example, in day-to-day electronic devices such as smartphones, smart watches, electronic readers, tablets, laptop computers and the like, but also provided as part of vehicular dashboard displays and entertainment systems, to name a few examples. The use of bifocals or progressive corrective lenses is also commonplace for individuals suffering from near and far sightedness.
The operating systems of current electronic devices having graphical displays offer certain “Accessibility” features built into the software of the device to attempt to provide users with reduced vision the ability to read and view content on the electronic device. Specifically, current accessibility options include the ability to invert images, increase the image size, adjust brightness and contrast settings, bold text, view the device display only in grey, and for those with legal blindness, the use of speech technology. These techniques focus on the limited ability of software to manipulate display images through conventional image manipulation, with limited success.
The use of 4D light field displays with lenslet arrays or parallax barriers to correct visual aberrations have since been proposed by Pamplona et al. (PAMPLONA, V., OLIVEIRA, M., ALIAGA, D., AND RASKAR, R. 2012. “Tailored displays to compensate for visual aberrations.” ACM Trans. Graph. (SIGGRAPH) 31.). Unfortunately, conventional light field displays as used by Pamplona et al. are subject to a spatio-angular resolution trade-off; that is, an increased angular resolution decreases the spatial resolution. Hence, the viewer sees a sharp image but at the expense of a significantly lower resolution than that of the screen. To mitigate this effect, Huang et al. (see, HUANG, F.-C., AND BARSKY, B. 2011. A framework for aberration compensated displays. Tech. Rep. UCB/EECS-2011-162, University of California, Berkeley, December; and HUANG, F.-C., LANMAN, D., BARSKY, B. A., AND RASKAR, R. 2012. Correcting for optical aberrations using multi layer displays. ACM Trans. Graph. (SiGGRAPH Asia) 31, 6, 185:1-185:12) proposed the use of multilayer display designs together with prefiltering. The combination of prefiltering and these particular optical setups, however, significantly reduces the contrast of the resulting image.
Finally, in U.S. Patent Application Publication No. 2016/0042501 and Fu-Chung Huang, Gordon Wetzstein, Brian A. Barsky, and Ramesh Raskar. “Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays”. ACM Transaction on Graphics, xx:0, August 2014, the entire contents of each of which are hereby incorporated herein by reference, the combination of viewer-adaptive pre-filtering with off-the-shelf parallax barriers has been proposed to increase contrast and resolution, at the expense however, of computation time and power.
This background information is provided to reveal information believed by the applicant to be of possible relevance. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art.
SUMMARYThe following presents a simplified summary of the general inventive concept(s) described herein to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the disclosure. It is not intended to restrict key or critical elements of embodiments of the disclosure or to delineate their scope beyond that which is explicitly or implicitly described by the following description and claims.
A need exists for a light field display, adjusted pixel rendering method and computer-readable medium therefor, and vision correction system and method using same, that overcome some of the drawbacks of known techniques, or at least, provide a useful alternative thereto. Some aspects of disclosure provide embodiments of such systems, methods, displays and optical element arrays.
In accordance with one aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display has an array of light field shaping elements (LFSE), the method comprising: digitally mapping the input image on a retinal plane of the user; for each given pixel of at least some of said pixels, digitally: projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping; rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the retinal plane is angled relative to the digital display. In one such embodiment, the retinal plane is modeled as a function of an input user pupil or eye location. In one such embodiment, the input user pupil or eye location is dynamically acquired via a digitally implemented pupil or eye tracker. In one such embodiment, the angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
In one embodiment, the angle is dynamically updated based on acquired display inclination data.
In one embodiment, the mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter.
In one embodiment, the designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
In one embodiment, the modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
In one embodiment, the digital display is defined by a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
In accordance with another aspect, there is provided a non-transitory computer-readable medium comprising digital instructions to be implemented by one or more digital processors to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display has an array of light field shaping elements (LFSE), by: digitally mapping the input image on a retinal plane of the user; for each given pixel of at least some of said pixels, digitally: projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping; rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the retinal plane is angled relative to the digital display.
In one embodiment, the retinal plane is modeled as a function of an input user pupil or eye location that is dynamically acquired via a digitally implemented pupil or eye tracker, wherein said angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
In one embodiment, the angle is dynamically updated based on acquired display inclination data.
In one embodiment, the mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter.
In one embodiment, the designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
In one embodiment, the modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
In one embodiment, the digital display is defined by a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust user perception of an input image to be rendered thereon, the device comprising: a digital display medium comprising an array of pixels and operable to render a pixelated image accordingly; an array of light field shaping elements (LFSE) to shape a light field emanating from at least some of said pixels and thereby at least partially govern a projection thereof from said display medium toward the user; and a hardware processor operable on pixel data for the input image to output adjusted image pixel data to adjust user perception of the input image as rendered by: digitally mapping the input image on a retinal plane of the user; for each given pixel of at least some of said pixels, digitally: projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping; rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the retinal plane is angled relative to the digital display.
In one embodiment, the device further comprises a digitally implemented pupil or eye tracker, wherein said retinal plane is modeled as a function of an input user pupil or eye location that is dynamically acquired via said digitally implemented pupil or eye tracker, wherein said angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
In one embodiment, the angle is dynamically updated based on acquired display inclination data.
In one embodiment, the mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter, wherein said designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
In one embodiment, the modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
In one embodiment, the digital display medium defines a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
In accordance with another aspect, there is provided a computer-implemented method, automatically implemented by one or more digital processors, to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display has an array of light field shaping elements (LFSE), the method comprising: computing an adjusted image on an adjusted image plane corresponding to a set of display pixel values, said adjusted image being defined as a set of adjusted image pixel values corresponding to a set of image pixels or portions, and wherein: for a trial set of display pixel values derived from said input image: for each given pixel of at least some of said pixels, digitally: calculating a vector between said given pixel and a user pupil location; approximating a direction of a light field emanated by said given pixel based on a given LFSE intersected by said vector; projecting an adjusted image ray trace between said given pixel and said given LFSE to intersect said adjusted image plane at a given adjusted image location given said direction; calculating a beam box region on said adjusted image plane centered on said adjusted image location characterizing the area illuminated by said pixel; identifying one or more image pixels overlapping with said beam box region, and; for each said overlapping image pixels: adding to said image pixel value a pixel value of said display pixel; calculating a cost function value between said adjusted image and said trial input image, said cost function value quantitatively characterizing the difference between said set of image pixel values and said input image; deriving a new set of display pixel values reducing said cost function value; repeating said computing an adjusted image, said calculating a cost function value and said deriving a new set of display pixel values, each time using said new set of display pixel values, until said difference parameter has been minimized; and rendering for each pixel in said set of pixels the corresponding pixel value from said set of display pixel values corresponding to said minimized difference parameter, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the pixel value is multiplied by a ratio of overlap between said overlapping image pixel and said beam box before being added to said image pixel value.
In one embodiment, the pixel value is multiplied by a relative radiant flux factor.
In one embodiment, the pixel value is multiplied by a pupil transmission factor characterizing how much light from said ray reaches the user's pupil.
In one embodiment, the adjusted image plane is a virtual image plane virtually positioned relative to the digital display at a designated distance from said user pupil location, and wherein said adjusted image ray trace comprises a virtual image vector between said given pixel and said given LFSE to intersect said virtual image plane.
In one embodiment, the adjusted image plane is designated as a user retinal plane, and wherein said adjusted image ray trace is projected to intersect with said user retinal plane by redirecting said adjusted image ray trace given said pupil location in accordance with an input user eye focus aberration parameter.
In one embodiment, the user pupil location is dynamically tracked via a pupil or eye tracker.
In one embodiment, the user perception is adjusted so to at least partially address the user's reduced visual acuity.
In accordance with another aspect, there is provided a non-transitory computer-readable medium comprising digital instructions to be implemented by one or more digital processors to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display has an array of light field shaping elements (LFSE), by: computing an adjusted image on an adjusted image plane corresponding to a set of display pixel values, said adjusted image being defined as a set of adjusted image pixel values corresponding to a set of image pixels or portions, and wherein: for a trial set of display pixel values derived from said input image: for each given pixel of at least some of said pixels, digitally: calculating a vector between said given pixel and a user pupil location; approximating a direction of a light field emanated by said given pixel based on a given LFSE intersected by said vector; projecting an adjusted image ray trace between said given pixel and said given LFSE to intersect said adjusted image plane at a given adjusted image location given said direction; calculating a beam box region on said adjusted image plane centered on said adjusted image location characterizing the area illuminated by said pixel; identifying one or more image pixels overlapping with said beam box region, and; for each said overlapping image pixels: adding to said image pixel value a pixel value of said display pixel; calculating a cost function value between said adjusted image and said trial input image, said cost function value quantitatively characterizing the difference between said set of image pixel values and said input image; deriving a new set of display pixel values reducing said cost function value; repeating said computing an adjusted image, said calculating a cost function value and said deriving a new set of display pixel values, each time using said new set of display pixel values, until said difference parameter has been minimized; and rendering for each pixel in said set of pixels the corresponding pixel value from said set of display pixel values corresponding to said minimized difference parameter, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the pixel value is multiplied by a ratio of overlap between said overlapping image pixel and said beam box before being added to said image pixel value.
In one embodiment, the pixel value is multiplied by a relative radiant flux factor.
In one embodiment, the adjusted image plane is a virtual image plane virtually positioned relative to the digital display at a designated distance from said user pupil location, and wherein said adjusted image ray trace comprises a virtual image vector between said given pixel and said given LFSE to intersect said virtual image plane.
In one embodiment, the adjusted image plane is designated as a user retinal plane, and wherein said adjusted image ray trace is projected to intersect with said user retinal plane by redirecting said adjusted image ray trace given said pupil location in accordance with an input user eye focus aberration parameter.
In one embodiment, the instructions are further executable to confirm that said adjusted image ray trace intersects with an input pupil area associated with said input user pupil location.
In one embodiment, the user pupil location is dynamically tracked via a pupil or eye tracker.
In one embodiment, the user perception is adjusted so to at least partially address the user's reduced visual acuity.
In accordance with another aspect, there is provided a digital display device operable to automatically adjust user perception of an input image to be rendered thereon, the device comprising: a digital display medium comprising an array of pixels and operable to render a pixelated image accordingly; an array of light field shaping elements (LFSE) to shape a light field emanating from at least some of said pixels and thereby at least partially govern a projection thereof from said display medium toward the user; and a hardware processor operable on pixel data for the input image to output adjusted image pixel data to adjust user perception of the input image as rendered by: computing an adjusted image on an adjusted image plane corresponding to a set of display pixel values, said adjusted image being defined as a set of adjusted image pixel values corresponding to a set of image pixels or portions, and wherein: for a trial set of display pixel values derived from said input image: for each given pixel of at least some of said pixels, digitally: calculating a vector between said given pixel and a user pupil location; approximating a direction of a light field emanated by said given pixel based on a given LFSE intersected by said vector; projecting an adjusted image ray trace between said given pixel and said given LFSE to intersect said adjusted image plane at a given adjusted image location given said direction; calculating a beam box region on said adjusted image plane centered on said adjusted image location characterizing the area illuminated by said pixel; identifying one or more image pixels overlapping with said beam box region, and; for each said overlapping image pixels: adding to said image pixel value a pixel value of said display pixel; calculating a cost function value between said adjusted image and said trial input image, said cost function value quantitatively characterizing the difference between said set of image pixel values and said input image; deriving a new set of display pixel values reducing said cost function value; repeating said computing an adjusted image, said calculating a cost function value and said deriving a new set of display pixel values, each time using said new set of display pixel values, until said difference parameter has been minimized; and rendering for each pixel in said set of pixels the corresponding pixel value from said set of display pixel values corresponding to said minimized difference parameter, thereby rendering a perceptively adjusted version of the input image.
In one embodiment, the pixel value is multiplied by a ratio of overlap between said overlapping image pixel and said beam box before being added to said image pixel value.
In one embodiment, the pixel value is multiplied by a relative radiant flux factor.
In one embodiment, the adjusted image plane is a virtual image plane virtually positioned relative to the digital display at a designated distance from said user pupil location, wherein said adjusted image ray trace comprises a virtual image vector between said given pixel and said given LFSE to intersect said virtual image plane, and wherein said designated distance comprises a minimum viewing distance designated such that said perceptively adjusted version of the input image is adjusted to accommodate a user's reduced visual acuity.
In one embodiment, the device is operable to adjust user perception of the input image for viewing by a viewer having reduced visual acuity such that said perceptively adjusted version of the input image at least partially compensates for the viewer's reduced visual acuity, wherein the device further comprises a user interface for dynamically adjusting said minimum viewing distance.
In one embodiment, the adjusted image plane is designated as a user retinal plane, and wherein said adjusted image ray trace is projected to intersect with said user retinal plane by redirecting said adjusted image ray trace given said pupil location in accordance with an input user eye focus aberration parameter.
In one embodiment, the device further comprises a user interface for dynamically adjusting said input user eye focus aberration parameter.
In one embodiment, the device further comprising a pupil or eye tracker or pupil or eye tracking interface operable to dynamically track and automatically accommodate for changes in said user pupil location.
Other aspects, features and/or advantages will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.
Several embodiments of the present disclosure will be provided, by way of examples only, with reference to the appended drawings, wherein:
Elements in the several figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be emphasized relative to other elements for facilitating understanding of the various presently disclosed embodiments. Also, common, but well-understood elements that are useful or necessary in commercially feasible embodiments are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure.
DETAILED DESCRIPTIONVarious implementations and aspects of the specification will be described with reference to details discussed below. The following description and drawings are illustrative of the specification and are not to be construed as limiting the specification. Numerous specific details are described to provide a thorough understanding of various implementations of the present specification. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of implementations of the present specification.
Various apparatuses and processes will be described below to provide examples of implementations of the system disclosed herein. No implementation described below limits any claimed implementation and any claimed implementations may cover processes or apparatuses that differ from those described below. The claimed implementations are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses or processes described below. It is possible that an apparatus or process described below is not an implementation of any claimed subject matter.
Furthermore, numerous specific details are set forth in order to provide a thorough understanding of the implementations described herein. However, it will be understood by those skilled in the relevant arts that the implementations described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the implementations described herein.
In this specification, elements may be described as “configured to” perform one or more functions or “configured for” such functions. In general, an element that is configured to perform or configured for performing a function is enabled to perform the function, or is suitable for performing the function, or is adapted to perform the function, or is operable to perform the function, or is otherwise capable of performing the function.
It is understood that for the purpose of this specification, language of “at least one of X, Y, and Z” and “one or more of X, Y and Z” may be construed as X only, Y only, Z only, or any combination of two or more items X, Y, and Z (e.g., XYZ, XY, YZ, ZZ, and the like). Similar logic may be applied for two or more items in any occurrence of “at least one . . . ” and “one or more . . . ” language.
The systems and methods described herein provide, in accordance with different embodiments, different examples of a light field display, adjusted pixel rendering method and computer-readable medium therefor, and vision correction system and method using same. For instance, the devices, displays and methods described herein may allow a user's perception of an input image to be displayed, to be adjusted or altered using the light field display. For instance, in some examples, users who would otherwise require corrective eyewear such as glasses or contact lenses, or again bifocals, may consume images produced by such devices, displays and methods in clear or improved focus without the use of such eyewear. Other light field display applications, such as 3D displays and the like, may also benefit from the solutions described herein, and thus, should be considered to fall within the general scope and nature of the present disclosure.
For example, some of the herein described embodiments provide for digital display devices, or devices encompassing such displays, for use by users having reduced visual acuity, whereby images ultimately rendered by such devices can be dynamically processed to accommodate the user's reduced visual acuity so that they may consume rendered images without the use of corrective eyewear, as would otherwise be required. As noted above, embodiments are not to be limited as such as the notions and solutions described herein may also be applied to other technologies in which a user's perception of an input image to be displayed can be altered or adjusted via the light field display.
Generally, digital displays as considered herein will comprise a set of image rendering pixels and a corresponding set of light field shaping elements that at least partially govern a light field emanated thereby to produce a perceptively adjusted version of the input image. In some examples, light field shaping elements may take the form of a light field shaping layer or like array of optical elements to be disposed relative to the display pixels in at least partially governing the emanated light field. As described in further detail below, such light field shaping layer elements may take the form of a microlens and/or pinhole array, or other like arrays of optical elements, or again take the form of an underlying light shaping layer, such as an underlying array of optical gratings or like optical elements operable to produce a directional pixelated output.
Within the context of a light field shaping layer, as described in further detail below in accordance with some embodiments, the light field shaping layer can be disposed at a pre-set distance from the pixelated display so to controllably shape or influence a light field emanating therefrom. For instance, each light field shaping layer can be defined by an array of optical elements centered over a corresponding subset of the display's pixel array to optically influence a light field emanating therefrom and thereby govern a projection thereof from the display medium toward the user, for instance, providing some control over how each pixel or pixel group will be viewed by the viewer's eye(s). As will be further detailed below, arrayed optical elements may include, but are not limited to, lenslets, microlenses or other such diffractive optical elements that together form, for example, a lenslet array; pinholes or like apertures or windows that together form, for example, a parallax or like barrier; concentrically patterned barriers, e.g. cut outs and/or windows, such as a to define a Fresnel zone plate or optical sieve, for example, and that together form a diffractive optical barrier (as described, for example, in Applicant's co-pending U.S. application Ser. No. 15/910,908, the entire contents of which are hereby incorporated herein by reference); and/or a combination thereof, such as for example, a lenslet array whose respective lenses or lenslets are partially shadowed or barriered around a periphery thereof so to combine the refractive properties of the lenslet with some of the advantages provided by a pinhole barrier.
In operation, the display device will also generally invoke a hardware processor operable on image pixel (or subpixel) data for an image to be displayed to output corrected or adjusted image pixel data to be rendered as a function of a stored characteristic of the light field shaping elements and/or layer (e.g. layer distance from display screen, distance between optical elements (pitch), absolute relative location of each pixel or subpixel to a corresponding optical element, properties of the optical elements (size, diffractive and/or refractive properties, etc.), or other such properties, and a selected vision correction or adjustment parameter related to the user's reduced visual acuity or intended viewing experience. While light field display characteristics will generally remain static for a given implementation (i.e. a given shaping element and/or layer will be used and set for each device irrespective of the user), image processing can, in some embodiments, be dynamically adjusted as a function of the user's visual acuity or intended application so to actively adjust a distance of a virtual image plane, or perceived image on the user's retinal plane given a quantified user eye focus or like optical aberration(s), induced upon rendering the corrected/adjusted image pixel data via the static optical layer and/or elements, for example, or otherwise actively adjust image processing parameters as may be considered, for example, when implementing a viewer-adaptive pre-filtering algorithm or like approach (e.g. compressive light field optimization), so to at least in part govern an image perceived by the user's eye(s) given pixel or subpixel-specific light visible thereby through the layer.
Accordingly, a given device may be adapted to compensate for different visual acuity levels and thus accommodate different users and/or uses. For instance, a particular device may be configured to implement and/or render an interactive graphical user interface (GUI) that incorporates a dynamic vision correction scaling function that dynamically adjusts one or more designated vision correction parameter(s) in real-time in response to a designated user interaction therewith via the GUI. For example, a dynamic vision correction scaling function may comprise a graphically rendered scaling function controlled by a (continuous or discrete) user slide motion or like operation, whereby the GUI can be configured to capture and translate a user's given slide motion operation to a corresponding adjustment to the designated vision correction parameter(s) scalable with a degree of the user's given slide motion operation. These and other examples are described in Applicant's co-pending U.S. patent application Ser. No. 15/246,255, the entire contents of which are hereby incorporated herein by reference.
With reference to
In the illustrated embodiment, the device 100 comprises a processing unit 110, a digital display 120, and internal memory 130. Display 120 can be an LCD screen, a monitor, a plasma display panel, an LED or OLED screen, or any other type of digital display defined by a set of pixels for rendering a pixelated image or other like media or information. Internal memory 130 can be any form of electronic storage, including a disk drive, optical drive, read-only memory, random-access memory, or flash memory, to name a few examples. For illustrative purposes, memory 130 has stored in it vision correction application 140, though various methods and techniques may be implemented to provide computer-readable code and instructions for execution by the processing unit in order to process pixel data for an image to be rendered in producing corrected pixel data amenable to producing a corrected image accommodating the user's reduced visual acuity (e.g. stored and executable image correction application, tool, utility or engine, etc.). Other components of the electronic device 100 may optionally include, but are not limited to, one or more rear and/or front-facing camera(s) 150, an accelerometer 160 and/or other device positioning/orientation devices capable of determining the tilt and/or orientation of electronic device 100, and the like.
For example, the electronic device 100, or related environment (e.g. within the context of a desktop workstation, vehicular console/dashboard, gaming or e-learning station, multimedia display room, etc.) may include further hardware, firmware and/or software components and/or modules to deliver complementary and/or cooperative features, functions and/or services. For example, in some embodiment, and as will be described in greater detail below, a pupil/eye tracking system may be integrally or cooperatively implemented to improve or enhance corrective image rending by tracking a location of the user's eye(s)/pupil(s) (e.g. both or one, e.g. dominant, eye(s)) and adjusting light field corrections accordingly. For instance, the device 100 may include, integrated therein or interfacing therewith, one or more eye/pupil tracking light sources, such as one or more infrared (IR) or near-IR (NIR) light source(s) to accommodate operation in limited ambient light conditions, leverage retinal retro-reflections, invoke corneal reflection, and/or other such considerations. For instance, different IR/NIR pupil tracking techniques may employ one or more (e.g. arrayed) directed or broad illumination light sources to stimulate retinal retro-reflection and/or corneal reflection in identifying a tracking a pupil location. Other techniques may employ ambient or IR/NIR light-based machine vision and facial recognition techniques to otherwise locate and track the user's eye(s)/pupil(s). To do so, one or more corresponding (e.g. visible, IR/NIR) cameras may be deployed to capture eye/pupil tracking signals that can be processed, using various image/sensor data processing techniques, to map a 3D location of the user's eye(s)/pupil(s). In the context of a mobile device, such as a mobile phone, such eye/pupil tracking hardware/software may be integral to the device, for instance, operating in concert with integrated components such as one or more front facing camera(s), onboard IR/NIR light source(s) and the like. In other user environments, such as in a vehicular environment, eye/pupil tracking hardware may be further distributed within the environment, such as dash, console, ceiling, windshield, mirror or similarly-mounted camera(s), light sources, etc.
With reference to
For the sake of illustration, the following embodiments will be described within the context of a light field shaping layer defined, at least in part, by a lenslet array comprising an array of microlenses (also interchangeably referred to herein as lenslets) that are each disposed at a distance from a corresponding subset of image rendering pixels in an underlying digital display. It will be appreciated that while a light field shaping layer may be manufactured and disposed as a digital screen overlay, other integrated concepts may also be considered, for example, where light field shaping elements are integrally formed or manufactured within a digital screen's integral components such as a textured or masked glass plate, beam-shaping light sources (e.g. directional light sources and/or backlit integrated optical grating array) or like component.
Accordingly, each lenslet will predictively shape light emanating from these pixel subsets to at least partially govern light rays being projected toward the user by the display device. As noted above, other light field shaping layers may also be considered herein without departing from the general scope and nature of the present disclosure, whereby light field shaping will be understood by the person of ordinary skill in the art to reference measures by which light, that would otherwise emanate indiscriminately (i.e. isotropically) from each pixel group, is deliberately controlled to define predictable light rays that can be traced between the user and the device's pixels through the shaping layer.
For greater clarity, a light field is generally defined as a vector function that describes the amount of light flowing in every direction through every point in space. In other words, anything that produces or reflects light has an associated light field. The embodiments described herein produce light fields from an object that are not “natural” vector functions one would expect to observe from that object. This gives it the ability to emulate the “natural” light fields of objects that do not physically exist, such as a virtual display located far behind the light field display, which will be referred to now as the ‘virtual image’. As noted in the examples below, in some embodiments, lightfield rendering may be adjusted to effectively generate a virtual image on a virtual image plane that is set at a designated distance from an input user pupil location, for example, so to effective push back, or move forward, a perceived image relative to the display device in accommodating a user's reduced visual acuity (e.g. minimum or maximum viewing distance). In yet other embodiments, lightfield rendering may rather or alternatively seek to map the input image on a retinal plane of the user, taking into account visual aberrations, so to adaptively adjust rendering of the input image on the display device to produce the mapped effect. Namely, where the unadjusted input image would otherwise typically come into focus in front of or behind the retinal plane (and/or be subject to other optical aberrations), this approach allows to map the intended image on the retinal plane and work therefrom to address designated optical aberrations accordingly. Using this approach, the device may further computationally interpret and compute virtual image distances tending toward infinity, for example, for extreme cases of presbyopia. This approach may also more readily allow, as will be appreciated by the below description, for adaptability to other visual aberrations that may not be as readily modeled using a virtual image and image plane implementation. In both of these examples, and like embodiments, the input image is digitally mapped to an adjusted image plane (e.g. virtual image plane or retinal plane) designated to provide the user with a designated image perception adjustment that at least partially addresses designated visual aberrations. Naturally, while visual aberrations may be addressed using these approaches, other visual effects may also be implemented using similar techniques.
In one example, to apply this technology to vision correction, consider first the normal ability of the lens in an eye, as schematically illustrated in
As will be appreciated by the skilled artisan, a light field as seen in
Following with the example of a microlens array,
Accordingly, upon predictably aligning a particular microlens array with a pixel array, a designated “circle” of pixels will correspond with each microlens and be responsible for delivering light to the pupil through that lens.
As will be detailed further below, the separation between the microlens array and the pixel array as well as the pitch of the lenses can be selected as a function of various operating characteristics, such as the normal or average operating distance of the display, and/or normal or average operating ambient light levels.
Further, as producing a light field with angular resolution sufficient for accommodation correction over the full viewing ‘zone’ of a display would generally require an astronomically high pixel density, instead, a correct light field can be produced, in some embodiments, only at or around the location of the user's pupils. To do so, the light field display can be paired with pupil tracking technology to track a location of the user's eyes/pupils relative to the display. The display can then compensate for the user's eye location and produce the correct virtual image, for example, in real time.
In some embodiments, the light field display can render dynamic images at over 30 frames per second on the hardware in a smartphone.
In some embodiments, the light field display can display a virtual image at optical infinity, meaning that any level of accommodation-based presbyopia (e.g. first order) can be corrected for.
In some further embodiments, the light field display can both push the image back or forward, thus allowing for selective image corrections for both hyperopia (far-sightedness) and myopia (nearsightedness).
In order to demonstrate a working light field solution, and in accordance with one embodiment, the following test was set up. A camera was equipped with a simple lens, to simulate the lens in a human eye and the aperture was set to simulate a normal pupil diameter. The lens was focused to 50 cm away and a phone was mounted 25 cm away. This would approximate a user whose minimal seeing distance is 50 cm and is attempting to use a phone at 25 cm.
With reading glasses, +2.0 diopters would be necessary for the vision correction. A scaled Snellen chart was displayed on the cellphone and a picture was taken, as shown in
Accordingly, a display device as described above and further exemplified below, can be configured to render a corrected image via the light field shaping layer that accommodates for the user's visual acuity. By adjusting the image correction in accordance with the user's actual predefined, set or selected visual acuity level, different users and visual acuity may be accommodated using a same device configuration. That is, in one example, by adjusting corrective image pixel data to dynamically adjust a virtual image distance below/above the display as rendered via the light field shaping layer, different visual acuity levels may be accommodated.
As will be appreciated by the skilled artisan, different image processing techniques may be considered, such as those introduced above and taught by Pamplona and/or Huang, for example, which may also influence other light field parameters to achieve appropriate image correction, virtual image resolution, brightness and the like.
With reference to
In some embodiments, as illustrated in
In yet some further or alternative embodiments, a pitch ratio between the microlens array and pixel array may be deliberately selected to further or alternatively alleviate periodic optical artifacts. For example, a perfectly matched pitch ratio (i.e. an exact integer number of display pixels per microlens) is most likely to induce periodic optical artifacts, whereas a pitch ratio mismatch can help reduce such occurrences. Accordingly, in some embodiments, the pitch ratio will be selected to define an irrational number, or at least, an irregular ratio, so to minimize periodic optical artifacts. For instance, a structural periodicity can be defined so to reduce the number of periodic occurrences within the dimensions of the display screen at hand, e.g. ideally selected so to define a structural period that is greater than the size of the display screen being used.
While this example is provided within the context of a microlens array, similar structural design considerations may be applied within the context of a parallax barrier, diffractive barrier or combination thereof.
With reference to
As illustrated in
The pupil location 1308, in one embodiment, is the three-dimensional coordinates of at least one the user's pupils' center with respect to a given reference frame, for example a point on the device or display. This pupil location 1308 may be derived from any eye/pupil tracking method known in the art. In some embodiments, the pupil location 1308 may be determined prior to any new iteration of the rendering algorithm, or in other cases, at a lower framerate. In some embodiments, only the pupil location of a single user's eye may be determined, for example the user's dominant eye (i.e. the one that is primarily relied upon by the user). In some embodiments, this position, and particularly the pupil distance to the screen may otherwise or additionally be rather approximated or adjusted based on other contextual or environmental parameters, such as an average or preset user distance to the screen (e.g. typical reading distance for a given user or group of users; stored, set or adjustable driver distance in a vehicular environment; etc.).
In the illustrated embodiment, the minimum reading distance 1310 is defined as the minimal focus distance for reading that the user's eye(s) may be able to accommodate (i.e. able to view without discomfort). In some embodiments, different values of the minimum reading distance 1310 associated with different users may be entered, for example, as can other adaptive vision correction parameters be considered depending on the application at hand and vision correction being addressed. In some embodiments, minimum reading distance 1310 may be derived from an eye prescription (e.g. glasses prescription or contact prescription) or similar. It may, for example, correspond to the near point distance corresponding to the uncorrected user's eye, which can be calculated from the prescribed corrective lens power assuming that the targeted near point was at 25 cm.
With added reference to
An exemplary ray-tracing methodology is described in steps 1110 to 1128 of
As illustrated in
The method then finds, in step 1114, the coordinates of the center 1416 of the LFSL optical element closest to intersection point 1411. This step may be computationally intensive and will be discussed in more depth below. Once the position of the center 1416 of the optical element is known, in step 1116, a normalized unit ray vector is generated from drawing and normalizing a vector 1423 drawn from center position 1416 to pixel 1409. This unit ray vector generally approximates the direction of the light field emanating from pixel 1409 through this particular light field element, for instance, when considering a parallax barrier aperture or lenslet array (i.e. where the path of light travelling through the center of a given lenslet is not deviated by this lenslet). Further computation may be required when addressing more complex light shaping elements, as will be appreciated by the skilled artisan. The direction of this ray vector will be used to find the portion of image 1306, and thus the associated color, represented by pixel 1409. But first, in step 1118, this ray vector is projected backwards to the plane of pupil 1415, and then in step 1120, the method verifies that the projected ray vector 1425 is still within pupil 1415 (i.e. that the user can still “see” it). Once the intersection position, for example location 1431 in
If this deviation is deemed to be too large (i.e. light emanating from pixel 1409 channeled through optical element 1416 is not perceived by pupil 1415), then in step 1122, the method flags pixel 1409 as unnecessary and to simply be turned off or render a black color. Otherwise, as shown in
In some embodiments, method 1100 is modified so that at step 1120, instead of having a binary choice between the ray vector hitting the pupil or not, one or more smooth interpolation function (i.e. linear interpolation, Hermite interpolation or similar) are used to quantify how far or how close the intersection point 1431 is to the pupil center 1417 by outputting a corresponding continuous value between 1 or 0. For example, the assigned value is equal to 1 substantially close to pupil center 1417 and gradually change to 0 as the intersection point 1431 substantially approaches the pupil edges or beyond. In this case, the branch containing step 1122 is ignored and step 1220 continues to step 1124. At step 1126, the pixel color value assigned to pixel 1409 is chosen to be somewhere between the full color value of the portion of image 1306 at intersection point 1423 or black, depending on the value of the interpolation function used at step 1120 (1 or 0).
In yet other embodiments, pixels found to illuminate a designated area around the pupil may still be rendered, for example, to produce a buffer zone to accommodate small movements in pupil location, for example, or again, to address potential inaccuracies, misalignments or to create a better user experience.
In some embodiments, steps 1118, 1120 and 1122 may be avoided completely, the method instead going directly from step 1116 to step 1124. In such an exemplary embodiment, no check is made that the ray vector hits the pupil or not, but instead the method assumes that it always does.
Once the output colors of all pixels have been determined, these are finally rendered in step 1130 by pixel display 1401 to be viewed by the user, therefore presenting a light field corrected image. In the case of a single static image, the method may stop here. However, new input variables may be entered and the image may be refreshed at any desired frequency, for example because the user's pupil moves as a function of time and/or because instead of a single image a series of images are displayed at a given framerate.
With reference to
Once parameters 1102 and variables 1104 have been set, this second exemplary ray-tracing methodology proceeds from steps 1910 to 1936, at the end of which the output color of each pixel of the pixel display is known so as to virtually reproduce the light field emanating from an image perceived to be positioned at the correct or adjusted image distance, in one example, so to allow the user to properly focus on this adjusted image (i.e. having a focused image projected on the user's retina) despite a quantified visual aberration. In
Referencing once more
From there, in step 1914, the coordinates of the optical element center 1416 closest to intersection point 1411 are determined. This step may be computationally intensive and will be discussed in more depth below. As shown in
Now referring to
The skilled artisan will note that any light ray originating from optical element center 1416, no matter its orientation, will also be focused onto focus point 2008, to a first approximation. Therefore, the location on retina plane (2012) onto which light entering the pupil at intersection point 1431 will converge may be approximated by drawing a straight line between intersection point 1431 where ray vector 1425 hits the pupil 1415 and focus point 2008 on focal plane 2006. The intersection of this line with retina plane 2010 (retina image point 2012) is thus the location on the user's retina corresponding to the image portion that will be reproduced by corresponding pixel 1409 as perceived by the user. Therefore, by comparing the relative position of retina point 2012 with the overall position of the projected image on the retina plane 2010, the relevant adjusted image portion associated with pixel 1409 may be computed.
To do so, at step 1927, the corresponding projected image center position on retina plane 2010 is calculated. Vector 2016 is generated originating from the center position of display 1401 (display center position 2018) and passing through pupil center 1417. Vector 2016 is projected beyond the pupil plane onto retina plane 2010, wherein the associated intersection point gives the location of the corresponding retina image center 2020 on retina plane 2010. The skilled technician will understand that step 1927 could be performed at any moment prior to step 1929, once the relative pupil center location 1417 is known in input variables step 1904. Once image center 2020 is known, one can then find the corresponding image portion of the selected pixel/subpixel at step 1929 by calculating the x/y coordinates of retina image point 2012 relative to retina image center 2020 on the retina, scaled to the x/y retina image size 2031.
This retina image size 2031 may be computed by calculating the magnification of an individual pixel on retina plane 2010, for example, which may be approximately equal to the x or y dimension of an individual pixel multiplied by the eye depth 1314 and divided by the absolute value of the distance to the eye (i.e. the magnification of pixel image size from the eye lens). Similarly, for comparison purposes, the input image is also scaled by the image x/y dimensions to produce a corresponding scaled input image 2064. Both the scaled input image and scaled retina image should have a width and height between −0.5 to 0.5 units, enabling a direct comparison between a point on the scaled retina image 2010 and the corresponding scaled input image 2064, as shown in
From there, the image portion position 2041 relative to retina image center position 2043 in the scaled coordinates (scaled input image 2064) corresponds to the inverse (because the image on the retina is inverted) scaled coordinates of retina image point 2012 with respect to retina image center 2020. The associated color with image portion position 2041 is therefrom extracted and associated with pixel 1409.
In some embodiments, method 1900 may be modified so that at step 1920, instead of having a binary choice between the ray vector hitting the pupil or not, one or more smooth interpolation function (i.e. linear interpolation, Hermite interpolation or similar) are used to quantify how far or how close the intersection point 1431 is to the pupil center 1417 by outputting a corresponding continuous value between 1 or 0. For example, the assigned value is equal to 1 substantially close to pupil center 1417 and gradually change to 0 as the intersection point 1431 substantially approaches the pupil edges or beyond. In this case, the branch containing step 1122 is ignored and step 1920 continues to step 1124. At step 1931, the pixel color value assigned to pixel 1409 is chosen to be somewhere between the full color value of the portion of image 1306 at intersection point 1423 or black, depending on the value of the interpolation function used at step 1920 (1 or 0).
In yet other embodiments, pixels found to illuminate a designated area around the pupil may still be rendered, for example, to produce a buffer zone to accommodate small movements in pupil location, for example, or again, to address potential inaccuracies or misalignments.
Once the output colors of all pixels in the display have been determined (check at step 1934 is true), these are finally rendered in step 1936 by pixel display 1401 to be viewed by the user, therefore presenting a light field corrected image. In the case of a single static image, the method may stop here. However, new input variables may be entered and the image may be refreshed at any desired frequency, for example because the user's pupil moves as a function of time and/or because instead of a single image a series of images are displayed at a given framerate.
As will be appreciated by the skilled artisan, selection of the adjusted image plane onto which to map the input image in order to adjust a user perception of this input image allows for different ray tracing approaches to solving a similar challenge, that is of creating an adjusted image using the light field display that can provide an adjusted user perception, such as addressing a user's reduce visual acuity. While mapping the input image to a virtual image plane set at a designated minimum (or maximum) comfortable viewing distance can provide one solution, the alternate solution may allow accommodation of different or possibly more extreme visual aberrations. For example, where a virtual image is ideally pushed to infinity (or effectively so), computation of an infinite distance becomes problematic. However, by designating the adjusted image plane as the retinal plane, the illustrative process of
While the computations involved in the above described ray-tracing algorithms (steps 1110 to 1128 of
With reference to
With reference to
For hexagonal geometries, as illustrated in
To solve this problem, the array of hexagonal tiles 1601 may be superimposed on or by a second array of staggered rectangular tiles 1705, in such a way as to make an “inverted house” diagram within each rectangle, as clearly illustrated in
Furthermore, while this particular example encompasses the definition of linearly defined tile region boundaries, other boundary types may also be considered provided they are amenable to the definition of one or more conditional statements, as illustrated below, that can be used to output a corresponding set of binary or Boolean values that distinctly identify a location of a given point within one or another of these regions, for instance, without invoking, or by limiting, processing demands common to branching or looping decision logics/trees/statements/etc.
Following with hexagonal example, to locate the associated hexagon tile center 1615 closest to the intersection point 1411, in step 1517, the method first computes the 2D position of the bottom left corner 1705 of the associated (normalized) rectangular tile element 1609 containing intersection point 1411, as shown in
{right arrow over (t)}=(floor(uvy),0)
{right arrow over (C)}corner=({right arrow over (uv)}+{right arrow over (t)})−{right arrow over (t)}
where {right arrow over (uv)} is the position vector of intersection point 1411 in the common frame of reference of the hexagonal and staggered rectangular tile arrays, and the floor( ) function returns the greatest integer less than or equal to each of the xy coordinates of {right arrow over (uv)}.
Once the position of lower left corner {right arrow over (C)}corner 1705 of the associated rectangular element 1814 containing the intersection point 1411 is known, three regions 1804, 1806 and 1807 within this rectangular element 1814 may be distinguished, as shown in
Continuing with the illustrated example, In step 1519, the coordinates within associated rectangular tile 1814 are again rescaled, as shown on the axis of
dx=2*(uvx−Ccorner
dy=3*(uvy−Ccorner
Thus, the possible x and y values of the position of intersection point 1411 within associated rectangular tile 1609 are now contained within −1<x<1 and 0<y<3. This will make the next step easier to compute.
To efficiently find the region encompassing a given intersection point in these rescaled coordinates, the fact that, within the rectangular element 1814, each region is separated by a diagonal line is used. For example, this is illustrated in
To finally obtain the relative coordinates of the hexagonal center associated with the identified region, in step 1523, the set of converted Boolean values may be used as an input to a single floating point vectorial function operable to map each set of these values to a set of xy coordinates of the associated element center. For example, in the described embodiment and as shown in
{right arrow over (r)}=(rx,ry)=(0.5+0.5*(caseR−case L),⅔−(caseR−caseL))
thus, the inputs of (1.0, 0.0), (0.0, 1.0) or (0.0, 0.0) map to the positions (0.0, −⅓), (0.5, ⅔), and (1.0, −⅓), respectively, which corresponds to the shown hexagonal centers 1863, 1865 and 1867 shown in
Now back to
The skilled artisan will note that modifications to the above-described method may also be used. For example, the staggered grid shown in
In yet other embodiments, wherein a rectangular and/or square microlens array is used instead of a nestled (hexagonal) array, a slightly different method may be used to identify the associated LFSL element (microlens) center (step 1114). Herein, the microlens array is represented by an array of rectangular and/or square tiles. The method, as previously described, goes through step 1515, where the x and y coordinates are rescaled (normalized) with respect to a microlens x and y dimension (henceforth giving each rectangular and/or square tile a width and height of 1 unit). However, at step 1517, the floor( ) function is used directly on each x and y coordinates of {right arrow over (uv)} (the position vector of intersection point 1411) to find the coordinates of the bottom left corner associated with the corresponding square/rectangular tile. Therefrom, the relative coordinates of the tile center from the bottom left corner are added directly to obtain the final scaled position vector:
{right arrow over (r)}=(rx,ry)=(floor(uvx)+0.5, floor(uvy)+0.5)
Once this vector is known, the method goes directly to step 1525 where the coordinates are scaled back into absolute units (i.e. mm) and rotated back to the original frame of reference with respect to the display to obtain the 3D positions (in mm) of the optical layer element's center with respect to the display's frame of reference, which is then fed into step 1116.
The light field rendering methods described above (from
In
To further illustrate embodiments making use of subpixel rendering, with reference to
In the example shown in
In the example shown in
In order to implement subpixel rendering in the context of light field image correction, in some embodiments, ray tracing calculations must be executed in respect of each subpixel, as opposed to in respect of each pixel as a whole, based on a location (x,y coordinates on the screen) of each subpixel. Beyond providing for greater rendering accuracy and sharpness, subpixel control and ray tracing computations may accommodate different subpixel configurations, for example, where subpixel mixing or overlap is invoked to increase a perceived resolution of a high resolution screen and/or where non-uniform subpixel arrangements are provided or relied upon in different digital display technologies.
In some embodiments, however, in order to avoid or reduce a computation load increase imparted by the distinct consideration of each subpixel, some computation efficiencies may be leveraged by taking into account the regular subpixel distribution from pixel to pixel, or in the context of subpixel sharing and/or overlap, for certain pixel groups, lines, columns, etc. With reference to
While this example contemplates a linear (horizontal) subpixel distribution, other 2D distributions may also be considered without departing from the general scope and nature of the present disclosure. For example, for a given digital display screen and pixel and subpixel distribution, different subpixel mappings can be determined to define respective pixel subcoordinate systems that, when applied to standard pixel-centric ray tracing and image correction algorithms, can allow for subpixel processing and increase image correction resolution and sharpness without undue processing load increases.
In some embodiments, additional efficiencies may be leveraged on the GPU by storing the image data, for example image 1306, in the GPU's texture memory. Texture memory is cached on chip and in some situations is operable to provide higher effective bandwidth by reducing memory requests to off-chip DRAM. Specifically, texture caches are designed for graphics applications where memory access patterns exhibit a great deal of spatial locality, which is the case of the steps 1110-1126 of method 1100. For example, in method 1100, image 1306 may be stored inside the texture memory of the GPU, which then greatly improves the retrieval speed during step 1126 where the color channel associated with the portion of image 1306 at intersection point 1423 is determined.
With reference to
In some embodiments, and as illustrated in
The general orientation of pupil plane 1415 may be parametrize, for example, by using the 3D location of pupil center 1417 and a corresponding normal vector. Such a normal vector may be taken to be, in some embodiments, equal to the gaze direction as measured by a gaze tracking system or similar, as will be discussed below.
Once the relative position and orientation of pupil plane 1415 is determined, the relative position/orientation of all remaining planes (parallel or non-parallel) may be determined and parametrized accordingly. Planes that are parallel share the same normal vector. From there, the methods of
In the illustrated example of
To extract normal vector 2470 of pupil plane 1415, the eye tracking methods and systems described above may be used or modified to further provide a measure of the eye's gaze direction (e.g. gaze tracking). As discussed above, there are many known eye tracking methods in the art, some of which may also be used for gaze-tracking. For example, this includes Near-IR glint reflection methods and systems or methods purely based on machine vision methods. Hence, in some embodiments, pupil plane 1415 may be re-parametrize using an updated 3D location of pupil center 1417 and an updated normal vector 2470 at each eye tracking cycle. In other embodiments, a hybrid gaze tracking/pupil tracking system or method may be used wherein gaze direction (e.g. normal vector 2470) is provided at a different interval than pupil center location 1417. For example, in some embodiments, for one or more cycles, only the 3D pupil center location 1417 may be measured and an old gaze direction vector may be re-used or manually updated. In some embodiments, an eye model or similar may be constructed to map a change in measured pupil center location 1417 to a change in the gaze direction vector without relying on the full capabilities of the gaze tracking system or method. Such a map may be based on one or more previous gaze tracking measurements. In any case, by measuring/determining the 3D pupil center location 1417 and normal vector 2470, the pupil plane may be parametrized accordingly.
Note that in
In some embodiments, one or more planes discussed above may be replaced with non-planar surfaces. For example, a curved light-field display may be used. In one non-limiting example, a curved pixel display 2501 and optical layer 2503 may be considered, as is shown in
With reference to
As an example, let x be the set of all pixel/subpixel values (e.g. sampled color/channel), and y be the set of corresponding values of image portions of image 1306. A transport or transfer function A may be constructed to simulate how the light emitted from an input set of values x generates the set of image values y (e.g. how a given set of x maps to a given set of y). This may be represented as the following system:
A·x=y.
In some embodiments, the currently described compressive lightfield rendering method is operable to find the best set of values x that generate the set of values y that are the closest to the digitized input image 1306 (e.g. yimage). The procedure to do so, in accordance with one embodiment, is illustrated as process 2600 in
At step 2605, an initial or guess input set of values x is used to start the rendering iteration. In general, this initial set of values may be anything but in practice, the closer the initial set of values x is to the solution, the faster finding the solution will be. Thus, in some embodiments, the input image 1306 may be used (e.g. x=yimage), or in other embodiments a single iteration of the ray tracing method 1100 may be used instead.
At step 2615, the current set of pixel/subpixel values x is used to generate the corresponding pixelated image values y. As mentioned above, this is done by modelling how a light beam from each pixel/subpixel is projected on the perceived image. FIGS. 27 and 31 illustrate how the beam mapping is calculated, in some embodiments, by using a modified version of the ray tracing methods 1100 or 1900, respectively.
As mentioned above, the beam of light emitted by a single pixel/subpixel may overlap multiple image portions. To determine how a given beam overlaps the image, a modified ray tracing method may be used, wherein the width of the beam is considered. To quantify which image portion the beam overlaps, it is useful to consider a digitized or pixelated image, either on the virtual plane or on the retina plane. For example, by defining an image resolution (which may or may not be the same as the pixel display resolution), the image size may be divided by this image resolution to define image pixels, each having a precise boundary on the relevant plane. By determining which of these (image pixels) overlaps with the beam, the value of the relevant pixel/subpixel may be added to this (image pixel). By adding the contribution of all pixel/subpixels to all (image pixel), the full set of values y may be determined. Furthermore, as will be explained below, these image pixels may either be on virtual image plane 1405—wherein they may be referred to as virtual pixels—or as projected on retina plane 2010—wherein they may be referred to as retina pixels).
Two methods of tracing the beam will now be explained, one based on a modified version of ray tracing method 1100 described in
With reference to
In some embodiments, as illustrated in
Note that while the size of each virtual pixel depends on the position of the virtual image plane (which dictates the scaling necessary so that the image fills out the display as perceived by the user), the size of the beam box is also a function of constant parameters 1102. Since the size of the beambox is fixed for a given light field rendering iteration, step 2701 may be done once, for example before step 2615 in
Once the position of the intersection point 1423 on image 1306 is found at step 1124, at step 2705, the contribution of the pixel/subpixel corresponding to the beambox is added to each virtual pixel at least partly inside the beambox. An exemplary implementation of step 2705 is shown in
In some embodiments, the set of values y may instead represent the actual image being generated on the user's retina (e.g. on retina plane 2010 instead of on virtual plane 1401). Thus, in some embodiments, step 2615 may instead be based on a modified version of the ray tracing method 1900 of
Furthermore, the skilled artisan will understand that step 2615 as described in either
Going back to
CF=Σ(A(xi)−yiimage)2
wherein xi is the (value) of pixel/subpixel i and yiimage is the corresponding virtual/retina pixel/subpixel value of input image 1306. However, the skilled technician will understand that different types of error functions may be used instead. For example, these may include, without restriction, absolute or mean absolute error functions, a Huber loss function, a log-cosh function, etc. Note that, in some embodiments, the chosen optimization algorithm may also require the gradient or the Hessian of the cost function. This may be computed at this step as well. The derivatives and/or Hessian values may be given in an analytical form, or be computed numerically.
As mentioned above, the optimized values of each pixel/subpixel of pixel display 1401 may be determined by minimizing the cost function described above using a numerical algorithm for solving unconstrained nonlinear optimization problems. Some examples of such algorithms include, without limitation, gradient/steepest descent, conjugate gradient, or iterative Quasi-Newton methods such as the limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS or LM-BFGS) algorithm or similar. These algorithms are generally iterative, wherein the value of x is changed incrementally in a direction that reduces the value of the error function. Thus, at step 2635, a minimization step is taken to obtain a new set of values x (e.g. new values for all pixels/subpixels).
At step 2645, the convergence of the minimization procedure is evaluated. In the case where it has indeed converged (e.g. the cost function has been minimized), the last set of values x is taken to be the finalized pixel display configuration and the light field image is rendered on pixel display 1401 at step 2655. If the minimization algorithm hasn't converged at this point, steps 2615 to 2635 are repeated until it does, or until a maximum number of iteration steps has been reached. The skilled technician will understand that different convergence criteria may be used. Similarly, the maximum number of minimization steps may be changed, depending on the constraints upon the rendering speed for example.
Unrestricted, the minimization procedure discussed above may produce values that are outside of the displayable range (e.g. outside of the range [0,1]). Thus, in some embodiments, the cost function may be modified so as to improve the probability that the minimization procedure results in values within the displayable range. For example, in some embodiments, a factor of (x−½)2 or abs|x−½| may be added to the cost function, for the range of values outside of [0, 1]. In some embodiments, a constant value of 0.25 may also be added for values inside this range.
Furthermore, method 2600 in general may, in some embodiments, be implemented to work on massively parallel processing devices, for example on a GPU or similar, as was discussed above.
The methods and systems described above were mainly discussed in the context of correcting vision problems such as nearsightedness, farsightedness and astigmatism. However, these methods and systems may equally be used to provide vision correction for higher order aberrations. Generally, it is common to describe higher order aberrations mathematically using so-called Zernike polynomials, which describe how the light wave front entering the eye is distorted by the aberration. For example, higher order aberrations such as spherical aberrations, coma, and trefoil may be represented by a second order Zernike polynomial function. In some embodiments, the light field rendering methods and systems described above may be used to generate a light field that compensates for such higher order aberrations. In some embodiments, this may include generating a curved or distorted image plane based on or derived from, in some instances, the corresponding Zernike polynomial function. Moreover, methods for generating a vision corrected light field image with curved surfaces were already described above. Thus, in some embodiments, the rendering methods 1100, 1900, or 2600 may be equally applied, but with the added characteristic that, for example, the virtual image is curved or distorted.
While the present disclosure describes various exemplary embodiments, the disclosure is not so limited. To the contrary, the disclosure is intended to cover various modifications and equivalent arrangements included within the general scope of the present disclosure.
Claims
1. A computer-implemented method, automatically implemented by one or more digital processors, to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display includes an array of light field shaping elements (LFSE), the method comprising:
- digitally mapping the input image on a retinal plane of the user;
- for each given pixel of at least some of said pixels, digitally:
- projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter;
- associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping;
- rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
2. The computer-implemented method of claim 1, wherein said retinal plane is angled relative to the digital display.
3. The computer-implemented method of claim 2, wherein said retinal plane is modeled as a function of an input user pupil or eye location.
4. The computer-implemented method of claim 3, wherein said input user pupil or eye location is dynamically acquired via a digitally implemented pupil or eye tracker.
5. The computer-implemented method of claim 4, wherein said angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
6. The computer-implemented method of claim 2, wherein said angle is dynamically updated based on acquired display inclination data.
7. The method of claim 1, wherein said mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter.
8. The method of claim 7, wherein said designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
9. The method of claim 1, wherein said modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
10. The method of claim 1, wherein the digital display is defined by a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
11. A non-transitory computer-readable medium comprising digital instructions to be implemented by one or more digital processors to automatically adjust user perception of an input image to be rendered on a digital display via a set of pixels thereof, wherein the digital display includes an array of light field shaping elements (LFSE), by:
- digitally mapping the input image on a retinal plane of the user;
- for each given pixel of at least some of said pixels, digitally: projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping;
- rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
12. The non-transitory computer-readable medium of claim 11, wherein said retinal plane is angled relative to the digital display.
13. The non-transitory computer-readable medium of claim 12, wherein said retinal plane is modeled as a function of an input user pupil or eye location that is dynamically acquired via a digitally implemented pupil or eye tracker, wherein said angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
14. The non-transitory computer-readable medium of claim 12, wherein said angle is dynamically updated based on acquired display inclination data.
15. The non-transitory computer-readable medium of claim 11, wherein said mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter.
16. The non-transitory computer-readable medium of claim 15, wherein said designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
17. The non-transitory computer-readable medium of claim 11, wherein said modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
18. The non-transitory computer-readable medium of claim 11, wherein the digital display is defined by a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
19. A digital display device operable to automatically adjust user perception of an input image to be rendered thereon, the device comprising:
- a digital display medium comprising an array of pixels and operable to render a pixelated image accordingly;
- an array of light field shaping elements (LFSE) to shape a light field emanating from at least some of said pixels and thereby at least partially govern a projection thereof from said display medium toward the user; and
- a hardware processor operable on pixel data for the input image to output adjusted image pixel data to adjust user perception of the input image as rendered by:
- digitally mapping the input image on a retinal plane of the user;
- for each given pixel of at least some of said pixels, digitally: projecting an adjusted image ray trace between said given pixel and a given LFSE to intersect said retinal plane at a given adjusted image location, given an estimated direction of a light field emanated by said given pixel given said given LFSE and a modeled redirection of said adjusted image ray trace in accordance with a designated eye focus parameter; associating an adjusted image value designated for said given adjusted image location with said given pixel based on said mapping; rendering each said given pixel according to said adjusted image value associated therewith, thereby rendering a perceptively adjusted version of the input image.
20. The device of claim 19, wherein said retinal plane is angled relative to the digital display.
21. The device of claim 20, further comprising a digitally implemented pupil or eye tracker, wherein said retinal plane is modeled as a function of an input user pupil or eye location that is dynamically acquired via said digitally implemented pupil or eye tracker, wherein said angle between said retinal plane and said digital display is dynamically updated based on data received from said pupil or eye tracker.
22. The device of claim 20, wherein said angle is dynamically updated based on acquired display inclination data.
23. The device claim 19, wherein said mapping is implemented by scaling the input image on said retinal plane as a function of said designated eye focus parameter, wherein said designated eye focus parameter is designated as a function of a quantified abnormal user eye focal length or a corrective eyewear prescription.
24. The device of claim 19, wherein said modeled redirection of said adjusted image ray trace is modelled in accordance with a non-linear eye focus parameter.
25. The device of claim 19, wherein said digital display medium defines a curved surface, and wherein said adjusted image ray trace is computed based on a vector normal to said curved surface for said pixel.
Type: Application
Filed: Aug 26, 2019
Publication Date: Apr 23, 2020
Inventor: Daniel GOTSCH (Kingston)
Application Number: 16/551,572
