GAZE DETECTION DEVICE

Abstract

Problem: The purpose is to reduce the size of a gaze detection device. Solution: A gaze detection device that is used with a head-mounted display and comprises an invisible light emitter device for illuminating the user's eye with invisible light, an invisible light mirror that reflects the invisible light and transmits visible light, a filter that transmits the invisible light but blocks visible light, and an imaging device that can capture images with the invisible light, wherein the invisible light mirror is positioned between a display and the user's eye, and between the display and the imaging device at an angle between the surface of the invisible light mirror and the image display surface of the display that is greater than or equal to 0 degrees and less than 45 degrees, reflecting the invisible light emitted by the invisible light emitter device and reflected from the user's eye, the filter is positioned between the display and the imaging device, the imaging device is positioned so that at least a part of the image display surface of the display is included in the field of view of the imaging device, allowing images of the user's eye to be captured using the invisible light reflected by the invisible light mirror.

Description

TECHNICAL FIELD

The present invention relates to a gaze detection device. In particular, the invention relates to a gaze detection device used in conjunction with a head-mounted display (HMD).

BACKGROUND ART

Conventional gaze detection devices that estimate a user's gaze direction from images of the user's eye comprise, as illustrated in FIG. 8, a lens 61, an infrared emitter device 62, a hot mirror 63 that reflects infrared light and transmits visible light, a camera 64, and a display 65. The hot mirror 63 is positioned at an angle of about 45 degrees relative to the optical axis 66 that is perpendicular to the surface of the display 65, with the angle θ between the surface of the display and the hot mirror being 45 degrees. The camera 64 is positioned below the optical axis 66 and is oriented perpendicular to and pointing towards the optical axis 66. The infrared light emitted by the infrared emitter device 62 reflects from the eye 60 of the user, then reflects from the hot mirror 63, and reaches the camera 64. The infrared light optical path connects the camera 64 and the user's eye 60 through a right angle reflection in the hot mirror 63. An example of the aforementioned structure is disclosed in FIG. 8 of patent document 1. Further, patent document 2 discloses a medical eye camera comprising a hot mirror that is tilted at an angle of 45 degrees. In contrast, as illustrated in FIG. 9 herein, patent document 1 discloses a configuration wherein the space wasted by the hot mirror is eliminated by making the angle y between the optical axis 66 and the dichroic mirror (hot mirror) 63 larger than 45 degrees and the angle θ between the display and the optical axis smaller than 45 degrees.

Technology for another known method for gaze detection is shown in FIG. 10, wherein a camera 64 is pointed towards an eye 60 of the user and an image of the user's eye 60 can be captured directly where the captured image is used for gaze detection.

Patent document 3 discloses a gaze detection device that uses a half mirror. The half mirror in patent document 3 changes the optical path by reflecting light incident from a specific direction and transmitting light incident from directions other than the specific direction (Patent Document 3 [0015]). In this gaze detection device, a lens is positioned in front of the user's eye, a liquid crystal display (LCD) is positioned orthogonally to the user's gaze direction, and a half mirror is oriented at a 45 degree angle with respect to the user's gaze direction. Light emitted from the LCD is reflected by the half mirror, reflected again by the lens, and reaches the eye of the user after being transmitted through the half mirror. Light from the LCD emitted towards the gaze detection device that faces the LCD is blocked by the half mirror and therefore cannot enter the gaze detection device.

Techniques for changing the video shown by the video display in a HMD in response to user's head movement are disclosed in patent document 4. Other technologies related to HMDs or gaze detection are disclosed in patent documents 5 to 7.

PATENT REFERENCES

Patent document 1: JP Application No. PH08-9205 (U.S. Pat. No. 6,018,630)

Patent document 2: JP Application No. PH05-076497

Patent document 3: JP Application No. 2001-108933

Patent document 4: JP Application No. PH08-179239

Patent document 5: JP Application No. 2014-21272

Patent document 6: JP Application No. 2006-163383 (USP 2006/0103591)

Patent document 7: JP Application No. P11-155152 (U.S. Pat. No. 6,611,283)

SUMMARY OF THE INVENTION

Problems Solved by the Invention

In existing technology described in FIG. 8, a hot mirror is tilted at an angle of 45 degrees relative to the optical axis 66 that is perpendicular to the display 65 surface. Therefore the large space required by the hot mirror is the main reason why the size of a head-mounted display cannot be

An existing technology to solve this problem, as illustrated in FIG. 9, is to positon the hot mirror at a larger angle than 45 degrees with respect to the optical axis 66. However, because the angle is restricted, it is impossible to achieve the desired reduction in size. In this structure, the optical axis of the camera 64 that captures images of the user's eye 60 is substantially horizontal with respect to the surface of the display. Light emitted from the display 65 adds a noise component to the image of the user's eye used for gaze detection, and it is thus necessary to orient the camera 64 in a direction where light emitted by the display 65 does not enter the camera. For this reason, even if the angle θ between the display 65 and the hot mirror 63 is made smaller than 45 degrees, the angle θ still has to be considerably large. Thus, the distance between the lens and the display still cannot be sufficiently reduced and miniaturization is impossible.

If a smaller hot mirror is used to obtain a more compact device, edges of the hot mirror between the user's eye and the display would create a visual disruption due to unintended reflection or refraction of light, disturbing the video presentation.

In a conventional configuration described in FIG. 10, the camera 64 can directly capture images of the user's eye. This means that the camera 64 is in the user's field of view, disrupting the user's immersion in the virtual realty world created by the head-mounted display.

In the configuration using a half mirror as described in patent document 3, the half mirror has to be position at an angle of 45 degrees to the gaze direction of the user, wasting space and making miniaturization of a head-mounted display impossible.

Therefore, the purpose of the present invention is to provide a gaze detection device that allows a head-mounted display (HMD) to be miniaturized, or a HMD or another system employing the same.

Means of Solving the Problem

The present invention has been made in view of the above considerations, and has the following features. The gaze detection device of the present invention is used in conjunction with a head-mounted display that is used while secured to the head of a user, wherein the head-mounted display comprises at least one display, at least one invisible light emitter illuminating an eye of the user with invisible light, at least one invisible light mirror that reflects the invisible light and transmits visible light, at least one filter that transmits the invisible light and does not transmit visible light, and at least one imaging device that can capture images using the invisible light, wherein at least one of the invisible light mirrors is positioned between at least one of the displays and an eye of the user and between at least one of the imaging devices and at least one of the displays, wherein at least one of the invisible light mirrors is positioned at an angle relative to the surface of at least one of the displays, wherein of the two perpendicular axes defined in the surface plane of at least one of the displays, the first rotation angle of at least one of the invisible light mirrors around one of the two axes, and the second rotation angle of the invisible light mirror around the other of the two axes, are both larger than or equal to 0 and less than 45 degrees, such that when invisible light emitted by at least one of the invisible light emitter devices illuminating an eye of the user is reflected from an eye of the user, at least one of the filters is positioned between at least one of the displays and at least one of the imaging devices, and one or more imaging devices are oriented so that at least a part of the image display surface of at least one of the displays is in the imaging range of the imaging device, at least one of the imaging devices can capture an image of an eye of the user by using invisible light reflected in at least one of the invisible light mirrors.

In one aspect of the present invention, one or more of the invisible light mirrors may be implemented as a combination of two invisible light mirrors, wherein one of the two invisible light mirrors is positioned between at least one of the displays and an eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from an eye of the user towards one of the one or more imaging devices, while the other of the two invisible light mirrors is positioned between at least one display and another eye of the user, and each of the first rotation angle and the second rotation angle of the invisible light mirror are set to reflection angles that direct invisible light reflected from the user's other eye towards the one imaging device.

Also, the gaze analysis device may be connected to at least one imaging device, wherein one or more imaging devices transfer the information related to the captured images of an eye of the user to the gaze analysis device, and the gaze analysis device may detect the user's gaze based on the information related to the captured images of the user's eyes.

Furthermore, if the non-visible light is infrared light, a hot mirror may be used as the invisible light mirror.

On or more lenses may be disposed between an eye of the user and one or more displays.

The distances between at least one lens, at least one display, and the user's eye may be determined based on at least the user's visual acuity and the focal length of the user's eyes.

The lens may be detachable from the device.

The gaze detection device of the present invention may be detachable from the head-mounted display.

At least one imaging device may be disposed at a position allowing imaging of the user's eye from a lower direction.

The head-mounted display of the present invention comprises the aforementioned gaze detection device.

Furthermore, the system of the present invention includes a head-mounted display described above, the head-mounted display comprises at least one display, the video control system comprises a gaze analysis device and a video control device connected to one or more displays, wherein the gaze analysis device transmits the user's gaze detection information to the video control device, and based on the user's gaze detection information received from the gaze analysis device, the video control device generates a video signal that is sent to one or more displays, and one or more displays display a video based on the video signal.

Furthermore, the image control device may be connected to a user input device, wherein the user input device transmits user inputs to the video control device and, based on the user input, the video control device may generate appropriate image information to be transmitted to the display.

Advantageous Effects of the Invention

In the aforementioned configuration of the present invention, the imaging device is positioned and oriented so as to allow the field of view of the imaging device to include at least a part of the display surface. The degree of freedom in selecting the angle between the invisible light mirror and the display is increased, making it possible to further miniaturize the head-mounted display. Further, since there is no need to use a smaller-sized invisible light mirror to miniaturize the device, there is no effect on the image presentation due to the edges of the invisible light mirror entering the field of view. Therefore, it is possible to miniaturize the device and maintain high video presentation quality.

In the present invention, when the user's eye is illuminated with infrared light from an infrared emitter device, the infrared light reflects from the eye of the user, is then further reflected by the invisible light mirror, and can be delivered to an imaging device that is positioned outside of the user's field of view. That is, by adopting such a configuration, since it is possible to position the imaging device outside of the user's field of view, the imaging device does not impede the visibility of the video image for the user and the immersion of the user in the virtual reality world created by the head-mounted display can be improved.

When the user slightly lowers an eyelid, the eyelid conceals the upper half of the eyeball. Since an imaging device installed above the eye acquires images of a user's eye from the upper direction an eyelid or eyelashes may obstruct imaging by a camera. In case the imaging device for capturing images of an eye of the user is positioned so as to capture images of an eye of the user from a lower direction, an eyelid or eyelashes have no effect and images of an eye of the user can be reliably captured.

By enclosing the gaze detection device of the present invention in a casing that is detachable from a head-mounted display, the gaze detection function can be added to a head-mounted display that does not have a built-in gaze detection device. Further, by enclosing in a casing a gaze detection device with an optical configuration tailored to a specific user, there is no need to purchase whole head-mounted display units for each user. Further, by setting the distances between at least one lens, at least one display and an eye of the user on the basis of the visual acuity and focal distance of the user's eye, head-mounted displays can be tailored to accommodate each user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic side view of the HMD according to the first embodiment of the present invention.

FIG. 2 is an explanatory diagram of the X-axis and Y-axis of the HMD.

FIG. 3 is a schematic plan view of a HMD according to the first embodiment of the present invention.

FIG. 4 is a schematic plan view of a HMD according to the second embodiment of the present invention.

FIG. 5 is a schematic plan view of a HMD according to the third embodiment of the present invention.

FIG. 6 is a schematic side view of the HMD according to the fourth embodiment of the present invention.

FIG. 7 is an example of a display screen according to the fourth embodiment of the present invention.

FIG. 8 is a schematic side view of a conventional gaze detection device.

FIG. 9 is a schematic side view of a conventional gaze detection device.

FIG. 10 is a schematic side view of a conventional gaze detection device.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The First Embodiment

A schematic side view of the head-mounted display according to the first embodiment is shown in FIG. 1. The head-mounted display 1 that can be secured to the head of a user contains a display unit 14 and the display unit 14 has an image display surface 16. Further, the gaze detection device installed in the head-mounted display comprises invisible light emitter devices 20 that illuminate the eye 10 of the user with invisible light, an invisible light mirror 13 that reflects the invisible light and transmits visible light, an imaging device 12 that can capture images with the invisible light, and a filter 15 that transmits the invisible light and does not transmit visible light. The gaze detection device further comprises a lens 11 that is positioned between the display unit 14 and the eye 10 of the user.

Although in the present embodiment infrared emitter devices 20-1˜20-4 are used as the invisible light emitter device 20, as long as the emitted light is invisible, the use of a light source with a different wavelength is possible as well. Further, as described in FIG. 1, in the present embodiment, the invisible light emitter device 20 is implemented as four infrared emitter devices 20-1˜20-4 that are positioned at the top, bottom, left, and right edges of a lens. However, as long as the emitted infrared light can reach the imaging device 12 after being reflected from the eye 10 of the user, other arrangements of the light emitter positions are also possible, while a smaller number of light emitters, for example one, may be used. The number of installed infrared light sources may be larger as well. The imaging device 12 in FIG. 1 is positioned in the lower part of the head-mounted display 1, but may be positioned in the upper part, sides, or elsewhere. In the present embodiment, the invisible light mirror 13 is a hot mirror that reflects infrared light and transmits visible light. In case the invisible light emitter device 20 is a light emitter with a wavelength outside of the infrared range, the invisible light mirror should reflects light at the operating wavelength of the invisible light emitter device. A hot mirror is disposed at a position where it is between an eye of the user and a display, and between the display and the imaging device. In this embodiment, a hot mirror 13 is positioned between the display 14 and the lens 11. As shown in FIG. 1, the angle θ between the hot mirror 13 and the image display surface 16 of the display unit 14 (the hot mirror side of the display 14) is larger than 0 degrees and less than 45 degrees, preferably in the range from 0 degrees to less than 20 degrees. Smaller angle θ leads to greater reduction of the head-mounted display thickness and weight. For example, the edges of the hot mirror 13 and the display 14 may be in contact.

As shown in FIG. 2, the angle between the hot mirror surface and the image display surface 16 of the display 14 can be defined by the rotation angle of the hot mirror 13 around the X-axis and the rotation angle of the hot mirror 13 around the Y-axis. Here, the axes are defined in the plane of the image display surface 16 of the display 14 such that when the head-mounted display is mounted on the head, the X-axis points along the left-right direction and the Y-axis points along the up-down direction. In the present embodiment, the Z-axis points in the direction of the user. The angle θ in FIG. 1 is the rotation angle of the hot mirror 13 around the X-axis. In the present embodiment, as shown in FIG. 3, the hot mirror 13 rotation angle α around the Y-axis is 0 degrees. The hot mirror 13 thus reflects the infrared light emitted by the infrared emitter device 20 and reflected from the eye 10 of the user towards the imaging device 12. When both angles α and θ are small, the wasted space can be reduced and the head-mounted display can be made smaller.

Filter 15 is positioned between the display 14 and the imaging device 12. The filter 15 is a visible light cut-off filter that transmits infrared light and blocks visible light. The filter 15 may be any filter that can block visible light and transmit the invisible light emitted by the invisible light emitter device 20 and reflected from the eye of the user. The imaging device 12 is oriented in a direction that places at least a part of the image display surface 16 in the field of view of the imaging device and the imaging device captures images of the user's eye using infrared light reflected by the hot mirror 13. That is, the imaging device 12 is positioned in a direction that allows light emitted from the image display surface 16 of the display 14 to reach the imaging device directly, while infrared light reaches the imaging device after being reflected in the hot mirror 13.

Next, the operation according to the present embodiment will be explained. First, the user secures the head-mounted display 1 to the head. The image display surface 16 of the display 14 is directed towards the eyes of the user and displays a video. The video may represent a computer-generated animation, a video game, a movie, etc. The image display surface 16 of the display 14 displays a video by emitting visible light. This emitted visible light is transmitted through the hot mirror 13, and reaches the eye 10 of the user through the lens 11. The lens 11 magnifies the image on the display 14 for viewing by the user. The lens can also contribute to vision correction for the user. The lens 11 is not essential for the operation of the gaze detection device.

The infrared emitter devices 20-1˜20-4 illuminate the eye 10 of the user with infrared light at a fixed illumination angle. The illuminating infrared light is reflected from the eye 10 of the user, passes through the lens 11, and reaches the hot mirror 13. The hot mirror 13 reflects the infrared light towards the imaging device 12. In the present embodiment, the hot mirror 13 is in the field of view 21 of the user and preferably covers the entire surface of the display 14. It is thereby possible to view the display without the edges of the hot mirror creating boundaries within the field of view 21. However, the hot mirror 13 does not necessarily have to cover the entire surface of the display 14. The imaging device 12 captures an image of the user's eye, wherein the image is formed by the infrared light reflected from the eye 10 of the user. It is possible to determine the user's gaze direction based on the captured image of the eye of the user. By reflecting the infrared light, the hot mirror 13 makes it possible for an imaging device located outside of the user's field of view 21 to acquire images of the user's eye. The user can view the video shown on the display without being obstructed by the imaging device.

In the present embodiment, using a plurality of infrared emitter devices 20-1˜20-4 allows the intensity of the infrared light reflected from the eye 10 of the user and reaching the imaging device 12 to be increased. It is possible to obtain sufficient received infrared light intensity with a single infrared emitter device as well. Increasing the infrared light emission intensity of the single infrared emitter device may be used to increase the intensity of the reflected infrared light reaching the imaging device 12. However, in that case strong infrared light irradiates a particular part of the user's eye, increasing the risk of injury to the eye of the user. By using a plurality of infrared emitter devices, the relatively weak infrared irradiation does not damage the eyes of the user, while sufficient infrared light intensity reaches the imaging device 12 for gaze detection.

Since the imaging device 12 is positioned such that at least a part of the surface of the display 14 is included in the field of view of the imaging device, visible light emitted from the image display surface 16 of the display 14 passes through the hot mirror 13 and illuminates the imaging device 12. However, the light emitted from the display 14 is visible light, which is blocked by the visible light cut-off filter 15, positioned between the display 14 and the imaging device 12, preventing the emitted light from reaching the imaging device 12. In contrast, the infrared light reflected by the hot mirror 13 passes through the visible light cut-off filter 15 and reaches the imaging device.

In existing technology, because visible light emitted from a display can enter the imaging device, noise is added to the user's gaze detection and it is therefore necessary to direct the imaging device in a direction that prevents visible light emitted by the display from entering the imaging device, preferably in a vertical direction. For this reason, the tilt angle of the hot mirror with respect to the display has to be large.

In contrast, by employing the configuration of the present invention, even though the imaging device 12 may be oriented so that at least a part of the image display surface of the display 14 is included in the field of view of the imaging device, the visible light emitted by the display 14 that reaches the imaging device 12 does not give rise to noise and the received infrared light can be used for capturing images of the user's eye. Therefore, there is greater freedom in the positioning direction of the imaging device, and also greater freedom in setting the angle θ between the hot mirror 13 and the display 14. For example, the angle θ may be set to 0 degrees. Accordingly, the head-mounted display can be made thinner and smaller.

Further, as shown in FIG. 1, the imaging device 12 is preferably arranged at a position that allows capturing images of an eye of the user from below. When imaging the eye of the user from below, without being obstructed by the eyelid or the eyelashes, reliable capturing of images of the user's eye is possible.

The Second Embodiment

FIG. 4 shows a schematic top-view diagram of the configuration of the head-mounted display according to the second embodiment. In the following description, explanations are omitted for components that are identical to those described in the first embodiment.

The configuration of the present embodiment is similar to the configuration of the first embodiment but applied for each eye of the user. In other words, for the right eye 10a of the user, the head-mounted display 1 comprises a display 14a, a hot mirror 13a that is positioned between the display 14a and the right eye 10a, and between the display 14a and the imaging device 12a, a lens 11a that is positioned between the display 14a and the user's right eye 10a, infrared emitter devices 20a-1˜20a-4 that illuminate the right eye 10a of the user with infrared light, an imaging device 12a that is oriented so that at least a part of the image display surface of the display 14a is included in the field of view of the imaging device, and a visible light cut-off filter 15a that is positioned between the imaging device 12a and the display 14a. In this embodiment, as in the first embodiment, illustrated in FIG. 1, the hot mirror 13a is rotated around an X-axis defined within the image display surface 16a plane of the display 14a, wherein the rotation angle θ (referred to as θ1 in this embodiment) between the surface of the hot mirror 13a and the display 14a is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees. On the other hand, as shown in FIG. 4, for a rotation of the hot mirror 13a around the Y-axis, the angle α between the image display surface 16a of the display 14a and the surface of the hot mirror 13a is 0 degrees.

Similar to the above, for the user's left eye 10b, the gaze detection device further comprises a display 14b, a hot mirror 13b, a lens 11b, infrared emitter devices 20b-1 20b-4, a visible light cut-off filter 15b, and an imaging device 12b. The hot mirror 13b is rotated around the X-axis defined within the image display surface 16b plane of the display 14b, wherein the rotation angle θ (referred to as θ2 in this embodiment) between the hot mirror 13b and the display 14b is greater than or equal to 0 degrees and less than 45 degrees, but preferably greater than or equal to 0 degrees and less than 20 degrees, while the rotation angle f between the surface of the hot mirror 13b and the image display surface 16b of the display 14b for a rotation of the hot mirror 13b around the Y-axis is 0 degrees. The angles θ1 and θ2 do not need to be equal to each other.

To apply the method described in the first embodiment to both, images formed by infrared light of the user's right eye 10a and the left eye 10b are captured by the infrared imaging devices 12a and 12b, respectively.

In the present embodiment, a display 14c and a display 14d are located on the sides of the front part of the head-mounted display. Since the user can see the video images on the displays 14c and 14d in addition to the displays 14a and 14b that are directly facing the user, the user may gain close to a real-world experience. The visible light cut-off filters 15a and 15b are disposed between the side-mounted displays 14c, 14d and the imaging devices 12a, 12b, blocking the visible light emitted from these displays, making it possible to properly image the motion of the user's eye. For example, a visible light cut-off filter that covers the lens of the imaging device may be used. It is also possible to use a plurality of visible light cut-off filters to block light from a plurality of displays.

Similarly, displays may be located along the top surface or the bottom surface of the front part of the head-mounted display 1.

The Third Embodiment

FIG. 5 shows a schematic top view of the configuration of the head-mounted display according to the present embodiment. The explanation will focus on differences from the second embodiment. For the sake of simplifying the explanation of the display, only a configuration with two front displays 14a and 14b will be considered. Further displays may be added on the sides and at the bottom. The displays 14a and 14b may be implemented as a single display unit.

In the present embodiment, a single common imaging device 12 is used for the left eye and the right eye. This single imaging device is positioned in a direction that allows at least a part of the image display surface of at least one of the displays 14a and 14b to be included in the field of view of the imaging device. Further, two hot mirrors 13a and 13b are used, each of which is positioned between one of the eyes of the user and one of the two displays 14a and 14b. As shown in FIG. 5, the angle α between the image display surface 16a of the display 14a and the surface of the hot mirror 13a is the rotation angle of the hot mirror 13a around the Y-axis of the display 14a.

A schematic side view of this embodiment has the same configuration as that shown in FIG. 1. For rotations of the hot mirror 13a around the X-axis defined in the plane of the image display surface of the display 14a, the rotation angle between the image display surface 16a of the display 14a and the surface of the hot mirror 13a is θ (referred to as θ3). The angles α and θ3 are set so as to reflect the infrared light reflected from the right eye of the user towards the common imaging device 12.

Similarly, the other hot mirror 13b is positioned between the display 14b and the user's left eye 10b. The angle between the hot mirror surface and the image display surface 16b of the display 14b, wherein the rotation angle of the hot mirror 13b around the Y-axis defined in the plane of the image display surface of the display 14b is β and the rotation angle of the hot mirror 13b around the X-axis is θ(referred to as θ4), is set so as to reflect infrared light reflected from the left eye of the user towards the imaging device 12. The angles α and β of this embodiment are set to predetermined values that are greater than 0. The angles α, β, θ3, and θ4 are not necessarily equal to each other.

The imaging device 12 receives infrared light reflected from each of the hot mirrors 13a and 13b, and captures images of both the right and the left eyes 10a and 10b of the user.

Since the imaging device is directed in a direction that allows at least a portion of the image display surface of a display to be included in the field of view of the imaging device, and the hot mirror is rotated not only around the horizontal axis of the head-mounted display, but also around the vertical axis to incline the hot mirror relative to the display, a single imaging device can be used to capture images of the gaze of both eyes. Therefore, the head-mounted display can be made more compact and manufactured at a lower cost.

The Fourth Embodiment

FIG. 6 shows a structure diagram of the video control system of the present embodiment. The figure shows a schematic side view of the head-mounted display 1a.

In this embodiment, the head mounted-display 1a further comprises a video control device 24. The head-mounted display 1a is a variation of the head-mounted display 1 described in the first embodiment. In addition to the configuration of the gaze detection device of the first embodiment, the gaze detection device of the head-mounted display 1a further comprises a gaze analysis device 22 that is connected to the imaging device 12. The gaze analysis device 22 is connected to a video control device 24, and the video control device 24 is connected to the display 14. Further, a user input device 25 for receiving input from the user is connected to the video control device 24. These connections may be wired or wireless. The video control device 24 of the present embodiment is a video game machine for controlling a video game, the user input device is a controller 25 connected to the video game machine.

The gaze analysis device of the present embodiment is placed in a casing 23 that is detachable from the head-mounted display 1a. Since the gaze detection device is detachable from the head-mounted display, it is possible to add the gaze detection function to an existing head-mounted display that does not include a gaze detection device.

The selection of the lens 11 and the distances between the lens 11, the display 14, and the user's eye 10 are preferably determined based on the user's visual acuity and the user's focal length. Since the optical system of the gaze detection device is designed based on the user's eye and the position of the lens, an entire head-mounted display unit would need to be purchased for each user to provide a customized configuration suitable for the user, placing a large burden on the user.

With the detachable gaze detection device of the present embodiment, the lens 11, and the distances between the lens 11, the display 14, and the user's eye 10 can be selected based on the user's eyesight and the user's focal length, which means that the user would only need to purchase the gaze detection device and mount it on a head-mounted display. Thus, it is possible to provide head-mounted displays with gaze detection devices suitable for each user at a lower cost.

Further, by adjusting the respective distances between the lens 11, the display 14, and user's eye 10 appropriately, it is possible to set an optimal optical configuration for each user in the gaze detection device. The adjustment can be done, for example, by fixing the distance between the lens and the display, and changing the distance between the lens and the user's eye. Further, if the distance between the lens and the user's eye is fixed, the adjustment can be done by moving the display. It is also possible to fix the distance between the display and the user's eye, and perform the adjustment by moving the lens in between. Specifically, as shown in FIG. 6, for example, the lens 11 may be attached to a rail 26 that is disposed between the eye 10 of the user and the display 14, wherein the lens can slide along the rail 26, allowing the lens 11 position to be adjusted to suit the user. The same configuration can be used for the display 14. Alternatively, the lens and the display may be configured to slide simultaneously.

For a head-mounted display, depending on the build of the cheekbones of the user and the shape of the user's nose, the distance between the lens 11 and the eye of the user may be significantly different for each user. By adopting the configuration described in this embodiment, it is possible to build a head-mounted display that fits every user. Further, it is possible to use a detachable lens to configure the gaze detection device for each user by replacing the lens.

Although in the present embodiment, the video control device 24 is described as being a device that is external to the head-mounted display, it is also possible to provide a device that is integrated in a head-mounted display. Further, although the gaze analysis device 22 has is described as being a part of the gaze detection device, it is also possible to implement it as an external device for a head-mounted display or a gaze detection device.

The imaging device 12 transmits the image data of the captured images of the user's eye to the gaze analysis device 22, and based on such image information, the gaze analysis device 22 analyzes the user's gaze, eye movement frequency, nystagmus, etc. In the present embodiment, the gaze analysis device 22 determines the gaze direction and speed, and transmits the gaze information to the video control device 24. Based on the user's gaze information received from the gaze analysis device 22, the video control device 24 generates the video information to be displayed on the display 14. In the present embodiment, the user plays a shooting game using the head-mounted display 1a. A video game machine 24, functioning as a video control device, displays the video of the shooting game on the image display surface 16 of the display 14. Based on the user's gaze information from the analysis by the gaze analysis device 21, a cursor is displayed on the image display surface of the display 14, marking the firing point in the vicinity of the user's gaze location, while the cursor moves in accordance with the gaze movement. For example, as shown in FIG. 7, a circular cursor 32 is shown on the display in the video image 30 around the position 31 of the user's gaze. When the user's gaze location moves to the right, the cursor 32 is also moved to the right. The position and the moving speed of the cursor are calculated on the basis of the position and movement speed of the gaze location. After the user moves the cursor to the desired position, the firing command is input from the controller 25, and according to this command, the video game machine 24 generate a video that is shown on the display 14. Instead of the user input received through the controller 25, if the user keeps the gaze fixed at one location for a predetermined period of time, the game machine 24 may decide that a user input action has occurred.

If instead of a video game machine the user is using a personal computer, the gaze location can be used to control the mouse pointer position. In this case the user input device 25 is a mouse. Also, the image control device 24 used for displaying a movie etc., in accordance with the movement of the gaze position, may performs a control function such as scrolling the image. In this case a user input device 25 is not necessarily required.

The above embodiments have been presented as examples for the purpose of explaining the present invention. The preset invention is not limited to these embodiments. The present invention, without departing from the gist thereof can be implemented in various forms.

REFERENCE LIST

• 1 HMD
• 10 User's eye
• 11 Lens
• 12 Imaging device
• 13 Invisible light mirror
• 14 Display
• 15 Filter
• 16 Image display surface
• 20 Invisible light emitter device
• 21 Field of view
• 22 Gaze analysis unit
• 23 Casing
• 24 Video control device
• 25 User input unit
• 26 Rail
• 30 Display surface plane
• 31 Gaze location
• 32 Cursor
• 60 User's eye
• 61 Lens
• 62 Infrared emitter device
• 63 Hot mirror
• 64 Camera
• 65 Display
• 66 Optical axis

Claims

1. A gaze detection device for use with a head-mounted display that comprises at least one display and is used while secured to the head of the user, comprising:

at least one invisible light emitter device for illuminating an eye of the user with invisible light;

at least one invisible light mirrors that reflects the invisible light and transmits visible light;

at least one filter that transmits the invisible light and does not transmit visible light; and

at least one imaging device that can capture images formed by the invisible light, wherein

at least one of the invisible light mirrors is disposed both between at least one of the displays and an eye of the user, and between the display and at least one of the imaging devices,

the angle between the image display surface of the display and the surface of the invisible light mirror is defined in reference to two orthogonal axes within the plane that includes the image display surface of the display, wherein the first rotation angle of the invisible light mirror around one of the two axes and the second rotation angle of the invisible light mirror around the other axis are both set to angles greater than or equal to 0 degrees and less than 45 degrees;

the invisible light mirror reflects invisible light emitted by at least one of the invisible light emitter devices and reflected from the user's eye;

at least one of the filters is disposed between at least one of the displays and at least one of the imaging devices;

at least one of the imaging devices is oriented so as to include at least a part of the image display surface of at least one of the displays in the field of view of the imaging device, allowing the imaging device to capture an image of the user's eye using the invisible light reflected by the invisible light mirror; and

the imaging device is positioned so as to make the crossing angle between the optical axis of the imaging device reflected from the invisible light mirror, and the ocular axis that is orthogonal to at least one of the one or more displays and passes through the corneal apex, greater than zero when the user has mounted the head-mounted display.

2. The device of claim 1, wherein at least one of the invisible light mirrors comprises two invisible light mirrors, one of which is disposed between at least one of the displays and one of the eyes of the user and oriented by setting each of the first rotation angle and the second rotation angle so as to reflect the invisible light reflected from the eye of the user towards one imaging device of the one or more imaging devices; and the other of the two invisible mirrors is disposed between at least one of the displays and the other eye of the user and oriented by setting each of the first rotation angle and the second rotation angle so as to reflect the invisible light reflected from the other eye of the user towards the same imaging device.

3. The device of claim 1 that further comprises a gaze analysis device connected to at least one imaging device, wherein one or more imaging devices capture images of the user's eye and send the image data to the gaze analysis device that detects the gaze of the user on the basis of the captured image data of the user's eyes.

4. The device of claim 1, wherein the invisible light is infrared light and the invisible light mirror is a hot mirror.

5. The device of claim 1 that further comprises at least one lens disposed between at least one of the displays and an eye of the user.

6. The device of claim 5, wherein the distances between at least one of the lenses, at least one of the displays and an eye of the user are set on the basis of at least the user's visual acuity and the user's focal length.

7. The device of claim 5, wherein the lens is detachable from the gaze detection device.

8. The device of claim 1, wherein the device is detachable from the head-mounted display.

9. The device of claim 1, wherein at least one of the imaging devices is disposed at a location that allows the user's eye to be imaged from a lower direction.

10. A head-mounted display comprising the device of claim 1.

11. A system that comprises

a head-mounted display of claim 10, wherein the head-mounted display comprises at least one display; and

a video control system comprising a video control device that is connected to the gaze analysis device and at least one display, wherein

the gaze analysis device sends the user's gaze detection information to the video control device;

the video control device generates video data for at least one display based on the users gaze information received from the gaze analysis device; and

at least one display shows a video based on the video data.

12. The system of claim 11 that further comprises a user input device connected to the video control device, wherein the user input device send the user inputs to the video control device and the video control device sends to the display video data based on the user input.

13. The device of claim 5 that further comprises an adjustment unit for adjusting the distance between the lens and the eye of the user.

14. The device of claim 1, wherein the device comprises a plurality of invisible light emitter devices.