Method and System for Real-time Rendering Displaying Virtual Reality (VR) On Mobile Using Head-Up Display Devices

Abstract

Disclosed a method and a system for real-time rendering displaying virtual reality (VR) using head-up display devices. The method comprises: obtaining relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion; creating a 3D model and obtaining an original coordinate data of the 3D model; obtaining first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model; performing lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data; rasterizing the second coordinate data to obtain pixel information; and drawing an image in accordance with a VR video data and the pixel information. According to the present disclosure, the lens distortion on the coordinate data is performed in the 3D model, so that video and immersive rendering can be realized in one processing, thereby improving rendering efficiency.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority and benefit of U.S. provisional application 62/441,936, filed on Jan. 3, 2017, which is incorporated herein by reference in its entirety.

BACKGROUND OF THE DISCLOSURE

Field of the Disclosure

The present disclosure relates to video processing technology, and more particularly, to a method and a system for real-time rendering displaying virtual reality (VR) using head-up display devices.

Background of the Disclosure

Virtual Reality (VR) is a computer simulation technology for creating and experiencing a virtual world. For example, a three-dimensional real-time image can be presented based on a technology which tracks a user's head, eyes or hand. For a network-based virtual reality technology, full-view video data is pre-stored on a server, and then transmitted to a display device, such as glasses. A video is displayed on the display device in accordance with a viewing angle of the user.

However, when the display device displays the video data, high-resolution video data needs to occupy a lot of computing resources, and as a result, the display device is required to has a high data processing capability. But currently, different types of display devices on market vary greatly in performance. In order to be compatible with these display devices, a VR playback system should be optimized as much as possible in terms of software, so as to reduce resource consumption, improve processing efficiency and meanwhile avoid degrading users's viewing experience.

SUMMARY OF THE DISCLOSURE

In view of this, the present disclosure provides a method and a system for real-time rendering displaying virtual reality (VR) using head-up display devices to solve the above problems.

According to a first aspect of the present disclosure, there is provided a method for real-time rendering displaying virtual reality (VR) using head-up display devices, comprising:

obtaining relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion;

creating a 3D model and obtaining an original coordinate data of the 3D model;

obtaining first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model;

performing lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data;

rasterizing the second coordinate data to obtain pixel information; and

drawing an image in accordance with a VR video data and the pixel information.

Preferably, the step of obtaining relevant parameters comprises:

obtaining parameters relevant to field of view in accordance with specification of a head-up display device and a screen size;

calculating the center position of lens distortion in accordance with the parameters relevant to field of view; and

calculate the projection matrix in accordance with the parameters relevant to field of view.

Preferably, the step of obtaining relevant parameters comprises:

obtaining an eye distance parameter based on specification of a head-up display device; and

calculate the camera matrix in accordance with the eye distance parameter.

Preferably, the camera matrix and the projection matrix is adjusted to achieve binocular-mode viewing effects.

Preferably, the step of obtaining first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model comprising:

calculating out camera matrices for left and right eyes in binocular mode by equations (1) to (4):

$\begin{array}{cc}\mathrm{half\_eye}\mathrm{\_ipd}=\frac{\mathrm{eye\_ipd}}{2}& \left(1\right)\\ \mathrm{translate}\left(X,Y,Z\right)=\left[\begin{array}{cccc}1& 0& 0& X\\ 0& 1& 0& Y\\ 0& 0& 1& Z\\ 0& 0& 0& 1\end{array}\right]& \left(2\right)\\ \mathrm{left\_view}\mathrm{\_matrix}=\mathrm{translate}\left(-\mathrm{half\_eye}\mathrm{\_ipd},0,0\right)*\mathrm{mat}4_{\mathrm{view}}& \left(3\right)\\ \mathrm{right\_view}\mathrm{\_matrix}=\mathrm{translate}\left(\mathrm{half\_eye}\mathrm{\_ipd},0,0\right)*\mathrm{mat}4_{\mathrm{view}}& \left(4\right)\end{array}$

wherein, left_view_matrix and right_view_matrix represent respectively a camera matrix for left eye and a camera matrix for right eye, mat4_view is the camera matrix which can be generated directly in accordance with rotation angles of a gyro, and eye_ipd represents the eye distance parameter;

calculating out the projection matrix mat4_projection in binocular mode by equation (5),

$\begin{array}{cc}\left[\begin{array}{cccc}\frac{2}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}& 0& -\frac{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)-\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}& 0\\ 0& \frac{2}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}& -\frac{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)-\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}& Y\\ 0& 0& \frac{\mathrm{far}}{\mathrm{near}-\mathrm{far}}& \frac{\mathrm{far}*\mathrm{near}}{\mathrm{near}-\mathrm{far}}\\ 0& 0& -1& 0\end{array}\right]& \left(5\right)\end{array}$

Wherein, fov_left, fov_right, fov_up, fov_down, far,near represent the parameters relevant to field of view in binocular mode;

setting mat4_model to be an identity matrix

calculating out the first coordinate data P_x,y,z^MVP by equation (6),

P_x,y,z^MVP=mat4_model*mat4_view*mat4_projection*P_x,y,z^original  (6)

wherein, P_x,y,z^MVP represents the first coordinate data, P_x,y,z^original represents the original coordinate data, mat4_model represents the model matrix, and mat4_projection represents the projection matrix, the camera matrices left_view_matrix and right_view_matrix for left and right eyes are respectively provided into equation (6) instead of mat4_view to obtain the first coordinate data P_x,y,z^MVP.

Preferably, the step of performing lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data comprises:

obtaining distortion parameters in accordance with following equations (7) and (8):

$\begin{array}{cc}{K}_1,K_2& \left(7\right)\\ f\left(K_1,K_2\right)=\{\begin{array}{c}{K}_1^{-1}=-K_1\\ K_2^{-1}=3K_1^2-K_2\end{array};& \left(8\right)\end{array}$

obtaining corrected image field coordinates (x_u,y_u) as the second coordinate data in accordance with the distortion parameters by using equations (9) and (10) in which all items containing p can be removed when tangential distortion correction is not performed,

x_u=x_d+(x_d−x_c)(K₁r²+K₂r⁴+ . . . )+(P₁(r²+2(x_d−x_c)²+2P₂(x_d−x_c)(y_d−y_c))(1+P₃r²+P₄r⁴+ . . . )  (9)

y_u=x_d+(y_d−y_c)(K₁r²+K₂r⁴+ . . . )+(2P₁(x_d−x_c)(y_d−y_c)+P₂(r²+2(y_d−y_c)²))(1+P₃r²+P₄r⁴+ . . . )  (10)

wherein (x_d,y_d) is distorted image field coordinates after lens projection, ie, the first coordinate data, (x_u,y_u) is the corrected image field coordinates, (x_c,y_c) is the center position of lens distortion, K_n is a nth radial distortion coefficient, P_n is a nth tangential distortion coefficient, r is a distance from pixels to optical axis.

Preferably, coordinate for the center position of lens distortion is obtained by the following steps,

performing linear interpolation between two vectors based on t using following equation:

$\begin{array}{cc}\mathrm{lerp}\left(t,x_l,x_h,y_l,y_h\right)=y_l+\left(t-x_l\right)\frac{y_h-y_l}{x_h-x_l}& \left(11\right)\end{array}$

(x_l,y_l) and (x_h,y_h) are two coordinate points in a plane;

calculating coordinate (x_center^window_pixel,y_center^window_pixel) of the center position of lens distortion in according to the projection matrix mat4_projection and the screen size width_widow*height_widow by using following equations:

(x_center^normal,y_center^normal)=mat4_projection*[00-10]

x_center^window_pixel=lerp(x_center^normal,−1,1,0,width_widow)

y_center^window_pixel=lerp(y_center^normal,−1,1,0,height_widow)  (12)

wherein the coordinate (x_center^normal,y_center^normal) is a point in the space coordinate axis of [−1,1].

Preferably, the method further comprising: adding a blackout mask.

Preferably, the method further comprising: acquiring real-time data from a gyros, and performing data smoothing and corner prediction while the VR video data is played to achieve anti-shake.

Preferably, the equation used for performing data smoothing is

θ_t+1=k(θ_t+ωΔt)+(1−k)Ø  (13)

where, θ_t is a fusion rotation angle based on time t, k is a fusion weight constant, and ω is an angular velocity read by an accelerometer, Ø is an angle read from the gyros, Δt is a difference between an output time moment and its previous time moment;

equations used for corner prediction is:

$\begin{array}{cc}{\theta }_{\Delta }=\{\begin{array}{c}\mathrm{angularSpeed}*\mathrm{predictionTimeS},\mathrm{angularSpeed}\ge \beta \\ 0,\mathrm{angularSpeed}\in \left[0,\beta \right]\\ \mathrm{null},\mathrm{other}\end{array}& \left(14\right)\\ {\theta }_{t+1}={\theta }_t+{\theta }_{\Delta }& \left(15\right)\end{array}$

wherein θ_t is a fusion rotation angle based on time t, angularSpeed is an angular velocity read by the accelerometer, predictionTimeS is a prediction time constant, and β is a rotation prediction threshold, the gyros and the accelerometer are provided on a head-up display device.

Preferably, the method further comprising: using relevant interfaces provided by OpenGL and WebGL to complete corresponding steps.

According to a second aspect of the disclosure, there is provided a system for real-time rendering displaying virtual reality (VR) using head-up display devices, comprising:

a parameter calculating unit configured to obtain relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion;

a model building unit configured to create a 3D model and obtain original coordinate data of the 3D model;

a coordinate calculating unit configured to obtain first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model;

a lens distortion unit configured to perform lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data;

a rasterization unit configured to rasterize the second coordinate data to obtain pixel information;

an image drawing unit configured to draw an image based on a VR video data and the pixel information.

According to the embodiment of the present disclosure, the binocular-mode VR immersive viewing effect is achieved by performing lens distortion on the coordinate data of the 3D model. Because the lens distortion on the coordinate data is performed in the 3D model, video and immersive rendering can be realized in one processing, thereby improving rendering efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure with reference to the following drawings, in which:

FIG. 1 is a diagram illustrating an example network of a VR playback system;

FIG. 2 is a flowchart diagram showing a method used in the VR playback system of FIG. 1;

FIG. 3 is a flowchart diagram of a method for real-time rendering displaying virtual reality (VR) using head-up display devices according to an embodiment of the present disclosure;

FIG. 4 is an example diagram of a head-up display device;

FIG. 5 is a specific flowchart diagram showing the step of obtaining relevant parameters mentioned in the method for real-time rendering displaying virtual reality (VR) using head-up display devices described in FIG. 3;

FIG. 6 is a schematic diagram of a parameter transfer process between a computer processor and a display chip; and

FIG. 7 is a schematic diagram of a system for real-time rendering displaying virtual reality (VR) using head-up display devices, according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE DISCLOSURE

Exemplary embodiments of the present disclosure will be described in more details below with reference to the accompanying drawings. In the drawings, like reference numerals denote like members. The figures are not drawn to scale, for the sake of clarity. Moreover, some well-known parts may not be shown.

FIG. 1 is a diagram illustrating an example network of a VR playback system. The VR playback system 10 includes a server 100 and a display device 120 which are coupled with each other through a network 110, and a VR device. For example, the server 100 may be a stand-alone computer server or a server cluster. The server 100 is used to store various video data and to store various applications that process these video data. For example, various daemons run on the server 100 in real time, so as to process various video data in the server 100 and to respond various requests from VR devices and the display device 120. The network 110 may be a selected one or selected ones from the group consisting of an internet, a local area network, an internet of things, and the like. For example, the display device 120 may be any of the computing devices, including a computer device having an independent display screen and a processing capability. The display device 120 may be a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a palmtop computer, a personal digital assistant, a smart phone, an intelligent electrical apparatus, a game console, an iPad/iPhone, a video player, a DVD recorder/player, a television, or a home entertainment system. The display device 120 may store VR player software as a VR player. When the VR player is started, it requests and downloads various video data from the server 100, and renders and plays the video data in the display device. In this example, the VR device 130 is a stand-alone head-up display device that can interact with the display device 120 and the server 100, to communicate the user's current information with the display device 120 and/or the server 100 through signaling. The user's current information is, for example, parameters relevant to users' field of view, positions of users' helmet, changes of sight of eyes. According to these information, the display device 120 can flexibly process the currently played video data. In some embodiments, when a user turns his head, the display device 120 determines that a core viewing region for the user has been changed and starts to play video data with high resolution in the changed core viewing region.

In the above embodiment, the VR device 130 is a stand-alone head-up display device. However, those skilled in the art should understand that the VR device 130 is not limited thereto, and the VR device 130 may also be an all-in-one head-up display device. The all-in-one head-up display device itself has a display screen, so that it is not necessary to connect the all-in-one head-up display device with the external display device. For example, in this example, if the all-in-one head-up display device is used as the VR device, the display device 120 may be omitted. At this point, the all-in-one head-up display device is configured to obtain video data from the server 100 and to perform playback operation, and the all-in-one head-up display device is also configured to detect a user's current viewing angle changing information and to adjust the playback operation according to the viewing angle changing information.

FIG. 2 is a flowchart diagram showing a method used in the VR playback system of FIG. 1. The method includes the following steps.

In step S10, a video data processing procedure is operated on the server.

In step S20, the display device obtains relevant information by interacting with the VR device.

In step S30, according to the relevant information, the display device requests the server to provide video data and receives the video data.

In step S40, the display device renders the received video data.

In this step, the video data obtained from the server is used to draw an image, i.e., the video data is played.

FIG. 3 is a flowchart diagram of a method for real-time rendering displaying virtual reality (VR) using head-up display devices according to an embodiment of the present disclosure. The method implements playing the video data in binocular mode. The method includes following steps.

In step S100, relevant parameters are obtained.

For example, the relevant parameters are calculated based on specification of a head-up display device and a screen size. The relevant parameters include parameters for field of view of left and right lenses, a camera matrix, a projection matrix, a model matrix and a center position of lens distortion. Referring to FIG. 4, FIG. 4 is an example diagram of a head-up display device. As shown in the figure, the head-up display device includes a stand and left and right lenses on the stand, and human eyes obtains images from left and right view areas through the left and right lenses. Because the left and right view areas provide images with difference, human mind, ater obtaining the information with difference, produces a three-dimensional sense. Different type of head-up devices have different specification and parameters, generally, the specification and parameters can be obtained by querying websites or querying built-in parameter files, and then the relevant parameters required in rendering process can be calculated in accordance with the specification and parameters.

In step S200, a 3D model is built, and the original coordinate data of the 3D model is obtained.

In this step, a suitable 3D model can be created in accordance with requirements. For example, a polygonal sphere can be created as the 3D model and the original coordinate data can be obtained based on the polygonal sphere.

In step S300, first coordinate data is obtained in accordance with the relevant parameters and the original coordinate data of the 3D model.

In step S400, lens distortion is performed on the first coordinate data based on the center position of lens distortion to obtain second coordinate data.

In step S300, vector calculation on the original coordinate data is performed in accordance with the camera matrix, the projection matrix and the model matrix to obtain the calculated coordinate data as the first coordinate data, and in step S400, the first coordinate data is further distorted to obtain the second coordinate data.

In step S500, the second coordinate data is rasterized to obtain pixel information.

In this step, the second coordinate data is processed into pixel information on a plane.

In step S600, an image is drawn based on a VR video data and the pixel information.

In the step, the VR video data downloaded from the server is decoded to obtain the pixel information therein, the pixel information are assigned in accordance with the pixel information, and finally the image is drawn.

In the embodiment, the original coordinate data in the 3D model is lens distorted and then the pixel information is assigned to the distorted coordinate data, so as to achieve binocular-mode viewing effects, because the lens distortion is performed during the time period of treatment of the 3D model, the video and binocular-mode rendering are implemented in one processing, which is equivalent to doubling the rendering efficiency of the existing scheme. Further, because the original coordinate data in the 3D model is lens-distorted in accordance with the relevant parameters obtained based on the information such as specification of the head-up display device, the screen size, and the like, the lens distortion effect can be adjusted by adjusting the relevant parameters to achieve better rendering effect.

In a preferred embodiment, in order to prevent a user from being dizzy due to immersive viewing, the above method further includes: obtaining real-time data of the gyros and performing data smoothing and corner prediction while the VR video data is played to achieve anti-shake.

In another preferred embodiment, the above method further includes adding a blackout mask. The blackout mask can be seen in FIG. 6, adding the blackout mask can improve immersive effect of VR viewing.

It should be noted that some steps described in the embodiments of the present disclosure may be implemented by calling relevant interfaces providing by OpenGL and/or WebGL. However, corresponding functions of OpenGL and WebGL are mainly implemented by the display chip, and calculation operations of the relevant parameters such as the projection matrix and the camera matrix are performed by computer processor, and thus, when the projection matrix and the camera matrix are transferred to openGL and/or WebGl, data transmission is required. The details can be understood with reference to FIG. 6.

FIG. 5 is a specific flowchart diagram showing the step of obtaining relevant parameters mentioned in the method for real-time rendering displaying virtual reality (VR) using head-up display devices described in FIG. 3. The method includes following steps.

In step S101, the parameters such as field of view and the like are obtained according to the specification of the head-up display device and the screen size.

In step S102, the eye distance parameter is obtained according to the specification of the head-up display device.

In step S103, the model matrix is obtained.

In step S104, the camera matrix is calculated.

In step S105, the center position of lens distortion is calculated.

In step S106, the projection matrix is calculated.

The center position of lens distortion and eye distance can refer to FIG. 4.

To further explain the above steps, a specific calculation step is provided in the following example.

Table 1 is a variable definition table.

TABLE 1
variable         meaning
Px, y, zoriginal original coordinate of each point in the 3D model
Px, y, zMVP      calculated coordinate
mat4model        model matrix
mat4view         camera matrix
mat4projection   projection matrix

The first coordinate data can be calculated by the following equation:

1) The camera matrices for left and right eyes in binocular mode can be calculated by equations (1) to (4):

$\begin{array}{cc}\mathrm{half\_eye}\mathrm{\_ipd}=\frac{\mathrm{eye\_ipd}}{2}& \left(1\right)\\ \mathrm{translate}\left(X,Y,Z\right)=\left[\begin{array}{cccc}1& 0& 0& X\\ 0& 1& 0& Y\\ 0& 0& 1& Z\\ 0& 0& 0& 1\end{array}\right]& \left(2\right)\\ \mathrm{left\_view}\mathrm{\_matrix}=\mathrm{translate}\left(-\mathrm{half\_eye}\mathrm{\_ipd},0,0\right)*\mathrm{mat}4_{\mathrm{view}}& \left(3\right)\\ \mathrm{right\_view}\mathrm{\_matrix}=\mathrm{translate}\left(\mathrm{half\_eye}\mathrm{\_ipd},0,0\right)*\mathrm{mat}4_{\mathrm{view}}& \left(4\right)\end{array}$

Among them, mat4_view represents a camera matrix, which can be generated directly in accordance with rotation angles of a gyro, left_view_matrix and right_view_matrix are respectively camera matrices for left and right eyes, eye_ipd represents the eye distance parameters;

2) the projection matrix mat4_projection in binocular mode is calculated by using equation (5):

$\begin{array}{cc}\mathrm{mat}4_{\mathrm{projection}}=\left[\begin{array}{cccc}\frac{2}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}& 0& -\frac{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)-\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{left}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{right}}\right)\end{array}}& 0\\ 0& \frac{2}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}& -\frac{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)-\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}{\begin{array}{c}\tan \left({\mathrm{fov}}_{\mathrm{up}}\right)+\\ \tan \left({\mathrm{fov}}_{\mathrm{down}}\right)\end{array}}& Y\\ 0& 0& \frac{\mathrm{far}}{\mathrm{near}-\mathrm{far}}& \frac{\mathrm{far}*\mathrm{near}}{\mathrm{near}-\mathrm{far}}\\ 0& 0& -1& 0\end{array}\right]& \left(5\right)\end{array}$

where, fov_left, fov_right, fov_up, fov_down, far,near represent parameters relevant to field of view in binocular mode.

3) the model matrix mat4_model is to be a unit matrix;

4) the first coordinate data P_x,y,z^MVP is calculated by using equation (6):

P_x,y,z^MVP=mat4_model*mat4_view*mat4_projection*P_x,y,z^original  (6)

P_x,y,z^MVP represents the first coordinate data, P_x,y,z^original represents the original coordinate data, mat4_model represents the model matrix, and mat4_projection represents the projection matrix, the camera matrices left_view_matrix and right_view_matrix for left and right eyes are respectively provided into the equation (6) instead of mat4_view to obtain the first coordinate data P_x,y,z^MVP.

The above step S400 can refer to the calculation steps in the following example.

TABLE 2
variable meaning
(xd, yd) distorted image field coordinate after lens projection
(xu, yu) corrected image field coordinates (i.e., using an ideal pinhole
         camera)
(xc, yc) distortion center coordinates (ie, the center position of lens
         distortion according to the disclosure)
Kn       the nth tangential distortion coefficient
Pn       the nth tangential distortion coefficient
r        the distance from pixeles to optical axis

1) distorted parameters is obtained based on the specification of lens of the head-up display device:

K₁,K₂  (7)

and it coordinates with an auxiliary equation (8) to obtain the distorted parameters and anti-distorted parameters,

$\begin{array}{cc}f\left(K_1,K_2\right)=\{\begin{array}{c}{K}_1^{-1}=-K_1\\ K_2^{-1}=3K_1^2-K_2\end{array};& \left(8\right)\end{array}$

2) lens distortion is performed based on the Brown model.

x_u=x_d+(x_d−x_c)(K₁r²+K₂r⁴+ . . . )+(P₁(r²+2(x_d−x_c)²+2P₂(x_d−x_c)(y_d−y_c))(1+P₃r²+P₄r⁴+ . . . )  (9)

y_u=x_d+(y_d−y_c)(K₁r²+K₂r⁴+ . . . )+(2P₁(x_d−x_c)(y_d−y_c)+P₂(r²+2(y_d−y_c)²))(1+P₃r²+P₄r⁴+ . . . )  (10)

When without tangential distortion correction, all items containing p can be removed.

The coordinate of the center position of lens distortion can be solved by the following equations.

For the two coordinate points (x_l,y_l) and (x_h,y_h) on a plane, the linear interpolation between two vectors can be performed in accordance with t by using the equation lerp.

$\begin{array}{cc}\mathrm{lerp}\left(t,x_l,x_h,y_l,y_h\right)=y_l+\left(t-x_l\right)\frac{y_h-y_l}{x_h-x_l}& \left(11\right)\end{array}$

by using the following equations, the coordinate (x_center^window_pixel,y_center^window_pixel) of the center position of lens distortion center can be solved according to the projection matrix mat4_projection and screen size width_widow*height_widow, where the coordinates (x_center^normal,y_center^normal) is a point in the space coordinate axis of [−1,1].

(x_center^normal,y_center^normal)=mat4_projection*[00-10]

x_center^window_pixel=lerp(x_center^normal,−1,1,0,width_widow)

y_center^window_pixel=lerp(y_center^normal,−1,1,0,height_widow)  (12)

The steps of data smoothing and corner prediction in the above embodiments can refer to the following description.

TABLE 3
variable meaning
θt       fusion rotation angle based on time t
k        fusion weight constant
ω        ω is an angular velocity read by an accelerometer ( the
         accelerometer is provided on the head-up display device)
Ø        angles read from the gyros
Δt       difference between the output time moment and the
         previous time moment

The equation for data smoothing is:

θ_t+1=k(θ_t+ωΔt)+(1−k)Ø  (13)

TABLE 4
variable        meaning
θt              fusion rotation angle based on time t
angularSpeed    an angular velocity read by an accelerometer
                (for example, the accelerometer is
                provided on the head-up display device))
predictionTimeS prediction time (a constant)
β               threshold value for rotation prediction

The equation for corner prediction is

$\begin{array}{cc}{\theta }_{\Delta }=\{\begin{array}{c}\mathrm{angularSpeed}*\mathrm{predictionTimeS},\mathrm{angularSpeed}\ge \beta \\ 0,\mathrm{angularSpeed}\in \left[0,\beta \right]\\ \mathrm{null},\mathrm{other}\end{array}& \left(14\right)\\ {\theta }_{t+1}={\theta }_t+{\theta }_{\Delta }& \left(15\right)\end{array}$

FIG. 7 is a schematic diagram of a system for real-time rendering displaying virtual reality (VR) using head-up display devices, according to an embodiment of the present disclosure.

The system includes a parameter calculating unit 701, a model building unit 702, a coordinate calculating unit 703, a lens distortion unit 704, a rasterization unit 705, and an image drawing unit 706.

The parameter calculating unit 701 configured to obtain relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion.

The model building unit 702 is configured to create a 3D model and obtain an original coordinate data of the 3D model. For example, the 3D model can be created based on WebGL and initialized to obtain UV coordinates.

The coordinate calculating unit 703 is configured to obtain first coordinate data according to the relevant parameters and the original coordinate data of the 3D model. The first coordinate data is obtained by performing calculation based on the relevant parameters and the original coordinate data of the 3D model.

The lens distortion unit 704 configured to perform lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data. That is, the first coordinate data is distorted according to the center positions of the left and right lenses to obtain the second coordinate data.

The rasterization unit 705 configured to rasterize the second coordinate data to obtain a pixel information.

The image drawing unit 706 configured to an image based on a VR video data and the pixel information.

According to the embodiment of the present disclosure, the binocular-mode VR immersive viewing effect is achieved by performing lens distortion on the coordinate data of the 3D model. Because the lens distortion on the coordinate data is performed in the 3D model, video and immersive rendering can be realized in one processing, thereby improving rendering efficiency.

Although the embodiments of the present disclosure have been described above with reference to the preferred embodiments, it is not intended to limit the claims. Any modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the present disclosure, Therefore, the protection scope of the present disclosure should be based on the scope of the claims of the present disclosure.

The foregoing descriptions of specific embodiments of the present disclosure have been presented, but are not intended to limit the disclosure to the precise forms disclosed. It will be readily apparent to one skilled in the art that many modifications and changes may be made in the present disclosure. Any modifications, equivalence, variations of the preferred embodiments can be made without departing from the doctrine and spirit of the present disclosure.

Claims

1. A method for real-time rendering displaying virtual reality (VR) using head-up display devices, comprising:

obtaining relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion;

creating a 3D model and obtaining an original coordinate data of the 3D model;

obtaining first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model;

performing lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data;

rasterizing the second coordinate data to obtain pixel information; and

drawing an image in accordance with a VR video data and the pixel information.

2. The method according to claim 1, wherein the step of obtaining relevant parameters comprises:

obtaining parameters relevant to field of view in accordance with specification of a head-up display device and a screen size;

calculating the center position of lens distortion in accordance with the parameters relevant to field of view; and

calculate the projection matrix in accordance with the parameters relevant to field of view.

3. The method according to claim 1, wherein the step of obtaining relevant parameters comprises:

obtaining an eye distance parameter based on specification of a head-up display device; and

calculate the camera matrix in accordance with the eye distance parameter.

4. The method according to claim 1, wherein the camera matrix and the projection matrix are adjusted to achieve binocular-mode viewing effects.

5. The method according to claim 1, wherein the step of obtaining first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model comprising: half_eye  _ipd = eye_ipd 2 ( 1 ) translate  ( X, Y, Z ) = [ 1 0 0 X 0 1 0 Y 0 0 1 Z 0 0 0 1 ] ( 2 ) left_view  _matrix = translate  ( - half_eye  _ipd, 0, 0 ) * mat   4 view ( 3 ) right_view  _matrix = translate  ( half_eye  _ipd, 0, 0 ) * mat   4 view ( 4 ) mat   4 projection = [ 2 tan  ( fov left ) + tan  ( fov right ) 0 - tan  ( fov left ) - tan  ( fov right ) tan  ( fov left ) + tan  ( fov right ) 0 0 2 tan  ( fov up ) + tan  ( fov down ) - tan  ( fov up ) - tan  ( fov down ) tan  ( fov up ) + tan  ( fov down ) Y 0 0 far near - far far * near near - far 0 0 - 1 0 ] ( 5 )

calculating out camera matrices for left and right eyes in binocular mode by equations (1) to (4):

wherein, left_view_matrix and right_view_matrix represent respectively a camera matrix for left eye and a camera matrix for right eye, mat4_view is the camera matrix which can be generated directly in accordance with rotation angles of a gyro, and eye_ipd represents the eye distance parameter;

calculating out the projection matrix mat4projection in binocular mode by equation (5),

wherein fovleft, fovright, fovup, fovdown, far, near represent the parameters relevant to field of view;

setting mat4model to be an identity matrix;

calculating out the first coordinate data Px,y,zMVP by equation (6), Px,y,zMVP=mat4model*mat4view*mat4projection*Px,y,zoriginal  (6)

wherein, Px,y,zMVP represents the first coordinate data, Px,y,zoriginal represents the original coordinate data, mat4model represents the model matrix, and mat4projection represents the projection matrix, the camera matrices left_view_matrix and right_view_matrix for left and right eyes are respectively provided into equation (6) instead of mat4view to obtain the first coordinate data Px,y,zMVP.

6. The method according to claim 1, wherein the step of performing lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data comprises: K 1, K 2 ( 7 ) f  ( K 1, K 2 ) = { K 1 - 1 = - K 1 K 2 - 1 = 3  K 1 2 - K 2; ( 8 )

obtaining distortion parameters in accordance with following equations (7) and (8):

obtaining corrected image field coordinates (xu,yu) as the second coordinate data in accordance with the distortion parameters by using equations (9) and (10) in which all items containing p can be removed when tangential distortion correction is not performed, xu=xd+(xd−xc)(K1r2+K2r4+... )+(P1(r2+2(xd−xc)2+2P2(xd−xc)(yd−yc))(1+P3r2+P4r4+... )  (9) yu=xd+(yd−yc)(K1r2+K2r4+... )+(2P1(xd−xc)(yd−yc)+P2(r2+2(yd−yc)2))(1+P3r2+P4r4+... )  (10)

wherein (xd,yd) is distorted image field coordinates after lens projection, ie, the first coordinate data, (xu,yu) is the corrected image field coordinates, (xc,yc) is the center position of lens distortion, Kn is a nth radial distortion coefficient, Pn is a nth tangential distortion coefficient, r is a distance from pixels to optical axis.

7. The method according to claim 1, wherein coordinate for the center position of lens distortion is obtained by the following steps, lerp  ( t, x l, x h, y l, y h ) = y l + ( t - x l )  y h - y l x h - x l ( 11 )

performing linear interpolation between two vectors based on t using following equation:

wherein, (xl,yl) and (xh,yh) are two coordinate points in a plane;

calculating coordinate (xcenterwindow_pixel,ycenterwindow_pixel) of the center position of lens distortion in according to the projection matrix mat4projection and the screen size widthwidow*heightwidow by using following equations: (xcenternormal,ycenternormal)=mat4projection*[00-10] xcenterwindow_pixel=lerp(xcenternormal,−1,1,0,widthwindow) ycenterwindow_pixel=lerp(ycenternormal,−1,1,0,heightwindow)  (12)

wherein the coordinate (xcenternormal,ycenternormal) is a point in the space coordinate axis of [−1,1].

8. The method according to claim 1, further comprising: adding a blackout mask.

9. The method according to claim 1, further comprising: acquiring real-time data from a gyros, and performing data smoothing and corner prediction while the VR video data is played to achieve anti-shake.

10. The method according to claim 1, wherein the equation used for performing data smoothing is θ Δ = { angularSpeed * predictionTimeS, angularSpeed ≥ β 0, angularSpeed ∈ [ 0, β ] null, other ( 14 ) θ t + 1 = θ t + θ Δ ( 15 )

θt+1=k(θt+ΩΔt)+(1−k)Ø  (13)

where, θt is a fusion rotation angle based on time t, k is a fusion weight constant, and ω is an angular velocity read by an accelerometer, Ø is an angle read from the gyros, Δt is a difference between an output time moment and its previous time moment;

equations used for corner prediction is:

wherein θt is a fusion rotation angle based on time t, angularSpeed is an angular velocity read by the accelerometer, predictionTimeS is a prediction time constant, and β is a rotation prediction threshold, the gyros and the accelerometer are provided on a head-up display device.

11. The method according to claim 1, further comprising: using relevant interfaces provided by OpenGL and WebGL to complete corresponding steps.

12. A system for real-time rendering displaying virtual reality (VR) using head-up display devices, comprising:

a parameter calculating unit configured to obtain relevant parameters including a camera matrix, a projection matrix, a model matrix and a center position of lens distortion;

a model building unit configured to create a 3D model and obtain original coordinate data of the 3D model;

a coordinate calculating unit configured to obtain first coordinate data in accordance with the relevant parameters and the original coordinate data of the 3D model;

a lens distortion unit configured to perform lens distortion on the first coordinate data based on the center position of lens distortion to obtain second coordinate data;

a rasterization unit configured to rasterize the second coordinate data to obtain pixel information;

an image drawing unit configured to draw an image based on a VR video data and the pixel information.