Multi-rate real-time players
A method of optimizing the scheduling of the drawing of graphical elements of a multi-player display (102) in an image processing environment. Each player (801 to 804) is capable of operating asynchronously and deriving its source from a different video, film or other image format. Draw commands from each player are combined into short sequences known as graphics requests (1002 to 1004). A scheduler (821) selects the next graphics request for rendering by a graphics thread (831) based on the requirement to minimize state changes within the graphics card (208) while meeting the due time requirement of the graphics request, that is dependent upon the frame rate.
Latest Autodesk Canada Inc. Patents:
- Editing video data
- Image data processing with color modification
- Data processing apparatus generates initialization schedule by sorting module order according to number of its dependencies for initializing data affecting execution of operational instructions
- Processing pipeline responsive to input and output frame rates
- Caching data
1. Field of the Invention
The present invention relates to rendering a plurality of asynchronous players on a single visual display, wherein each player is capable of rendering moving images at an arbitrary frame rate.
2. Description of the Related Art
The increasing use of digital systems to process images has resulted in the availability of systems that are capable of processing a wide variety of image types, including moving images from video and film, and virtual worlds generated from animations of three-dimensional geometric data. In a system that only has to deal with one particular data type, for example, video frames at a rate of thirty frames per second, it is possible to implement digital processing circuitry that operates synchronously with the frame rate, and considerable optimisations are thereby obtained.
As digital systems become increasingly complex, it has become customary to display several images on a visual monitor at the same time. These images are typically single frames from several clips, whose contents are to be edited together to form a final scene. Sometimes these clips are from several cameras viewing the same scene, and it is known to display moving images from several synchronous clips at the same time, so a director can make a best choice as to which camera angle to use without viewing each one separately.
In a typical compositing environment, clips may be derived from several diverse sources, including various film and video formats, having different frame rates and resolutions. Known image processing systems require that these be converted into a common format before they can be freely viewed, mixed and edited together during the video editing process. Displaying clips having different frame rates, in particular, is not done, due to excessive demands that would be made upon the graphics rendering process in order to achieve this.
Graphics rendering is typically performed by a highly optimised graphical signal processing circuit. This has many levels of pipelining in order to ensure that maximum performance is achieved for a given amount of silicon circuitry. However, switching between clips in different formats requires that pipelines in the graphics circuitry are cleared, and this will incur a prohibitive performance penalty on systems that attempt to freely display several different formats of image data. Furthermore, when several players are used to display clips having different frame rates, the necessity to switch between the different players rapidly increases this pipeline clearing overhead, making such a system impractical or very expensive to implement. In known systems, therefore, only one format is displayed at a time. Multiple formats can be represented by still frames, and these may then be previewed at high quality by adjusting the system to deal exclusively with that format while it is being viewed or edited. High quality combination of image types on the same display is only done by explicit conversion into a common format prior to editing.
BRIEF SUMMARY OF THE INVENTIONIt is an aim of the present invention to provide an improved method of simultaneously displaying multiple clips, each having potentially a different frame update rate, and thus providing an efficient mean of performing computational load balancing by establishing priority between the clips.
The invention will now be described by way of example only with reference to the accompanying drawings.
A system for processing image and other data is illustrated in
Instructions controlling the processing system 101 may be installed from a physical medium such as a CDROM disk 106, or over a network, including the Internet. These instructions enable the processing system 101 to interpret user commands from the keyboard 104 and the graphics tablet 105, such that moving image data, and possibly other media data, may be viewed, edited and otherwise processed to define a scene. A short sequence of video frames, for example from one particular camera, is known as a clip. A scene may be constructed from a number of clips, usually arranged sequentially, although certain effects may combine two or more clips simultaneously. Typically, the video material will include a mono or stereo soundtrack, which may be heard through loudspeakers contained in the video monitor 102 or through headphones. Other data, including camera motion tracking data, may be supplied to the processing system 101 via various hardware interfaces. Scene data, including clips and sound, may be loaded via a high capacity network, stored on the RAID 103, and then manipulated in real time in order to facilitate high quality preview and rendering of the final results of image processing.
The processing system 101 shown in
The memory controller 205 further facilitates connectivity between the aforementioned components of the processing system 101 and a high bandwidth non-blocking crossbar switch 207. The switch makes it possible to provide a direct high capacity connection between any of several attached circuits. These include a graphics card 208. The graphics card 208 generally receives instructions from the processors 201 and 202 to perform various types of graphical image rendering processes, resulting in images, clips and scenes being rendered in real time on the monitor 102. A high bandwidth SCSI bridge 209 provides an interface to the RAID 103, and also, optionally, to a Digital Data Tape recorder and playback apparatus. A high bandwidth SDI bridge 214 provides an interface to a Digital Video Tape recorder and playback apparatus. Digital tape is a commonly used medium for the storage of high quality video or cinematic clips. Images of this nature are typically transferred locally from a digital tape player or via a high speed network, onto the RAID 103, prior to image processing being performed. Once image processing has been completed, the resulting clips and scenes are transferred back to digital tape.
A second SCSI bridge 210 facilitates connection between the crossbar switch 207 and a CDROM drive 211. The CDROM drive provides a convenient way of receiving large quantities of instructions and data, and is typically used to install instructions for the processing system 101 onto a hard disk drive 212. Once installed, instructions located on the hard disk drive 212 may be fetched into main memory 206 and then executed by the processors 201 and 202. Furthermore, the hard disk drive may store important data structures representing scene data and or ways of manipulating video or film clips, which may be updated from one image processing session to the next. An input output (I/O) bridge 213 provides an interface for the graphics tablet 105 and the keyboard 104, through which the user is able to provide instructions to the processing system 101.
While processing image data using the system shown in
In the present embodiment, the image processing system shown in
An example of multiple players used for compositing is shown in FIG. 3. The monitor 102 has a display area that contains three players 301, 302 and 303. Each of these players provides a different view onto various aspects of a scene's construction. The first player 301 is being used by the operator to view high quality video frames at a rate of thirty frames per second. Transport controls are provided for rewind 311, reverse play 312, forward play 313, fast forward 314 and stop 315. An SMPTE standard timecode display 316 provides an indication of elapsed time of the clip being displayed in the window. The second player 302 is being used by the user to define the trajectory of a curved surface onto which the image being shown by the first player 301 is being projected. The third player 303 is being used by the user to define the shape of the surface being used for the projection shown by the second player 302.
To save processing time, the user has decided to preview frames on the second player 302 at a rate of fifteen frames per second. The view of the third player 303 needs only to be updated as often as possible, when processing resources become available. In the present embodiment, user interface components such as tape transport controls 311 to 316 may be considered as a degenerate form of player, which may have an update rate independent of the other players. Usually it is preferable to update user interface (UI) components (commonly referred to as widgets) at as high a rate as possible, but without interfering with the quality of an output render process.
The ability to display several different views asynchronously on the same screen provides many advantages to the user of such a system. However, it is extremely important that the results of image processing operations performed in this environment should be viewable at a high level of quality. High quality is essential if correct edit decisions are to be made. If the extra processing required by placing several asynchronous players on the same display becomes too high, this will result in the processing system 101 not being able to keep up with the demands being placed on it, resulting in dropped frames, and a significant loss in the quality of rendered images. If the quality of images cannot be maintained, the advantages of placing several players on the screen at the same time are outweighed by the inability of the operator to reliably view the results of his or her editing decisions.
Operations of the image processing system shown in
Image processing is performed at step 404. This includes compositing actions, editing, additional import and export of image and other media types, and generally all those steps that are required to form a finished clip or scene. At step 405 the finished clips are stored back onto digital tape and at step 406 the system is closed down and switched off.
The graphics card 208, first shown in
In order to meet the demands of common image processing requirements, the OpenGL standard has been developed. This defines a set of methods for controlling drawing processes that can be performed on a wide variety of graphics hardware. Interpretation of OpenGL drawing commands is performed by an OpenGL state machine. This may be implemented either by instructions executing on the CPUs 201 and 202, or, as in the preferred embodiment, in dedicated hardware components optimally configured to render OpenGL commands, located in the graphics card itself.
The OpenGL state machine is summarised in FIG. 6. The hardware components of the graphics card shown in
The OpenGL state machine performs processing upon two fundamental types of data. These are vertex data 601 and pixel data. Vertex data 601 defines points that are to be joined by lines and possibly areas that are to be shaded in some way. Pixel data 602 includes raw image data such as that supplied from digitised film clips, and also texture maps. The full details of the various components 601 to 610 of the OpenGL state machine may be found in chapter 1 of the OpenGL Programming Guide, Third Edition, published by Addison-Wesley, ISBN 0-201-60458-2, as well as several widely available sources on the Internet. Each of these components 601 to 610 has associated with it certain state variables, which define how that stage of the state machine is to operate. For example, a particular state variable may define whether or not to apply Gouraud shading to an area. A complete set of the state variable for the state machine shown in
Efficient hardware implementations of the OpenGL state machine shown in
In a typical OpenGL hardware implementation, pipelining techniques are extensively used. Once all the state variables are set correctly, and the assembly-line process is functioning in a predictable way, the efficiencies brought to graphics rendering by this technique are considerable. However, when one or several state variables are changed, the conditions of the assembly line are modified, and this requires a process of flushing a pipeline. As the depth of a pipeline is not fixed, and the pipeline may use first-in-first-out buffers, this flushing task can take a variable amount of time and a potentially large corresponding number of CPU clock cycles. If several concatenated pipelines have to be cleared, the performance penalty of pipeline flushing can be very severe. For this reason it is preferable to minimise the amount of changes that are made to the OpenGL state machine during the process of graphics rendering.
OpenGL was developed with the intention of improving the efficiency of rendering an image or a sequence of moving images. Once the state variables are correctly set, an OpenGL optimised graphics card can perform with extremely high efficiency, and render complex photorealistic scenes in real time. However, in the known art, if multiple players were displayed simultaneously on the same screen 102, using the same graphics card 208, the differing state requirements from player to player would result in the requirement to change OpenGL state variables, or possibly even complete OpenGL contexts at a rate that would significantly impair the capability of such a system to render moving images of a high quality. This reduction in efficiency would be sufficient to prevent such an arrangement being useful in a competitive image processing environment.
OpenGL is a widely adopted standard for communicating with Graphics cards, and as a result many graphics card are designed to implement the OpenGL state machine using customised silicon circuitry specifically tailored to the requirements of the OpenGL state machine. However, any efficient hardware implementation of graphics rendering functions will employ a level of pipelining, and will thus benefit from the invention that will be described herein. Furthermore, it is known to perform OpenGL rendering using software running on the main CPUs 201 and 202. Under these conditions the level of pipelining is typically much reduced. Nevertheless, the invention will provide benefits even when most of the rendering processes are performed on the general purpose CPUs of the processing system 101.
The contents of the main memory 206 shown in
The invention is summarised in FIG. 8. This summarises the objects that are created as a result of executing the image processing instructions 702. The three players are represented by instantiated objects 801, 802 and 803 that execute as separate threads. Each player thread 801 to 803 supplies graphics requests to a respective queue 811, 812 and 813. The queues 811 to 813 are interrogated by a scheduler 821, that decides which of the queues shall be serviced next. The UI thread 804 and the UI queue 814 are treated in the same way as the players 801 to 803 and queues 811 to 813. Once the scheduler has decided which queue to service next, it removes the next graphics request from that queue, and supplied this to an OpenGL thread.
Instructions for the scheduler include instructions for identifying the existing conditions of the graphics rendering process 831, comparing required conditions of queued graphics requests from the players 801 to 804 with the existing conditions, and scheduling a preferred graphics request in response to the comparison of conditions. These conditions are, in the present embodiment, the set of state variables of the OpenGL state machine. However, these could be any set of conditions that, when changed, reduce the efficiency of a graphics rendering process. The preferred graphics request is selected in response to a requirement for minimising state changes in the graphics pipeline. This is a form of scheduling, in that from the plurality of queued graphics requests from the players 801 to 804, a single graphics request is selected as being the next to be rendered. Thus, the order in time in which graphics requests are rendered is controlled by the scheduler, in order to minimise the state changes that are required for the requests to be processed.
A graphics request is a small collection of OpenGL draw instructions. In order to render a frame of a complex three dimensional scene, many hundreds of graphics requests would be required. By splitting up the task of drawing a frame of image data into small bundles of graphics commands, the scheduler gets the chance to continuously switch between players in order to optimally choose which graphics request to schedule as being next. Once the preferred graphics request has been scheduled in this way, it is removed from its respective queue 811 to 814 and supplied to the OpenGL thread 831.
Details of the scheduler 821 shown in
The player strategy is applied to the contents of a queue 811 to 814. In its simplest possible method of operation, each queue is a first-in-first-out (FIFO) buffer. However, re-ordering of queued graphics requests, even from the same player, can sometimes be beneficial, and so the player strategy can, if appropriate, cause each queue to yield an appropriate next graphics request, that is not necessarily the same as that which would be supplied by a FIFO buffer.
At step 903 the required state and estimated duration of the graphics request obtained at step 902 are identified. At step 904 a question is asked as to whether there is another player that remains for consideration. If so, control is directed to step 901. Alternatively control is directed to step 905.
At step 905 a scheduler strategy is applied to identify the player having the a preferred graphics request. Effectively this identifies the preferred graphics request from all the players 801 to 804. The scheduler strategy is a strategy distinct from the player strategy, but which operates in a similar way. At step 906 the preferred graphics request is removed from the queue of the player identified at step 905. In summary, graphics requests within a player's queue are sorted according to a player strategy. Each player thereby has a graphics request identified from its queue, that it prefers to be processed next. The scheduler strategy compares each of the graphics requests identified in this way, and also considers the current state of the OpenGL state machine. A graphics request is then removed from a player in accordance with the priority measured by the scheduler strategy. Once this has been done, control is directed back to step 901. Usually there will be a limit to the size of the queue for each player, and so a condition will eventually be reached either where some or all of the queues are empty for a portion of each monitor frame, or, alternatively, where the queues become full and the graphics card 208 is having difficulty in keeping up with the demand placed on it by the players 801 to 804.
The graphics queue 811 shown in
The due time 1011 is the frame number, offset from the present frame, by which the graphics request should be drawn in order to maintain the correct frame rate for the player. For example, if the monitor's refresh rate is one hundred hertz, and the frame rate for a player is twenty-five hertz, then due times will have values such as 02, 06, 10, 14 and so on, occurring at one in every four frames. If the due time for a particular graphics request is a long way off, then this makes it possible for the scheduler to select a graphics request that is more urgent, even if this requires a significant change of OpenGL state or a full context switch. The due time may be expressed as a frame number, or as being as-soon-as-possible, or whenever there is some spare processing capacity available. User interface widgets usually require updates as fast as possible, whereas some players may only need to be updated as a background process whenever there is some spare processing time on the graphics card.
The required state 1012 specifies the state that is required by the graphics request. This may be compared with the current state of the OpenGL thread 831, and a graphics request requiring the least amount of state change can be selected. The estimated duration 1013 is the estimated time for the graphics request to execute on the graphics card. This information can be used to optimise the scheduling of a graphics request under certain conditions. The graphics request itself is shown at 1014. This comprises a short sequence of OpenGL drawing commands.
The player strategy applied at step 902 in
Data flow associated with the player strategy 902 is summarised in FIG. 12. From each graphics request 1002, 1003, 1004 in a queue, the required state 1012 and due time 1011 are identified and supplied to the player strategy 902. The player strategy 902 also received the current state 1201 of the OpenGL state machine as an input. These data items are compared and this results in a single graphics request being selected from the queue, which is identified by means of a pointer 1202. This strategy is an example. Different types of player may be given different strategies, for example a user interface player may be given a different strategy 902 from a normal player. In the present embodiment the strategies are the same. However, it will be understood by those skilled in the art, that several differing strategies may be applied according to context, including selecting a different set of strategies that are optimised for a graphics card from a particular manufacturer. This also applies to the scheduler strategies to be described next.
The scheduler strategy applied at step 905 in
The data flow associated with the implementation of the scheduler strategy 905 is shown in FIG. 14. Each player 801 to 804 has an associated identified graphics request 1401, 1402, 1404. The estimated duration 1013, required state 1012 and due time 1011 of each of these identified graphics requests is supplied to the scheduler strategy 905, in combination with the current state 1201 of the OpenGL thread 831. The scheduler strategy selects a preferred graphics request on the basis of several comparisons, resulting in a pointer 1411 to a player identified as having the preferred graphics request in its queue. Once identified in this way, the scheduler 821 removes the preferred graphics from its queue and passes this on to the OpenGL thread 831.
The consequences and advantages of the invention are shown in the form of a timing diagram of graphics request processing performed by the graphics card 208. Player 801 has a required frame rate of thirty frames a second, and in this example it is assumed that the monitor refresh rate is sixty frames a second. Player 801 has a high priority and so graphics requests for player 801 tend to get rendered before any other graphics requests. Player 801 gets priority and its requests are shown in
The graphics requests are not of a fixed size, but they are always reasonably small, so that the scheduler 821 gets a chance to switch between players whenever this is appropriate. The invention facilitates multitasking of the graphics card 208 between multiple competing asynchronous players 801 to 804, and allocates its resources in accordance with a combination of due time and required state information. This minimises the unnecessary amount of state changes that are required to occur within the graphics card and its OpenGL state machine, while at the same time ensuring that multiple players having unrelated frame rates continue to update without interfering with each other. The players 801 to 804 may have similar or widely different required OpenGL states for their graphics requests, whichever condition arises, however, in response to user configuration of the players, the invention ensures that optimal use is made of the available graphics hardware.
The invention has been described without reference to double buffering, which is a technique that is used in all high quality image processing systems. Double buffering has been omitted from the description for the purpose of simplifying the description. However, those skilled in the art will understand that the preferred embodiment of the invention includes the use of double buffering techniques. This may be implemented in the following way. Once a player has supplied all the graphics requests required for a particular frame, the player sets a flag indicating that once all the queued requests for that frame have been passed to the graphics thread, then the front and back buffers for that player should be swapped. Each player has its own front and back buffers.
Claims
1. Apparatus for processing image data comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may operate asynchronously;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions include steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing required conditions of graphics requests from a plurality of said players with said existing conditions; and
- (c) scheduling a preferred graphics request in response to comparisons performed at step (b).
2. Apparatus for processing image data comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions include steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing said existing conditions with required conditions of player graphics requests;
- (c) selecting a preferred graphics request in response to said comparison of required conditions;
- (d) scheduling a preferred player by processing the due time of said preferred graphics requests of the players with said required and existing conditions; and
- (e) transferring a preferred graphics request from said scheduled player to said graphics rendering process.
3. Apparatus according to claim 1 or claim 2, wherein said players include a player for drawing user interface widgets.
4. Apparatus according to claim 2, wherein said due time for a frame is defined by its frame rate.
5. Apparatus according to claim 2, wherein said graphics rendering process is multi-threaded.
6. Apparatus according to claim 5, wherein individual threads in said graphics rendering process are allocated graphics requests in response to a load balancing strategy.
7. Apparatus according to claim 2, wherein said selecting step is performed so as to minimise the required change of state in said graphics rendering process.
8. Apparatus according to claim 2, wherein said scheduling instructions include frame control strategy instructions and or data specific to a hardware implementation of the graphics rendering process.
9. Apparatus for processing image data comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- (a) each frame for a player is defined by multiple graphics requests;
- (b) graphics requests are selected from players so as to minimise state changes while switching the rendering process from player to player; and
- (c) players running at higher frame rates have their graphics requests selected more frequently than those running at lower frame rates.
10. Apparatus according to claim 9, wherein said scheduling instructions include instructions such that each player is allocated a strategy for prioritising its graphics requests.
11. A method of processing image data in an image processing system comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may operate asynchronously;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions facilitate steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing required conditions of graphics requests from a plurality of said players with said existing conditions; and
- (c) scheduling a preferred graphics request in response to comparisons performed at step (b).
12. A method of processing image data in an image processing system comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions facilitate steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing said existing conditions with required conditions of player graphics requests;
- (c) selecting a preferred graphics request in response to said comparison of required conditions;
- (d) scheduling a preferred player by processing the due time of said preferred graphics requests of the players with said required and existing conditions; and
- (e) transferring a preferred graphics request from said scheduled player to said graphics rendering process.
13. A method according to claim 11 or claim 12, wherein said players include a player that draws user interface widgets.
14. A method according to claim 12, wherein said due time for a frame is defined by its frame rate.
15. A method according to claim 12, wherein said graphics rendering process is multi-threaded.
16. A method according to claim 15, wherein individual threads in said graphics rendering process are allocated graphics requests in response to a load balancing strategy.
17. A method according to claim 12, wherein said selecting step is performed so as to minimise the required change of state in said graphics rendering process.
18. A method according to claim 12, wherein said scheduling instructions include frame control strategy instructions and or data specific to a hardware implementation of the graphics rendering process.
19. A method of processing image data in an image processing system comprising storage means for storing image processing instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- (a) each frame for a player is defined by multiple graphics requests;
- (b) graphics requests are selected from players so as to minimise state changes while switching the rendering process from player to player; and
- (c) players running at higher frame rates have their graphics requests selected more frequently than those running at lower frame rates.
20. A method according to claim 19, wherein each said player is allocated a strategy for prioritising its graphics requests.
21. A computer-readable medium having computer-readable instructions executable by a computer configurable for image processing, said computer including storage means for storing image processing instructions derived from said computer-executable instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may operate asynchronously;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions include steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing required conditions of graphics requests from a plurality of said players with said existing conditions; and
- (c) scheduling a preferred graphics request in response to comparisons performed at step (b).
22. A computer-readable medium having computer-readable instructions executable by a computer configurable for image processing, said computer including storage means for storing image processing instructions derived from said computer-executable instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- said scheduling instructions include steps of:
- (a) identifying existing conditions of said graphics rendering process;
- (b) comparing said existing conditions with required conditions of player graphics requests;
- (c) selecting a preferred graphics request in response to said comparison of required conditions;
- (d) scheduling a preferred player by processing the due time of said preferred graphics requests of the players with said required and existing conditions; and
- (e) transferring a preferred graphics request from said scheduled player to said graphics rendering process.
23. A computer-readable medium according to claim 21 or claim 22, wherein said players include a player for drawing user interface widgets.
24. A computer-readable medium according to claim 22, wherein said due time for a frame is defined by its frame rate.
25. A computer-readable medium according to claim 22, wherein said graphics rendering process is multi-threaded.
26. A computer-readable medium according to claim 25, wherein individual threads in said graphics rendering process are allocated graphics requests in response to a load balancing strategy.
27. A computer-readable medium according to claim 22, wherein said selecting step is performed so as to minimise the required change of state in said graphics rendering process.
28. A computer-readable medium according to claim 22, wherein said scheduling instructions include frame control strategy instructions and or data specific to a hardware implementation of the graphics rendering process.
29. A computer-readable medium having computer-readable instructions executable by a computer configurable for image processing, said computer including storage means for storing image processing instructions derived from said computer-executable instructions, memory means for storing said image processing instructions during their execution and image data in the form of scene data for one or a plurality of animated scenes, and processing means; wherein
- said image processing instructions include player instructions for generating graphics requests to render an animated scene, and said processing means is configurable by said image processing instructions to instantiate a plurality of players, each of which comprises player data and at least one pointer to said player instructions, such that each player may have a different frame rate;
- said processing means is configurable by said player instructions such that graphics requests are generated by each said player in response to the scene data associated with each respective player, in order to draw frames of scene data for the player at its respective frame rate; and
- said image processing instructions include scheduling instructions such that said processing means is configurable to supply graphics requests generated by said players to a graphics rendering process;
- (a) each frame for a player is defined by multiple graphics requests;
- (b) graphics requests are selected from players so as to minimise state changes while switching the rendering process from player to player; and
- (c) players running at higher frame rates have their graphics requests selected more frequently than those running at lower frame rates.
30. A computer-readable medium according to claim 29, wherein said scheduling instructions include instructions such that each player is allocated a strategy for prioritising its graphics requests.
4500908 | February 19, 1985 | Mandeberg |
4876600 | October 24, 1989 | Pietzsch et al. |
5317686 | May 31, 1994 | Salas et al. |
5805148 | September 8, 1998 | Swamy et al. |
6154213 | November 28, 2000 | Rennison et al. |
6172669 | January 9, 2001 | Murphy et al. |
6661422 | December 9, 2003 | Valmiki et al. |
6762762 | July 13, 2004 | MacInnis et al. |
6798420 | September 28, 2004 | Xie |
WO98/10586 | March 1998 | WO |
- Mason Woo et al., OpenGL Programming Guide, Third Edition, The Official Guide to Learning OpenGL, Version 1.2, Chapter 1: Introduction to OpenGL, pp. 1-25, ©1999.
Type: Grant
Filed: Feb 7, 2001
Date of Patent: Jun 21, 2005
Patent Publication Number: 20020106184
Assignee: Autodesk Canada Inc. (Montreal)
Inventors: Benoit Belley (Montreal), Pierre P. Tremblay (St-Lambert)
Primary Examiner: Thai Tran
Attorney: Gates & Cooper LLP
Application Number: 09/779,252