System and method for simultaneously playing games and audio-visual content

Abstract

A handheld specific purpose combined player executes game software simultaneously with playing non-game audio, such as music. The combined player has a game processor that executes game software, outputs visual image data to a screen, and outputs audio. The combined player also has an audio processor that decodes audio data, the audio data separate from the game software. Connected to both the game processor and the audio processor is an audio mixer such that audio generated from the audio data and audio output by the game processor is combined to allow simultaneous playing of the game software and listening to audio generated from the audio data, the mixing controlled by inputs from a user.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to and the benefit of U.S. Provisional Patent Application Ser. No. 60/777,848, filed on Feb. 28, 2006, the entire contents of which are incorporated herein by reference.

SUMMARY OF THE INVENTION

A handheld specific purpose combined player executes game software simultaneously with playing non-game audio, such as music. The combined player has a game processor that executes game software, outputs visual image data to a screen, and outputs audio. The combined player also has an audio processor that decodes audio data, the audio data separate from the game software. Connected to both the game processor and the audio processor is an audio mixer such that audio generated from the audio data and audio output by the game processor is combined to allow simultaneous playing of the game software and listening to audio generated from the audio data, the mixing controlled by inputs from a user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a device for simultaneously playing games and audio according to an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

A hardware block diagram of an exemplary embodiment of the present invention is shown in FIG. 1. The combined player 100 includes a processor (CPU) 110 that is connected to a screen 120, input 130 with various keys and game memory 140. The combined player is a handheld, specific purpose device that allows a user to simultaneously play audio, such as music, being played back from an audio device, such as an MP3 player, and play a video game. In the embodiment shown in FIG. 1, processor 110 executes game software stored in game memory 140. As used for storing game software executed by the processor, in various embodiments game memory 140 is a removable solid state memory card, such as a ROM cartridge or flash memory card, a permanent ROM, a disk, such as an optical and/or magnetic disk, internal flash or conventional random access memory in which game software is stored, or any other know or developed hardware that stores game software. The user interacts with the game executed by the processor using input 130 and views the game on screen 120. Audio produced with respect to the execution of the game software is output on speaker 150 through audio mixer 160. In some embodiments, earphones, headphones or a transmitter (to, for example, wireless headphones, Bluetooth earpiece, FM receiver, etc.) are used in whole or in part to replace speaker 150.

Combined player 100 also includes an audio decoder DSP chip 170 that is also connected to audio mixer 160 and audio memory 180. In many embodiments, audio decoder DSP chip 170 is a hardware MP3 decoder that decodes MP3 music data stored in audio memory 180. In some embodiments, audio decoder DSP chip 170 is a hardware decoder for one or more of WMA, WAV, MIDI, or any other known or developed audio file format, in replacement or in addition to the MP3 decoder functionality. In some embodiments, audio memory 180 and game memory 140 are the same physical device, including all of the variations of game memory 140 discussed above. In such embodiments, the game software and audio data are stored in different portions of the same physical memory. In other embodiments, audio memory 180 is separate from game memory 140. In these embodiments, any one or more of the various types of memory usable for game memory are used for separate audio memory 180.

Audio mixer 160 mixes the music (or other audio) produced by audio decoder DSP chip 170 (e.g., music) with the audio output by processor 110 executing game software (e.g., sound effects) and sends the combined audio signal to speaker 150. As discussed more fully below, the user can select how the music and sound effects are mixed by the audio mixer 160, such as making one the background for the other or mixing to certain channels only, particularly in the case where one or more of the music and sound effects are mono. In some embodiments, one (music or sound effects) can be applied to the other to create audio effects (volume/loudness/pitch modulation, beats, surround sound, etc.). In some embodiments, the user can use music audio to replace the audio of games. In these embodiments, user inputs (e.g., joypad/firing actions) can modulate the music audio or the music audio “beats” or other features can be applied to the game video to create display effects. For example, “beats” can change color tone, intensity, or “screen flashing”, or modulate the compositing of pre-stored patterns to game video.

In most embodiments of the combined player 100, the selection keys normally found on music audio or portable video devices will be changed—to replace the traditional keys to have a joypad (direction pad) key placed in an ergonomic way for allowing game-playing on screen, such as the placements on traditional handheld games.

In some embodiments, combined player 100 does not have a separate audio decoder DSP 170. In some of these embodiments, the combined player has one single CPU. Multiple threads or processes are being computed by the one CPU via time-sharing. Some of the threads/processes implement the MP3 or other audio play-back and others implement the game playing. They share audio data via a data buffer (game memory 140 and/or audio memory 180) accessible by both groups of threads/processes.

In some embodiments, rather than a hardware audio decoder DSP 170, the combined player has a dedicated game chip or other hardware (not shown) that executes the game software. In these embodiments, the audio (music) is decoded by the audio decoder DSP 170, in which case the power of processor 110 can be drastically reduced or replaced altogether by, for example, an ASIC, or is decoded by the processor 110, in which case audio decoder DSP 170 can be removed. In some of these embodiments, the game software is stored on the dedicated game chip or other hardware and the game memory 140 is omitted.

In some embodiments, dual CPUs (not shown) are used. In such embodiments, audio decoder DSP may be removed and the audio decoding is performed mainly (mostly) or all on one CPU and the game processes run mainly (mostly) or all on the other CPU. In some embodiments, multiple CPUs (not shown) are used. These work similar to dual CPU embodiments, but may use more than one CPU for one or both of the audio decoding and game processing. In some embodiments, dual or multiple CPUs may be used an audio decoder DSP and/or dedicated game chip/hardware.

In these various embodiments, the dual use feature is implemented in the combined player at the software level, firmware level, and/or in dedicated hardware.

Returning to the audio mixer 160, in the combined player 100 shown in FIG. 1, combining and/or processing of sound: the audio track(s) generated by game and by the audio decoder (music, e.g., MP3s) are mixed or combined using a hardware audio mixer 160. While audio mixer 160 in FIG. 1 is a hardware device, it can be internal or external to other chips, such as the processor 110 or the audio decoder DSP 170. Alternatively, rather than using a hardware audio mixer, software, such as multiple processes using a shared buffer to process audio data stored therein before the audio data is sent for final play back, is used.

In many embodiments, the combined player provides a “pop-up” menu during game-playing to allow music audio play-back control (e.g., select song, MP3 configuration, pause, fast forward, etc.), so that user does not need to exit the game playing in order to configure the music audio playing. In most cases, when the pop-up menu is displayed, game-playing is put in pause. When the game is paused, the game state is saved before the pause as a protection in case of a crash so that such state can be restored. In many embodiments, lyrics of audio (music) played by the audio decoder DSP are show on the screen using superposition, if so selected by the user.

In some embodiments, the combined player also implements a digital video recorder device, that displays television and/or movie video. In some implementations, it has “dual-use” with video from the digital video recorder in that the video can be replace the game or the audio from the video can replace audio (music) in the combined player 100 “dual use” functionality. For example, the combined player may play music audio while watching a television video. The music audio can have similar effects on television/movie video as it has on games in as described above, and the pop-up menu can work for music audio also as described above. The mixing of game playing sounds can also be with the selected audio track(s) of television/movie video programming rather than music audio with the same alternatives and options as described above. In some embodiments, the digital video recorder functionality is replaced or supplemented by handheld digital television receiver functionality (for terrestrial digital television: DAB/DMB—digital audio/multimedia broadcasting, or DVB-H cellular, or in Korea, DBS).

In some embodiments, a “picture in picture”, split screen, or multiple-screen tiling effects is offered during game-playing: the game can be in PIP when playing music audio (which may be showing lyric or music video) or when playing TV/movie video or vice versa. As with the embodiments described above, a pop-up menu system allows user selection of video functions (change channel, etc.) or game functions (change game titles, levels, sound effects, etc.) when in dual-use mode.

In other embodiments, a dual screen is used. One screen plays the game or displays a television/movie video and the other screen shows MP3 lyric and/or music video. Alternative, the 2nd screen also shows menu and user interface. In another alternative, one screen plays a game and the other displays a television/movie video.

Alternative embodiments have both an FM (or other radio) receiver and a TV receiver co-existing, FM (or other radio) receiver and game playing co-existing, or audio decoder (e.g., MP3 player) and TV co-existing

In another alternative embodiment, the dual use device has wireless Internet or other network access, such as through a cellular or wi-fi connection. In this implementation, the music audio and/or audio-visual content may be supplied by streaming over the network or other network music delivery method. Similarly, a game may be played online, through, for example, a browser. In either of these cases, or a combination of both, the combination of music audio, audio-visual content and/or game playing is done at the dual use device as described above.

While certain exemplary embodiments have been described in detail and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive of the broad invention. It will thus be recognized that various modifications may be made to the illustrated and other embodiments of the invention described above, without departing from the broad inventive scope thereof. In view of the above it will be understood that the invention is not limited to the particular embodiments or arrangements disclosed, but is rather intended to cover any changes, adaptations or modifications which are within the scope and spirit of the invention.

Claims

1. A handheld specific purpose combined game and non-game audio player comprising:

a game processor that executes game software, outputs visual image data to a screen, and outputs audio;

an audio processor that decodes audio data, the audio data separate from the game software; and

an audio mixer connected to the game processor and the audio processor such that audio generated from the audio data and audio output by the game processor is combined to allow simultaneous playing of the game software and listening to audio generated from the audio data, the mixing controlled by inputs from a user.

2. The combined player of claim 1 wherein the audio mixer is comprised in the game processor.

3. The combined player of claim 1 wherein the audio mixer is comprised in the audio processor.

4. The combined player of claim 1 further comprising a combination processor that comprises the game processor and the audio processor.

5. The combined player of claim 4 wherein the audio mixer is comprised in the combination processor.