CONTEXT SENSITIVE REMOTE DEVICE
A user can make selections of one or more items, defined as a context, in a graphical user interface on a computer running software. The computer sends data to a connected wireless device, and the data is representative of the context. The wireless touch screen device displays one or more commands corresponding to the received context data and the commands can control the program on the computer. The wireless touch screen device then receives a user input on the wireless touch screen device to execute one of the one or more commands. The wireless touch screen device then sends the user command to the computer, wherein the computer executes the command and controls the program according to the user command. For example, this can allow remote control of a DAW, video-editing software, photo-editing software, and word processing software by a coupled wireless touch screen device.
Latest Apple Patents:
The following relates to remote devices for auxiliary input into a computer, and more particularly for context sensitive wireless touch screen devices for auxiliary input into a computer.
BACKGROUNDUsers can benefit from an auxiliary input to a computer running software, such as a digital audio workstation (DAW), video-editing software, photo-editing software, and word processing software.
Many users have at least one wireless touch screen device, such as a smart phone device or digital media player, in addition to a computer running software. Users can benefit from an integration of a computer running software with a wireless touch screen device, in order to use the wireless touch screen device as an auxiliary input into the computer running software.
However, current systems for utilizing a wireless touch screen device as an auxiliary input for a computer running software such as an additional input for a computer running software only allow one-way communication. For example, in a DAW environment, a wireless touch screen device can generate MIDI commands in response to a user command. A user must then typically configure the DAW to interpret the received MIDI command and execute a linked command in the computer software, for example, to play an arrangement upon receiving a C3 MIDI command. Users can benefit from a system utilizing a wireless touch screen device as an auxiliary input that engages in two-way communication with a computer running software such as a DAW. This two-way communication can allow the wireless device to present an appropriate command menu based on user selections made within a graphical user interface of the computer running software. The presented command menu can dynamically change as changes to the user selections are made within the graphical user interface.
SUMMARYProcessor-implemented methods, devices, and systems for using a wireless touch screen in two-way communication as an input device for a computer running software are disclosed. A user can make selections of one or more items in a graphical user interface on a computer running software. The one or more selections can be defined as a context. The computer sends data to a connected wireless touch screen device, wherein the data is representative of the context defined by the one or more selections in the graphical user interface. The wireless touch screen device displays one or more commands corresponding to the received context data and the commands can control the program on the computer. The wireless touch screen device then receives a user input on the wireless touch screen device to execute one of the one or more commands. The wireless touch screen device then sends the user command to the computer, wherein the computer executes the command and controls the program according to the user command. For example, this can allow remote control of a DAW, video-editing software, photo-editing software, and word processing software by a connected wireless touch screen device.
Many other aspects and examples will become apparent from the following disclosure.
In order to facilitate a fuller understanding of the exemplary embodiments, reference is now made to the appended drawings. These drawings should not be construed as limiting, but are intended to be exemplary only.
The method for coupling a wireless touch screen device and a personal computer can be implemented on a computer. The computer can be a data-processing system suitable for storing and/or executing program code. The computer can include at least one processor that is coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories that provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data-processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters. In one or more embodiments, the computer can be a desktop computer, laptop computer, or dedicated device.
The disclosed technology allows a smart phone with a touch-sensitive display to function as a remote control device for a computer running a software application, by wirelessly coupling the smart phone and computer. In response to a detected context, which is defined by user selections within a graphical user interface in the software application, the smart phone can display available menu commands, shortcut menu commands, key commands, or on-screen controls for the detected context. On-screen controls can include buttons, faders, text strings, and knobs. By manipulation of these on-screen controls, a user can cause the smart phone to send a user input to the computer running the software application, and control that application.
The displayed commands can be commands that a specific user engages most often for a detected context. In other words, the displayed commands can be “most frequently chosen” commands by a user for the detected context. The displayed commands can correspond, but are not limited to, to commands that the software application would display in response to a user “right click” or “control click” in a given context defined by one or more user selections in a graphical user interface.
In one embodiment of the disclosed technology, the smart phone will “follow” and assist the user when working with a software application. The smart phone will dynamically update its displayed screen controls based on user selections in the graphical user interface of an application.
Each of the displayed MIDI regions in the musical arrangement, as shown in
As shown in
The commands also include a selection to turn a metronome on or off 204. The commands further include a selection to turn “count-in” 206 for a metronome on or off. If this selection is turned on, the metronome will sound a set number of times, such as four beats, as a lead in for recording or other uses.
The commands also include transport control 208. The transport control 208 contains commands for record, rewind, forward, play, cycle region, and other information such as playhead position. By activating the commands shown in
The commands also include a selection to turn a metronome on or off 404. The commands further include a selection to turn “count-in” 406 for a metronome on or off. If this selection is turned on, the metronome will sound a set number of times, such as four beats, as a lead in for recording or other uses.
The commands also include transport control 408. The transport control 408 contains commands for record, rewind, forward, play, cycle region, and other information such as playhead position. By activating the commands in transport control 408, a user can control the arrangement on a DAW as shown in
The commands also include grid commands 606. The grid commands 606 include “Snap to” and “Value” data entry fields that allow a user to specify grid snap to values for the software instrument track 502 and region 504 as shown in
The commands also include grid commands 806. The grid commands 806 include “Snap to” and “Value” data entry fields that allow a user to specify grid snap to values for the software instrument track 702 and regions 704, 706 as shown in
The commands include 1004 to move notes by grid units. The move notes by grid units command 1004 allows a user to move selected notes 902, 904 backwards or forwards in the arrangement by a selected grid value, which can be set with the “Value” data entry field of the grid commands 1006. This would, for example if the grid value is set to quarter notes, allow a user to shift quarter notes on the first and second beat of a bar to the second and third beat of the bar by selecting the forward command.
The commands also include grid commands 1006. The grid commands 1006 include “Snap to” and “Value” data entry fields that allow a user to specify grid snap to values for the notes 902, 904 in region 908 on software instrument track 906 as shown in
The commands include commands 1204 to go to a previous or next instrument within a sound category. This would, for example, allow a user to cycle through available synthesizer software instruments in a synthesizer sound category.
The commands also include commands 1206. The commands 1206 allow a user to randomize the settings for the sound generator, the chorus, and/or the visual equalization for the selected track edit window 1100 corresponding to software instrument track 1102 as shown in
Although the exemplary device described herein employs the hard disk, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), a cable or wireless signal containing a bit stream and the like, may also be used in the exemplary operating environment.
To enable user interaction with the computing device 1300, an input device 1390 represents any number of input mechanisms such as a touch-sensitive screen for gesture or graphical input and a wireless receiver. The device output 1370 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display and wireless transmitter. In some instances, multimodal systems enable a user to provide multiple types of input to communicate with the computing device 1300. The communications interface 1380 generally governs and manages the user input and system output. There is no restriction on the disclosed technology operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
For clarity of explanation, the illustrative device embodiment is presented as comprising individual functional blocks (including functional blocks labeled as a “processor”). The functions these blocks represent may be provided through the use of either shared or dedicated hardware, including but not limited to hardware capable of executing software. For example the functions of one or more processors shown in
The method includes receiving a user input 1406 on the touch screen device to execute one of the displayed commands. The method includes sending the user input to the computer 1408, so that the computer executes the command. The command controls and modifies the program running on the computer. In this manner, the handheld wireless device acts as an auxiliary input for the computer running software and automatically presents commands appropriate for a user-selected context. The user selected context is defined by the selections made in the graphical user interface.
In one example, a context is defined by the selection of a real instrument track and no regions in an arrange window. Another context is defined by the selection of a software instrument track in an arrange window and no regions. An additional context is defined by the selection of a software instrument track and one region on the software instrument track in an arrange window. An additional context is defined by the selection in a graphical user interface of a software instrument track and two regions on the software instrument track in an arrange window.
Another context is defined by the selection in a graphical user interface of one or more notes in an editor window. Another context is defined by one or more selections in a software instrument track info edit window.
An additional context can be defined by the selection of one region on a real instrument track. Another context can be defined by the selection of multiple regions on a real instrument track in a graphical user interface.
The graphical user interface on the computer in the example method can correspond to an audio editing application program or digital audio workstation (DAW) program. In another example, it can correspond to a video editing application, a photo editing application, and a word processing application.
The personal computer 1504 receives one or more selections made in a graphical user interface, which can be defined as a context. The computer then sends data representative of the one or more selections to the wireless touch screen device 1502. The wireless touch screen device 1502 then displays pre-determined commands corresponding to the received data. The pre-determined commands are chosen by a software designer as commands appropriate for manipulating or controlling the selections made in the graphical user interface of personal computer 1504.
In this example, the wireless touch screen device 1502 is further configured to receive a user input to execute one of the displayed commands. In this example, a user can touch one of the icons displayed on touch screen device 1502 with a finger or appropriate stylus. Upon receiving this user input, wireless touch screen device 1502 will send a wireless signal to the personal computer 1504, wherein the computer is configured to execute the received command. This allows the wireless touch screen device 1502 to act as an additional input for personal computer 1504.
In the system of
Another context is defined by the selection by a user of one or more notes in an editor window. Another context is defined by the selection by a user of one or more items in a software instrument track info edit window.
The personal computer 1504 of system 1500 can run an application such as a digital audio workstation, digital video editor, digital photo editor, and word processor. A graphical user interface displayed by personal computer 1504 can correspond to the digital video editor, digital photo editor, or word processor application or programming running on personal computer 1504.
The technology can take the form of an entirely hardware-based embodiment, an entirely software-based embodiment, or an embodiment containing both hardware and software elements. In one embodiment, the disclosed technology can be implemented in software, which includes but may not be limited to firmware, resident software, microcode, etc. Furthermore, the disclosed technology can take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium (though propagation mediums in and of themselves as signal carriers may not be included in the definition of physical computer-readable medium). Examples of a physical computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Current examples of optical disks include compact disk read only memory (CD-ROM), compact disk read/write (CD-R/W), and DVD. Both processors and program code for implementing each as aspects of the technology can be centralized and/or distributed as known to those skilled in the art.
The above disclosure provides examples within the scope of claims, appended hereto or later added in accordance with applicable law. However, these examples are not limiting as to how any disclosed embodiments may be implemented, as those of ordinary skill can apply these disclosures to particular situations in a variety of ways.
Claims
1. A method for using a handheld wireless touch screen device as an input device for a computer comprising:
- receiving data from a computer running a program indicating a context defined by one or more selections in a graphical user interface;
- displaying one or more commands corresponding to the received context data on the wireless touch screen device, wherein the commands control the program on the computer;
- receiving a user input on the wireless touch screen device to execute one of the one or more commands; and
- sending the user input to the computer, wherein the computer executes the command and controls the program according to the user input.
2. The method of claim 1, wherein the one or more selections are selected from a group consisting of a real instrument track in an arrange window, a software instrument track in an arrange window, a software instrument track and one region on the software instrument track in an arrange window, a software instrument track and two regions on the software instrument track in an arrange window, a real instrument track and one region in an arrange window, and a real instrument track and multiple regions in an arrange window.
3. The method of claim 1, wherein the program is selected from the group consisting of a digital audio workstation, a video editing application, a photo editing application, and a word processing application.
4. The method of claim 1, wherein the one or more selections are selected from a group consisting of one or more notes in an editor window and one or more selections in a software instrument track info edit window.
5. The method of claim 1, wherein the displaying one or more commands corresponding to the received context data on the wireless touch screen device, wherein the commands control the program on the computer further comprises: displaying one or more commands stored in a frequently selected command list stored in memory for the received context.
6. A wireless touch screen device comprising:
- a processor configured to receive data from a computer running a program, wherein the data includes one or more selections made in a graphical user interface in the program;
- the processor configured to display, on a wireless touch screen display, one or more icons for commands corresponding to the received data, wherein the commands control the program on the computer;
- the processor configured to receive a user command on the wireless touch screen device to execute one of the one or more commands; and
- the processor configured to send the user command to the computer, wherein the computer executes the command and controls the program according to the user command.
7. The wireless touch screen device of claim 6, wherein the one or more selections are selected from a group consisting of a real instrument track in an arrange window, a software instrument track in an arrange window, a software instrument track and one region on the software instrument track in an arrange window, a software instrument track and two regions on the software instrument track in an arrange window, a real instrument track and one region in an arrange window, and a real instrument track and multiple regions in an arrange window
8. The wireless touch screen device of claim 6, wherein the one or more selections are selected from a group consisting of one or more notes in an editor window and one or more selections in a software instrument track info edit window.
9. The wireless touch screen device of claim 6, wherein the graphical user interface on the computer corresponds to a program selected from the group consisting of an audio editing application, a video editing application, a photo editing application, and a word processing application.
10. A system comprising:
- a computer including a graphical user interface; and
- a wireless touch screen device communicatively coupled to the computer,
- wherein the computer is configured to receive one or more selections in the graphical user interface and send data representative of the one or more selections to the wireless touch screen device and the wireless touch screen device is configured to display pre-determined commands corresponding to the received data.
11. The system of claim 10, wherein the wireless device is further configured to receive a user input to execute one of the displayed commands and send a signal to the computer, wherein the computer is configured to execute the received command.
12. The system of claim 10, wherein the one or more selections are selected from a group consisting of a real instrument track in an arrange window, a software instrument track in an arrange window, a software instrument track and one region on the software instrument track in an arrange window, and a software instrument track two regions on the software instrument track in an arrange window, a real instrument track and one region in an arrange window, and a real instrument track and multiple regions in an arrange window.
13. The system of claim 10, wherein the one or more selections are selected from a group consisting of one or more notes in an editor window and one or more selections in a software instrument track info edit window.
14. The system of claim 10, wherein the graphical user interface and displayed commands correspond with at least one application selected from the group consisting of a digital audio workstation, digital video editor, digital photo editor, and word processor.
15. A tangible computer readable medium storing instructions for displaying commands on a wireless touch screen device, the instructions comprising:
- receiving data from a computer indicating one or more selections;
- displaying one or more commands corresponding to the received data on the wireless touch screen device;
- receiving a user command on the wireless touch screen device to execute one of the one or more commands; and
- sending the user command to the computer.
16. The tangible computer readable medium of claim 15, wherein the one or more selections are selected from a group consisting of a real instrument track in an arrange window, a software instrument track in an arrange window, a software instrument track and one region on the software instrument track in an arrange window, a software instrument track and two regions on the software instrument track in an arrange window, a real instrument track and one region in an arrange window, and a real instrument track and multiple regions in an arrange window
17. The tangible computer readable medium of claim 15, wherein the one or more selections are selected from a group consisting of one or more notes in an editor window and one or more selections are a software instrument track info edit window.
18. The tangible computer readable medium of claim 15, wherein the graphical user interface on the computer corresponds to a program selected from the group consisting of an audio editing application, a video editing application, a photo editing application, and a word processing application.
Type: Application
Filed: Apr 9, 2010
Publication Date: Oct 13, 2011
Patent Grant number: 9021354
Applicant: Apple Inc. (Cupertino, CA)
Inventor: Jan-Hinnerk Helms (Hamburg)
Application Number: 12/757,426
International Classification: G06F 3/048 (20060101);