VIBROTACTILE CONTROL SOFTWARE SYSTEMS AND METHODS

Abstract

Methods and systems are disclosed to facilitate creating the sensation of vibrotactile movement on the body of a user. Vibratory motors are used to generate a haptic language for music or other stimuli that is integrated into wearable technology. The disclosed system in certain embodiments enables the creation of a family of devices that allow people such as those with hearing impairments to experience sounds such as music or other input to the system. For example, a “sound vest” or other wearable array transforms musical input to haptic signals so that users can experience their favorite music in a unique way, and can also recognize auditory or other cues in the user's real or virtual reality environment and convey this information to the user using haptic signals.

Description

CROSS REFERENCE

The present application is a continuation in part of U.S. patent application Ser. No. 14/713,908 entitled “Sound Vest,” filed on May 15, 2015. The entirety of the foregoing patent application is incorporated by reference herein.

BACKGROUND OF THE DISCLOSURE

Technical Field

The present invention relates to vibrotactile technologies, systems, and subsystems and, more specifically to systems to control vibrations to make it easy to create the sensation of vibrotactile movement on the body of a user. Also, this disclosure relates to a wearable vest designed to enable individuals such as hearing-impaired persons to experience sounds or other stimuli of various kinds, including but not limited to music, alarms, game events, and speech.

2. General Background

An aspect of the present disclosure is a system that uses vibratory motors to generate a haptic language for audio (or other stimumli) that is integrated into wearable technology. The inventive “sound vest” is intended as an assistive device for the hearing-impaired in certain embodiments. The disclosed system enables the creation of a family of devices that allow people, such as those with hearing impairments, to experience sounds such as music, or other inputs, to the system. The functionality of vests according to aspects of the present invention could include transforming sound/music/game input to haptic signals so that users can experience their favorite music in a unique way, and also systems that can recognize auditory cues in a user's everyday environment and convey this information to the user using haptic signals. Such pertinent auditory inputs could include a loud siren, someone calling out the user's name, etc.

It is desirable to address the limitations in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

By way of example, reference will now be made to the accompanying drawings, which are not to scale.

FIG. 1 is a block diagram of an illustrative embodiment,

FIG. 2 depicts aspects of certain embodiments of the present invention integrated into a wearable vest.

FIG. 3 is a photograph of a prototype in which a network of motors is stitched into a piece of fabric according to aspects of certain embodiments.

FIG. 4 is a flowchart of an algorithm according to aspects of the present invention in certain embodiments for converting audio data or other stimuli into signals for driving a network of vibrating motors incorporated into a wearable vest.

FIG. 5 is an exemplary diagram of a computing device that may be used to implement aspects of certain embodiments of the present invention.

FIG. 6 is an exemplary diagram of a main display of the vibrotactile control system VCS in any given computing device that may be used to implement aspects of certain embodiments of the present invention.

FIG. 7 is an exemplary diagram of a movement wheel device that may be used to implement aspects of certain embodiments of the present invention.

FIG. 8 depicts an exemplary timeline chart that enables the vibrotactile creator to draw intensities over time, according to certain embodiments of the present invention.

FIG. 9 is an exemplary diagram of a wave sound representation, which is a standard audio file representation that can be utilize as an audio guide, according to certain embodiments of the present invention.

FIG. 10 depicts an exemplary play/record bar that controls the system play and record engine, according to the embodiments of the present invention.

FIG. 11 depicts an exemplary tool selector to draw the desired shape on the timeline, according to the embodiments of the present invention.

FIG. 12 depicts an exemplary Brownian grain display according to embodiments of the present invention. The graininess created, adds a Brownian random generator that increases the seed size so the values wiggle around each given number instead of ramping up straight.

FIG. 13 is an exemplary diagram of a vibrotactile body display VBD showing the points of the body being vibrated at any given time at a given intensity, according to embodiments of the present invention.

FIG. 14 depicts an exemplary file export display that will export a file to a MPE MIDI file or an eight-channel .aiff file that may be used to implement aspects of certain embodiments of the present invention.

DETAILED DESCRIPTION

Those of ordinary skill in the art will realize that the following description of the present invention is illustrative only and not in any way limiting. Other embodiments of the invention will readily suggest themselves to such skilled persons, having the benefit of this disclosure, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. Reference will now be made in detail to specific implementations of the present invention as illustrated in the accompanying drawings. The same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts.

The data structures and code described in this detailed description are typically stored on a computer readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs or digital video discs), and computer instruction signals embodied in a transmission medium (with or without a carrier wave upon which the signals are modulated). For example, the transmission medium may include a communications network, such as the Internet.

The present disclosure in certain embodiments relates to a system, or “sound vest”, that uses vibratory motors to generate a haptic language for music or other sound, or other stimuli, that is integrated into wearable technology. A technical challenge to creating such a system is to design a system that decomposes auditory or other input into control signals that can be streamed out to a network of motors. The present inventors have designed a preliminary system that in certain embodiment performs entry-level signal processing techniques on the incoming sound or other stimuli in order to determine the spectral profile of the input. The motors are then powered based on the magnitude of the spectral power.

A preliminary design of the system in certain embodiments enables the use of up to 64 motors to represent the incoming audio or other stimuli. A revised design in certain embodiments utilizes 64 motors on each of the front and back sides of the vest, for a total of 128 motors. For example, each of M1, M2, M3, and M4 in FIG. 2 could represent up to 16 separate motors, for a total of 64 motors on the front side of the vest 40. A similar network of 64 motors could be deployed on the back side of the vest. The user's entire torso may be utilized to create a tono-topic map of the torso that is, vibratory motors on the left (L) side of the vest may be mapped to the left speaker, vibratory motors on the right (R) side of the vest may be mapped to the right speaker, vibratory motors on the bottom of the vest may be mapped to low frequencies, and vibratory motors on the top of the vest may be mapped to high frequencies.

FIG. 1 depicts the basic system in certain embodiments, including signal processor (20) that receives audio input (10) (e.g., from a microphone (50, see FIG. 2) or audio jack) and transforms the input audio signal into a haptic language for driving (30) a network of motors denoted M1, M2, M3, and M4. The signal processor may also include an analog-to-digital converter (ADC) (22) for digitizing real-time audio signals provided in analog form, memory or storage (24) for storing audio data, executable instructions, and the like; and a voice recognition module (26).

As shown in FIG. 2, the motors M1 through M4 may be integrated into a wearable vest (40) such that M1 and M2 are on the right side of the user's torso, and M3 and M4 are on the left side of the user's torso. Moreover, motors M1 and M3 may vibrate to represent the higher frequency components of the audio input, whereas motors M2 and M4 may vibrate to represent the lower frequency components. It should be understood that in a commercial implementation, there would likely be many more than four motors in certain embodiments.

As shown in FIG. 3, using conductive thread and relatively low-cost vibratory motors, an initial prototype was made by stitching thread into fabric, as illustrated.

Applicants are aware of information in the public domain relating to wearable technology with haptic feedback. Copies of this information are being submitted in conjunction with this application in Information Disclosure Statements (IDS). Some of the known prior art references translate sound to vibration, but the present disclosure is different in certain embodiments in that it goes beyond a simple sensory substitution. The brain is an amazingly “plastic” organ, and we will take advantage of its plasticity by giving the hearing impaired the opportunity to experience music through a haptic “language.” This difference lies in the real-time spectral analysis performed as the music streams into the micro-controller at the heart of the sound vest in certain embodiments—the audio streams in and is broken down to a representation of its basic frequency components. Then, each frequency domain is sent to a different part of the body (i.e., if the user is listening to Alvin and the Chipmunks, he will feel a lot of vibration up by his collarbones, and not much down low; listen to Barry White, and it will be the other way around due to the dominance of Mr. White's low frequency components). The inventive system in certain embodiments can also represent stereo by streaming to the left side of the body for the left speaker and right speaker to the right side.

Further Developments

During the course of further developing the system described above, we have discovered that the process of creating musical sensation though tactile stimuli can be improved in several ways in certain embodiments:

1. The audio signals or other stimuli can be improved by converting them into the MIDI (i.e., Musical Instrument Digital Interface) data format in certain embodiments, and then reducing the data to a small defined number of tracks, e.g., four (4) tracks representing drums, bass, guitars, and vocal. Other selections could be used as well, depending on the type of music. (Those skilled in the art understand that MIDI is a technical standard that enables a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another. A single MIDI link can carry up to sixteen channels of information, each of which can be routed to a separate device.)

2. Instead of mapping the audio signals to the motors as described above (i.e., mapping higher frequencies to the top of the vest and mapping the lower frequencies to the bottom of the vest), it may be advantageous in certain embodiments to map each of the 4 tracks to different parts of the vest. For example, the signals corresponding to vocals can be directed to the mid-section while the drums, bass, and guitar signals are directed to respective regions surrounding the mid-section. This mapping has been found to create less cross-over and less “muddiness” to the vibrations created by the motors.

3. If the system is unable to convert live audio to MIDI data in real time, it can be advantageous in certain embodiments to provide a mode in which the music data is first downloaded and then played back through the vest. In this way, the user can experience the music, albeit not in a real-time, “live” setting.

As shown in FIG. 4, an algorithm for converting audio data or other stimuli into signals for driving a network of vibrating motors incorporated into a wearable vest comprises the following steps in certain embodiments: First, in step S1, audio data in MIDI format is obtained. The data can either be downloaded to the system from a third party provider, or created using recorded audio and an audio production software tool. In step S2, the MIDI data is organized into 4 tracks representing vocals, drums, guitars, and bass. In step S3, the 4 tracks are mapped to different regions of the sound vest; and in step S4 the respective tracks of data are used to drive the motors in the different regions.

The system in certain embodiments may be enhanced by providing wireless links between the signal processor and the motors. In addition, a voice recognition module may be incorporated to enable the system to recognize specific spoken words for selective playback through the motors. For example, the user's name may be specifically recognized and used to signal the user through the motors.

FIG. 5 is an exemplary diagram of a computing device 500 that may be used to implement aspects of certain embodiments of the present invention. Computing device 500 may include a bus 501, one or more processors 505, a main memory 510, a read-only memory (ROM) 515, a storage device 520, one or more input devices 525, one or more output devices 530 (including vibrotactile actuators such as motors, as described herein), and a communication interface 535. Bus 501 may include one or more conductors that permit communication among the components of computing device 500. Processor 505 may include any type of conventional processor, microprocessor, or processing logic that interprets and executes instructions. Main memory 510 may include a random-access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 505. ROM 515 may include a conventional ROM device or another type of static storage device that stores static information and instructions for use by processor 505. Storage device 520 may include a magnetic and/or optical recording medium and its corresponding drive. Input device(s) 525 may include one or more conventional mechanisms that permit a user to input information to computing device 500, such as a keyboard, a mouse, a pen, a stylus, handwriting recognition, voice recognition, biometric mechanisms, and the like. Output device(s) 130 may include one or more conventional mechanisms that output information to the user, including a display, a projector, an A/V receiver, a printer, a speaker, and the like. Communication interface 535 may include any transceiver-like mechanism that enables computing device/server 500 to communicate with other devices and/or systems. Computing device 500 may perform operations based on software instructions that may be read into memory 110 from another computer-readable medium, such as data storage device 520, or from another device via communication interface 535. The software instructions contained in memory 510 cause processor 505 to perform processes that will be described later. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the present invention. Thus, various implementations are not limited to any specific combination of hardware circuitry and software.

In certain embodiments, memory 510 may include without limitation high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include without limitation non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 510 may optionally include one or more storage devices remotely located from the processor(s) 505. Memory 510, or one or more of the storage devices (e.g., one or more non-volatile storage devices) in memory 510, may include a computer readable storage medium. In certain embodiments, memory 510 or the computer readable storage medium of memory 510 may store one or more of the following programs, modules and data structures: an operating system that includes procedures for handling various basic system services and for performing hardware dependent tasks; a network communication module that is used for connecting computing device 510 to other computers via the one or more communication network interfaces and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on; a client application that may permit a user to interact with computing device 500.

Certain text and/or figures in this specification may refer to or describe flow charts illustrating methods and systems. It will be understood that each block of these flow charts, and combinations of blocks in these flow charts, may be implemented by computer program instructions. These computer program instructions may be loaded onto a computer or other programmable apparatus to produce a machine, such that the instructions that execute on the computer or other programmable apparatus create structures for implementing the functions specified in the flow chart block or blocks. These computer program instructions may also be stored in computer-readable memory that can direct a computer or other programmable apparatus to function in a particular manner, such that the instructions stored in computer-readable memory produce an article of manufacture including instruction structures that implement the function specified in the flow chart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow chart block or blocks.

Accordingly, blocks of the flow charts support combinations of structures for performing the specified functions and combinations of steps for performing the specified functions. It will also be understood that each block of the flow charts, and combinations of blocks in the flow charts, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.

For example, any number of computer programming languages, such as C, C++, C# (CSharp), Perl, Ada, Python, Pascal, SmallTalk, FORTRAN, assembly language, and the like, may be used to implement aspects of the present invention. Further, various programming approaches such as procedural, object-oriented or artificial intelligence techniques may be employed, depending on the requirements of each particular implementation. Compiler programs and/or virtual machine programs executed by computer systems generally translate higher level programming languages to generate sets of machine instructions that may be executed by one or more processors to perform a programmed function or set of functions.

In the descriptions set forth herein, certain embodiments are described in terms of particular data structures, preferred and optional enforcements, preferred control flows, and examples. Other and further application of the described methods, as would be understood after review of this application by those with ordinary skill in the art, are within the scope of the invention. The term “machine-readable medium” should be understood to include any structure that participates in providing data that may be read by an element of a computer system. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks and other persistent memory such as devices based on flash memory (such as solid-state drives, or SSDs). Volatile media include dynamic random access memory (DRAM) and/or static random access memory (SRAM). Transmission media include cables, wires, and fibers, including the wires that comprise a system bus coupled to a processor. Common forms of machine-readable media include, for example and without limitation, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD, or any other optical medium.

FIG. 6 is a grayscale version of an exemplary embodiment of a main view display 600, also called the vibrotactile control system VCS 610 herein. In the embodiment of FIG. 6, each section of the system represents different components of the main display that may serve to craft the vibrations that are coupled to a sound file. In certain embodiments of this invention, the sound file could comprise prerecorded sound or a live system. In both cases, the intention would be to replicate the vibrations of live instruments or sounds being play in real time.

The main view display 610 may facilitate understanding and interaction with the system in certain embodiments, and may be used to create an intuitive experience with little or no stiff learning curve, allowing the user to easily adopt the technology.

In the example of the main view display 610 shown in FIG. 6, starting from the top left corner, the embodiment of the present invention may include a movement wheel 710, a Brownian grain display 1210, and a file export display 1410. In the center, from top to bottom it may also include a play/record bar 1010, a wave sound representation 910, which may be a standard audio file representation, and a timeline chart 810. Finally, on the left portion of the main view display 610, an embodiment may show a vibrotactile body display 1310 and a tool selector 1110.

Embodiments of the present invention may be used to implement aspects that will create a surround body experience, in which the vibrotactile creator can craft and control vibrations to any part of the body at any given point in time.

FIG. 7 is a grayscale version of an exemplary embodiment of the movement wheel 700. In the embodiment of the present invention, the movement wheel 710 can be used to easily create or craft the sensation of vibrotactile movement on the body of a user, and it allows a vibrotactile creator to control movement and intensity.

Each section of the embodiment of FIG. 7 shows a different part of the body, such as the ribcage 730 section of the body, the spine 731, the mid back 732, the low back of the body 733, the left wrist 734, the right wrist 735, the left ankle 736, the right ankle 737 of a human body. Each module of the moment wheel 710 could be color coded, labeled or add a patter that would be associate to an specific part of the body described within the embodiment of the moment wheel 710.

In certain embodiments, the vibrotactile creator may simultaneously use the movement wheel 710 and the Brownian grain display 1210 to draw a path 740 that represents the movement from one part of the body to the other controlling the intensity of the line as it is drawn and the graininess of it. The thickness of the line is directly related to the intensity of the vibration. The thinner the line, the weaker the vibration and the thicker the line the stronger the vibration. Each line has the possibility of been drawn with different intensities as the user moves throughout the movement wheel 710. The movement varies with intensity in time.

For example, the user may start to draw the path 740 in the mid back 732 and move down to the low back 733. As the path moves down the vibrations could increase or decrease in intensity. In certain embodiments of this invention, the path 740 drawn by the vibrotactile creator may depict smooth lines 741 or dotted lines 742. Dotted lines 742 will indicate that in that specific vibration a degree of graininess is applied. Smooth lines 741 will indicate that in that specific part of the path 740 graininess was not applied.

In certain embodiments, the vibrotactile wheel 710 and the Brownian grain display 1210 may be visualized as a dynamic display that has to be operated at the same time to craft the vibrations as the sound is being played. Their operation could depend, for example, on the operation of a mouse or a touch screen to draw the path 740 of the movement and the intensity of the lines, and an expression pedal to select the level of graininess for the purpose of making the experience more dynamic.

In certain embodiments of the movement wheel 710, the user may lock adjacent positions of the body, toggling a switch with a feature called “lock group” herein. For example, the left ankle 736 may be locked with the right ankle 737 using the lock group option 722. Similarly, the left wrist 734 may be locked with the right wrist 735 with lock group option 721, and the low back position 733 may be locked with the mid back position 732 with lock group 720. A lock group may be use to automatically replicate the vibration created for one area into whichever area is locked with it. When a lock group feature is on, in certain embodiments one vibration triggers both parts of the body for symmetry.

FIG. 8 is a grayscale version of an exemplary embodiment of the timeline chart 800, which enables the vibrotactile creator to draw intensities over time for an specific part of the body.

In the embodiment of FIG. 8, the intensity values 840 range from 0 to 4095. These ranges of values are represented in the timeline 810 by horizontal lines 860 parallel to the x axis, and they show the intensity of the vibration. Time is measured along the x axis in intervals marked by vertical lines parallel to the y axis; these intervals are called tempo 861, and they represent the time that it is taking the vibration to move across intensities. The tempo 861 helps to divide the wave sound representation 910 into segments, in order to have a better visualization of the waveform at any given time. The time 850 in the timeline chart 810 can be adjusted in milliseconds 852 or beats per minute BPM 851. This visual aid may help to craft the vibration for the particular part of the piece that is being played.

In the embodiment of FIG. 8, the timeline 810 may feature the same body part names and/or colors as the movement wheel 710, but arranged in channels. In certain embodiments, channel 830 corresponds to the ribcage, channel 831 is the spine, channel 832 corresponds to the mid back, channel 833 to the low back, channel 834 and 835 correspond to the left and right wrist respectively, and channels 836 and 837 correspond to the left and right ankles.

Because the embodiment of the timeline 810 may feature the individual body parts, also seen in the embodiment of the movement wheel 710, the vibrotactile creator or user can concentrate in crafting the vibrations one part of the body at a time. Each body part has its own channel (830, 831, 832, 833, 834, 835, 836 and 837) in certain embodiments, which means that the user can mute all but one channel and begin the crafting process for an individual body part, or, if using the lock group feature (820, 821 and 822), for a set of body parts. These lock group features are also seen in the embodiment of the movement wheel 710.

The gray area shown in the ribcage timeline chart 830 is an exemplary depiction of an area called a loop point 870. This area may be selected to be repeated for purpose of editing and/or for playing back an audio file. The loop point 870 selection within a channel may be a particular part of the timeline or the entire timeline, the vibrotactile creator 620 can change the size of the loop point.

The waveforms or “vibrotactile forms” shown in the embodiment of FIG. 8 represent vibrations. In certain embodiments of the present invention, these vibrotactile waveforms allow the user of the sound vest system to feel different type of vibrations. For example, dots may feel different from line-based waveforms.

FIG. 8 depicts a visual example of the vibration that may occur in each channel. This example shows the vibrotactile forms for the vibrations on the ribcage 830, the spine 831, the mid back 832, the low back 833, the left wrist 834, the right wrist 835, left ankle 836 and the right ankle 837.

FIG. 9 is a grey scale version of a wave sound representation of a standard audio file 900. In the embodiment of FIG. 9, the wave sound representation 910 may be loaded (see LOAD AUDIO button 930) into the VCS 610 so it can be visualized in the display of the audio guide 920 to be used when crafting vibrations. The lines that divide the audio guide represent the tempo 940. The tempo lines 940 of the standard audio file align with the tempo lines 861 of the time line as shown in FIG. 8.

A grey scale representation of a bar 1000, shown in the embodiment of FIG. 10, controls the play and record engine of the system. The playback speed 1020 can be adjusted in the corresponding box. The negative values play the file backwards, and the positive values play the file forwards 1030. The loop mode 1040 may be set to: (1) off (i.e., circular, goes to the end and starts again); or (2) palindrome (i.e., goes to the end and plays backwards to the beginning).

For example, when a sound file is played in the play and record engine of the exemplary system 1010 shown in FIG. 10, the sound may be visualized as a wave sound representation 900. This visual match may help the vibrotactile creator 620 to craft the vibrations for the specific part of the piece that is being played, visualizing the vibrations in the channels of the timeline 810.

In the embodiment of the tool selector 1100 shown in FIG. 11, the user may select a tool from the tool selector 1110 to draw the desired shape on the timeline 810. The tool selector 1110 works like a photo editor or an audio distorted station, because it lets the user select what kind of line it would be used to craft the waveforms in the timeline 810.

The tool selector depicted in FIG. 11 may be used to implement embodiments of the present invention. For example, when the user works with the linear “fade in” part of the tool selector 1110, he or she may draw a part of the waveform that would be a straight line 1120. When 1121 is chosen, it depicts a movement that goes fast at first and then slows down at the end; 1123 would be the opposite to 1121. The s-curve 1122 shows a combination of the movements described in 1121 and 1123, where the waveform moves fast initially, then stabilizes in a plateau, then ends with a fast movement towards the maximum vibration. This type of moment may emulate what is called in music a psychoacoustic emotional effect. The user may feel a sudden burst of vibration followed by a plateau that will have him/her in suspense and then another burst of vibration. It may have an emotional effect.

In vibrotactile language according to certain embodiments, 1130 is called a click herein. In the display 1110, it is the equivalent of a staccato note in music. The click 1130 may be set 1131 from 10 to 50 milliseconds. After 50 milliseconds it turns into a small line because the vibration achieved is not a click per se but a short buzz. The click cannot last more than 50 milliseconds, its maximum value, otherwise it is received as a line, not a dot. But also, it cannot be shorter than 10 milliseconds because it may be hard to perceive due to the spinning limitation of the eccentric rotating mass vibration motors (ERM) used as vibrotacile actuators in certain embodiments. The ideal click using this technology may be between 30 and 50 milliseconds, based on experimental results.

In the embodiment of FIG. 11, the fade-out display that comprises 1140, 1141, 1142 and 1143 works opposite to the fade in display (1120, 1121, 1122 and 1123). Instead of fading in a vibration, it fades out.

The free-drawing display may allow the user to draw an uneven line. For example, if the main display timeline 810 is in a touch screen a user could create any line shape and it would create endless possibilities for the waveforms. Finally, the undo button 1160 is provided to remove any part of the wave form in the timeline 800 that the user does not want to use anymore.

When the tool selector 1100 fade-in and fade-out waveforms are visualized in the main display 600, the user can modify them by changing the time or the amplitude. This gives the craft of the vibration endless possibilities.

In the embodiment of FIG. 12, the graininess amount 1210 creates a vibration that is not smooth. It is a type of noise in the vibration, that would distort a smooth vibration, creating crackling. The graininess amount 1220 can be set from 0 to 100% as shown in exemplary embodiment of FIG. 12. The graininess may add a Brownian random generator effect 1200 that increases the seed size 1220 so the values “wiggle” around each given number instead of ramping up straight. The effect may feel like a vibrotactile distorted guitar; it may add expressiveness and interest to an otherwise clean signal.

The visual expression of graininess is given in the embodiment of the movement wheel 710. When graininess is applied the otherwise smooth lines 741 in the drawn path 740 will acquire spaces, like dotted lines 742 that will represent unevenness.

FIG. 13 presents an embodiment of the vibrotactile body display VBD 1310. This embodiment shows the points of the body being vibrated at any given time at a given intensity. The gray scale points (1330A, 1330B, 1331, 1332, 1333, 1334, 1335, 1336, 1337) may correspond to the gray scale colors on the movement wheel 710 and the grey scale colors in the timeline 610, for cohesiveness. The stronger a color is seen on the display, the stronger the vibration is on that point.

After the vibrotactile creator or user is finished crafting the experience, in certain embodiments he or she can then export the file to a MPE (multidimensional polyphonic expression) MIDI file 1410, as shown in FIG. 14. In the embodiment of FIG. 14, the user may export to a MPE (midi) file 1420, allowing the vibrotactile creator to control the video notes, this would typically not be possible without the MPE (midi) file. The user may also export into an eight channel .aiff audio file 1430. The user may turn on Open Sound Control OSC 1440 on and off to send the data in real time to other applications (e.g., audio, video and lighting software for example). The set port 1450 can be customized, traditionally to 7400-7500 (1455) for sending out the vibrations to an access point so it can be transmitted to the wearer's body wirelessly and finally the user can type in (see 1060) the Internet Protocol (“IP”) address 1465 needed for OSC output in the network, Internet, Ethernet, etc.

Certain embodiments implement methods and systems for creating and/or applying vibrations to a user's body, including simultaneous stimulation of multiple parts of a user's body during gaming sessions such as in virtual reality (“VR”) applications. According to aspects of the present invention, in certain embodiments, vibration stimuli matching the audio and/or visual effects found in VR environments are applied to a user's body, and in some of those embodiments, two or more such stimuli are applied simultaneously to different parts of a user's body.

Methods and systems implemented in certain embodiments use what is call “haptic panning” herein. This is similar to audio panning—when viewers watch a car moving from left to right on a screen they can also hear the sound following the image. The sound does not actually “move” with the car. The speakers placed at the left and right of the screen modulate the intensity of the sound according to well-known equations to create the illusion of movement of the sound source.

Likewise, various methods may be implemented similar to those known to skilled artisans for panning audio (e.g., constant power, vector-based amplitude panning (VBAP), ambisonics, etc.) to move vibrations around the body of a user. So, if it goal is to “move” the vibrations from left wrist to right wrist, for example, the following three different stages are implemented in certain embodiments: (A) “Left”, vibrations at 85% on Left Wrist, 10% on Ribcage, 5% on Right Wrist; (B) “Center,” vibrations at 10% on Left Wrist, 80% on Ribcage, 10% on Left Wrist; (C) “Right,” vibrations at 5% on Left Wrist, 10% on Ribcage, 85% on Right Wrist. Any desired “movement” can be implemented following variations of this technique, from any point A to point B, passing through the areas in between.

Methods and systems implemented in certain other embodiments use what is called “hit spreading” herein. For example, if a game character is punched in the stomach, the user should also feel the energy of the punch going all the way to the user's back. In a VR simulation, when an area is hit by a punch, a fireball, or any other of form of energy that hits the user's character in the virtual world, the energy will spread to other areas. If a virtual enemy launches a fireball and the character defends it with bracelets, in the real world the energy felt vibrating on the user's wrist that got hit will trigger vibrations on the ribcage with less intensity and on the back with even less intensity.

In certain other embodiments, for example, a game character may enter a zone with a “force field,” and in response in the real world, the vibrations get stronger in the area of the body that is closer to the field but other areas of the user's body will vibrate as well, although with less intensity.

In certain other embodiments, what are called the “Laser Sword Penetration” methods and systems herein can be understood as vibrations spreading on the areas that are being poked by a sword, but in order to make the effect more compelling, the vibrations also pulse in the neighboring areas, though with less intensity.

While the above description contains many specifics and certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art, as mentioned above. The invention includes any combination or sub combination of the elements from the different species and/or embodiments disclosed herein.

Claims

1. A system for transforming sensory stimulus to a haptic language, comprising:

a signal processor configured to receive a sensory stimulus and simultaneously generate a plurality of electrical driving signals according to a predefined mapping from signals comprising portions of said sensory stimulus to each of said plurality of electrical driving signals;

a wearable array of items comprising vibratory motors; and

a network comprising a plurality of said vibratory motors incorporated into said wearable array, wherein said plurality of electrical driving signals generated by said signal processor are used to drive said plurality of vibratory motors according to a predefined mapping of signals comprising portions of said sensory stimulus to a plurality of different regions of said wearable array.