Handheld or Wearable Device for Recording or Sonifying Brain Signals
A handheld device for sonifying electrical signals obtained from a subject is provided. The device can utilize at least one of several operations including (but not limited) digitizing signals from electrodes, adjusting the signals based on accelerometer input, filtering the signals, conditioning the signals according to conditioning parameters, modulating the signal according to sound synthesis parameters, and generating sound from the representations of the signals to accomplish sonification. The device can include an analog-to-digital (A/D) converter to digitize the one or more electrical signals and a processor that receives the one or more digitized electrical signals and produces a representation of an acoustic signal. The device further includes a speaker system that sonifies the representation of the acoustic signal.
This application is a continuation of U.S. patent application Ser. No. 15/159,759 filed May 19, 2016, which is incorporated herein by reference. U.S. patent application Ser. No. 15/159,759 claims priority from U.S. Provisional Patent Application 62/163,637 filed May 19, 2015, which is incorporated herein by reference.
FIELD OF THE INVENTIONThe disclosed embodiments relate generally to the field of sonifying signals detected from a living subject (e.g., electrical signals indicative of brain activity and/or heart activity), and in particular, to a handheld or wearable device for sonifying signals from a living subject.
BACKGROUND OF THE INVENTIONThe ability to measure signals from a living subject (e.g., relating to the living subject's bodily functions) is beneficial for medical and diagnostic applications. For example, from a diagnostic point of view, measuring brain signals helps to ascertain brain activity related to abnormal brain function, to monitor spatial and/or temporal progression of brain disease, to aid surgical or nonsurgical intervention by localizing disease-sites in the brain, and to monitor brain activity of a healthy subject or a subject of unknown health status when the subject experiences a variety of stimuli and lack of stimuli.
However, the use of electrical signals received from, for example, the brain (e.g., electroencephalography (EEG) signals) often requires a great deal of resources. Conventional EEG tests are typically performed at specialized centers (e.g., tertiary care centers), by specialized technicians, and the results are interpreted by specialized doctors (e.g., neurologists). Thus, conventional EEG is not typically available to, e.g., first responders in an acute emergency. Instead, the first responders must rely on external signs (e.g., level of consciousness or shaking) when deciding whether a patient may have a neurological problem. Because conventional EEG is beyond the resources of even some hospitals, a patient with suspected neurological problems will often be taken to a specialized center. Even at a specialized center, it may take hours to obtain EEG results and have the results interpreted by a neurologist.
Every year in the United States alone, about 10 million people are seen in emergency departments (ED) for evaluation of altered mental state (AMS). Additionally, 5 million patients with critical conditions are admitted to intensive care units (ICU). Some of these are admitted through EDs but a majority of the patients are either transferred directly from other hospitals or are cases with postsurgical complications. In these patients, electroencephalography (EEG) is the gold-standard test for detecting seizures. While there are many causes of AMS, seizures are one of the most frequently suspected. About 10-20% of ICU patients are subject to seizures, and 90% of seizures in ICUs are non-convulsive. Where EEG is available, physicians order it to rule in/out ongoing non-convulse status epilepticus (NCSE). If the diagnosis of NCSE is made quickly, it will precipitate appropriate acute management, and will reduce unnecessary diagnostic procedures, length of hospitalization, and morbidity.
In the US alone, about 20,000 patients die of NCSE. These patients have other severe co-morbidities but ongoing non-convulsive seizures will be a significant contributing factor to their extremely high mortality rate. In fact, NCSE has a mortality rate higher than the mortality rate of convulsive status epilepticus partly because of lack of obvious behavioral signs of seizures (e.g., convulsions), which delays detection and treatment. EEG is the only way to detect ongoing seizures. Early diagnosis of NCSE is life-saving for these patients and every hour of delay in diagnosis counts. Mortality of patients with NCSE treated with a delay of 20 hours is twice as high as those treated within 30 minutes.
Because EEG is one of the oldest diagnostic tools in neurology, and because it has shown promise in saving lives, one would assume that it is widely integrated into medical practice everywhere and one might think that it is equally available to all populations at risk. This is unfortunately not the case. Inequality of access exists even in the United States, and at a wider scale on the global stage. Many hospitals in the US cannot offer an EEG platform. In addition to purchasing expensive EEG platforms, a given hospital has to hire certified EEG techs and neurologists with training in clinical electrophysiology and maintain an on-call schedule leading to a substantial management cost. For hospitals without a large number of neurological cases, this is simply not a wise investment. If they have patients with AMS in whom they suspect seizures, the patient is transferred to a larger tertiary hospital. As a result, a large number of patients may be held in NCSE condition before they are transferred.
In hospitals with EEG capability, acquiring an EEG may take hours. There may also be a longer delay from the time the EEG is acquired to the time the ordering physician receives the diagnostic information. If the EEG tech is not on duty in the hospital premises, it may take even longer. There is also a significant financial waste in using the EEG platform where it is available (especially in the United States). Many more EEGs are ordered and acquired than the number of seizures detected by these EEGs. In a retrospective review of 300 EEGs, only 1% of EEGs acquired in the ED had ongoing seizures, yet more than 95% of EEGs were ordered to confirm or rule out seizures.
The current EEG practice in the outpatient ambulatory setting also suffers from significant limitations. Patients with paroxysmal events (e.g., transient confusional state or loss of awareness) are referred to 12-48 hours of EEG monitoring at home. The recording may yield useful information only if it captures one of the infrequent paroxysmal events. Moreover, patients with dementia cannot even be tested with continuous EEG for a long period of time given their inability to follow instructions. Patients with dementia have increased risk of subclinical seizures which may in part contribute to worsening of their memory loss. About 3 million people in the United States suffer from unprovoked seizures whose occurrence is unpredictable. When patients with seizure do not recover fully, there is always a concern that they may be having indolent subclinical seizures. Parents of infants and children with such seizures take them for evaluation in local hospitals. By the time they reach the hospital, seizures may have stopped.
Traditional approaches to measuring signals from a living subject (e.g., location-specific brain signals, or electrocardiography (ECG) signals corresponding to heart activity) typically involve recording and visually displaying electrical signals acquired from the brain or other organs. Moreover, these approaches are typically used for diagnostic or scientific purposes. When represented in visual or graphical form, subtle features and attributes—and subtle changes in features and attributes—of the electrical signals may not be easily discernible. However, when sonified or converted to auditory form, these subtle features and attributes can become more apparent. Furthermore, sonification methodologies that transform the signals acquired from the living subject into vocal patterns and vocal parameters—and changes in vocal patterns and vocal parameters—that resemble a human voice make it easier to discern, upon auditory inspection, subtleties in the underlying electrical signals that correspond to bodily functions.
Additionally, traditional approaches to measuring signals from a living subject have not focused on applications beyond diagnostics and scientific research. To that end, a method of sonifying signals obtained from a living subject is provided. In particular, in some embodiments, the method transforms signals acquired from the living subject into vocal patterns and vocal parameters that can be used for applications in entertainment as well as medical diagnostics and research.
Therefore, there is a need for devices and methods that increase the availability of the diagnostic benefits of analyzing electrical signals received from a living subject's brain. In addition, traditional methods of measuring and analyzing signals from a living subject have not focused on sonification (e.g., aural presentation) of the signals.
SUMMARY OF THE INVENTIONSystems and methods for sonification of electroencephalogram signals using sonification devices in accordance with embodiments of the invention are disclosed herein. In one embodiment, a device for sonifying signals includes an input port configured to receive at least one electroencephalogram signal produced by a plurality of electrodes, the electroencephalogram signals are indicative of brain activity, an analog-to-digital converter to digitize the at least one electroencephalogram signal; an accelerometer to indicate mechanical movement affecting the at least one electroencephalogram signal; a filter, where the filer is configured to filter non-seizure-related brain wave features from the at least one digitized signal; a processor that receives the at least one digitized signals and is directed by machine readable instructions to produce a sonification of the at least one electroencephalogram signal by performing a set of operations in real time. The set of operations include conditioning the at least one digitized signals according to conditioning parameters, and modulating the at least one digitized signals according to sound synthesis parameters to produce an audio signal. The device for sonifying signals further includes a speaker system configured to generate sound based upon an audio signal output by the processor. The processor of the device interrupts sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal above a particular threshold. The processor of the device resumes sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal has returned below the particular threshold.
In a further embodiment, the filter filters the at least one digitized signal utilizing filter bandpass cutoffs as part a dual-stage filter. In another embodiment, at least one of DC-bias, AC line contamination, and non-seizure-related brain wave features are rejected by the filter bandpass cutoffs. In a still further embodiment, the dual-stage filter comprises a first stage with a first-order pole-zero infinite impulse response DC-blocking filter and a second stage with a finite impulse response filter.
In still another embodiment, conditioning the at least one digitized signal brings the at least one digitized signals into range for sonification and enhances contrasts within the at least one digitized signal. In a yet further embodiment, modulating the at least one digitized signals according to sound synthesis parameters further includes continuously modulating vocal sound parameters according to sound synthesis parameters including at least one of pitch, loudness and vowel quality. In a further embodiment again, the processor sonifies the representation of the at least one electroencephalogram signal using at least one lookup table that consolidates formant pitch mapping and inverse pitch frequency mapping.
In another embodiment again, a device for sonifying signals includes an input port configured to receive at least one electroencephalogram signal produced by a plurality of electrodes, where the electroencephalogram signals are indicative of brain activity, an analog-to-digital converter to digitize the at least one electroencephalogram signal, an accelerometer to indicate mechanical movement affecting the at least one electroencephalogram signal, a filter, where the filer is configured to filter non-seizure-related brain wave features from the at least one digitized signal, a processor that receives the at least one digitized signals and is directed by machine readable instructions to produce a sonification of the at least one electroencephalogram signal by performing a set of operations in real time. The set of operations includes conditioning the at least one digitized signal by bringing the at least one digitized signal into a predetermined range for sonification and enhancing contrast between components within the at least one digitized signal, and modulating the at least one digitized signal by continuously modulating vocal sound parameters according to sound synthesis parameters including at least one of pitch, loudness and vowel quality to produce an audio signal. The device includes a speaker system configured to generate sound based upon an audio signal output by the processor. The processor interrupts sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal above a particular threshold. The processor resumes sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal has returned below the particular threshold.
In another embodiment again, a method for sonifying signals using a sonification device is provided, the method includes receiving at least one electroencephalogram signal produced by a plurality of electrodes using an input port of a sonification device, where the electroencephalogram signals are indicative of brain activity, digitizing the at least one electroencephalogram signal using an analog-to-digital converter of the sonification device, receiving an indication of mechanical movement affecting the at least one electroencephalogram signal using an accelerometer of the sonification device, filtering non-seizure-related brain wave features from the at least one digitized signal using a filter of the sonification device, producing a sonification of the at least one electroencephalogram signal by performing a set of operations in real time using a processor of the sonification device. The set of operations includes conditioning the at least one digitized signals according to conditioning parameters, and modulating the at least one digitized signals according to sound synthesis parameters to produce an audio signal. The method further includes generating sound based upon an audio signal output using a speaker system of the sonification device, interrupting sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal above a particular threshold, and resuming sonification of the at least one electroencephalogram signal when the accelerometer indicates mechanical movement affecting the at least one electroencephalogram signal has returned below the particular threshold.
In another additional embodiment, the filter of the sonification device filters the at least one digitized signal utilizing filter bandpass cutoffs as part a dual-stage filter. In a still yet further embodiment, at least one of DC-bias, AC line contamination, and non-seizure-related brain wave features are rejected by the filter bandpass cutoffs. In still yet another embodiment, the dual-stage filter comprises a first stage with a first-order pole-zero infinite impulse response DC-blocking filter and a second stage with a finite impulse response filter.
In a still further embodiment again, conditioning the at least one digitized signal brings the at least one digitized signals into range for sonification and enhances contrasts within the at least one digitized signal. In still another embodiment again, modulating the at least one digitized signals according to sound synthesis parameters further comprises continuously modulating vocal sound parameters according to sound synthesis parameters including at least one of pitch, loudness and vowel quality. In a still further additional embodiment, the processor sonifies the representation of the at least one electroencephalogram signal using at least one lookup table that consolidates formant pitch mapping and inverse pitch frequency mapping.
Like reference numerals refer to corresponding parts throughout the drawings.
DETAILED DESCRIPTIONTurning now to the drawings, handheld sonification devices that sonifiy signals (e.g., EEG signals received from electrodes on a living subject) in real time to audio signals for instant diagnostic evaluation in accordance with various embodiments of the invention are illustrated. The sonified signals help listeners differentiate seizures from non-seizures. Previous experimentation has shown greater than 85% to 97% accuracy by even lay individuals in differentiating seizures from non-seizures when utilizing sonification devices in accordance with some embodiments of the invention.
The sonification devices can include embedded software and/or hardware instructions that direct the sonification devices to perform a number of operations during sonification. Sonification devices in accordance with many embodiments utilize a unique combination of three core operations to produce human comprehendible sound that indicates the presence or lack thereof of seizure symptoms from electrical signals. The produced sounds have human recognizable vowel and tonal sound. These three core operations can include signal conditioning, parameter modulation, and sound generation. Moreover, a multitude of embodiments of particular importance expand upon these three core operations and can optionally further include new schemes for digitization of the signal, adjustments to the signal based on accelerometer input, and further techniques for filtration of the signal.
Signal conditioning can be an initial stage in some embodiments, which acquires raw sensor data before analyzing it. During signal conditioning, many embodiments process incoming real-time signals (e.g., EEG signals gathered from electrodes placed on a living subject) to bring the signals into range for sonification and enhancing contrasts. Signal conditioning can also include filtering out less important frequencies and DC-bias (i.e., the mean value of the waveform of the signal). In a number of embodiments, a combination of different filters, compressor processes, and upsampling processes are utilized as appropriate to the requirements of different sonification applications. Moreover, sonification devices in accordance with many embodiments of the invention utilize particular coefficient tunings for filtering, scaling and thresholds determined through exhaustive testing to measure changes in ranges of these values against the specificity and sensitivity of listeners ability to distinguish seizure from non-seizure states.
Parameter modulation uses the conditioned signal to continuously modulate vocal sound parameters according to matrices of sound synthesis parameters such as (but not limited to) pitch, loudness and vowel (timbral) quality. Parameter modulation can further include audification. Audification usually is thought to refer to time-base manipulation such as speeding up or slowing down playback rate so that inaudible low-frequency or high-frequency signals are shifted into our range of hearing. With this technique a wide range of time series data becomes audible (from slow seismic to ultra-fast physics). Previous audification methods rely on time-base shifting (e.g., by compressing the EEG signals ˜60 times) whereas various embodiments of the invention can utilize direct listening to low-frequency without distorting the temporal information.
By comparison to audification, “sonification” typically refers to translation of data to sound through manipulation of parameters in some sound-producing instrument. For example, a piano can be played by quarterly gross domestic product (GDP) values. Translation of these values into a sequence of piano notes can be accomplished by mapping the GDP range to a desired range of pitches. In several embodiments, the range of the EEG signal is mapped to a pitch range but without distorting the EEG signal by directly applying it as a pitch modulation. Some embodiment further map the same signal to loudness and vowel parameters, in parallel.
In a number of embodiments, the sound generation process produces audible sound via vocal synthesizers. The produced sound can be computer-generated sung vowels in real time. This end result is a continuous sound that can provide a listener with exactly what is needed to distinguish seizure and seizure-like states from non-seizure states. In certain embodiments, the sonification device is capable of executing these operations in real time on a handheld device to perform EEG-to-sound sonification.
As noted above, sonification devices in accordance with many embodiments of the invention can be handheld or wearable devices that sonify electrical signals obtained from a subject (e.g., a living subject such as a human or a non-human animal). In some embodiments, the device includes a plurality of electrodes configured to be placed at respective locations inside the brain during neurosurgical procedures. The plurality of electrodes produces one or more electrical signals corresponding to brain activity. In several embodiments, the device includes an input port configured to couple a plurality of electrodes to the device and to receive the one or more electrical signals produced by the plurality of electrodes. In certain embodiments, the plurality of electrodes includes a first electrode (or a first set of electrodes) that is configured to be placed at any of a plurality of locations on the subject's head. That is to say, the first electrode (or first set of electrodes) is capable of being moved (e.g., intended to be moved) to different locations on the subject's head (e.g., as described with reference to
In a number of embodiments, the device is intended for use by patients, patients' family members, emergency medical personnel and/or medical doctors who are not neurologists (e.g., emergency room physicians). As can readily be appreciated, however, the device can also be a vital tool for neurologists. In some embodiments, the device includes memory (and/or makes use of cloud-based memory external to the device) that stores the EEG data. The stored EEG data allows a specialist (e.g., a neurologist) to review the EEG data (and/or sonified data) after an acute episode has passed. For example, consider a patient who visits a neurologist complaining of occasional episodes of altered mental status (AMS). In a conventional medical test for AMS, a neurologist will send the patient home with adhesive electrodes (e.g., ten or more electrodes) applied to her head along with a device to record the data from the electrodes. This type of test is sub-optimal for a number of reasons. First, it is unpleasant for the patient to have the adhesive electrodes applied to their head, resulting in a maximum of twenty-four to forty-eight hours during which the electrodes can stay on the patient. Second, if the patient's episodes only occur on average once a month, there's a high probability that the portable EEG device will not capture an episode and the inconvenience will be for naught.
In accordance with some embodiments, the electrodes of the devices described herein are applied adhesively or non-adhesively (e.g., with wet or dry electrodes) by the patient when an episode begins, thus addressing both problems. Furthermore, in some circumstances, the patient is instructed to place the handheld device at a plurality of positions on her head. For example, the patient may be instructed by the neurologist to, upon noticing the onset of an episode, place the handheld device's electrodes on the left side of her head for a short period of time (seconds to minutes) seconds and then place the handheld device's electrodes on the right side of her head for the same amount of time. Alternatively, if the patient suffers from a condition that renders them unconscious, unresponsive, and/or unreliable during such episodes, a friend or family member can be instructed to do the same.
Consider, as another example, use of the handheld device by emergency medical personnel (e.g., emergency department physicians and/or field medical personnel such as emergency medical technicians). When a patient that is unconscious, unresponsive, and/or unreliable (more generally known as having an altered mental status) arrives in the emergency department or is discovered in the field, in some circumstances, the emergency medical personnel will apply the electrodes of the devices described herein to one or more locations on the patient's head (e.g., both sides of the patient's head). The sonified electrical signals obtained from various locations on the patient's head will manifest differently depending on the patient's brain condition. For example, bilateral silence and/or quiet are, in some embodiments, representative of brain death. Sonified electrical signals that are quieter on one side of the patient's head than the other in some circumstances represent a stroke. Rhythmic activity in the sonified electrical signals in some circumstances represents a seizure. Thus, the sonified electrical signals produced by the devices described herein, in accordance with some embodiments, provide early clues to medical providers that may help direct the patient to an appropriate medical center (e.g., a stroke center) and/or obviate the need for expensive, time-consuming, and potentially risky medical procedures (e.g., lumbar punctures, colloquially known as spinal taps). The devices described herein also provide an inexpensive option to medical facilities that do not have access to EEG capabilities.
In addition, sonification devices in accordance with various embodiments of the invention can be useful for assessing neurological and/or other conditions in non-human animals. Thus, a veterinarian may, in some circumstances, use the devices described herein to “auscultate” (e.g., by sonifying, as described below) brain signals from non-human animals.
Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described embodiments. However, the invention is optionally practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.
For ease of explanation, various embodiments are described below with reference to sonification of signals representing brain activity (e.g., electroencephalography (EEG) signals) of a living subject. However, one of skill in the art will recognize that signals representing other bodily functions (e.g., electrocardiography (ECG) signal, an electromyography (EMG) signal, or an electronystagmography (ENG) signal, a pulse oximetry signal, a capnography signal, a photoplethysmography signal), and/or any other detectable signal may be substituted, or used in addition to (e.g., in conjunction with), one or more signals representing brain activity.
A. Sonification DevicesSonification device 100 can include an input port 808 configured to couple to a plurality of electrodes (not shown) to sonification device 100 and to receive electrical signals produced by the plurality of electrodes. The electrical signals correspond to the subject's brain activity. Sonification device 100 can include an analog-to-digital (A/D) converter (e.g., analog-to-digital converter 608,
Sonification device 100 can be a “pocket sized” device. Sonification device 100 can include a power button 802 and a display 804 (e.g., an LCD display/touch screen). The electrodes can be tethered to the sonification device 100 through input port 808. Sonification device 100 can received electrical signals via the input port 808. The received electrical signals can correspond to a subject's brain activity, received through input port 808. The brain activity can include a first electrical signal corresponding to left-hemisphere brain activity and a second electrical signal corresponding to right-hemisphere brain activity. The output acoustic signal can include independently audible voices corresponding to each of the first electrical signal and the second electrical signal.
Display 804 can display a graphical user interface that can enable a user of the sonification device 100 to independently control the two voices (acoustic signals or acoustic signal portions) corresponding to the distinct sensors, one for each hemisphere of the subject's brain, and their time-domain signals. For example, display 804 includes display region 810-a corresponding to the left-hemisphere signal and display region 810-b corresponding to the right hemisphere signal. In some embodiments, display regions 810 include affordances (e.g., buttons, touch screen buttons) for controlling sonification of the respective signals (e.g., turning on/off the sonification of the respective signals and/or controlling other properties of the respective sonified signals). In the example shown in
Process 300 can include receiving (302) one or more signals. The received signals can be electrical signals produced by a plurality of electrodes and received via input ports to a handheld sonification device. The one or more electrical signals can correspond to a subject's brain activity. In some embodiments, the device includes a plurality of electrodes, while in several other embodiments the device includes an input port that is coupled to the plurality of electrodes (e.g., the electrodes are tethered to the device). In certain embodiments, a plurality of representations of acoustic signals is combined to produce a combined acoustic signal. Alternatively, a combined acoustic signal can be generated by combining acoustic signals corresponding to the plurality of representations of acoustic signals produced by the digital processor system. In yet another alternative, a plurality of acoustic signals, each corresponding to one or more of the aforementioned representations of acoustic signals, are recorded on distinct tracks, where the distinct tracks are configured to enable concurrent playback of the acoustic signals recorded in those tracks.
Process 300 can include digitizing (304) the received signal. The digitization can be accomplished on one or more electrical signals using an analog-to-digital converter. An analog-to-digital converter converts a continuous analog input signal (e.g., acoustic signals) to a digital number that represents the quantity's amplitude. Most embodiments of the invention, when implemented as a sonification device, include hardware analog-to-digital converters of sufficient quality and/or number of quantization levels to minimize errors introduced by the digitization. In several embodiments, the digitization process involves oversampling which refers to a process that samples at a rate that is significantly higher than the Nyquist frequency (i.e. twice the highest frequency component) of a bandlimited digitized signal.
In order to mitigate against subject mechanical movement, process 300 can include adjusting (306) the signals based on accelerometer input. In some circumstances, the mechanical movement is due to a medical condition of the patient (e.g., a seizure and/or head shaking). In some embodiments, the mechanical movement is due to an environment in which the device is being used (e.g., the device is being used in the back of a moving ambulance). The mechanical movement can introduce artifacts that are in fact rhythmic and thus the listener may mistake the artifact rhythms for seizures.
In some embodiments, the devices performing process 300 can include an accelerometer that produces the one or more signals indicative of mechanical movement of a subject (e.g., the accelerometer is within the device's housing). In some embodiments, the sonification device performing process 300 includes an accelerometer interface for receiving the one or more signals indicative of mechanical movement of a subject. In some embodiments, the sonification device performing process 300 includes a second input port (e.g., coupled with the accelerometer interface) for receiving one or more signals indicative of mechanical movement of the subject (e.g., the accelerometer is external to the devices performing process 300 and is tethered to the devices performing process 300 through the second input port). In some embodiments, the accelerometer is located on the subject (e.g., attached to the subject or attached to an article of clothing worn by the subject).
Process 300 can then adjust (306) the one or more digitized signals based on the one or more signals indicative of mechanical movement. In some embodiments, adjusting the one or more digitized electrical signals based on the one or more signals indicative of mechanical movement includes: in accordance with a determination that an amplitude of the one or more signals indicative of mechanical movement is above a predefined movement threshold, forgoing sonification of the acoustic signal (e.g., the devices performing process 300 rejects the EEG signals by forgoing sonification). In order words, process 300 interrupts and pauses sonification of the signal due to detecting mechanical movement from an accelerometer. Conversely, in accordance with a determination that the amplitude of the one or more signals indicative of mechanical movement is below the predefined movement threshold, the process 300 sonifies the acoustic signal (e.g., process 300 accepts the EEG signal). In some embodiments, process 300 displays an indication of whether the EEG signal is being accepted or rejected (e.g., on a display or using a blinking LED).
Process 300 can include filtering (306) the one or more signals. In several embodiments, filtering the one or more signals includes configuring a first filter (e.g., a high-pass filter) in accordance with the one or more signals indicative of mechanical movement and applying the configured first filter to the one or more digitized electrical signals. In some embodiments, process 300 detects frequencies in mechanical movement and configures a frequency response of the first filter to attenuate the detected frequencies within the digitized electrical signals (e.g., the devices performing process 300 reshape the frequency response of the first filter, so that there is a change in the response of at least a first frequency with respect to the response of a second frequency). In many embodiments, configuring the first filter includes adjusting a cutoff frequency of the first filter. In various embodiments, configuring the first filter includes detecting prominent frequency modes in the mechanical movement and configuring the first filter to reject the prominent frequency modes in the digitized electrical signals (e.g., rejecting a fixed number of frequency modes, such as 3-5 modes, or rejecting any mode that has a power spectral density value in the mechanical movement above a predefined power threshold).
In several embodiments, filtering (308) the one or more signals includes, in accordance with one or more predefined criteria, applying a low-pass filter (e.g., a second filter) to the one or more digitized electrical signals. In yet further embodiments, the low-pass filter is intended to remove artifacts that are the results of convulsive muscle movements in the range of 70 Hz. In various embodiments, the low-pass filter has a cutoff frequency selected to remove artifacts that are the result of the convulsive muscle movements (e.g., a cutoff frequency below 70 Hz). In still yet more embodiments, the one or more predefined criteria are met when a user selects an option, via a user interface (e.g., the graphical user interface shown in
In several embodiments, the filters employed by process 300 include EEG signal filter bandpass cutoffs. The EEG signal filter bandpass cutoffs can, according to some embodiments of the invention, be determined through testing. The EEG signal filter bandpass cutoffs can be utilized as part a dual-stage filter which is selective of only the signal features needed for recognition. DC-bias, AC line contamination, and non-seizure-related brain wave features can be rejected by employing EEG signal filter bandpass cutoffs. The first stage can be a first-order pole-zero Infinite impulse response (IIR) DC-blocker with a cutoff range of 0.1 Hz to 1.0 Hz. Experimentally, this configuration was found to be the optimal choice for rejecting analog front-end DC-bias. The second stage can be a 501-tap finite impulse response (FIR) filter with bandpass of 0.1-3.0 Hz to 5.0-15.0 Hz. An IIR equivalent can be less optimal because of resonance problems which created low-frequency ringing that was confused with signal features of interest. While particular embodiments and ranges and values were described herein with respect to specific filter values, reasonable variation on the values presented herein can be utilized as appropriate to the requirements of specific applications in accordance with various embodiments of the invention.
Process 300 can include conditioning (310) the signals according to conditioning parameters. Signal conditioning can be an initial stage in some embodiments, which acquires raw sensor data before analyzing it. During signal conditioning, many embodiments process incoming real-time signals (e.g., EEG signals gathered from electrodes placed on a living subject) to bring the signals into range for sonification and enhancing contrasts. Signal condition can include EEG rate processing according to particular signal conditioning parameters. EEG signal filter coefficients can be derived through offline testing on (actual, recorded) device signals and the best filter settings ported to the signal conditioning stage. Moreover, parameters specific to signal conditioning can be adapted according to various firmware configurations.
Other conditioning parameters of various other embodiments can include the following values as bases for ranges of success: thresholds of 1-25 μV, full-range scaling of 30-60 μV and compression of 1.5-3.0. Signal rectification i.e., absolute value can be applied to the filtered signal and then thresholding can be used to reject low-amplitude signal noise. Signal values below a threshold can be set to zero. Above a threshold signals can be scaled to create a fixed range of 0.0-1.0 with full range scaling and then can be compressed to raise the prominence of small features. Signal values can be boosted by taking the power-law exponent of the signal. In many instances, values which exceed full scale after compression can be clipped to full scale (1.0).
Process 300 can include modulating (312) the one or more signals according to sound synthesis parameters. Parameter modulation can include continuously modulating vocal sound parameters according to matrices of sound synthesis parameters such as pitch, loudness and vowel (timbral) quality. Parameter modulation refers to the matrix of sound synthesis parameters (pitch, loudness, vowel) and how a single conditioned EEG signal modulates each parameter in the group. Relationships can be expressed as offset and scale coefficients. The offset and scale coefficients can in many embodiments be determined ahead of device development in software-based research. The matrices determine the perceived identity of the vocal sound. While not necessarily a realistic replica of a singer, there is a recognizable “device identity” that can result from any particular matrix. The set of parameter coefficients chosen can make this identity invariant and almost trademark-like (for example, in the range of: “111”, “ahh”, “ehh”, “eee”, “ohh”, “ooo”). As an example, a sample matrix is provided herein will result in the following sonified output:
-
- Pitch offset in the range of 50-150 Hz—This sets the “hum” base pitch of the vocalist for below-threshold EEG:
- Pitch scale in the range of 110-440 Hz—This sets the pitch excursion for full-range EEG.
- Amplitude offset in the range of 0.0001-0.01)—This sets the “hum” amplitude for below-threshold EEG.
- Amplitude scale in the range of 0.05-2.0—This sets the maximum loudness for full-range EEG.
- Vowel offset in the range of 0.0-1.0—This sets the “hum” vowel for below-threshold EEG.
- Vowel scale in the range of 0.05-2.0—This sets the vowel interpolation amount for full-range EEG.
- Vowel lookup table—a combination of these 6 vowels are used.
Process 300 can include generating (314) sound from the one or more signals. Device sound generation can be real-time and heard via onboard loudspeakers of devices performing process 300. The loudspeaker and device enclosure can affect sound quality. Most embodiments utilize particular speaker types and enclosure for better acoustics linearity. For instance, a CUI Inc. MODEL: GF0401M can be utilized for good performance in several embodiments. This model provides for a speaker that is open to the front and to the inside of the enclosure. A different model, 1 cm smaller diameter, was 12 dB quieter and had a more nasal sound (the bottom octave of sound compared to the best choice was mostly missing). Enclosures which have been prototyped and tested included fully open, sandwich with open edges, and ported boxes. Ported boxes have been found to yield the best sound for most embodiments, the best sound being loud and resonant for rendering the vocalist “identity” as described above. As can readily be appreciated, the specific speaker and enclosure utilized in a sonification device largely depend upon the requirements of a given application.
Device sound generation is real-time and heard via an onboard loudspeaker. The loudspeaker and device enclosure affects sound quality. Real-time deadlines for digital sound generation are exacting. If the firmware misses a deadline, annoying clicks (brief sound dropouts) can be heard. Firmware in preferred embodiments was architected to support consistent audio on the relatively inexpensive processor chosen. Sound generation can take advantage of numerous hardware design techniques to speed up and enhance performance in numerous embodiments. The following discussion describes some of these techniques.
Sonification often requires mapping of data with a linear range into pitch frequencies of the human voice. The most common and natural way to do this is with the well-defined midi-to-frequency function: 440*2{circumflex over ( )}(midinote−69)/12 which requires both a power and a divide operation. Sonification process 300 creates formant audio that avoids fringing effects, discontinuity clicks, phase distortion, spectral modulation, and other problems with simpler algorithms. However, the operations of process 300 utilize a division to calculate each formant harmonic as shown in the midi-to-frequency function. Thus a two formant synthesis with interpolation of between upper and lower frequencies involve 4 division operations (typically multiple clock cycle processor operations) per sample. Process 400 can eliminate these division operations using an interpolated look-up table for the inverse of the pitch frequency (look up of 10), thus requiring only a multiply for each formant. Process 400 can be executed in conjunction with process 300 or as a sub-process of process 300.
Formant synthesis refers to creating audio signals that sound like a human singing voice or in more general terms, signals that have an acoustic resonance like the human vocal tract. Formant synthesis is useful in sonification applications where complex non-audio data is mapped to a human-like singing voice that can clarify or distinguish features in the data that may otherwise be difficult to detect. Process 400 provides for a fast method of synthesizing formant audio signals is described that achieves high quality real-time performance with a smaller, more efficient implementation in real-time embedded CPU/DSP processor firmware, FPGA, or ASIC devices. Process 400 receives (402) a signal for formant synthesis. This receipt can in many embodiments be in CPU/DSP processor firmware, FPGA, or ASIC devices.
Process 400 can include interpolating (404) signals using consolidated lookup tables. This consolidates interpolation for look-up into two tables for the formant pitch mapping and inverse pitch frequency mapping. The consolidation eliminates computationally expensive divide and power operations while maintaining synchronous operation and the interpolation of the pitch and formant frequencies from linear data.
Process 400 can include performing formant synthesis (406). High quality formant synthesis can in many embodiments be performed using a bank of linked oscillators, such as the phase-synchronous oscillators, where a single phasor is shared by the modulator and all carriers. In a typical implementation the bank is constructed with any number of harmonic outputs that are tapped off of a single common phasor. In practice, a bank of four (or more) carrier oscillators of this kind will can be used to generate a vocal sound. These can create human voice phonemes of 2 (or more) formants represented by a time-varying distribution pitch, harmonic, and amplitude. Process 400 can then complete sonification by generating (408) sound comparably to process 300 as discussed above.
Although specific processes for sonification of EEG signals are described above with reference to
In some embodiments, as shown in
In some embodiments, sensors 510 are provided to interface with a living subject's brain to obtain (e.g., sense and/or acquire) sensor time-domain signals (e.g., sensor time-domain signal 601,
As an example, signals (e.g., sensor time-domain signal 601,
In some embodiments, although not shown in
In some embodiments, heartbeat pulse sensors sense voltages corresponding to heart electrical activity. In alternative embodiments, heartbeat pulse sensors sense electrical currents corresponding to heart electrical activity. In some implementations, heartbeat pulse sensors sense differential voltages (e.g., differences in voltage values) between two measurement locations (e.g., between two sensing elements). For example, when a respective heartbeat pulse sensor includes two or more sensing elements (e.g., electrodes) positioned at respective positions external to the human body, the respective heartbeat pulse sensor senses differential voltages (e.g., bipolar voltages) between the two or more sensing elements located at the respective positions. In some implementations, a “twelve-lead electrocardiogram” is constructed by referencing each sensing element of a set of sensing elements to one or more other sensing elements to produce a corresponding set of differential voltage signals (e.g., a twelve-lead set of differential voltage signals), each of which is a respective sensor time-domain signal 601 (
In some embodiments, arrays of sensors (e.g., sensors 510) are designed to record intracranial EEG and produce a plurality of sensor time-domain signals (e.g., sensor time-domain signals 601,
In some implementations, analog front end 520 receives sensor time-domain signals (e.g., sensor time-domain signal 601,
As shown in
In some embodiments, a respective signal conditioning module 530 includes a data converter (e.g., an analog-to-digital converter for converting an analog filtered sensor time-domain signal obtained from sensors 510 to a corresponding digital representation), an up-sampler and a digital low-pass filter. In some implementations, signal modulators 540 receive the digitized time-domain signals output by signal conditioning modules 530, and concurrently generate a set of acoustic parameters, including a plurality of time-varying acoustic parameters from (e.g., using) the digitized time-domain signals. One or more of the plurality of time-varying acoustic parameters is modulated in accordance with at least the signal value of the time-domain signal (e.g., time-domain signal 618,
In some embodiments, a plurality of representations of acoustic signals is combined to produce a combined acoustic signal. Alternatively, a combined acoustic signal is generated by combining acoustic signals corresponding to the plurality of representations of acoustic signals produced by digital processor system 560. In yet another alternative, a plurality of acoustic signals, each corresponding to one or more of the aforementioned representations of acoustic signals, are recorded on distinct tracks, where the distinct tracks are configured to enable concurrent playback of the acoustic signals recorded in those tracks.
In some embodiments, analog front end 520 includes interface circuitry (e.g., interface analog circuitry 602) to interface with a respective sensor 510, for example, by way of providing bias voltages and/or currents to the respective sensor 510, buffering signals (e.g., using a buffer amplifier) received from sensors 510 and/or providing appropriate coupling conditions (e.g., providing appropriate input impedance) for interfacing with the signals received from sensors 510.
Alternatively, or in addition, according to some implementations, analog front end 520 includes one or more amplifiers 604 and/or filters 606 to pre-process (e.g., amplify and/or filter) sensor time-domain signals corresponding to brain electrical activity or heart electrical activity (e.g., sensor time-domain signal 601,
In some embodiments, the signal conditioning module (e.g., signal conditioning module 530) includes a data converter (e.g., analog-to-digital converter 608) for converting an analog filtered sensor time-domain signal obtained from sensors 510 (optionally after pre-processing by analog front end 520) to a corresponding digital representation with a predefined sampling rate (e.g., a sampling rate between 500 Hz and 2 kHz, such as 500 Hz; or more generally a sampling rate between 800 Hz to 4 kHz). Signal conditioning module 530 includes an up-sampler (e.g., up-sampler 612) to up-sample (e.g., increase the sampling rate of) the digital representation of the analog filtered sensor time-domain signal to produce a first intermediate signal (e.g., first intermediate signal 614). In some embodiments, the digital representation of the analog filtered sensor time-domain signal is up-sampled to produce a first intermediate signal having an audio sampling rate, for example, a sampling rate (e.g., 48 kHz) used in conventional audio applications. In some implementations, the first intermediate signal (e.g., first intermediate signal 614) produced by up-sampler 612 has a sampling rate of 48 kHz.
In some implementations, signal conditioning module 530 includes one or more digital low-pass filters (e.g., digital low pass filter 616) for filtering first intermediate signal 614 so as to produce time-domain signal 618. In some implementations, digital low pass filter 616 is a second order low-pass Butterworth filter with a 250 Hz corner frequency (also called a cutoff frequency). Digital low pass filter 616 filters first intermediate signal 614 to produce time-domain signal 618. In some embodiments, up-sampler 612 and digital low pass filter 616 are implemented in digital signal processor 610, sometimes called a DSP. In some other implementations, up-sampler 612 and digital low pass filter 616 are implemented in circuitry. Alternatively, up-sampler 612 and digital low pass filter 616 are implemented in software executed by a general-purpose processor. Without limitation, it is noted that up-sampling and then low pass filtering the digital representation of the analog filtered sensor time-domain signal may be used to convert the output of one or more sensors (e.g., extra-cranial sensors and/or heartbeat/pulse sensors) to a form that is suitable for use with a music or other audio synthesizer, while removing or limiting artifacts produced by the conversion process.
In some embodiments, signal modulator 540 includes a vibrato modulator (e.g., vibrato modulator 620-a) which generates a vibrato or frequency-control parameter (e.g., frequency-control parameter 622-a). In some implementations, the vibrato modulator (e.g., vibrato modulator 620-a) obtains a base frequency or pitch (e.g., a base frequency such as 50 Hz, 500 Hz, or any suitable frequency in the range of 50 Hz to 4 kHz) and modulates the base frequency in accordance with the signal value (e.g., amplitude, intensity and/or power) of the time-domain signal (e.g., time-domain signal 618). In other implementations, the vibrato modulator generates a vibrato or frequency-control parameter in accordance with the signal value of the time-domain signal (e.g., time-domain signal 618) that does not incorporate the base frequency or pitch. The amount of vibrato, as represented by the vibrato or frequency-control parameter, controls variations in frequency in the synthesized audio signal (i.e., the representation of an acoustic signal corresponding to the time-domain signal).
In some embodiments, signal modulator 540 includes a vowel modulator (e.g., vowel modulator 620-b) which generates a vowel-control parameter (e.g., vowel-control parameter 622-b). In some implementations, a vowel modulator (e.g., vowel modulator 620-b) selects a sequence of acoustic waveform patterns from a set of N (e.g., N is an integer in the range of 2 to 15, such as N=12) acoustic waveform patterns comprising a sequence of phoneme waveform patterns (e.g., phoneme patterns corresponding to sounds in spoken language). In some implementations, the phoneme patterns include a plurality of vowel waveform patterns, and optionally include phoneme patterns that are not vowel waveform patterns (e.g., “sss” or “vvv”). In some implementations, each of the phoneme patterns is distinguished from the other phoneme waveform patterns in the set with respect to acoustic characteristics such as formants. In some embodiments, vowel modulator (e.g., vowel modulator 620-b) modulates a rate at which the acoustic waveform (e.g., vowel waveform) patterns are sequentially selected in accordance with the signal value (e.g., amplitude, intensity and/or power) of the time-domain signal. For example, vowel modulator (e.g., vowel-control parameter 622-b) modulates a rate at which acoustic waveform patterns from a set of 12 acoustic waveform patterns are sequentially selected in accordance with the signal value (e.g., amplitude) of the time-domain signal (e.g., time-domain signal 618). For example, an increase in signal value (e.g., amplitude) of the time-domain signal (e.g., time-domain signal 618), causes vowel modulator (e.g., vowel-control parameter 622-b) to sequentially select acoustic waveform patterns from a set of 12 acoustic waveform patterns more rapidly or at an increased rate; and conversely, a decrease in signal value (e.g., amplitude) of the time-domain signal (e.g., time-domain signal 618), causes vowel modulator (e.g., vowel-control parameter 622-b) to sequentially select acoustic waveform patterns from a set of 12 acoustic waveform patterns more gradually (e.g., slowly) or at a decreased rate.
In some embodiments, signal modulator 540 includes an intensity modulator (e.g., intensity modulator 620-c) which generates an intensity-control parameter (e.g., intensity-control parameter 622-c). For example, an intensity modulator (e.g., intensity modulator 620-c) computes a time-varying amplitude value in accordance with the signal value (e.g., amplitude, intensity, and/or power) of the time-domain signal (e.g., time-domain signal 618) and generates a time-varying intensity-control parameter (e.g., intensity-control parameter 622-c) corresponding to the computed time-varying amplitude value. In some implementations, an increase in signal value (e.g., amplitude) of the time-domain signal (e.g., time-domain signal 618), causes the time-varying amplitude value—and corresponding time-varying intensity-control parameter (e.g., intensity-control parameter 622-c)—computed by intensity modulator (e.g., intensity modulator 620-c) to increase. Conversely, a decrease in signal value of the time-domain signal (e.g., time-domain signal 618), causes the time-varying amplitude value—and corresponding time-varying intensity-control parameter (e.g., intensity-control parameter 622-c)—computed by intensity modulator (e.g., intensity modulator 620-c) to decrease.
In some embodiments, signal modulator 540 includes a pitch modulator (e.g., pitch modulator 620-d) which generates a pitch-control parameter (e.g., pitch-control parameter 622-d). In some embodiments, pitch modulator (e.g., pitch modulator 620-d) selects a base frequency (e.g., corresponding to an acoustic pitch) in accordance with a spatial location of sensing the time-domain signal. In some embodiments, pitch modulator (e.g., pitch modulator 620-d) generates a time-varying pitch-control parameter in accordance with the selected base frequency and the signal value (e.g., amplitude, intensity and/or power) of the time-domain signal (e.g., time-domain signal 618,
In some embodiments, signal modulator 540 generates, obtains or otherwise provides one or more sonic identity parameters 622-e. In some embodiments, signal modulators 540 selects a sonic identity (for example, specific defining acoustic characteristics; e.g., acoustic characteristics associated with specific musical instruments) in accordance with a respective time-domain signal (e.g., a time-domain signal corresponding to a spatial location in the brain of sensing or a spatial location in the heart of sensing by way of sensors 510 located at different spatial locations in the brain and heart, respectively) and generates, obtains or otherwise provides one or more sonic identity parameters 622-e in accordance with the selected sonic identity. For example, for a time-domain signal obtained from the left hemisphere in the brain (e.g., obtained from sensors 510-5, 510-6 in
One or more of the plurality of time-varying acoustic parameters (e.g., frequency-control parameter 622-a, vowel-control parameter 622-b, and/or intensity-control parameter 622-c) are modulated in accordance with at least the signal value (e.g., amplitude, intensity, and/or power) of the time-domain signal (e.g., time-domain signal 618,
A synthesizer module produces a representation of an acoustic signal from the one or more acoustic parameters corresponding to a respective time domain signal 618. In some embodiments, a synthesizer module (e.g., synthesizer module 550) combines the concurrently generated set of acoustic parameters (e.g., the acoustic parameters produced by signal modulators 540 described above) to produce a representation of an acoustic signal (e.g., representation of acoustic signal 630) corresponding to the time-domain signal (e.g., time-domain signal 618,
As shown in
As shown in
Digital processor system 560 optionally includes sensor interfaces 770 for interfacing with sensors 510 (
In some embodiments, sensor interface 770 includes an impedance detector that detects whether sensors 510 (e.g., electrodes) are attached to the subject. In some embodiments, when DSP 560 determines that the impedance looking into sensors 510 is below an impedance threshold, DSP 560 determines that sensors 510 are attached to the subject. On the other hand, when DSP 560 determines that the impedance looking into the sensors 510 is above the impedance threshold, DSP determines that sensors 510 are not attached to the subject (e.g., the circuit formed by DSP 560 and sensors 510 is an open circuit).
Digital processor system 560 optionally includes an accelerometer 784 (e.g., a 3-axis accelerometer) that measures mechanical movement of the subject and/or the device (e.g., produces one or more electrical signals corresponding to mechanical movement of the subject and/or device).
Digital processor system 560 optionally (and typically) includes a battery 782 (e.g., a rechargeable battery) and charger 780, to provide power to digital processor system 560 and enable operation of digital processor system 560 without connection to an external power source (except to charge battery 782). In some embodiments, battery 782 is charged, via charger 780, when an external power source is connected to system 560 via a USB port or micro-USB port of the device.
Memory 710 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 710 optionally includes one or more storage devices remotely located from the CPUs 702. Memory 710, or alternately the non-volatile memory devices within memory 710, comprises a non-transitory computer readable storage medium. In some embodiments, memory 710, or the computer readable storage medium of memory 710 stores the following programs, modules and data structures, or a subset thereof:
-
- Operating system 712 that includes procedures for handling various basic system services and for performing hardware dependent tasks;
- Network communication module 714 that is used for connecting digital processor system 560 to other computers via the one or more communication network interfaces 709 (wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;
- User interface module 716 that receives commands from the user via one or more input devices 707 of user interface 705, generates user interface objects in display device 706, and optionally generates representations of signals corresponding to brain and/or heart activity, information corresponding to sensors and sensor interfaces, and information related to the configuration of body interface system 500 for display on display device 706;
- Parameter controller 720 that controls (e.g., executes instructions for) the generation of the set of acoustic parameters, including a plurality of time-varying acoustic parameters (such as a frequency-control parameter (sometimes called a vibrato parameter), a vowel-control parameter, an intensity-control parameter, a pitch-control parameter, and/or an identity-control parameter). Parameter controller 720 also interacts with sensor interfaces 770 to facilitate selection of parameters (e.g., any of the aforementioned parameters) and corresponding parameter values based on the sensors selected and sensor signals obtained (e.g., based on a spatial location in the brain and/or heart from which the time-domain signal is sensed). For example, sensor interface module 718 interfaces with parameter controller 720 to communicate a set of parameters, corresponding to one or more of pitch, vowel selection, vibrato, intensity (amplitude), and sonic identity parameter, selected in accordance the selected sensor, or in accordance with a spatial location in the brain and/or heart of sensing a time-domain signal;
- Stored control parameter sets 722 that include one or more sets of signal parameters or values corresponding to signal parameters (for example, one or more values of base frequencies, a set of acoustic waveform patterns corresponding to phoneme patterns, one or more sonic identities, etc.);
- Signal conditioning modules 530 up-sample and low pass filter the sensor time-domain signal to produce a time-domain signal representing brain and/or heart activity;
- Signal modulators 540 concurrently generate a set of acoustic parameters, including a plurality of time-varying acoustic parameters, for example, a frequency-control parameter (e.g., frequency-control parameter 622-a,
FIG. 6C ), a vowel-control parameter (e.g., vowel-control parameter 622-b,FIG. 6C ), a time-varying intensity-control parameter (e.g., intensity-control parameter 622-c,FIG. 6C ), a pitch-control parameter (e.g., pitch-control parameter 622-d,FIG. 6C ), and/or an sonic identity parameter (e.g., sonic identity parameters 622-e,FIG. 6C ); - Synthesizer modules 550 combine the concurrently generated set of acoustic parameters to produce a representation of an acoustic signal (e.g., representation of acoustic signal 630,
FIG. 6C ) corresponding to the time-domain signal (e.g., time-domain signal 618,FIG. 6B , produced by signal conditioning module 530); - Low-pass filter modules 580 configure and/or apply a low-pass filter to electrical signals received from sensor interface 770 to remove electrical noise that results from convulsive muscle movements (e.g., low-pass filter module applies a fixed-configuration low-pass filter, or a low-pass filter configured by system 560 (as discussed in more detail below), or a user-selectable low-pass filter, which is in addition to the low-pass filter applied by signal conditioning module 530);
- Accelerometer modules 590 process accelerometer information from accelerometer 784 and optionally configure and/or apply filters to remove mechanical movement artifacts in electrical signals received from sensor interface 770; and
- Optional local data storage 570 that stores data corresponding to the one or more electrical signals (e.g., data storage 570 stores raw EEG data and/or audio data so that the data can be reviewed later by, e.g., a specialist). In some implementations, data storage 570 includes a removable non-volatile memory card, such as a micro SD flash memory card (see “μSD” in
FIG. 7B , which represents a micro-SD card “reader” for receiving and interfacing a micro SD flash memory card to a microcontroller). As an alternative, or in addition to data storage 570, digital processor system 560 communicates with cloud-based storage (e.g., storage that is remote from the device) to store data corresponding to the one or more electrical signals.
Each of the above identified elements is optionally stored in one or more of the previously mentioned memory devices of digital processor system 560, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules is optionally combined or otherwise re-arranged in various embodiments. In some embodiments, memory 710 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 710 optionally stores additional modules and data structures not described above. For example, in some embodiments, memory 710 stores one or more EEG data analysis modules 724, for analyzing EEG data received by digital processor system 560 and conveying one or more results to a user of the device (e.g., via display 706 or speaker(s) 708), or to a remote device or user via communications interface 704. The one or more EEG data analysis modules 724, if provided, may use any of a number of seizure detection methods, including EEG data analysis methods previously developed or developed in the future.
Although
For ease of explanation, process 800 is described with reference to brain activity (e.g., process 800 is described as a method for sonifying one or more electrical signals obtained from a subject's brain). In some embodiments, however, process 800 is used to sonify signals representing other bodily functions (e.g., electrocardiography (EEG) signals).
The digital processor system obtains (802) a time-domain signal (e.g., time-domain signal 618,
The time-domain signal representing brain activity is obtained by conditioning (806) a sensor time-domain signal obtained from the one or more electrodes. For example, as shown in
In some embodiments, the conditioning includes (808) up-sampling the sensor time-domain signal to produce a first intermediate signal and low pass filtering the first intermediate signal to produce the time-domain signal representing brain activity or a signal corresponding to the time-domain signal representing brain activity. For example, as shown in
The digital processor system concurrently generates (810) a set of acoustic parameters (e.g., see operations 812-848), including a plurality of time-varying acoustic parameters. In this context, parameters are “concurrently generated” even if they are literally generated serially by single-threaded processor, when the resulting parameters are used or applied concurrently for generating an audio signal, or a representation of an audio signal. Typically, two or more concurrently generated parameters are generated or updated in response to a stream of digital signal values corresponding to the time-domain signal.
One or more of the plurality of time-varying acoustic parameters is modulated (812) in accordance with at least the signal value of the time-domain signal. For example, as explained above with reference to
In some embodiments, generating a plurality of acoustic parameters includes (814) generating a vibrato or frequency-control parameter (as described herein with respect to operations 816-820). For example, as described above with reference to
In some embodiments, the digital processor system obtains (816) a base frequency. In some embodiments, the digital processor system modulates (818) the base frequency in accordance with the signal value of the time-domain signal, to produce a signal-dependent variation of the base frequency. In some embodiments, the digital processor system generates (820) a frequency-control parameter corresponding to the signal-dependent variation of the base frequency. For example, as explained above, vibrato modulator (e.g., vibrato modulator 620-a,
In some embodiments, generating a plurality of acoustic parameters includes (822) generating a vowel-control parameter (as described herein with respect to operations 824-830). For example, as shown in
In some embodiments, a digital processor (e.g., digital processor 560) sequentially selects (824) acoustic waveform patterns from a ordered set of N acoustic waveform patterns, the set of N acoustic waveform patterns arranged in a predefined order, where N is an integer greater than 2. In some embodiments, the sequence of selected acoustic waveform patterns includes (826) a sequence of vowel waveform patterns. In some embodiments, a digital processor (e.g., digital processor 560) modulates (828) a rate of sequentially selecting acoustic waveform patterns in accordance with the signal value of the time-domain signal, to produce a signal-dependent rate of variation of acoustic waveform patterns. In some embodiments, a digital processor (e.g., digital processor 560) generates (830) a vowel-control parameter corresponding to the signal-dependent rate of variation of acoustic waveform patterns.
For example, as described above, vowel modulator (e.g., vowel-control parameter 622-b) modulates a rate of sequentially selecting acoustic waveform patterns from a set of 12 acoustic waveform patterns in accordance with the signal value of the time-domain signal (e.g., time-domain signal 618,
In some embodiments, the digital processor system generates (832) a plurality of acoustic parameters, as described herein with respect to operations 834-836. In some embodiments, the digital processor system computes (834) a time-varying amplitude value in accordance with the signal value of the time-domain signal. In some embodiments, the digital processor system generates (836) a time-varying intensity-control parameter corresponding to the computed time-varying amplitude value.
For example, as described above in relation to
In some embodiments, the digital processor system generates (838) the set of acoustic parameters, as described herein with respect to operations 840-842. In some embodiments, the digital processor system selects (840) a base frequency in accordance with a spatial location of sensing the time-domain signal. In some embodiments, the digital processor system generates (842) a time-varying pitch-control parameter in accordance with the signal value of the time-domain signal, and optionally in accordance with the selected base frequency. For example, as shown in
Without limitation with respect to other implementations, in some implementations the set of acoustic parameters are generated, in accordance with a set of instructions executed by one or more processors of a digital processor system, as described above. The following is an example of a pseudo-code representation of instructions for generating the set of acoustic parameters, once per time period (e.g., once every 10 milliseconds), where SigVal is the signal value for the current time period:
// amplitude
amplitude.param=max(0.0, c1+c2*SigVal);
//pitch
pitch.param=ConvertMidiToFreq(c3−c4*SigVal);
// vibrato
vibrato-gain.param=pitch.param * (2c5−1);
vibrato.param=vibrato.param+c6*SigVal;
vibrato.freq.param=max(0.0, min(c7, c8+vibrato.param));
//vowel
vow=vow+(c9 * SigVal);
vowel.param=integer (abs(vow)) modulo 12;
where, in one example, the following coefficient values are used: c1=0.1, c2=20, c3=45, c4=5, c5=0.05, c6=4, c7=8.0, c8=4.5, c9=20. Further, “ConvertMidiToFreq” is a function for converting a midi note to a frequency value, “max” is a function that outputs the maximum of its input values, “min” is a function that outputs the minimum of its input values, “abs” is a function that outputs the absolute value of its input, and “integer” is a function that outputs the integer portion of its input. In another example, in which two or more multiple time-domain signals are processed to produce a corresponding number of audio signals (sometimes called voices for ease of reference), one or more of the coefficients (e.g., c1 to c9 in the above example) are different for different ones of the audio signals, thereby producing audio signals that are audibly distinct. In one example, coefficients c3 (corresponding to base frequency) and c6 (corresponding to amount of vibrato) and c9 (corresponding to a rate at which the audio signal traverses a sequence of vowels or phonemes), have different values for each audio signal.
For example, as shown in
In some embodiments, the digital processor system generates (844) the set of acoustic parameters, as described with respect to operations 846-848. In some embodiments, the digital processor system selects (846) a sonic identity in accordance with a spatial location in the brain (or, alternatively, on the surface of the cranium) of sensing the time-domain signal. In some embodiments, the digital processor system generates (848) an identity-control parameter in accordance with the selected sonic identity. For example, as shown in
For example, as shown in
The digital processor system combines (850) the concurrently generated set of acoustic parameters to produce a representation of an acoustic signal corresponding to the time-domain signal. For example, as shown in
In some embodiments, the digital processor system applies (852) the concurrently generated set of acoustic parameters to a music synthesizer or music synthesizer module to produce the representation of an acoustic signal. In some embodiments, the music synthesizer or music synthesizer module uses (854) frequency modulation synthesis. For example, as shown in
In some circumstances, the acoustic signal, in audible form, manifests (856) one or more audibly discernible variations across a plurality of stages of a brain activity event. In some embodiments, the brain activity event is (858) an epileptic seizure, or other event corresponding to abnormal brain activity. For example, the acoustic signal corresponding to representation of acoustic signal 630, in audible form, manifests one or more audibly discernible variations (e.g., variations in vibrato, in rate of change of vowel, and/or in intensity) across a plurality of stages of a brain activity event. In some embodiments in which the brain activity event is an epileptic seizure, the acoustic signal in audible form manifests one or more audibly discernible variations (change in volume, pitch, rate of vowel change) across the stages (e.g., normal state, pre-ictal phase, seizure phase and postictal phase) of the epileptic seizure. For example, the acoustic signal is periodic and has higher amplitude during the seizure phase, and is chaotic (has lower or little periodicity) and has lower amplitude during the normal state.
In some implementations, the brain activity event for which brain electrical signals are sonified is not an epileptic seizure, and instead is a respective brain activity event that is the subject of analysis or monitoring. For example, in some implementations the brain activity event for which brain electrical signals are sonified comprises brain activity while the human subject performs various tasks (e.g., mental tasks, physical tasks, operating an apparatus, answering questions, playing a musical instrument, taking an exam, performing or attempting to perform multiple tasks or functions concurrently, etc.), brain activity associated with experiencing various external stimuli, brain activity associated with physiological functions, brain activity associated with various diseases, and the like.
E. Signal Reception DevicesIn various embodiments, device 900 may share any of the features described with reference to device 1000 (
In some implementations, device 900 includes a plurality of electrodes 902 (e.g., electrode 902-1 and electrode 902-2). In some embodiments, electrodes 902 are dry electrodes, while in other embodiments, electrodes 902 are wet electrodes. In some embodiments, various components of device 900 are incorporated into housing 904. In some embodiments, electrodes 902 are rigidly coupled with housing 904 (e.g., the plurality of electrodes includes dry or wet electrodes, at least a portion of which extend or protrude from (e.g., outside of) housing 904 of device 900). In some other embodiments, electrodes 902 are inserted in or held by a headband and coupled with circuitry in housing 904 via one or more electrical connectors.
In some embodiments, housing 904 includes handle 906. For example, handle 906 of device 900 is used to apply the device to the subject in a manner similar to how a physician would apply a stethoscope to a patient (e.g., handle 906 is gripped between the user's thumb and index finger). Thus, in some embodiments, device 900 is small enough to be comfortably held between a user's thumb and index finger. In some embodiments, housing 904 fits within a cube with dimensions of 10 centimeters (cm) by 7 centimeters by 7 centimeters. In some embodiments, housing 904 fits within a cube with dimensions of 7 centimeters (cm) by 5 centimeters by 5 centimeters. In some embodiments, housing 904 fits within a cube with dimensions 6 centimeters (cm) by 4 centimeters by 4 centimeters. In some embodiments, housing 904 fits within a sphere having a radius of 3 centimeters (cm).
Electrodes 902 are configured to be placed at respective locations on the subject's body. For example, in some embodiments, the electrodes of the plurality of electrodes 902 are configured to be placed at respective locations on the subject's head. The plurality of electrodes includes a first electrode 902-1 that is configured to be placed at any of a plurality of locations on the subject's head. That is to say, the first electrode is capable of being moved (e.g., intended to be moved) to different locations on the subject's head. As shown in the present example, device 900 is itself intended to be placed at any of a plurality of locations on the subject's head. For example, a physician (or the subject herself) might place device 900 on the left side of the subject's head to “listen” to brain activity in the left hemisphere of the subject's brain, then subsequently place device 900 on the right side of the subject's head to “listen” to brain activity in the right hemisphere of the subject's head. Thus, in some embodiments, device 900 allows the user to perform a bilateral aural comparison of the subject's brain activity.
In some embodiments, multiple electrodes are attached to the subject's head using a headband or adhesive and coupled with housing 904 via one or more electrical connectors. For example, a first set of electrical connectors may connect a first set of electrodes positioned near the subject's left hemisphere to circuitry in housing 904, and a second set of electrical connectors may connect a second set of electrodes positioned near the subject's right hemisphere to circuitry in housing 904. Using such embodiments, a physician (or the subject herself) can listen to brain activity in the left hemisphere of the subject and brain activity in the right hemisphere of the subject.
In some embodiments, the plurality of electrodes 902 is grouped into sensors 510 (e.g., first electrode 902-1 is grouped into sensor 510-9 with several other electrodes, which have not been labeled for visual clarity). In some embodiments, sensors 510 include only a single electrode. In some embodiments, sensors 510 each includes two or more electrodes that are electrically (e.g., conductively) coupled (e.g., the two or more electrodes within a respective sensor 510 provide a better electrical contact with the subject's skin). The plurality of electrodes produces one or more electrical signals corresponding to brain activity. In some embodiments, the one or more electrical signals are bipolar (e.g., differential) signals representing a voltage difference between two of the electrodes (e.g., electrode 902-1 and electrode 902-2).
Device 900 includes an analog-to-digital (A/D) converter (e.g., analog-to-digital converter 608,
Device 900 includes speaker system 908 that sonifies the representation of the acoustic signal. In some embodiments, speaker system 908 is positioned within housing 904. Alternatively, or in addition to speaker system 908, in some embodiments, device 900 includes a first output port through which to attach headphones. In some implementations, not shown, speaker system 908 is embodied as headphones/ear pieces (e.g., in a manner similar to conventional stethoscopes).
In some embodiments, the one or more electrical signals represent electroencephalography (EEG) data that are concordant with laboratory EEG data, such as laboratory EEG data obtained from one or more human test subjects. In some embodiments, the handheld device is sufficiently sensitive that it produces verified EEG data (e.g., rather than muscle tone data and/or a mixture of EEG and muscle tone data). In some embodiments, the laboratory EEG data is obtained from electrodes surgically implanted into one or more human test subjects' brains to verify that the handheld device produces EEG data that are concordant with what would be obtained in a laboratory setting.
In some embodiments, the processor (e.g., digital processor system 560,
For example, in some embodiments the device is configured to produce a first sound and/or first tactile feedback when it is receiving electrical signals that are not concordant with laboratory EEG data. The user thereby knows to change the position of the device's electrodes until the first sound and/or first tactile feedback stops being produced. Alternatively, in some embodiments, the device is configured to suppress the generation of a representation of an acoustic signal and/or the sonification of the acoustic signal until the received electrical signals are determined by the device to be concordant with laboratory EEG data. In such embodiments, the user thereby knows to change the position of the device's electrodes until the device produces “brain activity sonification” sounds, and furthermore the user can have a high degree of confidence that the sounds produced are in fact a sonification of the subject's EEG signals.
In some embodiments, device 900 includes memory (e.g., memory 710,
In some embodiments, device 900 includes a voice recorder that stores voice data in the device's memory (e.g., data storage 570 of memory 710,
In some embodiments, application of an electrode of the plurality of electrodes to, respectively, the left and right halves of the human subject's head yields audible indicia of the presence or absence of stroke. For example, in some embodiments, a stroke effecting the right (or left) side of the subject's head is indicated when the right (or left) side of the subject's head yields a substantially quieter sonified acoustic signal than the left (or right) side of the subject's head.
In some embodiments, the sonified representations of the acoustic signal include audible indicia of the presence or absence of seizure (e.g., indicated by rhythmic beating in the sonified acoustic signal), a postictal state, central nervous system depression, concussion, normal brain function, or brain death (e.g., indicated by total or near total quiet in the sonified acoustic signal).
Device 1000 is configured to be a “pocket” handheld device. To that end, device 1000 includes a housing 1004 that is shaped to more naturally fit within a user's pocket. In some embodiments, housing 1004 fits within a cube with dimensions of 10 centimeters (cm) by 7 centimeters by 3 centimeters. In some embodiments, housing 1004 fits within a cube with dimensions of 7 centimeters (cm) by 7 centimeters by 3 centimeters. In some embodiments, housing 1004 fits within a cube with dimensions of 12 centimeters (cm) by 6 centimeters by 3 centimeters. In addition, device 1000 includes a cap 1012 to cover sensors 510-10 and 510-11 to protect sensors 510 while device 1000 is in the user's pocket.
It should be noted that electrodes 902 are not necessarily incorporated into the devices described herein. In particular, as an alternative to the embodiments shown in
Device 1100 includes a plurality of electrodes 902. As mentioned above, these electrodes can be dry or wet electrodes. Electrodes 902 are configured to be placed at respective locations on the subject's head. For example, in some embodiments, electrode 902-3 and electrode 902-4 are positioned (placed) substantially at predefined locations when subject 1102 wears device 1100. The plurality of electrodes produces one or more electrical signals corresponding to brain activity. For example, device 1100 includes sensors 510-4 and 510-5 which produce an electrical signal corresponding to left hemisphere brain activity, and further includes sensors 510-6 and 510-7 which produce an electrical signal corresponding to right hemisphere brain activity. Device 1100 includes an analog-to-digital (A/D) converter to digitize the one or more electrical signals and a processor that receives the one or more digitized electrical signals and produces a representation of an acoustic signal. Device 1100 further includes a speaker system 908 that sonifies the representation of the acoustic signal. In some embodiments, the A/D converter, the processor, and the speaker system are incorporated into wearable housing 1104. In some embodiments, wearable housing 1104 is a headband, a helmet, or a hat. In some embodiments, wearable housing 1104 includes a headband that includes an adjustable strap or housing that is configured to fully wrap around the subject's head to stably hold the wearable housing on the subject's head. In some embodiments, device 1100 interfaces with a chest strap having one or more electrodes to measure a heartbeat signal concurrently with the brain signals.
Device 1100 is used in some circumstances for long-term monitoring of rarely (e.g., sparsely or infrequently) occurring conditions. For example, in some embodiments, device 1100 is used to produce diagnostics for neurology patients complaining of an altered mental state, such as dizziness, lightheadedness, or vertigo. To that end, device 1100 can be worn for prolonged periods of time without becoming awkward or uncomfortable. In addition, device 1100 can be easily removed for bathing and the like. This convenience allows device 1100 to monitor a patient for a month or longer, greatly increasing the likelihood that an episode will be measured by device 1100 and thus produce EEG data of an episode that is available for a neurologist to review. As another example, in some circumstances, device 1100 is worn by epileptics and/or patients with other types of diagnosed conditions to alert them of an on-coming episode. For example, an epileptic patient will wear device 1100 while driving. Device 1100 continuously monitors the epileptic patient for indicia of a pre-ictal state, which signifies that the patient is likely to start seizing. When the device detects indicia of an ictal state, the device alerts the patient using speaker 908, stating, e.g., “Pull Over! Pull Over! Seizure detected!”
F. Preliminary Results Using Sonification DevicesThe following discussion involves experimental results in accordance with several embodiments of the invention shown in the tables in
Moreover, as shown in table 1205 sonified EEG is comparable to Visual EEG in distinguishing seizures. A 15-second-long visual display of EEGs (recorded with conventional EEG system) from 56 ED and ICU recordings at Stanford Medical Center were taken as controls. These recordings were selected from a larger pool of ˜800 EEGs to include seizures (37%), lateralized periodic discharges (1%), generalized periodic epileptiform discharges (8%), triphasic waves (3%), diffuse or focal slowing (30%), and normal (18%). Four board-certified neurologists reviewed the EEG data. Each individual's diagnosis was compared in the audio file to the ⅔ majority visual EEG diagnosis. The sensitivity of audio diagnosis was found to be as good as current gold-standard visual EEG diagnosis (table 1205). It is worth noting that the audio diagnosis is highly consistent across different individuals (even across neurologists versus non-neurologists) whereas the visual diagnosis fluctuates more depending on the level of training. While all four neurologists had more than 15 years of experience in reviewing EEGs, non-neurologists and neurologists received equal amount (only 3.5 minute) of training.
G. Computer SystemComputer system 1300 may further include at least one output device 1308 such as a display unit, video hardware, or other peripherals (e.g., printer). At least one input device 1306 may also be included in computer system 1300 that may include a pointing device (e.g., mouse), a text input device (e.g., keyboard), or touch screen.
Communications interfaces 1314 also form an important aspect of computer system 1300 especially where computer system 1300 is deployed as a distributed computer system. Computer interfaces 1314 may include LAN network adapters, WAN network adapters, wireless interfaces, Bluetooth interfaces, modems and other networking interfaces as currently available and as may be developed in the future.
Computer system 1300 may further include other components 1316 that may be generally available components as well as specially developed components for implementation of the present invention. Importantly, computer system 1300 incorporates various data buses 1316 that are intended to allow for communication of the various components of computer system 1300. Data buses 1316 include, for example, input/output buses and bus controllers.
Indeed, the present invention is not limited to computer system 1300 as known at the time of the invention. Instead, the present invention is intended to be deployed in future computer systems with more advanced technology that can make use of all aspects of the present invention. It is expected that computer technology will continue to advance but one of ordinary skill in the art will be able to take the present disclosure and implement the described teachings on the more advanced computers or other digital devices such as mobile telephones or “smart” televisions as they become available. Moreover, the present invention may be implemented on one or more distributed computers. Still further, the present invention may be implemented in various types of software languages including C, C++, and others. Also, one of ordinary skill in the art is familiar with compiling software source code into executable software that may be stored in various forms and in various media (e.g., magnetic, optical, solid state, etc.). One of ordinary skill in the art is familiar with the use of computers and software languages and, with an understanding of the present disclosure, will be able to implement the present teachings for use on a wide variety of computers.
H. Doctrine of EquivalentsThose skilled in the art will appreciate that the foregoing examples and descriptions of various embodiments of the present invention are merely illustrative of the invention as a whole, and that variations in the steps and various components of the present invention may be made within the spirit and scope of the invention. Accordingly, the present invention is not limited to the specific embodiments described herein but, rather, is defined by the scope of the appended claims. Moreover, where processes, workflows, and/or techniques are described as being capable of being performed in accordance with embodiments of the invention, said embodiments may be freely combined, reordered, and/or substituted with each other without departing from the spirit and scope of the invention.
Although specific sonification processes are discussed above with respect to
It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first storage device could be termed a second storage device, and, similarly, a second storage device could be termed a first storage device, without changing the meaning of the description, so long as all occurrences of the “first storage device” are renamed consistently and all occurrences of the “second storage device” are renamed consistently. The first storage device and the second storage device are both storage devices, but they are not the same storage device. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. The present disclosure provides a detailed explanation of the present invention with detailed explanations that allow one of ordinary skill in the art to implement the present invention into a computerized method. Certain of these and other details are not included in the present disclosure so as not to detract from the teachings presented herein but it is understood that one of ordinary skill in the art would be familiar with such details.
Claims
1. A method for sonifying and playing bioelectrical signals, the method comprising:
- digitizing at least one bioelectrical signal;
- performing a set of operations to produce a sonification of the at least one bioelectrical signal, the set of operations including:
- (i)conditioning the digitized at least one bioelectrical signal by: (a) rejecting signal components with an amplitude below a threshold, (b) scaling signal components above the threshold to create a fixed range signal, and (c) compressing the fixed range signal to raise the prominence of small features, and
- (ii) producing an audio signal by using the conditioned and digitized at least one bioelectrical signal to modulate sound synthesis parameters; and
- generating sound based upon the audio signal using a speaker system.
2. The method of claim 1, wherein the at least one bioelectrical signal comprises at least one electroencephalogram signal (EEG) that is indicative of brain activity of a subject.
3. The method of claim 2, further comprising filtering non-seizure-related brain wave features from the at least one EEG signal.
4. The method of claim 3, wherein the non-seizure-related brain wave features are filtered using a dual stage filter comprising a first stage with a DC-blocking high pass filter and a second stage with a bandpass filter with a passband of 0.1-3.0 Hz to 5.0-15.0 Hz.
5. The method of claim 3, wherein at least one of DC-bias, AC line contamination, and non-seizure-related brain wave features are related by the dual stage filter.
6. The method of claim 1, wherein the sonification of the at least one bioelectrical signal is performed in real-time.
7. The method of claim 1, wherein the set of operations to condition the digitized at least one bioelectrical signal further includes boosting said signal to enhance contrast.
8. The method of claim 7, wherein said signal is boosted by taking the power-law exponent of said signal.
9. The method of claim 8, wherein absolute value rectification is applied to the at least one digitized signal prior to taking the power-law exponent.
10. The method of claim 7, wherein the set of operations to condition the digitized at least one bioelectrical signal further includes applying absolute value signal rectification to said signal to double signal frequency.
11. The method of claim 7, wherein said signal is boosted prior to (a) rejecting signal components with the amplitude below the threshold, (b) scaling signal components above the threshold to create the fixed range signal, and (c) compressing the fixed range signal to raise the prominence of small features.
12. The method of claim 1, wherein the sound synthesis parameters are modulated by continuously modulating vocal sound parameters according to at least one of pitch, loudness, and vowel quality.
13. The method of claim 1, wherein producing the audio signal comprises:
- performing a formant pitch mapping on the conditioned and digitized at least one bioelectrical signal using a midi-to-frequency function; and
- performing an inverse pitch frequency mapping on the at least one bioelectrical signal using an interpolated look-up table for the inverse of the pitch frequency.
14. The method of claim 1, wherein the fixed range signal is compressed by a factor of between 1.5 and 3.0.
15. The method of claim 1, wherein the audio signal is produced by performing at least one process selected from the group consisting of:
- applying a pitch offset in the range of 50-150 Hz;
- performing pitch scaling to a pitch scale in the range of 110-440 Hz;
- applying an amplitude offset in the range of 0.0001-0.01;
- performing amplitude scaling in the range of 0.05-2.0;
- applying a vowel offset in the range of 0.0-1.0;
- performing vowel scaling in the range of 0.05-2.0; and
- mapping the at least one digitized signal to a vowel lookup table comprising the sounds: “iii”, “ahh”, “ehh”, “eee”, “ohh”, and “ooo”.
16. The method of claim 14, wherein the audio signal is produced by performing the processes of:
- applying a pitch offset in the range of 50-150 Hz;
- performing pitch scaling to a pitch scale in the range of 110-440 Hz;
- applying an amplitude offset in the range of 0.0001-0.01;
- performing amplitude scaling in the range of 0.05-2.0;
- applying a vowel offset in the range of 0.0-1.0;
- performing vowel scaling in the range of 0.05-2.0; and
- mapping the at least one digitized signal to a vowel lookup table comprising the sounds: “iii”, “ahh”, “ehh”, “eee”, “ohh”, and “ooo”.
17. The method of claim 1, further comprising:
- detecting occurrence of a seizure by ear based upon the sound generated from the speaker system; and
- administering treatment in response to the detection of a seizure to interrupt the seizure.
18. The method of claim 1, wherein the at least one bioelectrical signal comprises at least one electrocardiogram signal (ECG) that is indicative of cardiac activity of a subject.
19. The method of claim 1, wherein the at least one bioelectrical signal comprises at least one electromyogram signal (EMG) that is indicative of muscle activity of a subject.
20. A system for sonifying and playing bioelectrical signals, the system comprising:
- an input port configured to receive the at least one bioelectrical signal;
- a processor configured to digitize the at least one bioelectrical signal and perform the set of operations to produce the sonification of the at least one bioelectrical signal according to claim 1; and
- a speaker system configured to generate sound according to claim 1.
Type: Application
Filed: Dec 2, 2019
Publication Date: May 7, 2020
Inventors: Josef Parvizi (Palo Alto, CA), Christopher D. Chafe (Woodside, CA), Xinghuan Chao (Palo Alto, CA), Ronald C. Eddington, JR. (Los Gatos, CA)
Application Number: 16/700,578
