AUTOMATIC PROCESSING CHAIN GENERATION

Abstract

Disclosed herein is a method for designing a processing chain of a sensor system, the method comprising receiving a desired application comprising at least one activity for a sensor system to monitor, the sensor system comprising at least one sensor capable of generating sensor data based on sensing the at least one activity, accessing a database comprising a plurality of raw sensor data and a plurality of annotations corresponding to the plurality of raw sensor data, the plurality of annotations identifying activities corresponding to the plurality of raw sensor data, and automatically generating a processing chain of the sensor system for executing the desired application based on the desired application and the plurality of raw sensor data, the processing chain for processing the sensor data and for extracting at least one feature from the sensor data for use in sensing the at least one activity.

Description

RELATED APPLICATION

This application claims priority to and the benefit of co-pending U.S. Provisional Patent Application 63/260,660, filed on Aug. 27, 2021, entitled “AI FRAMEWORK FOR SENSOR SYSTEMS,” by Abbas Ataya, having Attorney Docket No. IVS-1003-PR, and assigned to the assignee of the present application, which is incorporated herein by reference in its entirety

BACKGROUND

Mobile electronic devices often have limited resources for computing ability, so it is beneficial to design systems within the mobile device to be efficient. Processors with compilers tend to require more recourses than processors without compilers, such as battery life, memory, performance, etc. When using compilers, libraries for the compiler are required, and can result in slower computation speeds.

Decision node trees are built to account for as many decisions or scenarios that the developer may view as being relevant. However, a single user is unlikely to fully utilize every option of the decision tree. A way to streamline the decision tree would be beneficial as it would save on memory, processing power, and battery life.

When pre-processing data to be used in applications, human input is typically required to determine what pre-processing methods should be applied to the data to make it more closely resemble the expected phenomenon. The past experience and intuition of the human designer is generally the main input into making processing chains and understanding the system limitations.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and form a part of the Description of Embodiments, illustrate various embodiments of the subject matter and, together with the Description of Embodiments, serve to explain principles of the subject matter discussed below. Unless specifically noted, the drawings referred to in this Brief Description of Drawings should be understood as not being drawn to scale. Herein, like items are labeled with like item numbers.

FIG. 1 illustrates a block diagram of an example electronic device and training machine.

FIG. 2 illustrates a block diagram of the machine learning engine flow, according to some embodiments.

FIG. 3 illustrates a block diagram of the deployed model according to some embodiments.

FIG. 4 illustrates a flow diagram of an example process for a data processing method, according to some embodiments.

FIG. 5 illustrates a diagram of a decision tree model customization system, according to some embodiments.

FIG. 6 illustrates a flow diagram of an example process for modifying a trained classification model, according to some embodiments.

FIG. 7 illustrates an example output of the resource optimizer in the form of a table showing importance and cost of each feature, according to some embodiments.

FIG. 8 illustrates an example process for designing a processing chain of a sensor system, according to some embodiments.

DESCRIPTION OF EMBODIMENTS

The following Description of Embodiments is merely provided by way of example and not of limitation. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding background or brief summary, or in the following detailed description.

Reference will now be made in detail to various embodiments of the subject matter, examples of which are illustrated in the accompanying drawings. While various embodiments are discussed herein, it will be understood that they are not intended to limit to these embodiments. On the contrary, the presented embodiments are intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope the various embodiments as defined by the appended claims. Furthermore, in this Description of Embodiments, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present subject matter. However, embodiments may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the described embodiments.

Notation and Nomenclature

Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data within an electrical circuit. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present application, a procedure, logic block, process, or the like, is conceived to be one or more self-consistent procedures or instructions leading to a desired result. The procedures are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in an electronic device.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the description of embodiments, discussions utilizing terms such as “receiving,” “performing,” “generating,” “selecting,” “adjusting,” “comparing,” “prioritizing,” “modifying,” “adding,” associating,” “filtering,” “updating,” “forwarding,” “labeling,” or the like, refer to the actions and processes of an electronic device such as an electrical circuit.

Embodiments described herein may be discussed in the general context of processor-executable instructions residing on some form of non-transitory processor-readable medium, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.

In the figures, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by that block may be performed in a single component or across multiple components, and/or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, logic, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example mobile electronic device described herein may include components other than those shown, including well-known components.

Various techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described herein. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

The non-transitory processor-readable storage medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, other known storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer or other processor.

Various embodiments described herein may be executed by one or more processors, such as one or more motion processing units (MPUs), sensor processing units (SPUs), host processor(s) or core(s) thereof, digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), field programmable gate arrays (FPGAs), a programmable logic controller (PLC), a complex programmable logic device (CPLD), a discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein, or other equivalent integrated or discrete logic circuitry. The term “training processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. As it employed in the subject specification, the term “training processor” can refer to substantially any computing processing unit or device comprising, but not limited to comprising, single-core processors; single-processors with software multithread execution capability; multi-core processors; multi-core processors with software multithread execution capability; multi-core processors with hardware multithread technology; parallel platforms; and parallel platforms with distributed shared memory. Moreover, processors can exploit nano-scale architectures such as, but not limited to, molecular and quantum-dot based transistors, switches and gates, in order to optimize space usage or enhance performance of user equipment. A processor may also be implemented as a combination of computing processing units. Any code trained on the “training processor” may then be deployed on a reduced instruction set computer, or an “reduced instruction set computer (RISC) processor.”

In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured as described herein. Also, the techniques could be fully implemented in one or more circuits or logic elements. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of an SPU/MPU and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with an SPU core, MPU core, or any other such configuration.

Overview of Discussion

Discussion begins with a description of an example electronic device with which or upon which various embodiments described herein may be implemented. Examples of an electronic device and training machine are then described. Examples of the training phase and the deployment phase are then described. Examples of Model Customization are then described. Examples of Automatic Processing Chain Generation are then described.

Electronic mobile devices, in accordance with the described embodiments, may be used for a variety of purposes. In some cases, there are needs for the electronic mobile device to monitor a user and their activities (e.g., walking, running, sleeping, etc.). In such cases, it is useful for the electronic mobile device to have an efficient system that may adapt to each user.

Embodiments described herein receive data, process the data at a fixed code processing engine, wherein operation of the fixed code processing engine is controlled according to stored parameters, then classify the processed data at a fixed code classification engine, wherein operation of the fixed code classification engine is controlled according to the stored parameters.

Embodiments described herein provide an electronic device including at least one sensor device, a processor, and a memory device having processor-executable code stored thereon for execution by the processor. The code includes a fixed code processing engine for processing sensor data received from the at least one sensor device, a fixed code classification engine for classifying processed sensor data, where the fixed code processing engine and the fixed code classification engine are configurable according to a plurality of parameters that define operation of the fixed code processing engine and the fixed code classification engine.

Embodiments described herein include a method for modifying a trained classification model, the method including receiving feature data extracted from sensor data, classifying the feature data according to the trained classification model to identify a label corresponding to the feature data, wherein the trained classification model includes a decision tree including a plurality of decision nodes for feature identification fora plurality of features, tracking identified features of the plurality of features over a predetermined amount of time, and responsive to determining that a feature of the trained classification model does not satisfy a frequency of usage threshold over the predetermined amount of time, deactivating a decision node of the decision tree of the trained classification model corresponding to the feature such that a subsequent instance of the classifying does not consider a deactivated decision nodes for subsequently received feature data.

Embodiments described herein include a method for designing a processing chain of a sensor system, the method comprising receiving a desired application comprising at least one activity for a sensor system to monitor, the sensor system comprising at least one sensor capable of generating sensor data based on sensing the at least one activity, accessing a database comprising a plurality of raw sensor data and a plurality of annotations corresponding to the plurality of raw sensor data, the plurality of annotations identifying activities corresponding to the plurality of raw sensor data, and automatically generating a processing chain of the sensor system for executing the desired application based on the desired application and the plurality of raw sensor data, the processing chain for processing the sensor data and for extracting at least one feature from the sensor data for use in sensing the at least one activity.

Example Electronic Device and Training Machine

Turning now to the figures, FIG. 1 illustrates a block diagram of an example electronic device 100 and training machine 101, according to some embodiments. As will be appreciated, electronic device 100 may be implemented as a device or apparatus, such as a handheld mobile electronic device. For example, such a mobile electronic device may be, without limitation, a mobile telephone phone (e.g., smartphone, cellular phone, a cordless phone running on a local network, or any other cordless telephone handset), a wired telephone (e.g., a phone attached by a wire), a personal digital assistant (PDA), a video game player, video game controller, a Head Mounted Display (HMD), a virtual or augmented reality device, a navigation device, an activity or fitness tracker device (e.g., bracelet, clip, band, or pendant), a smart watch or other wearable device, a mobile internet device (MID), a personal navigation device (PND), a digital still camera, a digital video camera, a portable music player, a portable video player, a portable multi-media player, a remote control, or a combination of one or more of these devices. In other embodiments, electronic device 100 may be implemented as a fixed electronic device, such as and without limitation, an electronic lock, a doorknob, a car start button, an automated teller machine (ATM), etc. Training machine 101 may be implemented as a device or apparatus, for example a computer, such as a desktop computer, server rack, virtual machine, laptop, etc.

As depicted in FIG. 1, training machine 101 may include a host processor 110, a host bus 120, a host memory 130, and a sensor processing unit 170. Some embodiments of training machine 101 may further include one or more of a display device 140, an interface 150, a transceiver 160 (all depicted in dashed lines) and/or other components. In various embodiments, electrical power for training machine 101 is provided by a mobile power source such as a battery (not shown), when not being actively charged.

Host processor 110 can be one or more microprocessors, central processing units (CPUs), DSPs, general purpose microprocessors, ASICs, ASIPs, FPGAs or other processors which run software programs or applications, which may be stored in host memory 130, associated with the functions and capabilities of training machine 101.

Host bus 120 may be any suitable bus or interface to include, without limitation, a peripheral component interconnect express (PCIe) bus, a universal serial bus (USB), a universal asynchronous receiver/transmitter (UART) serial bus, a suitable advanced microcontroller bus architecture (AMBA) interface, an Inter-Integrated Circuit (I2C) bus, a serial digital input output (SDIO) bus, a serial peripheral interface (SPI) or other equivalent. In the embodiment shown, host processor 110, host memory 130, display 140, interface 150, transceiver 160, sensor processing unit (SPU) 170, and other components of training machine 101 may be coupled communicatively through host bus 120 in order to exchange commands and data. Depending on the architecture, different bus configurations may be employed as desired. For example, additional buses may be used to couple the various components of training machine 101, such as by using a dedicated bus between host processor 110 and memory 130.

Host memory 130 can be any suitable type of memory, including but not limited to electronic memory (e.g., read only memory (ROM), random access memory, or other electronic memory), hard disk, optical disk, or some combination thereof. Multiple layers of software can be stored in host memory 130 for use with/operation upon host processor 110. For example, an operating system layer can be provided for training machine 101 to control and manage system resources in real time, enable functions of application software and other layers, and interface application programs with other software and functions of training machine 101. Similarly, a user experience system layer may operate upon or be facilitated by the operating system. The user experience system may comprise one or more software application programs such as menu navigation software, games, device function control, gesture recognition, image processing or adjusting, voice recognition, navigation software, communications software (such as telephony or wireless local area network (WLAN) software), and/or any of a wide variety of other software and functional interfaces for interaction with the user can be provided. In some embodiments, multiple different applications can be provided on a single training machine 101, and in some of those embodiments, multiple applications can run simultaneously as part of the user experience system. In some embodiments, the user experience system, operating system, and/or the host processor 110 may operate in a low-power mode (e.g., a sleep mode) where very few instructions are processed. Such a low-power mode may utilize only a small fraction of the processing power of a full-power mode (e.g., an awake mode) of the host processor 110.

Display 140, when included, may be a liquid crystal device, (organic) light emitting diode device, or other display device suitable for creating and visibly depicting graphic images and/or alphanumeric characters recognizable to a user. Display 140 may be configured to output images viewable by the user and may additionally or alternatively function as a viewfinder for camera. It should be appreciated that display 140 is optional, as various electronic devices, such as electronic locks, doorknobs, car start buttons, etc., may not require a display device.

Interface 150, when included, can be any of a variety of different devices providing input and/or output to a user, such as audio speakers, touch screen, real or virtual buttons, joystick, slider, knob, printer, scanner, computer network I/O device, other connected peripherals and the like.

Transceiver 160, when included, may be one or more of a wired or wireless transceiver which facilitates receipt of data at training machine 101 from an external transmission source and transmission of data from training machine 101 to an external recipient. By way of example, and not of limitation, in various embodiments, transceiver 160 comprises one or more of: a cellular transceiver, a wireless local area network transceiver (e.g., a transceiver compliant with one or more Institute of Electrical and Electronics Engineers (IEEE) 802.11 specifications for wireless local area network communication), a wireless personal area network transceiver (e.g., a transceiver compliant with one or more IEEE 802.15 specifications for wireless personal area network communication), and a wired a serial transceiver (e.g., a universal serial bus for wired communication).

In some embodiments, electronic device 100 includes a display (not shown) similar to display 140. In some embodiments, electronic device 100 includes an interface (not shown) similar to interface 150. In some embodiments, electronic device 100 includes a transceiver (not shown) similar to transceiver 160.

As depicted in FIG. 1, electronic device 100 may include a general purpose sensor assembly in the form of integrated Sensor Processing Unit (SPU) 170 which includes sensor processor 172, memory 176, a motion sensor 178, and a bus 174 for facilitating communication between these and other components of SPU 170. In some embodiments, SPU 170 may include at least one additional sensor 180 (shown as sensor 180-1, 180-2, . . . 180-n) communicatively coupled to bus 174. In some embodiments, all of the components illustrated in SPU 170 may be embodied on a single integrated circuit. It should be appreciated that SPU 170 may be manufactured as a stand-alone unit (e.g., an integrated circuit), that may exist separately from a larger electronic device and is coupled to a host bus through an interface (not shown).

Some embodiments of electronic device 100 may further include one or more of a display device, an interface, a transceiver (all not shown) and/or other components. In various embodiments, electrical power for electronic device 100 is provided by a mobile power source such as a battery (not shown), when not being actively charged.

Sensor processor 172 can be one or more microprocessors, CPUs, DSPs, general purpose microprocessors, ASICs, ASIPs, FPGAs or other processors which run software programs, which may be stored in memory 176, associated with the functions of SPU 170. It should also be appreciated that motion sensor 178 and additional sensor 180, when included, may also utilize processing and memory provided by other components of electronic device 100.

In some embodiments, sensor processor 172 is a reduced instruction set computer (RISC). In this embodiment, there is an internal measurement unit which is run on the sensor processor 172.

Bus 174 may be any suitable bus or interface to include, without limitation, a peripheral component interconnect express (PCIe) bus, a universal serial bus (USB), a universal asynchronous receiver/transmitter (UART) serial bus, a suitable advanced microcontroller bus architecture (AMBA) interface, an Inter-Integrated Circuit (I2C) bus, a serial digital input output (SDIO) bus, a serial peripheral interface (SPI) or other equivalent. Depending on the architecture, different bus configurations may be employed as desired. In the embodiment shown, sensor processor 172, memory 176, motion sensor 178, and other components of SPU 170 may be communicatively coupled through bus 174 in order to exchange data.

Memory 176 can be any suitable type of memory, including but not limited to electronic memory (e.g., read only memory (ROM), random access memory, or other electronic memory). Memory 176 may store algorithms or routines or other instructions for processing data received from motion sensor 178 and/or one or more sensor 180, as well as the received data either in its raw form or after some processing. Such algorithms and routines may be implemented by sensor processor 172 and/or by logic or processing capabilities included in motion sensor 178 and/or sensor 180.

Motion sensor 178 is a sensor capable of sensing a form or type of motion, including without limitation a gyroscope or an accelerometer. A sensor 180 may comprise, without limitation: a temperature sensor, a humidity sensor, an atmospheric pressure sensor, an infrared sensor, a radio frequency sensor, a navigation satellite system sensor (such as a global positioning system receiver), an acoustic sensor (e.g., a microphone), another inertial or motion sensor (e.g., a gyroscope, accelerometer, or magnetometer) for measuring the orientation or motion of the sensor in space, or other type of sensor for measuring other physical or environmental conditions. In one example, sensor 180-1 may comprise an acoustic sensor, sensor 180-2 may comprise a temperature sensor, and sensor 180-n may comprise a motion sensor.

In some embodiments, motion sensor 178 and/or one or more sensors 180 may be implemented using a microelectromechanical system (MEMS) that is integrated with sensor processor 172 and one or more other components of SPU 170 in a single chip or package. Although depicted as being included within SPU 170, one, some, or all of motion sensor 178 and/or one or more sensors 180 may be disposed externally to SPU 170 in various embodiments.

Examples of Training Phase and Deployment Phase

FIG. 2 illustrates a block diagram of the machine learning engine flow 200, according to some embodiments. Training phase 202 takes place on training machine 101. Once the training phase 202 is complete, the trained machine learning (ML) classifier 208 is deployed on electronic device 100 in the deployment phase 204.

The training phase 202 revolves around training a ML classifier 208 to correctly identify patterns, and to make decisions from previous data. For example, if there is a large amount of data on a walking motion and a running motion, then after training the ML classifier 208 can receive a new set of data and correctly classify the new data as either walking or running. The data used for training may vary in the features included such that different models may be trained and deployed for different use cases.

In some embodiments, the training phase 202 is done offline. In some embodiments, the training phase 202 is not restricted in memory, speed, cost, etc., as the deployment phase 204 might be.

In the training phase 202, a labeled database 206 is used to train the ML classifier 208. The data within the labeled database 206 contains data that will undergo preprocessing to extract features, the extracted features then being input into the ML classifier which should be able to accurately classify and label the features. Features may include mean, variance, energy, number of peaks, peak distance, mean cross rate, dominant frequencies, spectral power, etc. The data within the labeled database 206 contains corresponding labels to what activity the data describes. For example, if the data is a set of motion data from a person running, then the data will be labeled as running. Other activities may include walking, sleeping, sitting, driving, exercising, etc.

First, the data will go through pre-processing 210. This stage may include eliminating noise in the data and putting the data in a more appropriate format for the following steps. Next, the data will go through the feature extraction 212 stage. Here, the data is analyzed (without the labels) for what activity the data might portray. Next, in the feature selection 214 stage, the less important features are filtered out, and the more important features are retained. The feature selection 214 may be done through a numerical threshold, human input, redundancy, or similar filtering methods.

Finally, the selected features are put into the ML classifier 208, which will take the features and label them. The ML classifier 208 labels may then be compared against the labels from the labeled database 206 for accuracy.

The training phase 202 is repeated until the ML classifier 208 is able to accurately label the features to a satisfactory degree. During these repetitions, ML classifier 208 is adjusted with the goal of being able to accurately label features. Once the ML classifier is judged to be ready, the deployment phase 204 is started.

Deployment phase 204 has the ML classifier 208 deployed on an electronic device (e.g., electronic device 100). In some embodiments, ML classifier 208 is deployed as a fixed code classification engine stored on memory 176 for execution by sensor processor 172. Once deployed, data sample 216 becomes the new input. In some embodiments, data sample 216 is received from at least motion sensor 178 or sensor 180. In some embodiments, data sample is a mix of preprocessed data and real time data.

After being received, data sample 216 will go through preprocessing 218 where noise is removed, and the data is formatted. Next, based on the feature extraction 212 and feature selection 214 stages of training phase 202, the data will go through the selected features extraction 220 phase. Here, the features that were determined to be important during the training phase 202 are extracted from the data. The extracted features will then go through ML classifier 208, where they receive a label 222 according to the activity performed in the feature (e.g., walking, running, sleeping, etc.).

In some embodiments, the described embodiments are run on a processor without a compiler during the deployment phase 204. Traditional systems with compilers require hard coded libraries for the processes and translation to machine language, which can slow down whatever process is being run. By using a processor without a compiler, such as a RISC processor, aspects such as performance, battery life, and cost are improved over general purpose processors.

FIG. 3 illustrates a block diagram of the deployed model 300. The deployed model has an input of data 316. Data 316 is run through processing engine 324, before being run though ML classifier 308.

In some embodiments, the deployed model 300 is deployed on a processor that does not have a compiler. The benefits of running on a processor with no compiler include being cost efficient, power efficient, and high performance. As the fixed code processing engine and the fixed code classification engine are fixed in the memory for execution by a processor without a compiler, the described embodiments are ideal for such a system due to its ability to edit the function with parameters.

Data 316 may also be referred to as input data, sensor data, or gathered data. In some embodiments, data 316 is collected from motion sensor 178. In some embodiments, data 316 is collected from sensors 180. In some embodiments, processing engine 324 is a fixed code processing engine 324. In some embodiments, ML classifier 308 is a fixed code classification engine 308.

In the processing engine 324 data 316 is filtered and features are extracted. The filtering of the sensor data and extraction of the features are controlled by computation engine parameters 326. Computation engine parameters 326 are updatable. In some embodiments, the computation engine also segments the data 316 into windows of a number of samples.

After passing through the processing engine 324, the processed sensor data is passed to the classification engine 308 where the data is classified. In some embodiments, the classification engine 308 determines and labels the processed sensor data.

In some embodiments, classification engine parameters 328 control the operation of classification engine 308. The parameters can affect the classification and labeling of the processed sensor data. Classification engine parameters 328 are updatable.

Classification engine parameters 328 and computation engine parameters 326 may collectively be referred to as the stored parameters. In some embodiments, an updated set of parameters is received, and the stored parameters are updated according to the updated parameters. In some embodiments, the stored parameters can automatically update based on the processed sensor data.

In some embodiments, the stored parameters can enable or disable a function of the processing engine 324. In some embodiments, the stored parameters can enable or disable a function of the classification engine 308. In some embodiments, the stored parameters are stored in the RAM. In some embodiments, the stored parameters can be manually updated. The stored parameters may also be referred to as a plurality of parameters.

FIG. 4 illustrates a flow diagram 400 of an example process for a data processing , according to some embodiments. Procedures of these methods will be described with reference to elements and/or components of various figures described herein. It is appreciated that in some embodiments, the procedures may be performed in a different order than described, that some of the described procedures may not be performed, and/or that one or more additional procedures to those described may be performed. The flow diagrams include some procedures that, in various embodiments, are carried out by one or more processors (e.g., a host processor or a sensor processor) under the control of computer-readable and computer-executable instructions that are stored on non-transitory computer-readable storage media. It is further appreciated that one or more procedures described in the flow diagrams may be implemented in hardware, or a combination of hardware with firmware and/or software.

At procedure 410 of flow diagram 400, data is received by the system. In some embodiments, the data is received from a sensor that collected the data.

At procedure 420, the data is processed at the fixed code processing engine, according to the stored parameters. Here, the data is filtered (e.g., for noise), and the features are extracted. The extracted features are determined based on the stored parameters. In one embodiment, the processing engine utilizes the computation engine parameters to control the operation of the processing engine. In one embodiment, the computation engine parameters control the filtering of data and extracting the features from the data. After this step, the data is referred to as processed data.

At procedure 430, the processed data is classified in the fixed code classification engine, according to the stored parameters. In one embodiment, the classification engine utilizes the classification engine parameters to control the operation of the classification engine. In one embodiment, the classification engine parameters control the classification and labeling of the processed data for a plurality of applications.

At procedure 440, after the processed data is classified, the classification engine determines and assigns a label to the data.

In some embodiments, at procedure 450, an updated set of parameters is received. In some embodiments this updated set of parameters is manually updated. In some embodiments the updated set of parameters are automatically generated.

In some embodiments, at procedure 460, the stored parameters are updated according to the updated set of parameters.

In some embodiments, the stored parameters comprise the computation engine parameters and the classification engine parameters. In some embodiments, the stored parameters can enable and disable a function of the fixed code processing engine. In some embodiments, the stored parameters can enable and disable a function of the fixed code classification engine.

Examples of Model Customization

FIG. 5 illustrates a diagram of a decision tree model customization 500 system, according to some embodiments. Shown in the figure is the offline training phase 530, generic model 532, the on-chip deployment phase 534, and the adapted model 536.

The system shown in at least FIG. 5 is capable of adapting to an individual user and their habits such that the device used in the on-chip deployment phase may be more efficient.

Offline training phase 530 takes place on a computer similar to that of training machine 101. In some embodiments, offline training phase takes place on a virtual machine. During offline training phase 530, a generic model 532 is created. In some embodiments generic model 532 is a decision tree comprising a plurality of nodes. In some embodiments, generic model 532 is a trained classification model, wherein the trained classification model identifies a label corresponding to the feature data, wherein the plurality of decision nodes are used for feature identification for a plurality of features.

In some embodiments the offline training phase 530 is done with machine learning. In some embodiments, offline training phase 530 is similar in structure to training phase 202.

Once the generic model 532 is determined to be of suitable quality, it is then moved to the on-chip deployment phase 534. In some embodiments, the on-chip deployment phase 534 takes place on a mobile electronic device similar to that of electronic device 100. In some embodiments, the on-chip deployment phase takes place on stationary electronic devices (for example, fitness equipment, household appliances, etc.).

On the first day of usage, the deployed model will be the same as the generic model. Over time, the model will disable nodes and paths in the decision tree that are rarely or never used, which results in adapted model 536.

Benefits of this adaptive system include freeing up memory, increasing processing speed, increasing battery life, etc.

In some embodiments, the nodes and branches are initially assigned a weight (or usage frequency) of zero. Each time a node or branch (i.e., a feature) is used, the weight of the feature is increased. This method will allow the system to keep track of which features are used most often, and which features are used rarely or if at all.

In some embodiments, the weight is checked periodically. In some embodiments, an input prompts the weights to be checked. In some embodiments, the weight is checked over set intervals of time. In some embodiments, there is an initial amount of time that must pass before the weights are regularly checked. In some embodiments, the weights are checked after a certain number of decisions are made. In some embodiments, the intervals over which the weights are checked steadily increase until a maximum interval is reached.

In some embodiments, any features with weights below a set threshold are deemed low usage and are removed from the decision tree. In some embodiments, any features with weights below a set threshold are deemed low usage and will be checked less often. If a low usage feature remains as such, then it will be removed from consideration entirely. In some embodiments, a low usage feature will be set to hibernation. In some embodiments, deactivated decision nodes allow the system to save on aspects such as battery life, processing power, computation load, etc. In some embodiments, once a feature is below a threshold it is no longer processed.

In some embodiments, a decision node can be reactivated responsive to a reset command. In some embodiments, the reset command is automatic. In some embodiments, the reset command is a manual input. In some embodiments, a path or branch of the decision tree can be reactivated responsive to a reset command.

In some embodiments, the weight starts at a set non-zero number and each time a feature is processed the weight decreases. In this embodiment, features with weights above a set threshold are considered low usage.

In some embodiments, a table is used to track each time a feature is processed. In some embodiments, the weights of features are compared to one another, and a feature is considered low usage if the weight is a set amount lower than an average. In some embodiments, a feature is considered low usage if its weight is a statistical outlier when compares to the other weights. In some embodiments, the threshold is a frequency of usage threshold.

In some embodiments, features commonly used in conjunction are tracked.

In some embodiments, if an intervening feature is determined to be a low usage feature, then the adapted model may remove that feature from the decision path while keeping the surrounding features. In some embodiments, new paths can be added between features that are often used together.

In some embodiments, a copy of the generic model 532 is stored separately in the memory. In this case, if the adapted model is determined to be unsatisfactory or no longer necessary (for instance, if a new person is to take ownership of the device) then the model can be reset. In some embodiments, multiple adapted models are stored for multiple purposes or users. In some embodiments, the multiple adapted models are tied to separate user accounts. For example, if multiple people use a single piece of fitness equipment, then the users can have customized models on the same piece of equipment.

In some embodiments, the electronic device receives an updated generic model 532 with new features. Upon receipt of an updated generic model 532, the process is restarted and the adapted model 536 is equivalent to the updated generic model. The new features would then undergo the same process of determining low use features. In some embodiments, upon receipt of an updated generic model 532 the frequency with which the weights are tracked is reset.

In some cases, a user might be in a situation where their habits or patterns change. In some embodiments, after being removed low usage features are periodically checked for potential usage. In some embodiments, removed features can be manually re-activated. In some embodiments, removed features can be automatically re-activated upon requirement of the feature. In some embodiments, an adapted model can be saved as a backup, or restore state. In some embodiments, a user can initiate the creation of a backup model. In some embodiments, a user can manually revert to a backup model. In some embodiments, a backup model is automatically generated. In some embodiments, an adapted model can be set as a backup model as the generic model is restored.

A threshold may also be referred to as a parameter. Parameters are not hard coded and may be changed or updated. Adjusting the parameters can change which features or branches are considered in the decision-making process.

In some embodiments, the on-chip deployed model is on a piece of hardware without a compiler. In these embodiments, parameters would be useful in allowing the decision tree to adapt without a compiler. In some embodiments, the on-chip deployed model is on a piece of hardware with a compiler.

FIG. 6 illustrates a flow diagram 600 of an example process for modifying a trained classification model, according to some embodiments. Procedures of these methods will be described with reference to elements and/or components of various figures described herein. It is appreciated that in some embodiments, the procedures may be performed in a different order than described, that some of the described procedures may not be performed, and/or that one or more additional procedures to those described may be performed. The flow diagrams include some procedures that, in various embodiments, are carried out by one or more processors (e.g., a host processor or a sensor processor) under the control of computer-readable and computer-executable instructions that are stored on non-transitory computer-readable storage media. It is further appreciated that one or more procedures described in the flow diagrams may be implemented in hardware, or a combination of hardware with firmware and/or software.

At procedure 610 of flow diagram 600, feature data is received and extracted from sensor data. At procedure 620, the feature data is classified using the trained classification model in order to identify the corresponding label. In some embodiments, the trained classification model is the generic model 532. In some embodiments, the trained classification model is a decision tree comprising a plurality of decision nodes for label identification using a plurality of features. The trained classification model may also be referred to as an adapted classification model.

In some embodiments, the identified features of the plurality of features are tracked over a predetermined amount of time.

At procedure 630 the usage rate of each feature is tracked over a predetermined amount of time. In some embodiments, for each instance of an identified feature a weight is applied to the identified feature of the plurality of features, and the weights for the plurality of features are aggregated over the amount of time.

In some embodiments, each instance of an identified feature has an initial weight of zero. Each time an instance of an identified feature is used, the weight of the feature is increased.

At procedure 640, the weights for each feature of the plurality of features are compared to the frequency of usage threshold over the amount of time, wherein the frequency of usage threshold corresponds to a minimum weight. In some embodiments, the weights used in procedure 640 are aggregated weights. In some embodiments, the frequency of usage threshold is a predetermined value. In some embodiments, the frequency of usage threshold is dependent on the usage of the device. In some embodiments, the frequency of usage threshold is a variable value.

At procedure 650, responsive to determining that a feature of the trained classification model does not satisfy a frequency of usage threshold over the predetermined amount of time, a decision node of the decision tree of the trained classification model corresponding to the feature is deactivated, such that a subsequent instance of the classifying does not consider a deactivated decision nodes for subsequently received feature data. In some embodiments, subsequent instance of the classifying that does not consider the deactivated decision node requires less computational load, power consumption, and memory than a prior instance of the classifying prior to the deactivating the decision node.

At procedure 660 a deactivated decision node of the decision tree is reactivated responsive to a reset command. In some embodiments, a reset command is a manual input. In some embodiments, a reset command automatically occurs after an amount of time.

In some embodiments, the reset command restores the decision tree to its original state. In some embodiments, the reset command is temporary, and the decision node will deactivate should it continue to not satisfy a frequency of usage threshold.

In some embodiments, a feature sensing device is implemented for use in feature identification for a user. In some embodiments the feature sensing device is configured for use by a plurality of users, such that each user of the plurality of users has an associated trained classification model and wherein the method is performed separately for each user of the plurality of users.

In some embodiments, multiple versions of the trained classification model can be stored. In some embodiments, each version of the trained classification model is connected to an account, or user profile. In some embodiments, one instance of the multiple versions of the trained classification model can be reset without affecting the other instances.

Examples of Automatic Processing Chain Generation

Determining what features to extract from data, and how to filter the data to get those features, usually comes from the intuition and knowledge of the designer. One benefit of the current machine learning system is that it can automatically build the processing chain with minimal intervention from a user.

In some embodiments, the machine on which the automatic processing chain generation takes place on is similar to training machine 101. In some embodiments, the automatic processing chain generation can be segmented into the performance optimization stage, and the resource optimization stage.

The performance optimization stage has the goal of taking the information of the desired application and determining which sensors are needed, which sensor axes are important, what preprocessing should be done to the raw sensor data, and what set of features will be useful to the desired application.

To start the performance optimization stage, the system will receive the desired application information. This is information that outlines what activity is to be monitored, and what data should be gathered to monitor that activity (for instance, monitoring heart rate for exercise, or gyroscope data for walking, etc.).

In some embodiments, the desired application information includes limitations on the resources the application is allowed to utilize. In some embodiments, power, memory, and size requirements (or a combination thereof) are included in the desired application information. In some embodiments, resource limitations are manually added to the desired application information. In some embodiments with resource limitations, the automatic processing chain generator will automatically alter the results based on the resource limitations. For instance, if a feature or sensor of low importance and high power requirements is included that goes above a power limitation, the feature or sensor will automatically be disabled.

After receiving the desired application information, the system will then access the annotated database to retrieve the sensor data and annotations. In some embodiments, the annotated database is similar to labeled database 206. Annotated database contains the data, and the annotations that note what features instances of data represent. In some embodiments, the data from annotated database is sensor data. In some embodiments, the data in annotated database is timestamped raw sensor data (for example, data from sensors such as an accelerometer, gyroscope, pressure, microphone, etc.) from at least one type of sensor. In some embodiments, the data from annotated database includes and axial signal values.

For example, if the desired application needs to detect if the person is walking or running, data from an accelerometer could be used. The annotated database will contain the timestamped raw data signals from the accelerometer, and an annotation label at each time to inform the system if the sensor signals correspond to walking or to running. In some embodiments, the annotations are used to check that the machine learning has the correct results.

In some embodiments, a user provides the raw data and the appropriate annotations. In some embodiments, there is an option for a user to search for what types of filters to apply to the raw data to extract the desired features. In some embodiments, if there is initially a large number of features to explore then the machine learning system will reduce the number of features to the ones pertinent to the desired application.

With the raw sensor data and annotations retrieved from annotated database the performance optimizer will execute the desired application based on the desired application and the plurality of raw sensor data. The performance optimizer will process the sensor data and extract at least one feature from the sensor data for use in sensing at least one activity.

In some embodiments, the performance optimizer will analyze the raw sensor data with the purpose of generating results suited for the desired application. In some embodiments, the performance optimizer will analyze what features need to be extracted, the minimum number of features that should be extracted, what filters should be applied to the raw data to make it better represent the features (e.g., removing noise), and the parameters and configurations required for the desired application.

In some embodiments, the meta-analysis includes time and spectral analysis, time frequency analysis, scripts, loss and cost functions, etc. In some embodiments, the performance optimizer will automatically apply filters to the data such as band pass filters, low pass filters, and high pass filters. In some embodiments, values such as mean, variance, peak to peak energy, dominant peaks, band power, distance between peaks, etc., are possible operations that can be applied to the raw sensor data. In some embodiments, the performance optimizer will analyze if a sensor is important to the application. If a sensor is deemed important, then the performance optimizer will analyze which axes of the sensor are most pertinent.

In one embodiment, the initial processing block requires nothing from the user but will analyze the data to identify aspects such as the value of the cut off frequencies, what axis to use, if a sensor is needed (e.g., gyroscope), if all the known features need to be extracted or if a subset is enough, etc.

In some embodiments, the performance optimizer analyzes what sensors are being used, which axes from the sensors are the most pertinent and contain most valuable information that will help in generating the desired application (for example, detecting if a person is driving, walking, running, speaking, what words are being spoken, etc.). In some embodiments, the performance optimizer analyzes what are the preprocessing techniques and filters that will help clean the raw data, accentuate, and extract the most useful information for the desired application. In some embodiments, the performance optimizer analyzes what features should be extracted to best represent the application, with the most valuable and pertinent amount of information to help generate the desired application. In some embodiments, the performance optimizer analyzes what is the best classifier to maximize performance.

In some embodiments, the performance optimizer automatically selects a sensor for use in the sensing of the desired application. In some embodiments, the performance optimizer automatically selects at least one axis of the sensor for use in the sensing of the desired application. In some embodiments, performance optimizer automatically selects the at least one feature to extract from the sensor data for use in sensing the at least one activity. In some embodiments, the performance optimizer automatically selecting at least one preprocessing operation to apply to the sensor data for extracting the at least one feature. In some embodiments, the performance optimizer automatically selects at least one filter to apply to the sensor data for extracting the at least one feature.

In some embodiments, the optimal data segment length and the inference rate are variables that may be automatically selected by the performance optimizer. In some embodiments, the optimal data segment length is the amount of time over which a segment of data is collected from at least one sensor, after which the segment of data goes to the ML system for preprocessing and feature extraction. In some embodiments, the inference rate is the amount of time between two decisions of the ML system. In some embodiments, the decision made by the ML system involves collecting a new segment of data. For example, if an optimal data segment length is set to collect data for three seconds, with an inference rate of one second, then the two consecutive segments have an overlap of two second.

FIG. 7 illustrates an example output of the resource optimizer in the form of a table 700 showing the importance and cost of each feature, according to some embodiments.

After optimizing the performance of the desired application in the data processing stage, the resource optimization stage assists in the understanding of how the desired application will run on the final platform. The goal of the resource optimization stage is to find a balance between the features included and the performance of the deployed system.

In the resource optimization stage, an offline model is made to model the processor's memory, power, and MIPS usage in function of enabled processing blocks and generated models to be able to continuously keep track of those key elements when building the desired application offline. In some embodiments, the offline model is made on a computer system more powerful than the final platform. In some embodiments, the offline model is made on a computer similar to training machine 101.

In some embodiments, the offline model will keep track of the requirements and make sure the desired application runs properly when deployed to the embedded device. Based on the resource requirement for the different features (e.g. in time or frequency, or time-frequency domain), the best features can be selected, and the best feature settings determined to get optimal performance based on sensor resources. The offline model can determine the required resources for each feature block, and then select the best combination of features. This combination may be dependent on the application. The offline model can also determine which sensors to use, and how the adding or remove of a sensor affects aspects such as performance, memory, cycles, and power. In some embodiments, based on these determined rules, the actual sensor can set the sensor configuration.

In some embodiments, the offline model automatically selects the optimal features and sensors to retain for the desired application. In some embodiments, a user can review the potential sensors and features through the resource optimizer.

Table 700 allows for a user to understand the cost of each feature when deployed on the final platform (e.g. size, computation time, power, etc.). Table 700 also allows a user to see the relative importance of a feature. Using this table, a user could determine whether or not they wish to keep a feature active. In some embodiments, a user choosing to deactivate a feature is similar to deactivating a node in the decision tree customization model 500. In some embodiments, the cost and importance values are tallied or weighed into a single value, In some embodiments, the importance of a feature depends on the classifier and desired application since it shows how important the feature is in the classifying and identifying.

In some embodiments, the automatic processing chain generator is independent of the classifier. In some embodiments, the automatic processing chain generator can be applied to systems with decision trees, neural networks, random forest, or any other classification method. In some embodiments, the classifier and its parameters are subjects that can be searched and optimized with the automatic processing chain generator.

Through the resource optimization phase, a user can keep the performance of the device intact by running less complicated models and know the resulting performance change. The user is also able to understand how much of the system resources and power the desired application will consume on the target platform, without having to deploy the model on the platform.

For instance, from the results in table 700 a user might decide that sensor 2 is unnecessary and might deactivate it. Alternatively, a user might decide to keep features even at the cost of performance. In other words, a user is able to make tradeoffs in the structured processing chain. For example, if the user prefers having lower power (or cycles, or memory, or combination) at the expense of performance, the user can make modifications to the solution processing chain to accomplish their goal before deploying the desired application to the target platform.

In some embodiments, the offline model tracks system requirements of at least one feature for sensing at least one activity of the desired application. In some embodiments, the system requirements include memory consumption for at least one sensor, computation consumption for at least one sensor, and power consumption for at least one sensor. In some embodiments, the system requirements include memory consumption for at least one feature, computation consumption for at least one feature, and power consumption for at least one feature.

In some embodiments, the sensing of an activity utilizes a plurality of features. In some embodiments, a relative importance of the plurality of features used in sensing the at least one activity of the desired application is determined. In some embodiments, the results of the system requirements for the at least one sensor for sensing the at least one activity of the desired application are then aggregated.

In some embodiments, the results of the system requirements for the plurality of features for sensing the at least one activity of the desired application are then displayed. FIG. 7 shows an example of how the results can be displayed; however other formats are easily applicable.

With the results being displayed, in some embodiments a user reviews the results and is able to select features or sensors for deactivation. In some embodiments, after a feature or sensor is deactivated, the results will update to no longer include the deactivated feature or sensor and the new results are displayed.

In some embodiments, a deselected feature is still displayed in the updated results but has an indicator to show that the feature is inactive. In some embodiments, a separate set of inactive features and sensors is available for review.

In some embodiments, where resource limitations were included in the desired application information, the offline model will automatically account for the resource limitations in the displayed results. In some embodiments, resource limitations are considered ideal but not required. In some embodiments, resource limitations are considered hard limits. In some embodiments, a user can specify upon input if a resource limitation is a required limitation or not. In some embodiments, the automatic processing chain automatically disables features or sensors based on the resource limitations. In some embodiments, it is automatically determined which features or sensors should be disabled based on resource limitations, but user confirmation is requested before disabling the feature or sensor.

FIG. 8 illustrates an example process for designing a processing chain of a sensor system, according to some embodiments. Procedures of these methods will be described with reference to elements and/or components of various figures described herein. It is appreciated that in some embodiments, the procedures may be performed in a different order than described, that some of the described procedures may not be performed, and/or that one or more additional procedures to those described may be performed. The flow diagrams include some procedures that, in various embodiments, are carried out by one or more processors (e.g., a host processor or a sensor processor) under the control of computer-readable and computer-executable instructions that are stored on non-transitory computer-readable storage media. It is further appreciated that one or more procedures described in the flow diagrams may be implemented in hardware, or a combination of hardware with firmware and/or software.

At procedure 810, a desired application comprising at least one activity for a sensor system to monitor is received. In some embodiments, the sensor system includes at least one sensor capable of generating sensor data based on sensing at least one activity. In some embodiments, the at least one activity is a feature such as walking, running, sleeping, etc. In some embodiments, the at least one sensor is a sensor such as, for example, an accelerometer, gyroscope, pressure, microphone, etc.

At procedure 820, a database including raw sensor data and annotations corresponding to the raw sensor data is accessed. In some embodiments, the database is referred to as an annotated database. In some embodiments, the raw sensor data is a plurality of raw sensor data. In some embodiments, the annotations are a plurality of annotations. In some embodiments, the plurality of annotations identifies activities corresponding to the plurality of raw sensor data. In some embodiments, the raw sensor data includes timestamped data and axial signal values for a plurality of sensors.

Next, starting at procedure 830, a processing chain is automatically generated. The processing chain is of the sensor system for executing the desired application based on the desired application and the plurality of raw sensor data. The processing chain will process the sensor data and extract at least one feature from the sensor data for use in sensing the at least one activity.

At procedure 830, a sensor is automatically selected from the at least one sensor to be used in the sensing of the desired application. In some embodiments, at least one axis of the sensor is automatically selected for use in the sensing of the desired application. In some embodiments, the optimal data segment length and inference length are automatically selected for use in the sensing of the desired application.

At procedure 840, at least one feature is automatically selected to be extracted from the sensor data for use in sensing the at least one activity. In some embodiments, at least one preprocessing operation is automatically selected to be applied to the sensor data for extracting the at least one feature. In some embodiments, at least filter is automatically selected to be applied to the sensor data for extracting the at least one feature. In some embodiments, the preprocessing operation is used to reduce noise in the data and can be, for example, a low pass filter, a high pass filter, a bandwidth filter, etc. in some embodiments, the preprocessing operation is used to make the data better represent the feature that is to be extracted.

At procedure 850, an offline model is created and the system requirements of the at least one feature for sensing the at least one activity of the desired application are tracked. In some embodiments, the offline model replicates the system requirements of the target platform. In some embodiments, the system requirements include memory consumption for the at least one sensor, computation consumption for the at least one sensor, and power consumption for the at least one sensor. In some embodiments, the system requirements include memory consumption for the at least one feature, computation consumption for the at least one feature, and power consumption for the at least one feature. In some embodiments, the sensing of at least one activity utilizes a plurality of features.

At procedure 860, the relative importance of the plurality of features used in sensing the at least one activity of the desired application is determined. In some embodiments, the system requirements for the at least one sensor for sensing the at least one activity of the desired application are then aggregated.

At procedure 870, the system requirements for the plurality of features for sensing the at least one activity of the desired application are displayed. In some embodiments, a feature of the plurality of features can be selected for deselection from sensing the activity. In other words, a feature of the plurality of features can be deactivated. After at least one feature is deactivated, the displayed plurality of features is updated to remove the deselected feature from the plurality of features for sensing the activity. The updated system requirements for the plurality of features for sensing the at least one activity of the desired application is then displayed without considering the deselected feature.

In some embodiments, a deselected feature is still displayed in an updated system requirements but has an indicator to show that the feature is inactive. In some embodiments, a separate set of inactive features is available for review.

In some embodiments, at least one resource limitation is received. In some embodiments, the at least one resource limitation is received at the start of the process. In some embodiments, the at least one resource limitation assists in automatically determining a relative importance of the plurality of features used in sensing the at least one activity of the desired application based on the at least one resource limitation. In some embodiments, a feature is automatically deselected based on the resource limitation.

CONCLUSION

The examples set forth herein were presented in order to best explain, to describe particular applications, and to thereby enable those skilled in the art to make and use embodiments of the described examples. However, those skilled in the art will recognize that the foregoing description and examples have been presented for the purposes of illustration and example only. Many aspects of the different example embodiments that are described above can be combined into new embodiments. The description as set forth is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Reference throughout this document to “one embodiment,” “certain embodiments,” “an embodiment,” “various embodiments,” “some embodiments,” or similar term means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of such phrases in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics of any embodiment may be combined in any suitable manner with one or more other features, structures, or characteristics of one or more other embodiments without limitation.

Claims

1. A method for designing a processing chain of a sensor system, the method comprising:

receiving a desired application comprising at least one activity for a sensor system to monitor, the sensor system comprising at least one sensor capable of generating sensor data based on sensing the at least one activity;

accessing a database comprising a plurality of raw sensor data and a plurality of annotations corresponding to the plurality of raw sensor data, the plurality of annotations identifying activities corresponding to the plurality of raw sensor data; and

automatically generating a processing chain of the sensor system for executing the desired application based on the desired application and the plurality of raw sensor data, the processing chain for processing the sensor data and for extracting at least one feature from the sensor data for use in sensing the at least one activity.

2. The method of claim 1, wherein the raw sensor data comprises timestamped data and axial signal values for a plurality of sensors.

3. The method of claim 1, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data comprises:

automatically selecting a sensor of the at least one sensor for use in the sensing of the desired application.

4. The method of claim 3, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data further comprises:

automatically selecting at least one axis of the sensor for use in the sensing of the desired application.

5. The method of claim 1, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data comprises:

automatically selecting the at least one feature to extract from the sensor data for use in sensing the at least one activity.

6. The method of claim 5, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data comprises:

automatically selecting at least one preprocessing operation to apply to the sensor data for extracting the at least one feature.

7. The method of claim 5, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data comprises:

automatically selecting at least filter to apply to the sensor data for extracting the at least one feature.

8. The method of claim 1, wherein the automatically generating a processing chain of a sensor system for executing the desired application based on the desired application and the plurality of raw sensor data comprises: automatically selecting the optimal data segment length and inference length.

9. The method of claim 1, further comprising:

tracking system requirements of the at least one feature for sensing the at least one activity of the desired application.

10. The method of claim 9, wherein the system requirements comprise memory consumption for the at least one sensor, computation consumption for the at least one sensor, power consumption for the at least one sensor.

11. The method of claim 9, wherein the sensing the at least one activity utilizes a plurality of features.

12. The method of claim 11, wherein the tracking system requirements of the at least one feature for sensing the at least one activity of the desired application comprises:

determining a relative importance of the plurality of features used in sensing the at least one activity of the desired application.

13. The method of claim 12, wherein the tracking system requirements of the at least one sensor for sensing the at least one activity of the desired application comprises:

aggregating the system requirements for the at least one sensor for sensing the at least one activity of the desired application.

14. The method of claim 9, further comprising:

displaying the system requirements for the plurality of features for sensing the at least one activity of the desired application.

15. The method of claim 14, further comprising:

receiving a selection of a feature of the plurality of features for deselection from sensing the activity.

updating the plurality of features to remove the deselected feature from the plurality of features for sensing the activity; and

displaying the system requirements for the plurality of features for sensing the at least one activity of the desired application without considering the deselected feature.

16. The method of claim 9, wherein the system requirements comprise memory consumption for the at least one feature, computation consumption for the at least one feature, power consumption for the at least one feature.

17. The method of claim 9, wherein the tracking system requirements of the at least one feature for sensing the at least one activity of the desired application comprises:

receiving at least one resource limitation;

determining a relative importance of the plurality of features used in sensing the at least one activity of the desired application based on the at least one resource limitation; and

automatically deselecting at least one feature based on the at least one resource limitation.