VIRTUAL SENSOR SYSTEM
A sensing system includes a sensor assembly and a back end server system. The sensor assembly includes a collection of sensors in communication with a control circuit. The sensors are each configured to sense one or more physical phenomena in an environment of the sensor assembly. The control circuit of the sensor assembly is configured to identify one or more selected sensors of the collection of sensors whose data corresponds to an event occurring in the environment of the sensor assembly and transmit data to the back end server system. The back end server system is configured to generate a first order virtual sensor by training a machine learning model to detect the event based on the data from at least one of the selected sensors and detect the event using the trained first order virtual sensor and data from the selected sensors.
This application is a continuation of U.S. Nonprovisional patent application Ser. No. 15/961,537, titled VIRTUAL SENSOR SYSTEM, filed Apr. 24, 2018, which claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 62/602,487, titled SYNTHETIC SENSORS, filed Apr. 24, 2017; U.S. Provisional Patent Application No. 62/602,543, titled GENERAL PURPOSE SYNTHETIC SENSOR SYSTEM, filed Apr. 27, 2017; and U.S. Provisional Patent Application No. 62/605,675, titled SECURE UBIQUITOUS SENSING SYSTEM, filed Aug. 22, 2017; the disclosure for each of which is hereby incorporated by reference in its entirety.
GOVERNMENT LICENSE RIGHTSThis invention was made with government support under grant CNS1526237 awarded by the National Science Foundation. The government has certain rights in the invention.
BACKGROUNDThe background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
The promise of smart environments (e.g., the “smart home”) and the Internet of Things (IoT) relies on robust sensing of diverse environmental facets. Traditional approaches rely on measuring one particular aspect of an environment with special-purpose sensors. Regardless of the approach taken, the goal remains the same: to apply sensing and computation to enhance the human experience, especially as it pertains to physical contexts (e.g., home, office, workshop) and the amenities contained within. Numerous approaches have been attempted and articulated, though none have reached widespread use to date.
One option is for users to upgrade their environments with newly released “smart” devices (e.g., light switches, kitchen appliances), many of which contain sensing functionality. However, this sensing is generally limited to the appliance itself (e.g., a smart light sensing whether it is on or off) or single parameter associated with its core function (e.g., a smart thermostat sensing whether the room is occupied). Likewise, few smart devices are interoperable, forming silos of sensed data that thwart a holistic experience. Instead of achieving a smart home, the best one can currently hope for are small islands of smartness. This approach also carries a significant upgrade cost, which so far has proven unpopular with consumers, who generally upgrade appliances in a piecemeal manner.
A variety of different sensing modalities have been described in the context of environmental sensing, including special-purpose sensing systems, distributed sensing systems, infrastructure-mediated sensing systems, and general-purpose sensing systems. These sensing modalities can be organized according to the number of sensors that they utilize and the number of facets or parameters that they sense. In particular, special-purpose sensing systems utilize a single sensor, infrastructure-mediated and general-purpose sensing systems utilize one or a few sensors, and distributed sensing systems utilize many sensors. Further, special-purpose sensing systems sense a single facet, infrastructure-mediated sensing systems tend to sense one or a few facets, general-purpose sensing systems tend to sense many facets, and distributed sensing systems can sense anywhere from a single facet to many facets of an environment.
However, currently existing sensing systems typically transfer all the sensed data to a backend server for processing and/or storage system leading to problems relating to, for example, bandwidth usage and processing speed.
SUMMARYIn one general aspect, the present invention is directed to a ubiquitous sensing system utilizing one or more sensors that are capable of directly or indirectly detecting events in the environment surrounding the sensor assembly. While the sensing system is configured for indirect sensing, such that each and every object and/or person in the environment need not be instrumented in a location in order to sense their state or events associated with them, the sensors may also be coupled to objects and/or humans for direct sensing without any modifications. The sensing system includes a sensor assembly that can be positioned within an environment or location and that is capable of communicating with a server or other type of computer system for processing. The sensing system may optionally process sensor data locally and transmit the processed data to the server. The server utilizes machine learning to characterize received sensor data and/or training data in association with an event or events to learn to detect the occurrence of the designated event(s). In one aspect, the user can annotate the sensor data stream to indicate when certain events occurred and the machine learning algorithm then learns what characteristics of the data stream correlate to the event, allowing the sensing system to then detect future occurrences of the event. In another aspect, the sensing system utilizes deep machine learning to determine when events have occurred and what characteristics of the sensor data stream correlate to those events. In yet another aspect, the server can have a library of previously trained machine learning models and/or may train machine learning models from prior data collection steps, crowd sourcing, or the like, for different activities and events, and the sensing system can directly send sensor data and have the server determine what events have occurred. The server thus can define a set of machine learning-trained “virtual sensors” that are each capable of detecting events from combinations of sensor data that are correlated with the occurrences of the events, but that are not necessarily provided by sensors that are directly affixed or otherwise associated with the object(s) being sensed. More specifically, these types of virtual sensors can be referred to as “first order” virtual sensors. The server can further implement higher order virtual sensors that are capable of detecting events or conditions from a combination of data from lower order virtual sensors (e.g., second order virtual sensors detect an event from the output of first order virtual sensors).
In that connection, in one embodiment, the sensing system comprises a sensor assembly with processing and communication capabilities and a back end server system. The sensor assembly comprises a control circuit and one or more sensors. Each of the sensors senses one or more different physical phenomenon in an environment of the sensor assembly. The back end server system, which comprises at least one server, is in communication with the sensor assembly. Further, the control circuit of the sensor assembly is configured to, among other things: (i) extract features from raw sensor data from the plurality of sensors; and (ii) transmit data packets to the back end server system, wherein the data packets comprise featurized data from the plurality of sensors. The at least one server of the back end server system is configured to implement the first order virtual sensors, where each of the first order virtual sensors is trained through machine learning to detect, based on the featurized data transmitted from the sensor assembly, an event or condition in the environment of the sensor assembly.
In another general embodiment, the back end server system is programmed to receive the featurized sensor data from the sensor assembly; determine an occurrence of an event via the featurized sensor data; train, via machine learning, a virtual sensor implemented by the server to detect the event by characterizing the featurized sensor data for the plurality of sensors that are activated in association with the event; and monitor, via the virtual sensor, for subsequent occurrences of the event.
Thus, various embodiments of the present invention provide a highly capable sensor that can directly or indirectly monitor a large environment. These and other benefits of the present invention will be apparent from the description that follows.
The features of various aspects are set forth with particularity in the appended claims. The various aspects, however, both as to organization and methods of operation, together with further objects and advantages thereof, may best be understood by reference to the following description, taken in conjunction with the accompanying drawings as follows.
Certain aspects will now be described to provide an overall understanding of the principles of the structure, function, manufacture, and use of the devices and methods disclosed herein. One or more examples of these aspects are illustrated in the accompanying drawings. Those of ordinary skill in the art will understand that the devices and methods specifically described herein and illustrated in the accompanying drawings are non-limiting examples aspects and that the scope of the various aspects is defined solely by the claims. The features illustrated or described in connection with one aspect may be combined with the features of other aspects. Such modifications and variations are intended to be included within the scope of the claims. Furthermore, unless otherwise indicated, the terms and expressions employed herein have been chosen for the purpose of describing the illustrative aspects for the convenience of the reader and are not to limit the scope thereof.
The sensing system 100 is configured to train and implement one or more virtual sensors 118, which are machine learning based classification systems or algorithms trained to detect particular events to which the virtual sensors 118 are assigned as correlated to the data sensed by the sensors 110 of the sensor assembly 102 and/or other virtual sensors 118. The training and implementation of various aspects of the virtual sensors 118 are described in more detail below.
The sensing system 100 may be configured to be, without limitation, a special-purpose sensing system, a distributed sensing system, infrastructure-mediated sensing system, and/or a general-purpose sensing system.
In an aspect, special-purpose sensing systems may include a single sensor assembly 102 configured to monitor a single facet of an environment. For example, a sensor assembly 102 including a microphone can be affixed to a faucet so that water consumption can be inferred (which, in turn, is used to power behavior-changing feedback). As another example, a sensor assembly 102 including a temperature sensor and/or an occupancy sensor can be placed in a room to sense environmental data that can be used by a heating, ventilation, and air conditioning (HVAC) system to manage the HVAC system.
In an aspect, infrastructure-mediated sensing systems may include one or more sensor assemblies 102 installed within a structure at strategic infrastructure probe points. For example, sensor assemblies 102 can be coupled to a building's power lines to detect “events” caused by electrical appliances. Since home electrical lines are shared, a single sensor assembly can observe activities across an entire house. Infrastructure-mediated sensing systems may also be coupled to, e.g., HVAC, plumbing, natural gas lines, and electric lighting. For example, a sensing assembly including one or more sensors may be installed at a probe point, enabling the sensing system to monitor aspects of the building. For example, a plumbing-attached sensor assembly may be configured to detect sink, shower, and toilet use. Infrastructure-mediated sensing systems may include one sensor assembly and/or a plurality of sensor assemblies utilized to monitor a few facts of an environment.
In an aspect, distributed sensing systems may include many sensor assemblies 102 deployed in an environment that are networked together. Such a sensing system may be used to enlarge the sensed area (e.g., occupancy sensing across an entire warehouse) or increase sensing fidelity through complementary readings (e.g., sensing seismic events utilizing sensors deployed across an area). The distributed sensor assemblies 102 can be homogenous (e.g., an array of identical infrared occupancy sensors) and/or heterogeneous. Also, the array can sense one facet (e.g., fire detection) or many facets (e.g., appliance use). For example, a home security system is a heterogeneous distributed system, where one or more sensor assemblies may include door sensors, window sensors, noise sensors, occupancy sensors and even cameras work together to sense a single facet of the environment: “Is there an intruder in the home?” As another example, a homogenous array of sensor assemblies comprising magnetic sensors can be utilized to detect object interactions throughout an entire house. Thus, distributed sensing systems may be configured to include as many sensor assemblies utilized to monitor anywhere between a single facet to many facets of an environment, depending upon the particular implementation of the distributed sensing system.
In an aspect, a general purpose sensing system may include a wide variety of underlying sensor assemblies 102 that can be utilized flexibly such that they can be attached to a variety of objects and can sense many facets without any modification to the sensor assembly 102.
In certain aspects, the sensing system 100 may be a direct sensing system and/or an indirect sensing system. For direct sensing, a sensor assembly 102 is physically coupled to an object or infrastructure of interest and may provide excellent signal quality. Some direct sensing systems may include utilize batteries or other power sources to power the sensor assembly. Indirect sensing systems seek to sense state and events indirectly, without having to physically couple to objects. For example, a sensor assembly including an electromagnetic sensor (EMI sensor) can be installed near an appliance and/or it's power source to detect usage of the appliance because when an appliance is in different modes of operation (e.g., refrigerator compressor running, interior lights on/off), the object and/or the power source emits characteristic electromagnetic noise that can be captured and recognized. As another example, a sensor assembly including an acoustic sensor can be installed in a workshop to recognize tool usage according to the detected acoustic characteristics of each tool. Example sensors to be included in the sensor assembly 102 that are configured for indirect sensing can include, without limitation, noncontact thermometers, rangefinders, motion sensors, EMI sensors, acoustic sensors, vibration sensors, magnetic field sensors, cameras, ultrasonic sensors, laser based sensors (e.g., lidar), or the like. Indirect sensing systems have greater flexibility in sensor placement which allows for sensors to be better integrated into the environment or even hidden. Further, it may be possible to place the sensor assembly of an indirect sensing system at a nearby wall power outlet, eliminating the need for batteries.
Referring back to
The featurization module 112 can determine or extract various features from, for example, the time domain and/or the frequency domain representations (e.g., by transformation of the time domain representations) of the sensor data. The features from the raw data from the sensors 110 can be extracted utilizing a number of different techniques, which can vary according to sample rate at which the sensor data was collected or the particular type of sensors 110. Furthermore, the number and/or types of features extracted from the raw sensor data and/or transmitted to the computer system 104 by the featurization module 112 can be based on, for example, the sample rate, the types of sensors, user input, or the like. In various aspects, the number and/or types of features extracted by the featurization module 112 can be controlled by the featurization module 112 itself, the computer system 104, a client 106 (
In one aspect depicted in
After the data is processed by the featurization module 112, the featurized data can be processed and/or analyzed by a machine learning module 116 of the computer system 104 included in the sensing system 100. In one aspect, the machine learning module 116 can generate a machine learning model to detect correlations between the data and events that have occurred. In one aspect, the machine learning module 116 generates a classifier, which is an algorithm that is trained via a machine learning model to assign an input to one or more categories based upon the training that the classifier received. In this aspect, the classifier can be trained to identify the occurrence of a given event based upon the grouped, featurized data that is provided to the machine learning module 116 as training data. In training the classifier to identify an event, the machine learning module 116 can assess the informational power of different sensor channels and may select appropriate thresholds for optimal accuracy in characterizing the training data. The training by the machine learning module 116 causes the classifier to learn what sensor data streams are associated with an event type and, further, what characteristics of those data streams identify the event type with particularity. Once trained to identify an event, a virtual sensor 118 can output a notification and/or signal when the event is detected that causes a graphical user interface 500, an example of which is depicted in
The machine learning module 116 can utilize supervised learning, unsupervised learning, and/or both techniques in training the classifier. The advantage of using both supervised and unsupervised methods may be that it is an effective method for correlating different types of features from multimodal data. Using both supervised and unsupervised methods may also be advantageous because it enables fine tuning of unsupervised training with supervised training results. Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consists of a set of training examples. In supervised learning, each example is a pair consisting of an input object, typically a vector, and a desired output value or target. The goal is to learn a general rule that maps inputs to outputs. A supervised method may be advantageous because a supervised learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples. Further, an unsupervised method tries to find hidden structure in unlabeled data and includes an algorithm with no target value, i.e., there is no error or reward signal to evaluate a potential solution. Instead, the algorithm has a halting criterion. Examples of halting criteria include, but are not limited to, precision, recall, accuracy, number of cycles, and time. An unsupervised method may be advantageous for use in model training when the only data available is unlabeled data.
In an aspect, the machine learning module 116 can utilize now or hereafter known machine learning methods to detect correlations between the data and events that have occurred, such as various deep learning algorithms, clustering, etc. In other aspects, after the data is processed by the featurization module 112, the featurized data can be processed by other classification modules, such as a logistic regression module, a clustering module (e.g., k-means, spectral, density based spatial clustering of applications with noise (DBSCAN) and mean-shift), a decision tree module, or a random forest module. In one aspect, the machine learning module 116 comprises an ensemble classification model utilizing, e.g., an algebraic combination technique or a voting (plurality) combination technique. Ensemble classifications models can promote robustness against false positives, while supporting the ability to detect simultaneous events. In one aspect, the machine learning module 116 comprises use base-level support vector machines (SVMs) trained for each virtual sensor, along with a global (multi-class) SVM trained on all sensors. In embodiments where the computer system 104 implements numerous first or higher order virtual sensors 118, the virtual sensors 118 could all use the same machine learning technique or they could use different machine learning techniques. For example, some virtual sensors 118 could use support vector machines, some decision trees, some neural networks, etc. In an aspect, the featurized data may be organized as feature vectors and the feature vectors are fed into the machine learning module 116 as the training data for the classifiers.
In some aspects, after the raw sensor 110 data is processed by the featurization module 112, the featurized data can optionally be processed by an activation group module prior to being transmitted to and/or prior to being processed by the machine learning module 116. In an aspect, the activation group module further processes and converts featurized data from the featurization module 112 into various forms of processed data, as discussed below. In one aspect, the activation group module can be executed on the sensor assembly 102, i.e., prior to the featurized data being transmitted to the computer system 104. In another aspect, the activation group module can be executed by the computer system 104 after the featurized data has been received thereby. Additionally and/or alternatively, the activation group module can be a part of a node between the sensor assembly 102 and the computer system 104 (e.g., a gateway). In aspects including such an intermediate node, the intermediate node can be considered part of the computer system 104 as described herein. In an aspect, the activation group module may also process raw sensor data without featurization.
In an aspect, the activation group module can determine which of the sensors 110 have been triggered at a given time or within a given time window and extract a subset of the data from the sensors as activation group data corresponding to only the activated sensors. Determination of which sensor channels have been activated may reduce the effects of environmental noise on the received sensor data. The activation group module can determine which of the sensors 110 have been triggered or activated by, for example, determining a baseline or background profile of the environment as a calibration routine and using the baseline or background profile to determine which sensors or sensor channels are “activated” by subtracting or otherwise removing the baseline or background profile from the featurized sensor data. In one aspect, the activation group module determines whether a given sensor 110 has been activated by utilizing an adaptive background model for each sensor channel (e.g., rolling mean and standard deviation). In various aspects, all received data streams can be compared against the background profile using, e.g., a normalized Euclidean distance metric. Sensor channels that exceed the baseline by a predetermined threshold (which may be unique for each sensor) are tagged as “activated.” In one aspect, the activation group module can further utilize hysteresis to avoid detection jitter. Thresholds can be, e.g., empirically obtained by running the sensors 110 for several days while tracking their longitudinal variances or set by the user or system administrator. In one aspect, the background profile refers to data sensed by a sensor based on ambient condition of an environment that are not related to events of interest, and which the sensing system 100 can obtain when a sensor assembly 102 is initially activated or deployed in a new environment. Alternatively and/or additionally, the sensing system 100 can periodically obtain the environmental background profile for each sensor assembly 102. Obtaining an environmental background profile helps with reducing false positives for the same activity as the baselines change (e.g., for detecting the sound of a particular machine in a factory setting, the constant drone of a fan or other such consistent sounds may be subtracted from the featurized data as a baseline or background profile).
Further, the activation group module can create data sets by subtracting the baseline or background profile from any sensor signals detected by the sensors 110. Such data sets will require less bandwidth for transmission (if the activation group module is part of or executed by the sensor assembly 102) to the computer system 104 for training by the machine learning module 116. Still further, the activation group module can tag the data sets with identification information corresponding to the activated sensors such that the machine learning module 116 knows which sensor streams to consider, and which sensor streams to ignore, when training the machine learning model (e.g., a classifier). This assists in classification by reducing the feature space in which the classifier is trained. The identification of a particular grouping of sensors 110 that have been activated in association with an event can serve as useful metadata to describe the event, which can in turn assist in classifying the event type that has been detected. For example, a boiling kettle can activate infrared, vibration, and/or acoustic sensors of the sensor assembly 102 and a determination by the activation group module that infrared, vibration, and/or acoustic sensors have been activated from amongst a group of sensors 110 of the sensor assembly 102 can itself be used as a feature to assist in classifying the event as a kettle boiling within the detection area of the sensor assembly 102. In some aspects, the activation group module can, optionally, assemble an amalgamated feature vector of the featurized data from the activated sensors, which is then provided to the machine learning module 116.
Referring back to the machine learning module 116, in one aspect where the machine learning module 116 utilizes supervised learning, the sensing system 100 can be configured to provide labels for the featurized data. The labels can be provided by users and/or generated by the sensing system 100.
In aspects where the labels are provided by users, the sensing system 100 can include an interface for users to indicate when and what types of events have occurred, which can then be correlated to the data sensed by the sensor assembly 102. For example,
In aspects where the labels are generated by the sensing system 100, the sensing system 100 can automatically generate the labels, by, for example, clustering, deep learning, or other now or hereafter known methods that can be implemented by the machine learning module 116. In one aspect, a user could use the user interface 500 to verify whether any labels automatically generated by the sensing system 100 are correct or incorrect. The machine learning module 116 could then adjust the training of the machine learning model being used to generate the labels to avoid characterizing such false positives. In another aspect, a user could use the user interface 500 to supplement the labels automatically generated by the sensing system 100 or otherwise apply additional labels to the sensor data streams, as described above. The machine learning module 116 could then adjust the training of the machine learning model being used to generate the labels to properly characterize such false negatives. Additionally and/or alternatively, the feature vectors along with their associated labels may be fed into the machine learning module 116 as the training data for the classifiers.
In another aspect where the machine learning module 116 utilizes unsupervised learning, the machine learning module 116 can comprise a deep neural network configured to perform deep learning. As used herein, “deep learning” refers to a form of machine learning that utilizes multiple interconnected neural network layers along with feedback mechanisms or other methods to improve the performance of the underlying neural network. Deep learning systems are usually based on several interconnected layers of a convolution neural network, among other layers, interconnections, or feedback mechanisms. There are many variants of neural networks with deep architecture depending on the probability specification and network architecture, including, but not limited to, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural network (RNN)-enhanced models capable of sequential data pattern learning, and autoencoders. Deep-learning models may be trained to learn representations of data using supervised and/or unsupervised learning. From a computational standpoint, the methods used in deep learning involve several mathematical calculations of matrix-to-matrix and matrix-to-vector calculations. The number and nature of these calculations makes them essentially impossible for a human to perform the calculation by-hand or by manual process, within any practical amount of time. In one such aspect, the machine learning module 116 uses a two-stage clustering process. First, the machine learning module 116 reduces the dimensionality of the data set using a multi-layer perceptron configured as an autoencoder. The autoencoder can have, e.g., multiple nonoverlapping sigmoid functions in the hidden layer(s). Because the output of the autoencoder is the same as the input values, the hidden layer(s) will learn the best reduced representation of the feature set. Second, this reduced feature set is used as input to an expectation maximization (EM) clustering algorithm. In other embodiments, the machine learning module 116 can comprise a decision tree, a logistic regression model, a random forest, etc. In one aspect wherein the machine learning module 116 utilizes deep learning, some or all of the featurization or feature extraction may be accomplished automatically using learning from the training data. However, since the accuracy of a deep learning system depends, at least in part, on the sequence in which training data is provided to the deep learning system, pre-processing of the training data (e.g., using featurization by the featurization module 112, using the activation group module, by providing labels, etc.) and selection of training data may be used to improve accuracy of the model. Selection of training data includes, for example, using domain-specific knowledge to improve performance of the machine learning system. Domain expertise, as used herein, provides a context in which a deep learning system can operate and can be used to select elements of training data, the order in which the training data is presented to the deep learning system, and certain sorts of invariances.
In one aspect, the computer system 104 can be further programmed to perform featurization (i.e., featurization at the computer system 104), in addition to and/or as an alternative to the onboard featurization performed by the sensor assembly 102. The additional featurization can include extracting features that would require computationally expensive processing for the sensor assembly 102 hardware to handle or that would be too large to transmit to the computer system 104. The additional features can be computed by the computer system 104 (e.g., the cloud or a remote server) or an intermediate node between the sensor assembly 102 and the computer system 104, such as a gateway. In aspects including such an intermediate node, the intermediate node can be considered part of the computer system 104 as described herein. In various aspects, the computer system 104 (including, potentially an intermediate node) can be configured to compute additional features from data corresponding to one or more high sample rate sensors and/or one or more low sample rate sensors of the sensor assembly 102. The additional features computable by the computer system 104 can include, without limitation, band ratios, fractional harmonics, first or second order signal derivatives, MFCCs, and/or statistical features (e.g., min, max, range, mean, median, mode, sum, standard deviation, and centroid) from raw data from the acoustic, EMI, vibration, or other sensors 110, and/or from already featurized data from the featurization module 112. In one aspect, the computer system 104 can be configured to normalize data from other sensors 110. This server-side featurized sensor data can then be fed, either alone or in combination with the data featurized onboard the sensor assembly 102, to the machine learning module 116 (or, in some aspects, to the activation group module, which in turn feeds into the machine learning module 116) for classification, as described above.
The machine learning module 116 of the computer system 104 is also configured to train one or more virtual sensors 118. In one aspect, the classifier trained by the machine learning module 116 on the provided training data and/or sensor data associated with a given event can be considered a virtual (or synthetic) sensor 118 for that event. For example, a classifier trained by the machine learning module 116 to recognize a boiling kettle according to featurized data from various combinations of infrared, vibration, and/or acoustic sensors of the sensor assembly 102 can be defined as a “kettle boiling” virtual sensor 118. As another example, a classifier trained by the machine learning module 116 to recognize the movement of a door according to featurized data from acoustic and/or vibration sensors 110 can be defined as a “door movement” virtual sensor 118. It should be noted that because each virtual sensor 118 is trained on data collected by a sensor assembly 102 or combinations of sensor assemblies 102, and because such sensor assemblies 102 will be located in a number of different types of environments when utilized in the field, the subset of sensors 110 included in each sensor assembly 102 that are correlated with an event type can vary. Furthermore, the machine learning associated with the same event and same subset of sensors 110 can also vary depending on the environment of the sensor assemblies 102 (e.g., if the background profile is different for the different environments). Specifically, the machine learning model for each virtual sensor 118 could also have different parameters and/or weights for an event based on the environment in which the sensor assemblies 102 are located. Therefore, different virtual sensors 118 (i.e. virtual sensors 118 utilizing different subsets of sensors 110 or having different parameters and/or weights assigned to the event) may be implemented to detect the same event in different locations, different environments, or even over different time periods in the same location. In other words, each virtual sensor 118 will be uniquely trained to detect the occurrence of an event according to the sensor data unique to the particular environment(s) in which the sensor assemblies 102 are located and/or at a particular time. For example, a “door movement” virtual sensor 118 in a first environment and/or during a first time period (e.g., during working hours) could be trained to identify the movement of a door based on a combination of acoustic and vibration data and/or a first machine learning model. However, a “door movement” virtual sensor 118 in a second environment and/or during a second time period (e.g., at night) could be trained to identify a door closing based solely upon acoustic data and/or a second machine learning model. Further, the machine learning model for each virtual sensor 118 could also have different parameters and/or weights based on the environment in which the sensor assemblies 102 are located. The sensing system 100 does not utilize any pre-established restrictions on the training of the virtual sensors 118, thus each virtual sensor 118 will be uniquely trained to detect events according to its environment.
While the above description describes a different virtual sensor 118 (i.e., a classifier or a machine learning model) for each set of unique conditions associated with an event, a single generic virtual sensor 118 could be trained for each event that takes into account a wide variety of conditions related to different environments and over a period of time, without deviating from the principles of this disclosure.
As depicted in
In one aspect depicted in
Although
Furthermore, although
The second order virtual sensors 124 could also subscribe to and/or receive other, non-first order virtual sensor data. For example, in some embodiments, a second order virtual sensor 124 could receive data from at least one first order virtual sensor 120, as well as featurized data from one or more of the sensors 110 of the sensor assembly 102, in order to make its classification. This applies to higher order sensors as well. A xth order virtual sensor could receive data from at least one (x−1)th order sensor, as well as either (i) data from lower order sensors (e.g., (x−2)th, (x−3)th, etc.) and/or (ii) featurized data from one or more of the sensors 110 of the sensor assembly 102, in order to make their classification.
In some aspects, the higher order virtual sensors can include algorithms that, for example and without limitation, count the number of occurrences or duration of an event detected by a lower order virtual sensor, algorithms that smooth the outputs of lower order virtual sensors (and, in some cases, the sensors 110 of the sensor assembly 102), algorithms that combine the outputs of multiple lower order virtual sensors and/or sensors 110 (featurized and/or raw data), or the like. As an example of a higher order virtual sensor that can combine the outputs of lower order sensors, a second order virtual sensor 124 could indicate whether an occupant is present within a home by analyzing the outputs of multiple human activity-related first order virtual sensors 120, such as a “washing dishes” first order virtual sensor 120, a “movement in the kitchen” first order virtual sensors 120, and so on. In another example, a third order virtual sensor could output an alarm if the second order virtual sensor 124 determines that a home owner is not present (e.g., by determining that for a threshold period of time the first order virtual sensors 120 have indicated that no lights have not been turned on) and another first order virtual sensor 120 detects a fire event. In these aspects, the higher order virtual sensors can receive the outputs of one or more lower order virtual sensors and/or sensors 110 of the sensor assembly and then make its corresponding classification accordingly.
In some aspects, the higher order virtual sensors can, as with the first order virtual sensors 120, include classifiers trained by a machine learning module on the output from one or more lower order sensors to identify the occurrence of a trained-for event or condition. The higher order virtual sensors can be trained on the outputs of at least one immediately lower order of virtual sensor in the hierarchical structure, rather than strictly on the outputs of the sensors 110 of the sensor assembly 102. For example,
Although
In one aspect, the first order virtual sensors 120 can produce a binary output (e.g., are binary classifiers). For example, a first order virtual sensor 120 trained to identify whether a faucet is running or whether someone is at their desk working could produce a continuous, time-stamped binary “yes” or “no” outputs. In this aspect, higher order virtual sensors can further produce nonbinary outputs, such as state (of an object or environment), count, and duration. For example, the sensing system 100 could implement five separate first order virtual sensors 120 that track five separate aspects of a microwave: whether the microwave is running, the keypad has been pressed, the door has been opened, the door has been closed, and the completion chime has sounded. From these time-stamped binary outputs of the first order virtual sensors 120, a second order virtual sensor 124 could generate a nonbinary output of the states of the microwave: available, door ajar, in-use, interrupted, or finished. In one implementation, when the completion chime is detected (i.e., the “completion chime” first order sensor 120 is activated), the microwave state output of the second order virtual sensor 124 can change from “in-use” to “finished.” The microwave state output can stay as “finished” until a “door closed” event is detected (i.e., the “door closed” first order virtual sensor 120 is activated), after which the items inside the microwave are presumed to have been removed and the microwave state output of the second order virtual sensor 124 is changed to “available.” Second order virtual sensors 124 need not be connected to multiple first order virtual sensors 120 to produce nonbinary outputs though. As another example, the sensing system 100 could implement a first order virtual sensor 120 that detects when a door is opened and a second order virtual sensor 124 that counts the number of times that the first order virtual second 120 has been activated. As yet another example, the sensing system 100 could implement a first order virtual sensor 120 that detects when a faucet is running and a second order virtual sensor 124 that tracks the time duration that the first order virtual sensor 120 is activated. That way, an approximation of the total amount of water used could be computed. These are examples of just a few of the first and second order virtual sensors that can be implemented in various embodiments of the present invention. In other aspects, the first order virtual sensors 120 can produce nonbinary outputs. For example, the first order virtual sensors 120 could include multi-class classifiers trained to output one of several labels. As described above, the first and second (or higher order) virtual sensors could be trained to detect other, binary or nonbinary, conditions, states, durations, etc.
By having lower order virtual sensors feed into higher order virtual sensors, the sensing system 100 can infer increasingly richer details about the environment in which the sensor assembly 102 is located. Further, multiple sensor assemblies 102 can be communicably connected to the computer system 104 and the data feeds from the multiple sensor assemblies 102 can be combined to provide additional data that can be processed by machine learning to infer information about the environment from correlated data from the sensor assemblies 102. For example, one or more appliance-level second order virtual sensors could feed into a kitchen-level third order virtual sensor, which could in turn feed into a house-level fourth order virtual sensor, and so on. A house-level virtual sensor drawing on multiple lower order sensors (whether they are virtual sensors or actual sensors disposed on one of the sensor assemblies within the house) across many rooms can classify complex facets like human activity. Tracking human activities accurately can be very useful in a variety of contexts, such as with smart homes, healthcare tracking, managed care for the elderly, and security and safety of human occupants.
The outputs of the various virtual sensors 118 can further be fed into applications executed by the computer system 104, an external client 106 (
The sensor assembly 102 is communicably connectable to the computer system 104 (e.g., one or number of networked servers) such that the computer system 104 can receive the signals or data generated by the sensors 110 for processing thereon, as described above. In the depicted example, each sensor assembly 102 is communicably connectable to the computer system 104 via a data communication network 108, such as the Internet, a LAN, a WAN, a MAN, or any other suitable data communication network. In this aspect, the sensor assembly 102 can include an appropriate network interface for connecting to the data communication network 108 such as, for example, a W-Fi network interface controller. In other aspects, the sensor assembly 102 can communicably connect to the computer system 102 utilizing other wired or wireless communication protocols or other communication networks (e.g., a cellular telecommunication network or Ethernet). The network interface controller of the sensor assembly 102 may include a network interface controller suitable to implement wireless or wired communication utilizing a variety of communication protocols and/or access methods, such as cellular, Bluetooth, ZigBee, RFID, Bluetooth low energy, NFC, IEEE 802.11, IEEE 802.15, IEEE 802.16, Z-Wave, HomePlug, global system for mobile (GSM), general packet radio service (GPRS), enhanced data rates for GSM evolution (EDGE), code division multiple access (CDMA), universal mobile telecommunications system (UMTS), long-term evolution (LTE), LTE-advanced (LTE-A), LoRa (or another lower power wide-area network communication protocol), or any other suitable wired and/or wireless communication method or combination thereof. The network 108 may include one or more switches and/or routers, including wireless routers that connect the wireless communication channels with other wired networks (e.g., the Internet). The data communicated in the network 108 may include data communicated via short messaging service (SMS), multimedia messaging service (MMS), hypertext transfer protocol (HTTP), direct data connection, wireless application protocol (WAP), email, smart energy profile (SEP), ECHONET Lite, OpenADR, or any other protocol that may be implemented with the sensor assemblies 102, physical hubs, cloud sever communication, or gateway modules.
In certain aspects, one or more of the sensor assemblies 102 may also be communicably connected to each other, and/or to a client 106 (e.g., a user device) via the network 108 and/or via a separate network. For example, such a network may be a local network established by a local router or a local switch. Optionally, the sensor assemblies 102 may be a peer-to-peer (P2P) network, and may communicate with each other directly. In a peer-to-peer network, service discovery schemes can multicast the presence of nodes, their capabilities, and group membership. The peer-to-peer devices can establish associations and subsequent interactions based on this information.
A sensing assembly 102 may implement one or more application-layer communication protocols. Examples include constrained application protocol (CoAP), message queue telemetry transport (MQTT), OPC UA, HTTP, REST APIs and the like for implementing a respective messaging protocol. Sensing assembly 102 may also implement lower-layers communication protocols which may implement layers of a communication protocol stack lower than the application-layer. Example layers implemented may include one or more of the physical, data link, network, transport, session, internet, and presentation protocols. Example protocols implemented include one or more of: Ethernet, Internet Protocol, Transport Control Protocol (TCP), protocols for the 802.11 standard (e.g., PHY, Medium Access Control, Logical Link Control, and the like), and the like.
In one embodiment, the computer system 104 may be a virtual machine. The virtual machine may be any virtual machine, while in some embodiments the virtual machine may be any virtual machine managed by a Type 1 or Type 2 hypervisor, for example, a hypervisor developed by Citrix Systems, IBM, VMware, or any other hypervisor. In some aspects, the virtual machine may be managed by a hypervisor, while in aspects the virtual machine may be managed by a hypervisor executing on a server or a hypervisor executing on a user device.
In some embodiments, the client 106 may display application output generated by an application remotely executing on a server or other remotely located machine (for e.g., for controlling, communicating and/or accessing data from a sensor assembly 102 and/or controlling and/or communicating with various objects of an environment being sense). In these embodiments, the client device may execute a virtual machine receiver program or application to display the output in an application window, a browser, or other output window.
As mentioned above, in one aspect, the sensors 110, connector 126, microcontroller 121, and various other components of the sensor assembly 102 can be supported upon a printed circuit board (PCB) substrate 128. In one aspect, the sensors 110 can be disposed on a first surface of the PCB substrate 128 and the connector 126 can be disposed on a second, opposing surface of the PCB substrate 128 so that the sensors 110 are oriented outwardly towards the environment when the connector 126 is plugged into or connected to a corresponding socket. In other aspects, the sensors 110 can be mounted on various layers of the PCB substrate 128. For example, sensors 110 such as, without limitation, an EMI sensor configured to measure the electro magnetic interference in the line voltage of a power circuit caused by an electrical device may be included in a first one or more layer(s) of the PCB substrate 128 and other sensors 110 may be included in a different layer of the PCB substrate. In one aspect, the sensor assembly 102 further includes a housing enclosing the various components. That is, the housing can house the PCB substrate 128 and the sensors 110 connected thereto. The housing can protect against physical damage and electrostatic discharge. Further, the housing can be designed to accommodate sensors 110 that require line of sight and access to the environment's air by, for example, having access cutouts for the relevant sensors 110. The housing could be constructed from, for example, laser cut cast acrylic and/or constructed via injection molding or 3D printing processes. In other aspects, the sensor assembly 102 could comprise multiple PCB substrates, with the sensors 110 on different PCB substrates. In such embodiments, the housing can enclose all of the sensors 110 and PCB substrates.
The sensors 110 can include various combinations of sensing devices that are configured to detect various different physical or natural phenomena. In one aspect, the sensors 110 include an infrared radiation sensor 130 (e.g., a Panasonic Grid-EYE AMG8833 infrared array sensor), an ambient light color and/or intensity sensor 132 (e.g., a TAOS TCS34725 RGB sensor), a magnetic field sensor 134 (e.g., a Freescale Xtrinsic MAG3110 magnetometer), a temperature sensor 136, an ambient pressure sensor, a humidity sensor (e.g., all part of a Bosch BME280 environmental sensor), an air quality or air composition sensor (e.g. a Bosch BME680 sensor for sensing the presence of certain volatile organic compounds), a vibration sensor 138 (e.g., an InvenSense MPU-6500 accelerometer six-axis accelerometer and gyroscope motion tracking sensor, which can detect vibrations through the structure when the sensor assembly 102 is secured to an electrical outlet), an external device detection sensor 140 (e.g., a 2.4 GHz network interface controller for detecting the presence and/or activity of external electronic devices connected to the Wi-Fi network or a Bluetooth LE sensor for detecting the presence of external electronic devices in the vicinity of the sensor assembly 102), a motion sensor 142 (e.g., a Panasonic AMN21111 PIR motion sensor), an acoustic sensor 144 (e.g., an Analog Devices ADMP401 microphone), and/or an EMI sensor 146 (e.g., a 100 mH inductor to capture over-air EMI and/or a passive RC network to sense EMI changes in the line voltage of the power source to which the sensor assembly 102 is connected). Various implementations of the sensor assembly 102 can utilize any number and combination of the aforementioned sensors and any other types of sensors 110 for detecting physical or natural phenomena. Further, the sensors 100 can be analog or digital sensors. Preferably, the sensor assembly 102 does not comprise a high-resolution camera (i.e., higher resolution than a thermal imager, such as an infrared radiation sensor 130). As such, the sensing system 100 can make the detections and classifications described herein without use of a camera, which decreases the cost and power consumption of the sensor assembly 102. It also decreases the amount of data that needs to be featurized onboard and transmitted to the computer system 104 since there is no image data to featurize and transmit. Further, not employing a camera reduces privacy concerns and image data of the environment of the sensor assembly 102 are not captured. Even when a high-resolution camera is employed on the sensor assembly 102, the image data can be featurized onboard the sensor assembly 102 so that privacy is maintained in the data sent to the computer system 104.
In one aspect, the sensor assembly 102 can further include one or more interfaces that can be utilized to connect to or communicate with additional sensors external to the sensor assembly 102. The interfaces can include, for example, Serial Peripheral Interface (SPI), Inter-Integrated Circuit (I2C), General Purpose Input/Output pins (GPIOs), and/or universal asynchronous receiver-transmitter (UART). The interfaces allow additional external sensors to be connected to the sensor assembly 102 in order to supplement and/or extend the functionality of the sensor assembly 102. Additional sensors that could be modularly connected to the sensor assembly 102 via the interfaces could include, for example, motion sensors (e.g., Doppler radar sensors), EMI sensors configured to detect the transients caused in the line voltage of the power source directly to which the sensor assembly 102 is connected, a lidar sensor, an ultrasonic sensor, and/or an active noise management system.
The sensors 110 of the sensor assembly 102 can include passive sensors and/or active sensors. A passive sensor is a sensor that simply detects or senses various physical or natural phenomena of an environment. Examples of such passive sensors are described above and may include, without limitation, vibration sensors, microphones, EMI sensors, infrared radiation sensors, acoustic sensors, temperature sensors, humidity sensors, camera, motion sensors (e.g., accelerometer, gyroscope, etc.), electric field sensors, chemical sensors, photo sensors, or the like. An active sensor is sensor used for measuring signals transmitted by the sensor that were reflected, refracted or scattered by an object of the environment and/or disturbances caused by the transmitted signals in the environment. Examples of such active sensors include, without limitation, sonar sensors (e.g., Doppler sensors), ultrasonic sensors, radar sensors, lidar sensors, acoustic sensors, infrared cameras, active IR sensors, indoor positioning systems, x-ray sensors, seismic sensors, active sound measurement systems, light emitting systems, or the like. In an aspect, an output device (described below) of a sensor assembly 102 may be configured to transmit a signal that may be reflected, refracted or scattered by an object of the environment and/or may cause disturbances in the environment, where such reflection, refraction, scattering, and/or disturbance is subsequently sensed by a sensor 110 of the sensor assembly 102, thereby forming an active sensor assembly 102 without an actual active sensor 110. The data from such active sensors could be featurized and used to detect events/conditions by the first or second (or higher) order virtual sensors. The active sensors could also be used to calibrate a space in which the sensor assembly 102 is located, as described further below. The data from such active sensors could be featurized and used to detect events/conditions by the first or second (or higher) order virtual sensors, just like the passive sensors as described above. The active sensors could also be used to calibrate a space in which the sensor assembly 102 is located, as described further below. Furthermore, an active sensor can be used for authentication of an object and/or a person as described below.
In aspects wherein the sensor assembly 102 includes an active sensor, such as an acoustic sensor (e.g., a microphone), the acoustic sensor can be utilized via an active sound management system that transmits a sound signal and receives the reflected, refracted and/or refracted signal to determine, for example, the sensor assembly's 102 position relative to walls or other structures within its vicinity and calibrate the acoustic sensor (and/or other sensors 110) accordingly. Such calibrations can be utilized to, for example, compensate for echoes or other audio artifacts that could interfere with the detection of certain events. The audio artifacts can be compensated for by, for example, signal processing techniques executed onboard the sensor assembly 102 to reduce errors. In still other aspects wherein the sensor assembly 102 includes an ultrasonic sensor, the ultrasonic sensor can be utilized to emit sound waves in order to calibrate other sensor assemblies 102 that are within the detection distance. Such audio signals can be utilized to pair sensor assemblies 102 together and/or allow the sensing system 100 to determine the spatial orientation of the various sensor assemblies 102 relative to each other within an environment. In yet another aspect, wherein the sensor assembly includes a speaker, the sensor can output particular sound pattern (e.g., a frequency sweep tone from configurable low frequency values to high frequency values) and have either the microphone on the same sensor assembly 102, or a different sensor assembly 102 in the vicinity, detect the audio signal using a microphone sensor to actively measure and calibrate for the environment.
In one aspect, the sampling rate of each of the sensor assembly's 102 sensors 110 can be automatically varied according to the sensor 110 type or the property or phenomena being sensed. For example, the vibration sensor 138 could have a high sample rate and the temperature sensor 136 could have a low sample rate (because temperature generally changes relatively slowly). Varying the sampling rate according to the property being sensed by the sensors 110 allows data to be collected at the rate needed to capture environmental events, without unnecessary fidelity and the accompanying processing and transmission requirements. In an example, the temperature sensor 136, humidity sensor, ambient pressure sensor, light color and/or light intensity sensors 132, magnetic field sensor 134, electronic device sensor 140, infrared radiation sensor 130, and motion sensor 142 are each sampled at, for example, about 8 Hz to about 12 Hz, and preferably at about 9 Hz, 10 Hz, or 11 H; the vibration sensor 138 is sampled at, for example, about 3 kHz to about 5 kHz, and preferably at about 3.8 kHz, 3.9 kHz, 4 kHz, 4.1 kHz, or 4.2 kHz (e.g., each axis of a three-axis accelerometer is sampled at, for example, 8 kHz, 3.9 kHz, 4 kHz, 4.1 kHz, or 4.2 kHz); the acoustic sensor 144 is sampled at, for example, about 15 kHz to about 19 kHz, about 16 kHz to about 18 kHz, or at about 17 kHz; and the EMI sensor 146 is sampled at, for example, about 250 kHz to about 750 kHz, about 490 kHz to about 510 kHz, about 495 kHz to about 505 kHz, or at about 500 kHz. It should be noted when accelerometers are sampled at high speed, they can detect minute oscillatory vibrations propagating through structural elements in an environment (e.g., dry-wall, studs, and joists).
Referring back to
The computer system 104 can be accessible via a client 106, such as a personal computer, laptop or mobile device, through a console user interface or a graphical user interface (GUI), such as a web browser or mobile app. When the client 106 connects to the computer system 104, the computer system 104 can permit the client to access the data from the sensor assemblies 102. In one aspect, the client 106 may only access the data registered to the user account through which the client 106 has accessed the computer system 104 or otherwise allow the user of the client 106 to access the data from the sensor assemblies 102 associated with the user. In one aspect, a user can visualize the featurized data transmitted from the sensor assembly 102 to the computer system 104 through the GUI. The GUI can provide spectrograms, line charts, and other graphical and/or numerical formats for viewing the received data. Further, sensor streams could be separated into time and frequency domain components. In one aspect, the GUI can be customized to visualize only a subset of the featurized sensor streams, as desired by the user. For example,
In some aspects, the computer system 104 can further implement a management module that allows firmware and/or software updates to be transmitted to the sensor assemblies 102. This management module can be controlled via an interface of the client 106, e.g. the GUI. Further, the management module of the interface could allow custom code to be deployed at each sensor assembly 102, as desired by the user. Still further, the management module could collect and store telemetry information, such as uptime of the sensors 110 and/or sensor assembly 102, data rates for the sensors 110, reboots of the sensor assembly 102, and so on. The management module could further allow users to adjust the sampling rates of the sensors 110 of the sensors assemblies 102 (e.g., on a sensor-by-sensor basis and/or on a categorical basis across all of the sensor assemblies 102). Still further the management module can instruct the sensor assembly as to which features should be extracted for a particular and at what rate.
In order to assist in the understanding of the presently described sensing system, illustrative implementations of the sensing system will now be described. The following examples are intended for representative purpose only and should not be interpreted as limiting in any way.
For example, a virtual sensor 116 could be trained to detect when a person is awake according to a combination of the motion data 212 and the ambient light color data 222, which detect the movement of the occupant and the occupant turning on a lamp, respectively. As another example a virtual sensor 116 could be trained to detect when the occupant is showering according to the humidity data 216. As another example, a virtual sensor 116 could be trained to detect when the occupant is streaming TV according to variations in the ambient light color data 222 and the electronic device data 218, which in this case is a W-Fi sensor configured to detect when electronic devices are being utilized according to the Received Signal Strength Indicator (RSSI) of the W-Fi. As yet another example, a virtual sensor 116 could be trained to detect when the occupant has come home according to a combination of the motion data 212 and/or the temperature data 214 (wherein the rising temperature could result from the occupant increasing the thermostat when he or she comes home). As still yet another example, a virtual sensor 116 could be trained to detect when the microwave was being utilized according to the EMI data 220, which can detect the EMI spike from the microwave being activated.
For example, a virtual sensor 116 could be trained to detect day and night cycles according to the ambient light color data 244 and ambient light illumination data 246. As another example, a virtual sensor 116 could be trained to detect when the occupant is present and active within the apartment according to some combination of the motion data 232, the ambient light color data 244 (because the ambient light color sensor 132 can detect when a lamp and the kitchen lights are on), and/or the temperature data 234 (because the motion data 232 correlates to a temperature increase). As another example, a virtual sensor 116 could be trained to detect when the microwave was being utilized according to the EMI data 220. As yet another example, a virtual sensor 116 could be trained to detect when the occupant is streaming TV according to variations in the ambient light color data 222 and the electronic device data 218. The humidity data 236 and the ambient pressure data 238 could be utilized to detect additional longer term environmental changes, such as the weather.
For example, a virtual sensor 116 could be trained to detect rain, as depicted in
It should be noted that sensor assemblies 102 with different combinations or arrangements of sensors 110 can be utilized for different applications or locations. For example, the sensor assemblies 102 described in connection with
For example, a virtual sensor 116 could be trained to detect when the automobile is approaching a highway according to the acceleration data 292, which indicates that the automobile has been gradually accelerating for an extended period of time. As another example, a virtual sensor 116 could be trained to detect when a window has been lowered according to some combination of the acoustic data 294 (which detects an increase in the amount of noise within the automobile), the temperature data 296 (which detects a temperature drop), the ambient humidity data 298 (which detects a humidity increase), and/or the ambient pressure data 300 (which detects a pressure drop). Further, a virtual sensor 116 could likewise be trained to detect when the window has been closed according to these same data streams. Additionally, a second order virtual sensor 124 could be trained from first order virtual sensors 120 to track the state of the automobile window. Instead of outputting a binary output as with the first order virtual sensors 120 (e.g., “Is the window closed? Yes or no?” or “Is the window open? Yes or no?”), the second order virtual sensor 124 could be trained from the outputs of the first order virtual sensors 120 to provide a nonbinary output directed to the window's state (e.g., “Is the window open, being opened, partially opened, closed, or being closed?”) based on this data. As another example, a virtual sensor 116 could be trained to detect heading of the vehicle according to the magnetic field data 302. Similarly to the above example, the magnetic field data 302 could train a number of first order virtual sensors 120 (e.g., “Is this vehicle heading north?” or “Is the vehicle heading west?”) and a second order virtual sensor 124 could be trained from the output of the first order virtual sensors 120 to provide a nonbinary output directed to the vehicle's state (e.g., “What direction is the vehicle heading in?”). As yet another example, a virtual sensor 116 could be trained to detect the degree of cloudiness according to the ambient light illumination data 306, which can indicate the number and length of the instances that the sun is obscured during the course of the vehicle's trip. The ambient light color data 304 could be utilized to detect additional events or parameters associated with the vehicle or the vehicle's environment, such as what time of day it is, as discussed above in the context of other examples, or whether the vehicle is proceeding through a tunnel.
For example, a variety of virtual sensors 116 could be trained to detect when the faucet is running, a urinal has been flushed, a kettle has been put on the stove, and/or various tools are being utilized according to a combination of vibration data 312 and acoustic data 314. It should be noted that although certain events can be detected utilizing the same combinations of featurized data streams, they are nonetheless detectably discernible because the different events have different patterns or characteristics within the sensor data streams. The different patterns or characteristics exhibited in the data streams for the sensors 110 activated by each event can be utilized by the machine learning of the sensing system 100 to characterize that event to generate a virtual sensor 116 that can reliably identify future occurrences of the event. In these particular examples, a faucet running, a urinal flushing, an electric saw running, and the other annotated events each generate a unique signature in the vibration data 312 and/or the acoustic data 314 that can be characterized by the machine learning of the sensing system 100 to identify those events.
As another example, a virtual sensor 116 could be trained to detect when the microwave door is opened or closed according to the acoustic data 314. Further, a virtual sensor 116 could be trained to detect when the microwave has completed a heating cycle according to the acoustic data 314 (by detecting the microwave's completion chime). Further, a virtual sensor 116 could be trained to detect when the microwave is running according to the EMI data 316. These virtual sensors 116 can represent first order virtual sensors 120 detecting binary properties of the microwave. The outputs of these first order virtual sensors 120 can be fed into a second order virtual sensor 124 trained to track the state of the microwave.
Referring back to
In one aspect, the sensor assemblies 102 can transmit or stream the sensor data to a local computer or a computer external to the computer system 104. In one aspect, the local computer can include a client 106 that is capable of executing the interface for visualizing the data from the sensor assemblies 102 and/or controlling the functions of the sensor assemblies 102, as described above. In another aspect, the local computer can be executing the machine learning module 116, as described above. The local computer to which the data is streamed can be, for example, behind a system configured to perform network address translation (NAT), common in residential settings with a single public IP address shared by many computers. In this way, the data streams from the sensor assemblies 102 do not necessarily need to be transmitted all the way to the computer system 104 and then back to the interface for visualization on the client 106 and/or the machine learning module 116 for processing of the featurized data. In one aspect, the computer system 104 can control whether the sensor assemblies 102 are streaming data to a local computer according to whether there is a substantial distance between the sensor assemblies 102 and the computer system 104, whether the communication roundtrip time exceeds a particular threshold, or whether the available bandwidth falls below a particular threshold. In another aspect, a user can control whether the data from the sensor assemblies 102 is streamed to a local computer via the interface described above. In one aspect, the sensor assemblies 102 could also be programmed to automatically locate the nearest server (e.g., of the computer system 104) to which to stream its data based on a variety of metrics, such as distance, roundtrip communication time, and/or bandwidth.
Users can access or log into the computer system 104 via a client 106 to view the data from the sensor assemblies 102 associated with their user account, modify features or settings of the sensor assemblies 102, and update their security preferences. In one aspect, users can selectively permit other users to view, access, and/or modify their associated sensor assemblies 102. This could permit, for example, all family members to view the data and events detected by the sensor assemblies 102 within the family's home. The permissions provided to the invited users can be controlled from a master account, for example.
In various aspects, in order to properly capture, model, and classify events in an environment, sensor streams from the sensors 110 within a single sensor assembly 102 and the sensors 110 across multiple sensors assemblies 102 within the environment are preferably temporally correlated and/or synchronized. For example, a “door closing” event typically causes a synchronous increase in air pressure and a structural vibration, which could be detected across a number of sensor assemblies 102 located throughout the building. This co-occurrence of signals that are detectable with different sensors 110 and across different sensor assemblies 102 is what enables the virtual sensors 116 to robustly detect and classify events, such as a door closing in the above example. Even a minor temporal decorrelation between sensors 110 and/or sensor assemblies 102 could reduce the segmentation confidence of the classifier (i.e., virtual sensor 116). In one aspect, the sensing system 100 can be configured to temporally synchronize or correlate the data streams from sensors 110 both within a single sensor assembly 102 and across multiple sensor assemblies 102 to allow events detected by different data streams to be temporally associated together.
In one aspect, the computer system 104 utilizes a Network Time Protocol (NTP) to synchronize its own clock periodically, which is then used to keep all of the clocks 127 of all of the sensors assemblies 102 connected to the computer system 104 in synchronization with the computer system 104. Each sensor assembly 102 can include, for example, a quartz clock to keep track of time between these time synchronizations to minimize any clock drift between the different sensor assemblies 102. The sensor assembly 102 timestamps all sensor data with the system epoch time from its synchronized clock 127 to, e.g., millisecond granularity. Further, synchronizing the sensor assemblies 102 and timestamping all sensor data addresses any reordering problems from the sensors 110 being sampled asynchronously and any processing or transmission delays before the data packet reaches the computer system 104.
In one aspect, all data for each sensor stream is continuously buffered in an onboard buffer (e.g., a buffer onboard the sensor assembly 102), with each sensor reading timestamped according to the clock 127 synchronized across the sensing system 100 network. If communication between the sensor assembly 102 and the computer system 104 is lost or congested, the sensor assembly 102 can continually attempt to re-establish a communication channel with the computer system 104. The system firmware 129A of the sensor assembly 102 can be configured to periodically check for W-Fi connectivity and an active connection to the computer system 104 (or a server or other computer thereof). If the system firmware 129A determines that the communication channel is lost or congested, the sensor assembly 102 can execute an exponential back-off algorithm to periodically attempt to reconnect to the communications channel (e.g., W-Fi network). If the sensor assembly 102 is unable to reconnect to the computer system 104 for a particular length of time (e.g., one hour), the system firmware 129A can be configured to reboot and then once again attempt to reconnect to the computer system 104. Upon the sensor assembly 102 reconnecting to the computer system 104, the sensor assembly 102 can then upload all of the buffered, timestamped sensor data to the computer system 104, which can then reorganize the data with the data from other sensor assemblies 102 as necessary.
In one aspect, the sensor assembly 102 comprises a software watchdog to monitor the status of all of the sensors 110. If any sensor 110 does not report new data within a configurable period (e.g., 60 seconds), the software watchdog executed by the system firmware 129A can automatically restart the sensor assembly 102. In another aspect, the sensor assembly 102 comprises a hardware watchdog that reboots the sensor assembly 102 if the application or the system firmware 129A fails to respond in a timely manner (e.g., within one minute). After reset, the sensor assembly 102 re-initializes all the sensors 110 and resumes operation.
In one aspect, the sensor assembly 102 can further include one or more output devices, such as a light emitting device assembly including one or more LEDs, microphones, vibration motors, displays, and other audio, visual, and/or haptic indicators (not shown here). The output devices can be utilized to provide alerts or feedback to the user when various events occur. For example, the sensor assembly 102 can cause an LED assembly to illuminate or flash, a vibration motor to vibrate, a speaker to emit an audible alert, or the like when, for example, a particular event has been detected. The computer system 104 can detect the occurrence of the event via an appropriate virtual sensor 118 and then transmit a command or signal to the sensor assembly 102 to cause the corresponding output device to provide feedback, as previously described. Such audible or visual feedback can be utilized to provide notifications to hearing or vision-impaired individuals that an event has occurred (e.g., a kettle is boiling) or otherwise alert users that an event that the user may wish to be aware of has occurred. These alerts can be configured via the interface, for example. Users can thus access the computer system 104 via a client 106 and then program or configure the desired alerts in accordance with virtual sensors 118 and/or the trigger action rules described above. In yet another aspect, the sensor assembly 102 can include a wireless communication circuit (e.g., a Bluetooth LE transmitter, WiFi, Zigbee, Z-Wave) for transmitting alerts (e.g., push notifications) to the user (e.g., the client 106) when a selected event has been detected.
In aspects where the sensor assembly 102 includes one or more output devices, the output devices can also be utilized to confirm or authenticate the identity and/or location of particular sensor assemblies 102. For example, a user visualizing the data streams from a number of sensor assemblies 102 within a sensing system 100 can (e.g., via a client device 106) cause the output device of a particular sensor assembly 102 to begin emitting an alert so that the user can confirm the identity and location of the sensor assemblies 102. For example, a user could select “sensor assembly #3” and cause it to emit an alert. Thereafter, the computer system 104 can transmit a command to the sensor assembly 102 corresponding in identity to “sensor assembly #3” to cause it to emit a sequence of flashes by the light source or a sequence of beeps from the speaker. The user can then enter the sequence into the client device 106 to authenticate the client device 106 to the sensor assembly 102. In various aspects, the output may also be provided to a user mobile device.
In aspects wherein the sensor assembly 102 includes a wireless communication circuit (e.g., Wi-Fi, Bluetooth, radio frequency identification, or the like), the sensing system 100 can authenticate a user based upon their mobile electronic devices (e.g., smart phone, wearables, and other such devices). For example, the sensing system 100 can utilize the wireless communication circuit to determine whether human activity detected by the sensor assemblies 102 is being performed by a user of interest, based upon whether the wireless communication circuit is able to detect the mobile electronic devices of the user or other authorized individuals present within the vicinity of the relevant sensor assemblies 102. The mobile electronic devices of the user and/or other authorized individuals can be, for example, pre-registered with the sensing system 100 or pre-paired with the sensor assemblies 102. Various applications and/or trigger rules could then be programmed to send alerts to an authorized user and/or take other actions if human activity detected by the sensing system 100 at certain locations, at certain times, or according to other such constraints, is not being performed by the users of interest. Such aspects could be utilized to, for example, detect when an unauthorized individual is in the user's home at night or while the user is at work. Such aspects could also be utilized to, for example, confirm the identity of an individual making changes to the configurations or settings of the sensing system 100 via a client 106 according to their proximity to a sensor assembly 102 of the sensing system 100.
In addition to utilizing machine learning to train virtual sensors 116 to automatically characterize and detect various events within the detection range of each sensor assembly 102, a client 106 can also be utilized to access the computer system 104 to define conditions (rules) and associated actions to take in response to those conditions via an interface. In other words, users can program explicit actions for the sensor assembly 102 and/or computer system 104 to take when certain conditions have been satisfied. For example, a user could define a condition where if the motion sensor 142 is triggered and the time is after midnight, then send a text message to a particular cellular number. The trigger action rules can be defined with multiple conditions (triggers) and multiple actions. Triggers may capture specific conditions on sensor values (e.g., temperature>20° C.) or demonstrated behaviors detected by a virtual sensor 116 (e.g., window is open). Both sensors and actions can either refer to specific devices (e.g., temperature on sensor assembly #25) or to locations (e.g., kitchen). For instances where a trigger and/or action is specified to a location, the computer system 104 can compute the average value of all of the sensor assemblies 102 from a given sensor channel (e.g., temperature) for that specific location (e.g., kitchen).
In one aspect, the virtual sensors 118 can output data in a format that is consistent with one or more known API architectures, such as Representational State Transfer (REST) or publish-subscribe APIs. Such outputs formats can utilize appropriate authentication, access control primitives, and other controls. Outputting data from the virtual sensors 118 in accordance with a known API can allow apps to seamlessly make use of wide variety of data (whether it be raw sensor data, featurized sensor data, or higher order inferences) generated by the sensing system 100.
While the process 600 is described for the sake of convenience and not with an intent of limiting the disclosure as comprising a series and/or a number of steps, it is to be understood that the process does not need to be performed as a series of steps and/or the steps do not need to be performed in the order shown and described with respect to
A computing or data processing system 1700 suitable for storing and/or executing program code may take many forms and in one embodiment may include at least one processor 1702, which may be or be part of a controller, coupled directly or indirectly to memory devices or elements through a system bus, as shown in
According to various approaches, the nonvolatile memory 1704 may comprise any type of nonvolatile memory such as, but not limited to, electrically erasable programmable read only memory (EEPROM), flash programmable read only memory (PROM), battery backup RAM, hard disk drives, etc. The nonvolatile memory 1704 is typically used to hold the executable firmware and any nonvolatile data containing programming instructions that can be executed to cause the processor 102 to perform certain functions.
In some embodiments, the I/O interface 1705 may include a communication interface that allows the processor 1702 to communicate with devices external to the controller. Examples of the communication interface may comprise, but are not limited to, serial interfaces such as RS-232, Universal Serial Bus (USB), Small Computer Systems Interface (SCSI), RS-422, or a wireless communication interface such as Bluetooth, near-field communication (NFC) or other wireless interfaces. The computing system 1700 may communicate with an external device via the communication interface 1705 in any communication protocol such as Automation/Drive Interface (ADI).
The sensing system 100 described herein can be utilized in a number of different contexts. For example, the sensing system 100 can be utilized to assist in monitoring patients and providing effective healthcare to patients. With a rapidly aging population, providing care at home for this population will become a necessity. A key aspect that caregivers need is to track the Activities of Daily Living (ADL) for people, and be able to detect anomalies when these activities deviate from the norm for each individual. The sensing system 100 including virtual sensors 118, as described herein, can be used to provide a comprehensive system to track ADLs, such as bathroom usage, movement within the house, daily chores like cooking and eating, adherence to medications, detecting falls, all without needing instrumentation and intrusive sensing. In the future, virtual sensors to track various contexts within a person's home can be also used for predictive interventions (e.g. predict an impending fall), and not just for reactive events, particularly by customizing them to each individual.
As another example, the sensing system 100 can be utilized in industrial settings. In industrial settings with mechanical and electrical machinery, preventative maintenance and predictive analytics can be of huge help to increase equipment lifetime, as well as reduce downtime due to failures. Most older equipment does not have any sensing capability and newer equipment may have limited purpose specific sensing. The sensing system 100 including virtual sensors 118, as described herein, can be utilized to learn different states of industrial equipment by training a variety of virtual sensors 118 and then building applications and notifications to help with usage tracking, tracking error conditions, scheduling maintenance operations automatically, and other such tasks.
As another example, the sensing system 100 can be utilized in environmental sustainability efforts. Over two-thirds of the electricity generated in the US and over 30% of the potable water is used by human occupants of buildings, both commercial and residential. tracking resource usage, i.e. water and energy, at a fine granularity and notifying human occupants as well as building managers of wastage can lead to significant reduction in the usage of these natural resources. The sensing system 100 including virtual sensors 118, as described herein, can be utilized to train a variety of virtual sensors 118 in home and buildings to track individual appliance usage and water consumption and correspondingly provide comprehensive user interfaces and notifications to promote behavioral changes.
As another example, the sensing system 100 can be utilized for managing facilities. Given that humans spent over one third of their lives inside a commercial building (i.e., their work place), methods to make them more performative to serve occupant need can lead to improved productivity, comfort, and happiness. While sensing systems do get deployed in modern buildings for heating, ventilation, air-conditioning (HVAC), and lighting management, they are all purpose-specific, costly to deploy and maintain, and not easy to repurpose. The sensing system 100 including virtual sensors 118, as described herein, can be utilized to provide a uniform sensing substrate for all things related to smart building management, including control of HVAC systems, space utilization, power consumption tracking, occupancy and people movement tracking, fault detection, etc.
As another example, the sensing system 100 can be utilized for home-based consumer applications. With the advent of the IoT and integration with voice assistants, such as Amazon Alexa and Google Home, the presently described sensing system 100 can be utilized in a number of different ways as part of a “smart home.” For example, the sensing system 100 can implement virtual sensors 118 to track the usage of consumables like toilet paper and soap to notify users when they are running low (or even directly order replenishments) or notify users about the status of appliances in the home (e.g., a dishwasher or laundry machine). The sensing system 100 can also implement virtual sensors 118 trained to detect meta events, such as any movements or sounds within the home, for security purposes. The sensing system 100 can also implement virtual sensors 118 trained to non-intrusively detect sleep duration and patterns, without the user(s) being required to wear any device(s).
As another example, the sensing system 100 can be utilized in a variety of implementations for smart cities. There is a major push across the US and the globe to make cities smarter by adding sensing to street lights and other public infrastructure, such as buses, trolleys, and roads. The sensing system 100 including virtual sensors 118, as described herein and suitably outfitted for outdoor environments, can sense a wide variety of facets of the city environment. In addition, a number of virtual sensors 118 can be trained to detect events of interest that have distinct signatures. For example, virtual sensors 118 can sense a traffic jam, an accident, a gun shot, traffic estimation, street illumination, environmental quality, etc. A key advantage of the sensing system 100 implementing virtual sensors 118 is to do all the processing and featurization at the sensor assembly 102 itself, thereby addressing many of the privacy concerns in a smart city environment and also reducing the data that needs to be transmitted at the scale of a city.
In one general aspect, therefore, the present invention is directed to a sensing system comprising a sensor assembly and a back-end server system. The sensor assembly comprises a control circuit and one or more sensors, where each of the sensors is configured to sense one or more physical phenomena in an environment of the sensor assembly that are indicative of events. The back end server system comprises at least one server that is in communication with the sensor assembly. The control circuit of the sensor assembly is configured to extract a plurality of features from raw sensor data collected by the one or more sensors to form featurized data and to transmit data packets to the back end server system, where the data packets comprise the featurized data. The at least one server of the back end server system is configured to implement one or more first order virtual sensors, where each of the one or more first order virtual sensors is trained through machine learning to detect, based on the featurized data, an event in the environment of the sensor assembly.
In another general aspect, the back end server system comprises at least one server that comprises a processor and a memory for storing instructions that, when executed by the processor, cause the server to: (i) receive the featurized data from the sensor assembly; (ii) determine an occurrence of one or more events via the featurized data; (iii) train, via machine learning, one or more first order virtual sensor implemented by the server to detect the one or more events based on the featurized data; and (iv) monitor, via the virtual sensor, for subsequent occurrences of the one or more events based on featurized data from the sensor assembly.
According to various implementations, the events detected by the one or more first order virtual sensors are not directly sensed by any of the one or more sensors of the sensor assembly. Also, the sensor assembly may be in wireless communication with the back end server system.
Additionally, the at least one server of the back end server system may be further configured to implement one or more second order virtual sensors, wherein the one or more second order virtual sensors are trained to detect, based on, at least in part, outputs of one of more of the first order virtual sensors, a second order condition in the environment of the sensor assembly. At least one of the one or more second order virtual sensors may produce a non-binary output and the first order virtual sensors may produce binary outputs, non-binary outputs, or a set of labels. For example, the first, second and/or higher order virtual sensors may comprise machine-learned classifiers that are trained to detect events, conditions, durations, etc. in the environment of the sensor assembly. The classifiers could be support vector machines or deep learning algorithms/networks, for example, that may be trained through supervised or unsupervised learning. Labeled data for supervised learning may be collected from annotations of events by a user that are captured via a user interface provided by the back end server system.
The sensors may comprise passive and/or active sensors. Examples of passive sensors are an infrared radiation sensor, an ambient light color sensor, an ambient light intensity sensor, a magnetic field sensor, a temperature sensor, an ambient pressure sensor, a humidity sensor, a vibration sensor, an external device communication sensor, a motion sensor, an acoustic sensor, an indoor air quality sensor, a chemical sensor, a vision sensor, and an electromagnetic interference sensor. Examples of active sensors are a sonar sensor, an ultrasonic sensor, a light emitting sensor, a radar based sensor, an acoustic sensor, an infrared camera, an active infrared sensor, an indoor positioning system, an x-ray based sensor, a seismic sensor, and an active sound measurement system. The sensor assembly may also comprise an output feedback device, such as a speaker, a light source, and a vibration source. Additionally, the sensor assembly may be positionally stationary. And it need not include a high-resolution camera.
In various implementations, the back-end server system is configured to transmit a notification to the sensor assembly when a particular event is detected by the one or more first order virtual sensors. In turn, the sensor assembly may transmit a notification to a user via the output feedback device in response to receiving the notification from the back end server system that the particular event was detected.
In various implementations, the sensor assembly comprises one or more circuit boards, where the control circuit and the one or more sensors are connected to the one or more circuit boards. The sensor assembly may further comprise a housing that houses the one or more circuit boards, the one or more sensors, and the control circuit. In particular, the sensor assembly may comprise a single circuit board and a housing. The control circuit and the or more sensors may be connected to the single circuit board and the housing may house the single circuit board, the one or more sensors, and the control circuit.
In yet other implementations, a first sensor of the one or more sensors may have an adjustable sampling rate. In such an embodiment, the at least one server of the back end server system may be further configured to transmit an adjustment for the adjustable sampling rate for the first sensor to the sensor assembly.
Additionally, the featurized data for a sensor of the sensor assembly may comprise a statistical measure of raw sensor data for the sensor over a time window. The statistical measure may include; the minimum value over the time window; the maximum value over the time window; the range over the time window; the mean over the time window; the median over the time window; the mode over the time window; the sum of the raw sensor values over the time window; the standard deviation over the time window; and/or the centroid of the raw sensor values over the time window.
Furthermore, the control circuit of the sensor assembly may be configured to transmit periodic data packets to the back end server system, where the data packets comprise concatenated featurized data for two or more sensors of the plurality of sensors. The data packets may also be encrypted by the sensor assembly prior to being transmitted.
In various implementations, the sensor assembly further comprises a wireless communication circuit for communicating wirelessly with a user device. The wireless communication circuit may comprises a wireless communication circuit selected from the group consisting of a Bluetooth circuit, a WiFi circuit, a Z-Wave circuit, a Zigbee circuit, a RFID circuit, a LoRA radio circuit and a LoRAWAN radio circuit. Additionally, the back-end server system may be configured to transmit a notification to the sensor assembly when a particular event is detected by the one or more first order virtual sensors. In turn, the sensor assembly is configured to transmit a notification to the user device via the wireless communication circuit in response to receiving the notification from the back end server system that the event was detected. Also, the back-end server system may be configured to transmit a notification to a remote computer-based system when a particular condition is detected by a first, second or higher order virtual sensor.
In another general aspect, the sensing system may include a plurality of such sensor assemblies that are distributed throughout an environment or location. In such an embodiment, the first, second, and/or higher order virtual sensors may use data from sensors on more than one of the sensor assemblies to detect their corresponding events, conditions, durations, etc. that they are trained to detect throughout the environment or location.
In another general aspect, the present invention is directed to a method that comprises the steps of (i) sensing, by a sensor assembly that comprises one or more sensors, one or more physical phenomena in an environment of the sensor assembly; (ii) extracting a plurality of features from raw sensor data collected by the one or more sensors to form featurized data; and (iii) detecting, by a machine-learning first order virtual sensor of a back end server system, based on the featurized data, an event in the environment of the sensor assembly.
In various implementations, the sensor assembly extracts the plurality of features from the raw sensor data and the method further comprises the step of transmitting, by the sensor assembly, the featurized data to the back end server system. The method may also comprise the step of, prior to detecting the event, training the first order virtual sensor to detect the event from featurized data. The method may also comprise the step of receiving, by the back end server system via a user interface, annotations of occurrences of the event to use as the labeled data for the supervised training. In various embodiments, the back end server system comprises a plurality of machine-learning first order virtual sensors, and the detecting step comprises detecting, by each of the plurality of machine-learning first order virtual sensors, based on the featurized data, a different event in the environment of the sensor assembly. Also, the back end server system may further comprise a machine-learning second order virtual sensor that is trained through machine learning to detect, based on output from at least one of the plurality of first order virtual sensors, a second order condition in the environment of the sensor assembly, in which case the method may further comprise the step of detecting, by the machine-learning second order virtual sensor, the second order condition in the environment of the sensor assembly based on the output from at least one of the plurality of first order virtual sensors. In various implementations, one of the sensors has an adjustable sampling rate, in which case the method may further comprises the step of transmitting, by the back end server system, an adjustment for the sampling rate to the first sensor.
In various implementations, the sensor assembly further comprises an output feedback device. In such an embodiment, the method may further comprise the step of outputting, by the output feedback device, a code for authentication of the sensor assembly to a user device. Also, the sensor assembly may further comprise a wireless communication circuit for communicating wirelessly with a user device. In such an embodiment, the method may further comprise the steps of (i) transmitting, by the back-end server system, a notification to the sensor assembly when the event is detected by the first order virtual sensor; and (ii) transmitting, by the sensor assembly, a notification to the user device via the wireless communication circuit in response to receiving the notification from the back end server system that the event was detected.
While several forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such detail. Numerous modifications, variations, changes, substitutions, combinations, and equivalents to those forms may be implemented and will occur to those skilled in the art without departing from the scope of the present disclosure. Moreover, the structure of each element associated with the described forms can be alternatively described as a means for providing the function performed by the element. Also, where materials are disclosed for certain components, other materials may be used. It is therefore to be understood that the foregoing description and the appended claims are intended to cover all such modifications, combinations, and variations as falling within the scope of the disclosed forms. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.
The foregoing detailed description has set forth various forms of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, and/or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as one or more program products in a variety of forms, and that an illustrative form of the subject matter described herein applies regardless of the particular type of signal bearing medium used to actually carry out the distribution. For example, the various virtual sensors 118, 120, 124 described herein may be implemented with software stored in primary and/or secondary memory of the computer system 104, that when executed by a processor(s) of the computer system, causes the processor(s) to perform virtual sensor classifications as described herein. Similarly, the activation group and machine learning modules 116 (see
Instructions used to program logic to perform various disclosed aspects can be stored within a memory in the system, such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, the instructions can be distributed via a network or by way of other computer readable media. Thus a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to, floppy diskettes, optical disks, compact disc, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or a tangible, machine-readable storage used in the transmission of information over the Internet via electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, the non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).
As used in any aspect herein, the term “control circuit” may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor comprising one or more individual instruction processing cores, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA)), state machine circuitry, firmware that stores instructions executed by programmable circuitry, and any combination thereof. The control circuit may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc. Accordingly, as used herein “control circuit” includes, but is not limited to, electrical circuitry having at least one discrete electrical circuit, electrical circuitry having at least one integrated circuit, electrical circuitry having at least one application specific integrated circuit, electrical circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program, which at least partially carries out processes and/or devices described herein, or a microprocessor configured by a computer program, which at least partially carries out processes and/or devices described herein), electrical circuitry forming a memory device (e.g., forms of random access memory), and/or electrical circuitry forming a communications device (e.g., a modem, communications switch, or optical-electrical equipment). Those having skill in the art will recognize that the subject matter described herein may be implemented in an analog or digital fashion or some combination thereof.
As used in any aspect herein, the term “logic” may refer to an app, software, firmware and/or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets and/or data recorded on non-transitory computer readable storage medium. Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices.
As used in any aspect herein, the terms “component,” “system,” “module,” and the like can refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution.
As used in any aspect herein, an “algorithm” refers to a self-consistent sequence of steps leading to a desired result, where a “step” refers to a manipulation of physical quantities and/or logic states which may, though need not necessarily, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common usage to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities and/or states.
Unless specifically stated otherwise as apparent from the foregoing disclosure, it is appreciated that, throughout the foregoing disclosure, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
One or more components may be referred to herein as “configured to,” “configurable to,” “operable/operative to,” “adapted/adaptable,” “able to,” “conformable/conformed to,” etc. Those skilled in the art will recognize that “configured to” can generally encompass active-state components and/or inactive-state components and/or standby-state components, unless context requires otherwise.
It is worthy to note that any reference to “one aspect,” “an aspect,” “an exemplification,” “one exemplification,” and the like means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. Thus, appearances of the phrases “in one aspect,” “in an aspect,” “in an exemplification,” and “in one exemplification” in various places throughout the specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more aspects.
Any patent, publication, or other disclosure material, in whole or in part, that is said to be incorporated by reference herein is incorporated herein only to the extent that the incorporated materials do not conflict with existing definitions, statements, or other disclosure material set forth in this disclosure. As such, and to the extent necessary, the disclosure as explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material, or portion thereof, that is said to be incorporated by reference herein, but which conflicts with existing definitions, statements, or other disclosure material set forth herein will only be incorporated to the extent that no conflict arises between that incorporated material and the existing disclosure material.
Claims
1. A sensing system comprising:
- a sensor assembly comprising a collection of sensors in communication with a control circuit, wherein each of the sensors in the collection of sensors is configured to sense one or more physical phenomena in an environment of the sensor assembly; and
- a back end server system, comprising at least one server, that is in communication with the sensor assembly, wherein:
- the control circuit of the sensor assembly is configured to: identify one or more selected sensors of the collection of sensors whose data corresponds to an event occurring in the environment of the sensor assembly, and transmit data from the one or more selected sensors to the back end server system, and
- the at least one server of the back end server system is configured to: generate a first order virtual sensor by training a machine learning model to detect the event based on the data from at least one of the one or more selected sensors, and detect the event using the trained first order virtual sensor and data from the one or more selected sensors.
2. The sensing system of claim 1, wherein the event detected by the first order virtual sensor is directly or indirectly sensed by any of the sensors in the collection of sensors of the sensor assembly.
3. The sensing system claim 1, wherein the at least one server of the back end server system is further configured to generate a second order virtual sensor to detect, based on, at least in part, outputs of the first order virtual sensor, a second order condition in the environment of the sensor assembly.
4. The sensing system of claim 3, wherein the back end server system is configured to transmit a notification to a remote computer-based system when a particular condition is detected by the first or the second order virtual sensor.
5. The sensing system of claim 1, wherein the collection of sensors comprise at least one passive sensor selected from the group consisting of an infrared radiation sensor, an ambient light color sensor, an ambient light intensity sensor, a magnetic field sensor, a temperature sensor, an ambient pressure sensor, a humidity sensor, a vibration sensor, an external device communication sensor, a motion sensor, an acoustic sensor, an indoor air quality sensor, a chemical sensor, a vision sensor, and an electromagnetic interference sensor.
6. The sensing system of claim 1, wherein the collection of sensors comprise at least one active sensor selected from the group consisting of a sonar sensor, an ultrasonic sensor, a light emitting sensor, a radar based sensor, an acoustic sensor, an infrared camera, an active infrared sensor, an indoor positioning system, an x-ray based sensor, a seismic sensor, and an active sound measurement system.
7. The sensing system of claim 1, wherein:
- the sensing system further comprises an output feedback device selected from the group consisting of a speaker, a light source, a vibration source and a wireless communication device;
- the back end server system is configured to transmit a notification to the sensor assembly when a particular event is detected by the first order virtual sensor; and
- the sensor assembly is configured to transmit a notification to a user via the output feedback device in response to receiving the notification from the back end server system that the particular event was detected.
8. The sensing system of claim 1, wherein the first order virtual sensor comprises a classifier that is trained to detect the event in the environment of the sensor assembly.
9. The sensing system of claim 8, wherein the classifier is trained using at least one of supervised learning or unsupervised learning.
10. The sensing system of claim 1, wherein:
- the sensor assembly comprises a housing;
- the collection of sensors are connected to one or more circuit boards; and
- the housing houses the one or more circuit boards, the collection of sensors, and the control circuit.
11. The sensing system of claim 1, wherein the control circuit of the sensor assembly is configured to identify the one or more selected sensors of the collection of sensors whose data corresponds to the event by:
- determining a baseline profile of the environment; and
- using the baseline profile to determine the one or more sensors from the collection of sensors that are correlated to the event.
12. The sensing system of claim 1, wherein the at least one server of the back end server system is configured to transmit to the sensor assembly an instruction for identifying the one or more sensors in the collection of sensors.
13. The sensing system of claim 1, further comprising, by the control circuit, extracting a plurality of features from raw sensor data collected from one or more of the selected sensors before transmission to the back end server system.
14. The sensing system of claim 1, further comprising, by the at least one server of the back end server system, transmitting the trained first order virtual sensor to a datastore for storage for use by another back end server system.
15. The sensing system of claim 1, wherein:
- the sensor assembly is one of a plurality of sensor assemblies distributed throughout a location, wherein each of the plurality of sensor assemblies is in communication with the back end server system; and
- each of the plurality of sensor assemblies comprises a collection of sensors connected to a control circuit, wherein each of the sensors in the collection of sensors is configured to sense one or more physical phenomena in a local environment of the sensor assembly, wherein the control circuit of that sensor assembly is configured to: identify one or more selected sensors of the collection of sensors whose data corresponds to the event occurring in the environment of the sensor assembly, and transmit data from the one or more selected sensors to the back end server system, and
- the first order virtual sensor is trained through machine learning to detect, based on the data transmitted from the plurality of sensor assemblies, the event in the location.
16. A method comprising:
- sensing, by a sensor assembly that comprises a collection of sensors in communication with a control circuit, each of which configured to detect one or more physical phenomena in an environment of the sensor assembly;
- identifying, by the sensor assembly, one or more selected sensors of the collection of sensors whose data corresponds to an event occurring in the environment of the sensor assembly;
- generating, by a back end server system communicably connected to the sensor assembly, a first order virtual sensor by training a machine learning model to detect the event based on data from the one or more selected sensors; and
- detecting, by the trained first order virtual sensor, based on the data from the one or more selected sensors, the event in the environment of the sensor assembly.
17. The method of claim 15, further comprising
- extracting, by the sensor assembly, a plurality of features from raw sensor data collected by the one or more selected sensors; and
- transmitting, by the sensor assembly, the featurized data to the back end server system that comprises at least one server.
18. The method of claim 15, wherein:
- generating the first order virtual sensor comprises training the machine learning model through supervised training with labeled data; and
- the method further comprises the step of receiving, by the back end server system, via an user interface, annotations of occurrences of the event to use as the labeled data for the supervised training.
18. The method of claim 16, wherein generating the first order virtual sensor comprises training the machine learning model through unsupervised learning using a deep learning algorithm.
19. The method of claim 16, wherein identifying the one or more selected sensors of the collection of sensors whose data corresponds to the event comprises:
- determining a baseline profile of the environment; and
- using the baseline profile to determine the one or more sensors from the collection of sensors that are correlated to the event.
20. The method of claim 19, wherein the at least one server of the back end server system is configured to transmit to the sensor assembly an instruction for identifying the one or more sensors in the collection of sensors.
Type: Application
Filed: Oct 3, 2019
Publication Date: Jan 30, 2020
Inventors: Yuvraj AGARWAL (Pittsburgh, PA), Christopher HARRISON (Pittsburgh, PA), Gierad LAPUT (Pittsburgh, PA), Sudershan BOOVARAGHAVAN (Pittsburgh, PA), Chen CHEN (Pittsburgh, PA), Abhijit HOTA (Pittsburgh, PA), Bo Robert XIAO (Pittsburgh, PA), Yang ZHANG (Pittsburgh, PA)
Application Number: 16/591,987
