Contextually Aware Mobile Device
A method for determining if a mobile device is in a vehicle includes monitoring an acceleration of the mobile device for a time interval during a period of time. The method also includes calculating one or more characteristics of the acceleration of the mobile device during the time interval and comparing the one or more characteristics of the acceleration to a set of threshold values. The method further includes determining the state of the mobile device based on the comparison of the one or more characteristics of the acceleration to the set of threshold values, wherein the state of the mobile device includes an in-vehicle state, a non-vehicle state.
This application is a continuation-in-part of application Ser. No. 13/469,254, filed May 11, 2012, entitled Contextually Aware Mobile Device, by Amit Goyal, which in turn claims the benefit of U.S. Provisional Application Ser. No. 61/485,164 filed May 12, 2011; Serial No. 61/546,361 filed Oct. 12, 2011; and Ser. No. 61/580,635 filed Dec. 27, 2011.
BACKGROUNDThe present invention relates to systems and methods for determining the context of a mobile device, and more particularly is related to determining the context of a mobile device in an energy efficient manner.
Various attempts at developing applications to determine the context of a mobile device have been previously made. Generally, the context of the mobile device can include whether the mobile device is stationary or in motion and if the mobile device is in motion if the mobile device is located inside a vehicle.
Determining if a mobile device is inside of a vehicle is of particular value for a wide array of mobile applications. Some applications that would benefit from accurate context awareness include the ability to change the state of phone to prevent unfavorable user behavior like texting while driving. Another application may be configured to determine vehicle congestion and communicate that information to a traffic server. Yet another sample application would change the state of the phone into “driver mode” that would make certain functions more easily accessible while others could be less easily accessible.
To date, most methods for determining that a mobile device is in a vehicle have been through the use of location technologies such as GPS, A-GPS, Wifi Position System (WPS), and cellular phone tower triangulation. Using location based technologies to determine mobile device context has numerous drawbacks. For example, determining the speed and location of the device in this manner has bandwidth, power, and privacy implications. In addition, certain vehicle situations such as a vehicle inside a city become substantially more difficult to identify since the vehicle does not travel very far within a fixed amount of time.
Furthermore, power is a major resource constraint in mobile devices. In situations where the mobile device is not able to be easily charged, the device must rely on a battery, which is a limited power source. Continuously determining a mobile device's location and speed using a power intensive technology, such as GPS, greatly reduces the battery life of the mobile device. In addition, calculating speed and transmitting location data wastes energy and bandwidth.
SUMMARYAn exemplary embodiment includes a method for determining if a mobile device is in a vehicle includes monitoring an acceleration of the mobile device for a time interval during a period of time. The method also includes calculating one or more characteristics of the acceleration of the mobile device during the time interval and comparing the one or more characteristics of the acceleration to a set of threshold values. The method further includes determining the state of the mobile device based on the comparison of the one or more characteristics of the acceleration to the set of threshold values, wherein the state of the mobile device includes an in-vehicle state, a non-vehicle state.
An additional exemplary embodiment includes a method for determining a location of user associated with a mobile device. The method includes monitoring an acceleration of the mobile device for a time interval during a period of time and calculating one or more characteristics of the acceleration of the mobile device during the time interval. The method also includes comparing the one or more characteristics of the acceleration to a set of threshold values and determining the state of the mobile device based on the comparison of the one or more characteristics of the acceleration to the set of threshold values, wherein the state of the mobile device includes an in-motion state, an in-vehicle state and a stationary state. Based on determining a change of the state of the mobile device, the method includes calculating and updating the location of the mobile device.
A further exemplary embodiment includes a method for determining if a mobile device is in a vehicle. The method includes monitoring an acceleration and a rotation of the mobile device for a time interval during a period of time and comparing the acceleration and the rotation of the mobile device to a set of threshold values. The method also includes determining a state of the mobile device based on the comparison of the acceleration and the rotation to the set of threshold values, wherein the state of the mobile device includes an in-vehicle state, a non-vehicle state, a stationary state, and a traffic state.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with the advantages and the features, refer to the description and to the drawings.
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The forgoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
Referring now to
In exemplary embodiments, the I/O devices 32 may include input devices, for example but not limited to, a touch screen, a keyboard, mouse, scanner, microphone, or other input device. Furthermore, the I/O devices 32 may also include output devices, for example but not limited to, a display, or other output devices. The I/O devices 32 may further include devices that communicate via both inputs and outputs, for instance but not limited to, a modulator/demodulator, a radio frequency (RF), wireless, or other transceiver, a telephonic interface, a bridge, a router, or other devices that function both as an input and an output. I/O devices are used to transmit data from the vehicle to the central server. In exemplary embodiments, when the mobile device 10 is in operation, the processor 12 is configured to execute the software stored within the memory 20, to communicate data to and from the memory 20, and to generally control operations of the mobile device 10 pursuant to the software. The software and the O/S, in whole or in part, but typically the latter, are read by the processor 12 and then executed. Functionality of the mobile device 10 can be implemented in software, firmware, hardware, or a combination thereof. In an embodiment, a portion of the mobile device is implemented in software, as an executable program, and is executed by a special or general-purpose digital computer.
The mobile device 10 includes an accelerometer 40 for sensing acceleration of the mobile device. The accelerometer 40 is capable of detecting acceleration and deceleration of a vehicle in which the mobile device is positioned. In exemplary embodiments, the accelerometer 40 may be configured to provide acceleration data in three dimensions relative to the orientation of the mobile device. In addition, the accelerometer 40 may be capable of being sampled at a configurable rate. In exemplary embodiments, the sensors 93 may include a gyroscope for sensing the orientation of the mobile device. The gyroscope is capable of detecting rotation movements of a vehicle in which the mobile device is positioned.
In exemplary embodiments, the sensors 93 of the mobile device 10 may include a Global Positioning System (GPS) sensor. The mobile device 10 may be configured to activate the GPS sensor and calculate the latitude, longitude, heading and speed of the mobile device based upon the received GPS data. In exemplary embodiments, the mobile device 10 may be configured to use assisted GPS and or differential GPS techniques to determine the latitude, longitude, heading and speed of the mobile device 10.
In exemplary embodiments, the communication port 42 allows communication between the mobile device and one of many different devices. As an example, the communication port is capable of allowing the mobile device to communicate with the OBD or OBDII connection port of a vehicle. As a result, the communication port is capable of communication via one or more of many different communication protocols. It should be noted that in accordance with the above-described embodiment of the invention, a different means of communication is used for communication with the OBD or OBDII than for communication with the central server. While this is the case, one having ordinary skill in the art will appreciate that in accordance with an alternative embodiment of the invention, a similar means of communication may be used for both communications with a vehicle and with the central server.
Referring now to
Referring now to
Next, as shown at block 104, the method 100 includes calculating one or more characteristics of the acceleration of the mobile device during the time interval. In exemplary embodiments, the one or more characteristics of the acceleration may include a mean acceleration, a standard deviation of acceleration, a minimum acceleration, a maximum acceleration or the like. In addition, in exemplary embodiments, the acceleration data monitored and characteristics of the acceleration calculated may include acceleration data in three dimensions and the characteristics may be calculated separately for each dimension or across the three dimensions. For example, if the acceleration data includes data in three dimensions, the mean acceleration may be calculated to be a single variable which includes the monitored acceleration data across all three dimensions. In another example, if the acceleration data includes data in three dimensions, the mean acceleration may be calculated to include three variables which represent the monitored acceleration data in each of the three dimensions. In exemplary embodiments, the one or more characteristics of the acceleration may include a rotation of the mobile device, which can be calculated from the change in the three dimensional acceleration data. In other exemplary embodiments, the rotation of the mobile device may be received from a sensor such as a gyroscope.
As illustrated at block 106, the method also includes comparing the one or more characteristics of the acceleration to a set of threshold values. In exemplary embodiments, the set of threshold values may include, but are not limited to, one or more mean accelerations, one or more standard deviations of the acceleration, and one or more maximum accelerations and one or more minimum accelerations, or the like. Next, as shown at block 108, the method 100 includes determining a state of the mobile device based on the comparison of the one or more characteristics of the acceleration to the set of threshold values. In exemplary embodiments, the state of the mobile device can include a stationary state and an in-motion state (also referred to as a non-vehicle state), which is indicative of the device being in motion but not in a vehicle (i.e., that the user of the mobile device is walking, running, or the like). In addition, the state of the mobile device can further include an in-vehicle state which is indicative of the device being in a vehicle. In exemplary embodiments, during the stationary state the acceleration data indicates that mobile device is not moving; during the in-motion state the acceleration data indicates that mobile device is likely moving but not in a vehicle; and during the in-vehicle state the acceleration data indicates that mobile device is likely inside a vehicle.
Referring now to
Referring now to
Referring now to
In addition to the examples shown in
In exemplary embodiments, once the state of the mobile device is determined to be in-vehicle, the mobile device can be configured to activate its GPS sensor periodically to determine its location, heading and speed. In addition, if desired, one or more features of the phone may be limited, such as the ability to compose a text or email. In exemplary embodiments, the mobile device may also use the battery level to determine if, and how frequently, the GPS sensor will be used. Since using the GPS sensor requires substantial energy, the GPS sensor may not be used during in-vehicle state at all if the battery level of the mobile device is below a minimum value, for example twenty percent of battery life remaining
In exemplary embodiments, once the state of the mobile device is determined to be in-vehicle, the mobile device may increase the duration of the time interval during a period of time. For example, once a mobile device is determined to be inside of a vehicle, the mobile device may increase the percentage of time that it is monitoring the acceleration data of the mobile device to be better able to determine if the vehicle is in traffic. For example, if during a stationary state the time interval is set to be approximately ten percent of the time period, once the mobile device is determined to be in a vehicle, the time interval may be set to fifty percent of the time period. In exemplary embodiments, the sampling rate of the accelerometer may also be increased once the mobile device is determined to be in an in-vehicle state.
In exemplary embodiments, once the mobile device is determined to be in a vehicle, the mobile device may compare the one or more characteristic of acceleration to a second set of threshold values and responsively determine if the vehicle is in traffic, as illustrated in block 110 of
In exemplary embodiments, once the mobile device is determined to be in an in-vehicle state it will remain in the in-vehicle state until a non-vehicle state is detected. Accordingly, once the mobile device is determined to be in an in-vehicle state, if the mobile device is determined to be stationary, it will remain in an in-vehicle state. This particular condition will be indicative of the mobile device being in a vehicle that is stationary. Accordingly, the in-vehicle state can be divided into a stationary vehicle state and a moving vehicle state. By monitoring the transitions from the moving vehicle state to the stationary vehicle state a determination that the mobile device is likely in a vehicle that is in traffic can be made and the device state can be updated to a traffic state. In exemplary embodiments, the mobile device may be configured to activate a GPS sensor to calculate the speed and location of the mobile device and to use the calculated speed data to verify that the mobile device is in fact in a vehicle that is in traffic, prior to changing the state of the mobile device to a traffic state.
In exemplary embodiments, once the state of the mobile device is determined to be in a vehicle in traffic the mobile device is responsively set to a traffic state. During the traffic state, the mobile device may activate the GPS sensor with a greater frequency than while in vehicle state and the mobile device may be configured to report its location and speed to a traffic management system. In exemplary embodiments, the mobile device may be configured to automatically launch a navigation application during traffic state to alert the user to the severity of the traffic and provide the user with an option of re-routing around the traffic.
In exemplary embodiments, the mobile device may be configured to take specific actions when the state of the mobile device changes. One common state change will occur after the current state of the mobile device is set to an in-vehicle state. In this case, the next state of the mobile device will either be a traffic state or in an in-motion state. The in-motion state will occur once the user exits the motor vehicle and begins walking or running Another common state change is the change from an in-motion state to a stationary state; that is, once the user of the mobile device stops walking and becomes stationary. In exemplary embodiments, the mobile device may be configured to activate the GPS sensor each time a change in the state of the mobile device occurs and to update a stored current location of the mobile device.
Tracking a user's movements based on the motion of a mobile device can be used to provide insight into the user's context. Understanding when a user has entered a vehicle, exited a vehicle, is walking, is running, is moving in some other way or has stopped can be used to provide insight into activities of the user. For example, based on how long a user is outside a vehicle and stopped indicates the timeframe the individual is in a fixed location. User tracking applications, like social networking applications and loyalty programs, that would like to know the timeframe a user is at a particular location can benefit through user updates. One example of a social networking application is Foursquare™. Because users may not manually check-in to locations, much less upon every occurrence of the user visiting the location, a need exists for an accurate, low-battery power consuming automated check-in solution.
Activating GPS or another location based technology like Wifi positioning based on accelerometer and/or gyroscope movements can provide the needed accurate, low-battery power consuming automated check-in solution. Based on the change in user context identified through accelerometer data, the location technology can be activated and the time and location can be used to update the user tracking application. One example of a user tracking application is Google Latitude™. In exemplary embodiments, various methods can be used to accurately determine the context of a user based on the detected motion of the mobile device. For example, one method may include monitoring the state of a mobile device and after the mobile device remains in a stationary state for a fixed amount of time before updating the user's location. The fixed amount of time can be used to ensure that the user has arrived at a destination and not temporally stopped and can also be used to prevent an excessively high number of check-ins. In exemplary embodiments, the user may be able to adjust the fixed amount of time that the mobile device must be in the stationary state before auto check-in occurs. In addition, the user may be presented with various options to always, never, or display user prompt for checking-in at specific locations. For example, a user may select to always auto check-in at a favorite restaurant or shop, to never auto check-in at home or at work, and to have the mobile device prompt him before checking-in at unknown or new locations.
In exemplary embodiments, acceleration data may be used in combination with rotation data to determine the state of the mobile device. The rotational data may be received directly from a gyroscope or it may be calculated from the acceleration data received from an accelerometer. By using both rotation and acceleration, it is possible to determine the state of the mobile device with very high accuracy. Using acceleration and rotation data, it is possible to simulate every motion of an object in three dimensional space. Based on a digital simulation, it is possible to associate movements with the state of the mobile device.
In exemplary embodiments, the time required to accurately determine the state of the mobile device may be less than ten seconds. Higher confidence levels can be associated to the state of the device as the time interval of the acceleration of the mobile device is increased. In exemplary embodiments, accurately determining that the mobile device is in an in-vehicle state requires more time than determining that the mobile device is in an in-motion state. In addition, because mobile devices have battery power constraints even a low power algorithm may need to be optimized. Periodically running an algorithm for a period of time (for example 20 seconds every 5 minutes) has tremendous battery power implications. The longer the timeframe this algorithm runs to determine the device state, the more battery power it uses. During testing, it was observed that running a state identification algorithm as disclosed herein periodically for ten seconds every five minutes versus twenty seconds every five minutes resulted in a difference of more than two percent overall battery power lifespan for a mobile device through the course of a day. Because users use mobile devices heavily, even slight impacts on battery power may have huge implications.
Assuming the device is inside a vehicle, the magnitude of the acceleration may be observed to infer the context of the vehicle. For example, if the acceleration of the device is approximately the acceleration of gravity, it can be inferred with a high degree of certainty that the vehicle is stationary. If the acceleration is substantially more than or less than the acceleration of gravity (again, without rotation) then this implies the vehicle is moving. In another embodiment, if the vehicle oscillates back and forth between stationary and moving for a period of time, this implies a traffic event. The greater the times associated with this back and forth idle and moving states, the greater the intensity of traffic. In another embodiment, if the vehicle is moving for a substantial period of time, it can be inferred that the vehicle is no longer in traffic.
In another embodiment, if the device is left inside the vehicle, then the excessive rotation associated with it leaving that environment will not take place. In such a situation, the device will reflect a stationary vehicle for an excessive period of time. Beyond some reasonable threshold for no movement, then in another embodiment of the invention it can be sufficiently inferred that the vehicle is in fact parked and not in an idle or traffic event.
More generally speaking, mobile devices such as mobile phones have become integral to the day-to-day lives of people around the globe. Yet the integration of phones to the activities, needs and dispositions of the user remains primitive. In fact a truly smart device should be able to check the weather report and remind the user as she leaves her house to take an umbrella, and, at her request, inform her friends which pub she is at. The reason that game-changing applications such as these have not taken off is simple because it has thus far been difficult to infer the context of the operator/owner. How does the phone know that you are leaving your home, or whether you were walking when your entered a building? These are the questions that we answer with the context layer. By leveraging the context layer, application providers will be able to write apps that integrate seamlessly with the habits and needs of users.
The context of the user isn't an easy attribute to discover because it depends on a complex combination of circumstances. The word “context” can itself become a difficult term to define: are we referring to a lower-level context such as “leaving the house” or a higher-level context such as “leaving the house in a hurry?” Fortunately, we can guess lower-level contexts with some certainty by combining information gathered from the phone with information gathered from the environment. Essentially, what we propose is the combination of mobile data with world state to draw conclusions about the context of the user.
In terms of device states, there are several sensors and actuators on the mobile device that together provide a great deal of information. Several examples include but are not limited to: use-states, sensor states, and accessory states.
In terms of use-states, the device has several use-states that can be detected by querying the operating system (i.e., Android, Symbian, etc.). The phone can be in idle state, ringing state and call state. Within the call state, it can be in ordinary state, or in call-waiting state and/or mute state. The display can be on or off. Different apps can be running The app that is in the fore ground is also an aspect of state. Together, these user states provide a great deal of information about the disposition of the phone and its interaction with the user.
In terms of sensor states, mobile devices also have a surprising number of sensors. The microphone is one of the most important, and its primary purpose is phone voice pickup. However, when not in use in a phone call, the microphone can also be used to determine external signals as Shazam™ or ShopKick does. However, there are a number of other sensors. The accelerometer can detect the rate of motion. The GPS gives latitude and longitude. In fact location is a combination on GPS and cell-tower triangulation. Before a mobile phone moves inside a building, the location may be accurately known because the number of satellites acquired can be high. However, the moment a device moves indoors, cell-tower or wi-fi tracking can be used. The time of transition between these modes can actually inform a system of when and where a user entered a building.
There are many more sensors. The gyroscope can give detect rotation, and rate of change of rotation. The compass can give orientation. The camera can be used for a number of applications including capturing images and detecting glare. The state of charge of the battery is also measured with an internal sensor. In the future, software sensors such as an activity monitor which detects how much the CPU is being used and how much CPU power is being drawn will also become common. The “bars” on a mobile phone also indicate bandwidth from any source, including cellular or WiFi.
In terms of accessory states, most mobile devices also interact with a number of peripherals. Charging usually done with a USB power cord, and whether the battery is being charged can be determined. The use of a Bluetooth headset can also be detected. In the future, other accessories such as NFC readers will become prevalent and may also be detected.
In terms of world State, the mobile device, obviously, exists in a geospatial and temporal context which we refer to as the world state. The GPS location on a device places it in a geospatial location which, when correlated with a GIS system, can inform whether the device is in a park, in a parking lot or in the wilderness. Enhanced with WiFi-based mapping technologies, the location of the device can also be pinpointed within a building or a mall. Time is also a dimension that is useful. Universal time is obviously known by the device, but in conjunction with location, the time zone is also known (often communicated to the device by the cell tower.) Together, this information can be tied to such details as weather reports and emergency reports.
In terms of combining state information, this state information, along with state history, can be combined to yield rich insights into the behavior and disposition of the user of the cell phone. Consider, for example, combining states to warn a user to pick up an umbrella: 1. The phone was charging for several hours 2. The time when this occurred was 10:00 pm-6:00 am 2a. The location where this occurred, if it happened several times, is probably home. The home location has now been detected. 3. The phone detects motion in the morning. It is a weekday. It is being used, and will probably be taken out. 4. The location of the phone can be correlated to a weather report. Rain is expected. 5. If the user has enabled weather based alerts, the phone informs the user that rain is expected and may suggest that the user to take her umbrella, particularly if the phone knows the user walks to work.
In this example, the several categories of state were combined to detect a user disposition and deliver information to the user. Information of this sort can also be combined across users to create a powerful crowdsourcing platform. Further confirmation can be derived by matching the location with a map—was the person in a train or in a car? By sourcing this information, it will be possible to recreate a clear impression of movement in a location.
Combining state information to create a coherent view of the user is not easy. There are many aspects to such algorithms. First, such algorithms need to be rapid so that the conclusions are available real-time. Second, the algorithms must be sensitive to the constraints of the mobile device. Power is almost always a premium, and compute cycles are an important concern. For example, use of the GPS sensor is very power consuming, and of several possible algorithms for detecting location, the one that minimizes the amount of time the GPS sensor is on will be optimal. Third, bandwidth is a concern not just as a limitation, but also because of the impact it has on power use and other resources. It is important to note that some resources cost different amounts for certain users, such as data consumption, which may be cheaper for those with larger data plans because of volume pricing. Algorithms must minimize the number of times they upload or download data. Finally, privacy is always a concern, and algorithms that safeguard consumer information are often optimal.
One embodiment of a context layer determines, stores and provides context information to applications. The context layer can provide and collect information at different time constants. Real-time systems must provide results immediately. They may use recent history and other data. Historical context is more retrospective, and will enable a user, for example, to understand her own behavior in retrospective: how much time did I spend at Starbucks? Predictive information may be based on learning and can be used to guess the next behavior of the user. For example, the user is taking 195—and 80% of the times that he has used that highway; he has taken Exit 25 to Main Street. Therefore, let's update him on a discount in the nearby area.
The computation of context in real-time may need to be on the phone but there is no reason why heavier calculations cannot be performed on the cloud. This type of interchange of information though requires the algorithms to respect the bandwidth, power and computation limits of the phone—in other words, the benefits should outweigh the costs. A carefully designed algorithm will minimize the exchange of information to a few small packets of data to maximize performance. In addition, the device may wait until an ideal time to do heavier data exchange. For example, the device may wait until the middle of the night when it is normally plugged in and has an active wi-ft connection.
One embodiment relates to a system and method for determining the context of a user using a computer. Traditional computing has three primary layers: a hardware layer, an operating system, and an application layer. Generally speaking these layers each serve a dedicated purpose: the hardware layer provides the infrastructure for computing, the operating system manages the computer's resources, and the application layer provides capabilities to the end user. In traditional computing, these layers generally lack an understanding of the user's context. The context layer may reside in different layers of computing to facilitate different types of computation.
Numerous applications have attempted to derive user context. However, this approach requires these applications to constantly “reinvent the wheel” and implement logic that others may have already implemented. In addition, very few of these applications constantly run on the device, limiting the applications ability to provide constant contextual monitoring. Finally, these applications are unaware of what other applications are doing, resulting in their operations to often deplete valuable resources like battery power.
As mentioned previously, Contextual Awareness is captured through the intersection of Device State and World State, which implies User State as shown in
In one embodiment the device would determine its micro-location context. Micro-location context refers to the immediate contextual state of the device. For example, if the device is on the user, the device could be positioned in a variety of places, including but not limited to inside a bag held by a user, inside the user's pocket, in the user's hand, next to the user's ear, etc. Another example of micro-location context is if the device is not on the user. In this scenario, the device may be in a variety of places, including but not limited to on a desk, on the floor, on a couch, and on a dashboard in a car. The purpose of highlighting these contextual states is not to limit the various scenarios describing micro-location but rather to demonstrate examples of the micro-location of a device that include other similar embodiments.
In one embodiment the device would determine its macro-location context. Macro-location context refers to the larger contextual state of the device. For example, the device may be lost or the device may be in a situation where the owner knows its location. In either scenario, the device may be in a At Destination State or a In Transit State. If in a At Destination State, the device may be in a variety of places, including but not limited to Home, Work, Gym, Restaurant, Theater, etc. If in a In Transit State, the device may be in a variety of places, including but not limited to with a user or without a user. If the device is with a user, that user could be walking, running, bicycling, inside a motorized vehicle, etc. The device could also be in a motorized vehicle without a person. The motorized vehicle could include a car, airplane, bus, train, elevator, or other location. The purpose of highlighting these contextual states is not to limit the various scenarios describing macro-location but rather to demonstrate examples of the macro-location of a device that include other similar embodiments.
In one embodiment the device would determine its power state. For example, the device may be charging or the device may not be charging. In either scenario, the device can determine its battery state, which would include but not be limited to low battery, full battery, and sufficient battery states. In addition, the device could determine the specific amount of battery level left assuming a specified usage level. The purpose of highlighting these contextual states is not to limit the various scenarios describing the Power State but rather to demonstrate examples of the Power State of a device that include other similar embodiments.
In one embodiment the device would determine its Usage state. For example, the device may be off or on. If the device is on, it could be either locked or unlocked. If the device is locked, the screen is either being viewed or not being viewed. If the device is unlocked, the screen is either being viewed or not being viewed. If the device is unlocked and the screen is being viewed, the phone may be in a state when it is being used to browse the web, to look at contacts, to dial a number, to game, to watch a video, to text, to talk, etc. Regardless of whether the device is locked or unlocked, the phone may be in an active state or inactive state. An active state implies the device is still performing some action, including but not limited to playing music, alarm sound, indicating that messages are available, etc. The purpose of highlighting these contextual states is not to limit the various scenarios describing the Usage State but rather to demonstrate examples of the Usage State of a device that include other similar embodiments.
As mentioned previously User State can be inferred through the intersection of the World State and Device State in numerous ways, including but not limited to Motion State, Location State, Consumption State, and Emotional State. In one embodiment the User Context Layer would determine the user's Motion State. For example, the user may be performing a variety of activities, including but not limited to walking, running, taking stairs, dancing, jumping, sitting, awake, sleeping, moving, stopped, taking transportation. If the user is taking transportation, the User Context Layer would distinguish between a variety modes of transport, including but not limited to car, bus, elevator, bicycle, train, and airplane. The purpose of highlighting these contextual states is not to limit the various scenarios describing physical motion but rather to demonstrate examples of physical motion of a user that include other similar embodiments.
In one embodiment the device would determine its user's location. Location context refers to a variety of contextual states, including but not limited to being at a Restaurant, Work, Home, Friend's Home, Gym, Mall, Store, Theater, etc. The purpose of highlighting these contextual states is not to limit the various scenarios describing location but rather to demonstrate examples of the user's location that include other similar embodiments.
In one embodiment the device would determine its user's consumption context. For example, the device may determine the user is eating breakfast, lunch, dinner, dessert, etc. The device may determine the user is shopping. The device may determine the user is entertaining oneself through being at a movie, play, concert, or sports show. The purpose of highlighting these contextual states is not to limit the various scenarios describing the Consumption State but rather to demonstrate examples of the User's Consumption State that include other similar embodiments.
In another embodiment, the device may prompt the user to determine the user's emotion state. For example, the user's emotional state could be hungry, mad, bored, happy, playing, working, free, busy, etc. The purpose of highlighting these contextual states is not to limit the various scenarios describing the Consumption State but rather to demonstrate examples of the User's Emotion State that include other similar embodiments.
There are many ways to represent External/World Context, including but not limited to time, weather, traffic, relative user location, and promotions. In one embodiment the device would determine context related to world time. For example, the device may determine the time is generally Morning, Afternoon, Evening, Night, Day, Weekday, and Weekend. The device may also maintain the length of time the current state has not changed.
In one embodiment, the device would determine relative user locations. For example, the original user's device may determine the relative distance between another target user's device and guide the original user to the target user. In another example, the device may update a central system and determine human congestion, such as individuals being at a night club.
In one embodiment the device would provide promotions based on user context. For example, context specific promotions could be offered in a wide variety of situations, including but not limited to the following situations: 1) User enters Grocery Store 2) User exits a vehicle at a mall 3) User is getting breakfast, lunch, or dinner 4) User is driving home after work. The purpose of highlighting these contextual states is not to limit the various scenarios describing promotions but rather to demonstrate examples of the user's location that include other similar embodiments.
The context layer has numerous dimensions across which it can provide context, including but not limited to Spatial, Temporal, Activity, and Two-Way Communication. In one embodiment, the contextual layer provides attributes across a variety of spatial attributes. These spatial attributes include but are not limited to geographic location as well as relative position. For example, a contextual state may be attributed by specific longitude and latitude coordinates or more general references like: city, rural, highway, surface road, house, indoors, outdoors, etc. In another example, a contextual state may be attributed by relative spatial references such as: inside bag, on table, in pocket, in hand, next to ear, etc. etc. The spatial dimensions may be inclusive, such as inside a bag. Or they can be exclusive, such as outside bag.
In one embodiment, the contextual layer provides attributes across a variety of temporal attributes. These temporal attributes include but are not limited to point in time as well as relative time. For example, a contextual state may be attributed by a specific time or more general references like: morning, afternoon, evening, day, night, etc. In another example, a contextual state may be attributed by relative temporal references such as: yesterday, next week, next year, etc.
In one embodiment, the contextual layer provides attributes across a variety of activity attributes. These activity attributes include but are not limited to: at the mall, at a restaurant, walking during lunch time, leaving work, going home, travelling, etc.
In one embodiment, the contextual layer would change state based on two-way communication with both the Internet as well as other devices. The context layer may be implemented across numerous dimensions including but not limited to real-time, historical, predictive, and local vs. central. In addition, the context layer may be optimized across a variety of dimensions, including but not limited to accuracy, battery power, bandwidth, and privacy.
In one embodiment, the contextual layer computes context based only on real-time information. For example, certain real-time data may indicate the user is riding inside a motorized vehicle or walking In one embodiment, the contextual layer computes context based on real-time, historical, and predictive information. The addition of historical context allows the context layer to compute in a more intelligent manner based on previous contextual analysis. For example, understanding that the user is inside a car allows the system to seek a major location update upon vehicle context, since it can be assumed that a vehicle likely moves the user substantial distances.
In another embodiment, the contextual layer computes context based on predictive information based on historical trends. For example, it can be inferred that based on historical trends that if a user has never driven between the hours of 4 am and 5 am, then the likelihood of such activity going forward is highly unlikely. This information allows the contextual layer to forego calculating potential vehicle context during these times with high confidence. In another example, predictive algorithms can allow the context layer to determine expected behavior and only report exceptions to such expected behavior. For example, if a user regularly enters a vehicle to go home at 8 pm from work, then on a particular day when the user stays at work late the context layer can signal an alert for exception based behavior. In this situation, an application may text the user's spouse to inform of the exception.
In one embodiment, the contextual layer computes based on local data only. In another embodiment, the contextual layer computes based on aggregate data provided from a central system. In one embodiment, the contextual layer computes with full access to all resources. However, this embodiment may consume a large amount of battery power and reduce the usefulness of the overall system. In another embodiment, the contextual layer optimizes across a variety of dimensions including but not limited to accuracy, battery power, bandwidth, and privacy.
In one embodiment, the Context Layer is informed by History, Current Situation, or Predictive Algorithms. In one embodiment, the mobile device automatically initiates an application based on the previous context. Examples of phone context modes include but are not limited to: car mode, quite mode, airplane mode, theater mode, school mode, work mode, home mode, sleep mode, battery saver mode, walking mode, running mode, bicycle mode, music mode, phone talk mode, phone with person mode, phone not with person mode. Examples of output modes include but are not limited to screen on, screen off, loud speaker active, loud speaker inactive, internal speaker active, internal speaker inactive, vibrate active, vibrate inactive, LED active, LED inactive, Flash active, and Flash inactive. Examples of input modes include but are not limited to camera active, camera inactive, microphone active, microphone inactive, touch screen active, touch screen inactive, swipe active, swipe inactive, tap active, tap inactive, motion active, motion inactive, shake active, and shake inactive.
In one embodiment, the mobile device is adaptive and intelligently samples to conserve battery power. It is possible to modulate numerous dimensions of the device, including but not limited to the sampling frequency, current sampling window, future sampling window, and sensors. In one embodiment, it is possible to use historical contextual patterns and time to inform current and future analyses. For example, in one embodiment the mobile device would learn that a user typically drives to work in the morning at 8am and therefore based on real-time data weather data anticipates a traffic delay tomorrow and informs the user to leave early for work the night before, to reschedule a meeting that would otherwise be likely missed, or to take the meeting from an alternative location.
In a preferred embodiment, an application would automatically determine the user is inside a vehicle. Upon exiting the vehicle, the application would automatically observe the user's location and mark it. The user, upon a desire to find the location of the vehicle, could then access the application and request the location of the vehicle.
In another embodiment, the application would be a smartphone application that would continuously run in the background of the phone determining the user's status of whether the user the is inside a vehicle. Upon determining the user is inside a vehicle, the application would wait until the user has exited to mark the location of the vehicle. This application's graphical user interface would be brought to the foreground whenever the user would access the application to observe the location in a graphical manner.
In another embodiment, the application would run on a wearable device, such as a watch. In another embodiment, the application would run in the onboard computer system of the vehicle and update a website that could be accessed by the user. In another embodiment, the application would run on a device wirelessly connected to the vehicle. In another embodiment, the application would be running on a portable device, including but not limited to a smartphone, a watch capable of running applications, a tablet computer, and a personal laptop. In another embodiment, the application would be running on a device physically connected to the vehicle, such as a smartphone or a personal navigation device.
Automatically determining vehicle entry and vehicle exit, precursors for determining when the application should mark the location of the vehicle, are components to the embodiment. Determining the appropriate method to observe car exit and obtain a location fix depends on the device on which the application is running In one embodiment, the application would run on the onboard computer system of a vehicle. In this application, the system would mark the location of the vehicle each time the vehicle is turned off. The application would then update an external system, such as a website server, so that this information could be later accessed by the user.
In one embodiment, the application would run on a smartphone. Using a smartphone, the application could determine vehicle context using a variety of methods including but not limited to using only acceleration data, acceleration data and rotation data, and acceleration data combined with rotation data combined with location data. Using only acceleration data, it is possible to observe the data directly and perform a Fast Fourier Transform on the data to identify frequencies that represent vehicle mode of transport. Using both acceleration data and rotation data, it is possible to observe acceleration magnitude above a threshold and rotation below a threshold to infer vehicle context. Using only location data, it is possible to infer vehicle context by the speed of the device. If the device is moving above a threshold, vehicle context can be inferred. Periods of continued vehicle context interposed with short periods where the device is moving below a threshold can be determined to be a continuous vehicle journey. Extended periods below a threshold can imply vehicle exit. Combinations of these methods can also be used to increase accuracy of the vehicle entry and vehicle exit. Vehicle exit could also be inferred by observing a walking behavior or a running behavior.
In another embodiment, the device is physically connected to the vehicle. Examples of this embodiment include but are not limited to a smartphone being charged by the vehicle and a personal navigation device being charged by the vehicle. In this embodiment, one method for determining vehicle entry and vehicle exit is the power state of the vehicle. If the vehicle is running (i.e. in a turned on state), it can be implied that the user is driving the vehicle and has entered the vehicle. If the vehicle is not running (i.e. powered off), it can be inferred that the user is no longer driving the vehicle and has exited.
In another embodiment, the device is wirelessly connected to the vehicle. In this example, one method for determining vehicle entry and vehicle exit is the state of the wireless connection. A connected wireless state would imply the user has entered the vehicle. A disconnected wireless state would imply the user has exited the vehicle.
In another embodiment, combinations of the previous embodiments would be used to more accurately determine vehicle entry and vehicle exit. Determining Vehicle Uniqueness and Ownership: In another embodiment, the system would distinguish between different vehicles. Vehicles have different acceleration profiles based on the engine. In particular, it is possible to take the Fast Fourier Transform of a vehicle's acceleration to observe the frequency of the engine. By performing this method upon entering a vehicle, it is possible to uniquely identify one vehicle from another. In addition, a user typically enters an owned vehicle more often than a vehicle owned by another person. As such, after identifying multiple vehicles, it is possible to identify the particular vehicle owned by the user. It should be noted that the sampling rate for identifying a vehicle will likely need to be greater than determining vehicle exit. As such, another embodiment temporarily increases the sampling rate of the accelerometer but then decreases it to optimize for power consumption.
In another embodiment, the local system would attempt to mark the location of the vehicle but fail to obtain the position via a GPS fix. Such a failure is indicative of when there is an overhead obstruction. In this embodiment, the local system prompts the user to ask for user input to enter information manually marking the location of the vehicle. This feature could be enabled/disabled by the user.
Referring now to
For these LBS providers, manual Check-ins and Checkouts have started resulting in a phenomenon called “Check-in Fatigue”. The concept of Check-in Fatigue implies that users are tiring of having to repeatedly remove their phones from their pockets to perform the act of Check-in/Checkout, especially at their favorite locations where they perform this function repeatedly.
To overcome the issue of Check-in fatigue, technologists have attempted to develop automated Check-in/Checkout solutions. However, the solutions that have been developed thus far suffer from numerous weaknesses.
One Check-in solution developed by developer Tim Sears and presented in his mobile application Future Check-in maintains a list of favorite locations and automatically performs the act of Check-in on behalf of the user whenever the user arrives within 300 meters of the location. The technology presumably used to facilitate this capability is cell tower location, which has an accuracy of approximately this amount.
The weaknesses of Sears' “Future Check-in” solution are substantial. For starters, a user may simply be passing by the location in a vehicle, and the system may inaccurately perform the Check-in. Immediate Checkouts would also be identified, but it would be difficult to determine if the user actually went inside the venue or not. As a result of these weaknesses, it is also possible for users to intentionally mislead the system to obtain rewards associated with it. Inaccurate Check-ins and Checkouts, such as into other nearby venues within a dense urban environment, substantially reduce the value of the overall system by reducing the confidence of merchants in participating in a rewards based system for users that frequent their venue.
Another solution implemented by the company Shopkick deploys external hardware in merchant venues. This hardware emits a special signal which is identified by mobile devices running their mobile application, whenever these devices come within a certain range. While the benefit of this solution is that their technology can be tuned to limit the range of the automated Check-in/Checkout, their solution is expensive and only works with merchants that have deployed their hardware.
What is needed is a system that accurately performs automated Check-ins and Checkouts. In addition, the system must function effectively while minimizing power consumption and bandwidth of any mobile solution it is a part of since both are limited resources. The solution must also maintain privacy of the user and allow the user to manage the extent to which automated Check-ins and Checkouts occur. For example, a user may want to be automatically checked in and out at some locations and not at others, shown in
The general algorithm for enabling Automated Check-in and Automated Checkout is to determine when the user is moving to a new location (from here on referred to as the In Transit State or Moving State) and when the user is remaining in the same general location (from here on referred to as the At Destination State or Fixed State). When the user changes state from Moving to Fixed States, the location and time of the change is marked as a Check-in. When the user changes state from a Fixed to Moving State, the location and time is marked as a Checkout. Generally speaking, the act of a Check-In or Check-Out is also referred to throughout this document as a “registration” or “registering.”
It is important to note that although the system is capable of automatically registering locally and in a third party system, such as one in the cloud, application context may require the user to approve these Check-ins and Checkouts before registering this data with a third party system. Furthermore, the concept of Automated Checkouts is new per this writing. Given the accuracy of the solution proposed, it is possible to distinguish Check-ins from Checkouts which previously could not be achieved.
The following are sample embodiments of determining In-Transit state. In one embodiment, the In Transit State is determined through analysis of the mobile device's acceleration. As shown in
In one embodiment, the In Transit State is determined through analysis of the mobile device's rotation. If the phone's rotation exceeds a threshold over a period of time, it can be inferred that the phone and associated user are in a In Transit State. For example, the amount of rotation experienced by the phone when the user is walking or running is substantially more than when the user is sitting. In one embodiment, rotation is sensed by the device's Gyroscope. In another embodiment, rotation is calculated from the Acceleration of an accelerometer that captures Gravity.
In another embodiment, the In Transit State is determined through a change in temperature beyond a threshold. For example, when the user goes from an indoor environment to an outdoor environment, the change in associated temperature could be used to identify a In Transit State.
In another embodiment, the In Transit State is determined through a change in ambient sound. For example, when a user enters a vehicle the ambient noise is typically very different than the ambient sound prior to entering the vehicle and the change could be used to identify a In Transit State.
In another embodiment, the In Transit State is determined through a change in ambient light. For example, when a user moves from a dark indoor environment to a sunny outdoor environment, the change in ambient light could be used to identify a In Transit State.
In another embodiment, the In Transit State is determined through identification of an RFID tag. For example, when a user moves from an outdoor environment absent of RFID tags into an indoor environment with products marked by RFID tags, the sensing of tags could be used to identify a In Transit State. In a similar manner, if tags are being sensed and the environment changes such that those tags are no longer sensed, such as going from an Indoor environment where there are tagged products to an outdoor environment where there are no longer any tagged products, this would also imply a In Transit State.
In another embodiment, the In Transit State is determined through a change in bearing beyond a threshold determined from the Compass/Magnetic Field Sensor. For example, a user walking often changes direction and the change in compass bearing could imply a In Transit State.
In another embodiment, the In Transit State is determined through a change in visual recognition. For example, the camera could be activated to determine the setting using visual recognition techniques and a substantial change in environment, such as an indoor environment to an outdoor environment, could imply a In Transit State.
In another embodiment, the In Transit State is determined through connectivity changes. Mobile devices have numerous methods of connecting to the external environment, including but not limited to Cell Towers, Wifi Networks, Bluetooth devices, NFC devices, and Charging State (electric connectivity). Substantial changes in any of these could imply a In Transit State. For example, in one embodiment, different Cell Tower Ids connected to by the device imply the phone and user are now in a In Transit State. In another embodiment, Cell Tower signal strength changes beyond a threshold imply the phone and user are in a In Transit State.
In another embodiment, different Wifi network ids imply the phone and user are now in a In Transit State. In another embodiment, Wifi Network signal strength changes beyond a threshold imply the phone and user are in an In Transit State.
In another embodiment, different Bluetooth devices sensed by the Bluetooth receiver imply the phone and user are now in a In Transit State. In another embodiment, Bluetooth device signal strength changes beyond a threshold imply the phone and user are in a In Transit State.
In another embodiment, different NFC devices are sensed and imply the phone and user are now in a In Transit State. In another embodiment, NFC device signal strength changes beyond a threshold imply the phone and user are in an In Transit State. In another embodiment, a change in Charging State could imply a Moving State Change. For example, users must unplug their devices before leaving home. The disconnecting of the electrical connectivity facilitating a Charging State could imply an In Transit State.
Numerous location technologies could be used to identify an In Transit State, including but not limited to GPS, a Wifi based positioning system, and Cell Tower Location data.
In another embodiment, the In Transit State is determined through a change in location determined by GPS. For example, an activated GPS sensor could indicate a change in location beyond a threshold, implying the phone and user are in an In Transit State.
In another embodiment, the In Transit State is determined through a change in location determined by a Wifi Positioning System. For example, an activated WiFi Location request could indicate a change in location beyond a threshold, implying the phone and user are in a In Transit State.
In another embodiment, the In Transit State is determined through a change in location determined by a Cell Tower Positioning System. For example, a Cell Tower Location request could indicate a change in location beyond a threshold, implying the phone and user are in a In Transit State.
In another example, a context aware database that implies changes in location could imply an In Transit State. For example, a change from Restaurant to Beach could imply an In Transit State.
Combinations of these embodiments are additional embodiments. Example 1. To determine that the user is in the In Transit State, it is possible to combine detection algorithms. Doing so may have multiple benefits, from reduced power consumption to faster and more accurate state detection. For example, if the phone is moving above a threshold and WiFi ids associated with the device have changed substantially, then it may be determined that the User is in the In Transit State. In subsequent state determinations, further refinements may also be achieved. For example, if the user is in the In Transit State, then simply checking the accelerometer briefly to see if the phone is still moving beyond a certain time threshold, such as every 2 minutes, may be sufficient to confidently determine the User is still in the At Destination State. In addition, other algorithms may further be used on a less frequent basis, such as every hour, to periodically check the Wifi Ids to confirm the state previously determined by the Acceleration.
The following are sample embodiments of determining At-Destination State. In one embodiment as depicted in
In one embodiment, the At Destination State is determined through analysis of the mobile device's rotation. If the phone's rotation fails to exceed a threshold over a period of time, it can be inferred that the phone and associated user are in an At Destination State. For example, the amount of rotation experienced by the phone when the user is sitting is substantially less than when the user is walking or running In one embodiment, rotation is sensed by the device's Gyroscope. In another embodiment, rotation is calculated from the Acceleration of an accelerometer that captures Gravity.
In another embodiment, the At Destination State is determined through a change in temperature below a threshold. For example, when the user remains in an indoor environment, the change in associated temperature is typically relatively small over the course of time.
In another embodiment, the At Destination State is determined through a relatively constant ambient sound within certain thresholds. For example, when a user is inside a vehicle there tends to be a minimum level of ambient noise. In another example, when the user is outside in a Forest, there tends to be a relatively constant low ambient sound.
In another embodiment, the At Destination State is determined through a relatively constant ambient light that remains within certain thresholds. For example, when a user is in a sunny outdoor environment, the relatively constant ambient light could be used to identify a At Destination State.
In another embodiment, the At Destination State is determined through identification or lack of identification of RFID tags. For example, when a user is in an outdoor environment there tends to be an absence of RFID tags. In a similar manner, if tags are being sensed this may imply an indoor environment and could imply an At Destination State.
In another embodiment, the At Destination State is determined through a relatively constant bearing below a threshold determined from the Compass/Magnetic Field Sensor. For example, a user sitting at a desk often does not change direction and the lack of change in compass bearing could imply a At Destination State.
In another embodiment, the At Destination State is determined through a lack of change in visual recognition. For example, the camera could be activated to determine the setting using visual recognition techniques and a lack of substantial change in environment, such as a continued presence in an indoor environment, could imply a At Destination State.
In another embodiment, the At Destination State is determined through lack of connectivity changes. Mobile devices have numerous methods of connecting to the external environment, including but not limited to Cell Towers, Wifi Networks, Bluetooth devices, NFC devices, and Charging State (electric connectivity). Lack of substantial changes in any of these could imply an At Destination State.
For example, in one embodiment, the same Cell Tower Ids connected to by the device imply the phone and user are in an At Destination State. In another embodiment, if the Cell Tower signal strength does not change beyond a threshold implies the phone and user are in an At Destination State. In another embodiment, same Wifi network ids imply the phone and user are now in a At Destination State. In another embodiment, Wifi Network signal strength changes below a threshold imply the phone and user are in an At Destination State.
In another embodiment, the same Bluetooth devices sensed by the Bluetooth receiver imply the phone and user are in a At Destination State. In another embodiment, Bluetooth device signal strength changes below a threshold imply the phone and user are in a At Destination State.
In another embodiment, the same NFC devices are sensed and imply the phone and user are now in a At Destination State. In another embodiment, NFC device signal strength changes below a threshold imply the phone and user are in an At Destination State.
In another embodiment, Charging State could imply an At Destination State. For example, user often keep their devices plugged in at home. The constant electrical connectivity facilitating a Charging State could imply a At Destination State. Location Technologies: Numerous location technologies could be used to identify an At Destination State, including but not limited to GPS, a Wifi based positioning system, and Cell Tower Location data.
In another embodiment, the At Destination State is determined through a lack of change in location determined by GPS. For example, an activated GPS sensor could indicate a lack of change in location beyond a threshold, implying the phone and user are in an At Destination State.
In another embodiment, the At Destination State is determined through a lack of change in location determined by a Wifi Positioning System. For example, an activated WiFi Location request could indicate a lack of change in location beyond a threshold, implying the phone and user are in an At Destination State.
In another embodiment, the At Destination State is determined through a lack of change in location determined by a Cell Tower Positioning System. For example, a Cell Tower Location request could indicate a lack of change in location beyond a threshold, implying the phone and user are in a At Destination State.
In another example, a context aware database that implies lack of changes in location could imply an At Destination State. For example, a request to this database that continues to respond with Restaurant could imply an At Destination State. This context aware database could be based on many technologies, including but not limited to Wifi Ids, Cell Tower Ids, GPS coordinates, Coordinates determined from WiFi, etc.
Combinations of these embodiments are additional embodiments. Example 1. To determine that the user is in the At Destination State, it is possible to combine detection algorithms. Doing so may have multiple benefits, from reduced power consumption to faster and more accurate state detection. For example, if the phone is moving below a threshold and WiFi ids associated with the device have not changed substantially, then it may be determined that the User is in the At Destination State. See
Using Time, Historical Patterns, and Predictive Behavior. In another embodiment, historical data can dramatically improve accurate real-time contextual awareness and facilitate predictive algorithms that increase accuracy of contextual awareness. For purposes of enabling a more intelligent methodology for enabling Automatic Check-in and Automatic Checkout, historical patterns can be identified that both influence accuracy of situational analysis and rate of determination.
For example, in one embodiment, if a user regularly performs a Check-In at a venue prior to going to work, then less accurate and less power intensive methods are necessary to verify and perform the Check-In. For example, in one embodiment, rather than activating GPS at the location the system could merely look for tangential characteristics of that particular venue that identify it, such as a specific ambient noise, temperature, visual pattern, cell tower ids, wifi network ids or mac addresses, etc.
In another embodiment, predictive algorithms can be used to reduce battery consumption. For example, if it is known the user rarely performs a Check-in to a venue during the hours of 9 am to 5 pm because he is at work, then sampling algorithms to confirm the user's location can be dramatically reduced.
In another embodiment, algorithms that identify behavior opposite of the expected behavior can alert the system to abnormal behavior and initiate a different set of algorithms. For example, if the user typically awakes in the morning and performs a Automatic Check-In at a nearby restaurant, but on a particular day fails to awaken, then the traditional Automatic Check-In algorithms based on historical patterns could recognize the exception and initiate a different set of algorithms, for example more aggressive algorithms that would be more sensitive to user movements to determine granular state.
In another embodiment, users can mark a list of their favorite locations where they want to perform Automatic Check-in. In another embodiment, from their previous Check-in behaviors, a favorites list can be automatically inferred and created. This favorites list can be used to reduce the frequency of the more battery intensive Automatic Check-in algorithms until only when the user is close to an approved location, determined for example by cell tower signal, for example. In computing parlance, a favorites list may be considered a white list and similarly, users may also have black lists for excluding Check-ins in all the same ways that a white list is used for including Check-ins.
In another embodiment, users can mark the time periods in which they are comfortable allowing the system to perform Automatic Check-in. In another embodiment, the system can suggest times during which Automatic Check-ins will be performed to reduce battery consumption. In another embodiment, the system can be initiated into battery conservation mode where the times which most Check-ins occur are automatically identified and other times are intentionally avoided.
In another embodiment, users can dismiss Check-in events at certain locations for a single instance or permanently.
In another embodiment, users can choose whether to publicly disclose Check-ins. In another embodiment, a user may choose to Check-in to a location and not publicly disclose the event. For example, a user may choose to Check-in to a location for purposes of obtaining loyalty points but desire to not share this publicly to her social circle.
Although it is possible to have all sensors active and operating at the highest sampling rate at the same time, doing so is extremely inefficient use of a scarce power resource. A system and method to reduce power consumption is to intelligently use sensors based on context to optimize for power usage.
In one embodiment, the algorithm for determining the device is in either a At Destination State or a In Transit State runs for a short period of time and then shuts off for another period of time. For example, the algorithm for At Destination State determination may run for 10 seconds. If the algorithm determines the device is in a At Destination State, it shuts off all sensors, enters a “sleep” state, and remains asleep for 5 minutes. After 5 minutes, the system “wakes up,” activates the necessary sensors again for 10 seconds, and then determines if the device is for example now inside a vehicle. This process repeats until the algorithm determines the device is in a In Transit State. Upon determining In Transit State, it continues to remain active until another At Destination State is observed.
In another embodiment, the system reduces the sampling rate of a particular sensor in one context and then increases the sampling rate in another context, when greater frequency of samples is required. For example, the local system may sample the accelerometer at the lowest frequency when the system is attempting to determine that the device is in a At Destination State, but once it determines that it is in a In Transit State, it increases the sampling rate of the accelerometer to more accurately identify more granular states, including but not limited to walking, running, driving, etc. The local system may optionally increase the sampling rate even more to calculate even more granular data, such as speed from the local system.
In another embodiment, the system activates one type of sensor that consumes less power and then after determining an appropriate context activates another sensor that consumes greater power for additional visibility into the context. For example, when the device is in a In Transit State, the system may only run the accelerometer until situations that indicate a At Destination State arise. This may be, for example, when a user is sitting (not inside a vehicle) for an extended period of time. In such a situation, the system would then activate the more power hungry GPS sensor to obtain location and upload this information to the central system as an Automated Check-In.
In another embodiment, the system determines a charging state and then activates all sensors since the power constraint has become significantly relaxed. For example, users often power their phones inside vehicles. In such a case, the power constraint is significantly relaxed since energy is now temporarily abundant.
In another embodiment, the system reduces context computation based on geography. For example, if the system determines local context within 100 meters within the user's home, it could stop determining local state. It should be noted that the instructions to reduce computation in this regard could be created automatically or otherwise by the local system or an external system.
In another embodiment, the system reduces context computation based on time. For example, the system could eliminate local context computation in the morning to reduce power consumption. It should be noted that the instructions to reduce computation in this regard could be created automatically or otherwise by the local system or an external system.
In another embodiment, the system activates one type of sensor for a short period of time and then after determining an appropriate context determines if it should continue running or stop itself automatically. For example, when the device is in a At Destination State, the system may only run the accelerometer for a single second and determine no movement has occurred. In this situation, the system may choose to stop itself because it may determine that in this context additional information does not add much value. However, in an In Transit State, the system may run for a single second and determine because there is movement that it wants to try to determine the mode of transport, possibly by running much longer. These are merely examples of the embodiment and not meant to be limiting in any way.
In one embodiment, the central system and local system may initiate communication with each other. In another embodiment, the local system may only initiate communication with the central system. In another embodiment, the central system may only initiate communication with the local system. In one embodiment, the local system obtains updates from the central system. In another embodiment, the central system obtains updates from the local system. See
In one embodiment, an update comprises data including but not limited to check-in and check-out frequency profiles, user data access profiles, geography profiles, and update frequency. For example, the frequency profiles specify the maximum and minimum window thresholds for which a user is allowed to check-in and check-out. The frequency profiles may also have time windows in which check-ins and check-outs may occur. In another example, the user data access profiles specify which users have access to the check-ins and check-outs. In another example, the geography profiles specify which geographic areas check-ins and check-outs occur. In another example, the geography profiles specify the types of contextual locations in which check-ins and check-outs occur. In another example, the update frequency specifies how often updates will occur and which system initiates the update.
In another embodiment, updates are compressed to optimize bandwidth use.
In another embodiment, the central system may dynamically command the local system to report back check-ins with high frequency during the times surrounding a particular event or recurring events. For example, a friend might want to track a user's movements on Friday night.
When performing local computation to determine context, it is possible to intelligently activate sensors to help correlate data and reduce noise in computation purposes. The general concept is to use sensor data from one sensor to confirm or deny the calculated state by another sensor.
In one embodiment, the local system determines a device is in a At Destination State using acceleration data, and then uses GPS to confirm the At Destination State by obtaining the location of the device.
In one embodiment, the same algorithms will be used for identifying context on multiple devices. In another embodiment, different algorithms will be used for identifying context on multiple devices. Different devices, even of the same model number, sometimes exhibit different sensing properties. In particular, the degree of noisiness or accuracy of the sensor can vary. To accommodate for this difference, different algorithms must be developed. These algorithms may be based on the noisiness of the device but may also be based on other factors, such as sensor capabilities. To determine the noisiness of the device, the device is calibrated.
In another embodiment, the device is calibrated based on the model number. This implies a variety of factors under consideration for the calibration, including but not limited to sensor sampling rates, capabilities of the sensors, and even the existence of the sensors themselves. For example, one device may have a gyroscope while another may only have an accelerometer.
In another embodiment, the device is calibrated by listening to the noise during a period of likely known behavior. For example, at 4am in the morning, it is likely the device is still because the owner is sleeping. During this time, the sensors may be activated and the noise may be sampled to enable threshold calibration.
In another embodiment, thresholds can be applied to the sensor data to ensure the device is being calibrated to noise and not movements associated with user interaction. For example, acceleration beyond a threshold or rotation beyond a threshold imply the device is moving due to user movements and not noise alone.
In another embodiment, the state of the device is observed to ensure the user is not interacting with the device. For example, if the touch screen is being engaged by the user or it is moving substantially, the device can determine user interaction and share that state with the algorithm to inform the calibration algorithm to wait until a time period when there is no user interaction.
After a need to Check-In or Check-Out has been determined, the act of registering the location requires the use of a location technology to associate the location with a particular local context. In one embodiment GPS would be used to generate the location data needed to associate local context. In another embodiment Cell Tower Location data would be used to generate the location data needed to associate local context. In another embodiment Wifi Data would be used to generate location data needed to associate local context. In another embodiment, some future location technology would be used to generate the location data needed to associate local context.
In another embodiment, the automated check-in and automated check-out system behaves differently based on the accuracy of the location data. For example, the system may register the user with a specific local context or prompt the user. For example, registering a location that has an uncertainty of 100 meter radial range and 20 possible locations would generate a prompt to the user. Registering a location that has an uncertainty of 10M radial range and 2 possible locations would generate a prompt to the user. Registering a location that has an uncertainty of 5 meter radial range and 1 possible location would automatically register the user.
Based on historical check-in patterns, it is possible to improve the ability of the registration system to improve its registration capabilities. For example, if the accuracy of the registration system identifies 20 possible locations in a radius of 100M but the user consistently only checks into 3 locations, then rather than prompting the user with a general prompt like in the previous example, the system would provide the three options that are typically registered by the user.
In one embodiment, periodic location registrations must be performed while the user is in the In Transit State to facilitate one user tracking another user. For example, the case may arise where one user is in a In Transit State for an extended period of time. In such a scenario, the system would periodically register the location to facilitate a secondary user tracking a primary user.
In another embodiment of the application, the time for which the secondary user is tracking the primary user is limited. For example, it could be in terms of time windows or on a recurring basis. For example, weekends only. For example, one time only 30 min increments.
In another embodiment of the application, the accuracy of the location for which the secondary user is tracking the primary user is limited. For example, a generic friends list may only receive state-level accuracy, whereas one's boyfriend may receive the most accurate location available.
There are numerous methods to determine the context of a device. In one sample embodiment, pocket context is determined using proximity data associated with the lack of light through the light sensor. Additional pocket context can be inferred using acceleration data and orientation data indicative of motion such as a user sitting or a user walking In another sample embodiment, hand context can be determined using proximity data associated with the presence of light through the light sensor. Hand context can also be determined using acceleration data, acceleration data and orientation data, and acceleration data and rotation data.
In another embodiment, sleeping context can be determined through acceleration data, charging state, or time of day. For example, acceleration below a threshold for an extended period of time may imply a user is sleeping. Charging may also imply a user is sleeping. Time of day may also imply a user is sleeping. Combinations of these methods may further increase the accuracy of sleeping context determination.
In another embodiment, work or school context can be determined through time of day or location data. In another embodiment user listening to music context can be determined through sound data, either being played by the device itself or being heard through the microphone.
One embodiment also relates to a system and method for computing vehicle related context using mobile devices. One embodiment uses mobile device sensor data to identify power-saving techniques for understanding the context of both the device and its associated user. Contextual situations can include, for example, identifying the user is inside a motor vehicle, identifying the motor vehicle is in traffic, and identifying the user has exited the vehicle. This information is used to generate, amongst other types of knowledge, highly accurate real-time vehicle traffic data and the location of parked vehicles.
Determining mode of transportation is of particular value for a wide array of mobile applications. Some valuable applications that would benefit from accurate context awareness include the ability to change the state of phone to prevent unfavorable user behavior like texting while driving. Another sample application would determine vehicle congestion and communicate that information to a traffic server. Yet another sample application would change the state of the phone into “driver mode” that would make certain functions more easily accessible while others could be less easily accessible.
To date, most methods for determining vehicle awareness have been through the use of location technologies such as GPS, A-GPS, Wife Position System (WPS), and Cell Phone Tower triangulation. Historically, the general approach has been to calculate periodically the location of a phone and then determine the speed of movement. Assuming fairly high speed, one can infer the device has been inside a moving vehicle.
Using location based technologies to determine vehicle context has numerous weaknesses. Accuracy and frequency of location information has bandwidth, power, and privacy implications. In addition, certain vehicle situations such as a vehicle inside a city become substantially more difficult to identify since the vehicle does not travel very far within a fixed amount of time. Finally, the delay in determining context has substantial implications for applications that desire to implement solutions based on this knowledge.
Others have attempted to determine mode of transportation through pure acceleration analysis. Smartphones have accelerometers that sense acceleration movements, and based on their movements, context can often be determined. Typical approaches using this information leverage the magnitude of acceleration and patterns of movement embedded in the data to determine the context. The greatest challenge with only using acceleration of a device is that modes can often be mistaken. For example, bicycling can be mistaken for being inside a motorized vehicle, running can easily be mistaken for walking, etc.
Beyond simple mode of transportation determination, there are many other finer benefits that can be determined. In particular, high quality traffic data can be generated by the local device. Historically, the quality of traffic data has been substantially delayed and at times highly inaccurate. The technologies used to generate this traffic data have been inefficient or too expensive in terms of cost, battery power, or privacy intrusion. In addition, where traffic data is available, the quality of the information currently tends to be average speeds along a road segment, instead of speeds associated with specific vehicles. As a result, the quality of traffic data generated has suffered substantially along with the quality of the recommendations associated with it.
While tremendous research has been done solely to use location data (i.e. GPS data, WPS data, Cell Phone Tower data, etc.) for traffic computation purposes, most approaches to date have been to use this information in a centralized location, such as a traffic server hosted in the Internet cloud.
There are numerous reasons for why this has been the case. A requirement of a traffic data system is that traffic data must be real-time, so delaying uploads to facilitate bundling and compression over data over extended periods of time to optimize bandwidth and power consumption, such as sending updates daily, poses a challenge since the data loses value over time. It is for this reason that the data is currently being sent with such high frequency.
Central computation of traffic data has numerous problems. Sending location data with vehicles to a centralized location for traffic computation purposes requires significant bandwidth due to the vast amount of data that must be processed. Sending the data itself requires a device that consumes significant power from the frequent uploads.
Finally, allowing the location data to be aggregated in a central location is a privacy risk for the associated users, since the routes taken by vehicles could be tracked if the data is not correctly anonymized. As an example, Google uses this approach and crowdsources traffic data by having phones report the speed of the device.
Power is a major resource constraint. In situations where the communication device is not being charged by the vehicle, the device must rely on a battery, which is a limited power source. Mobile devices, like smartphones, cannot continuously report locations and speeds using GPS since obtaining GPS fixes are power intensive. In addition, calculating speed and sending data is power hungry and dramatically reduces the battery's lifetime. In addition, the bandwidth requirement of a solution that continuously reports data is prohibitive, since bandwidth is a limited resource.
It should be noted that the current quality of traffic alert notifications and navigation recommendations is associated with navigation devices rerouting users along routes with known end destinations. In particular, this mostly occurs on highways, since the quality of traffic data on surface roads is poor such that recommendations along surface roads are not typically provided.
In addition to understanding mode of transport, understanding the point at which mode of transport changes is also of particular value. For example, numerous applications charge for helping users determine the location of a user's vehicle. The general workflow requires a user to mark the location of the vehicle upon its exit. To do so, the user must open the application, input a request to mark the location, and then the application must obtain a location reference, typically a GPS fix. In so doing, the application has now marked the user's vehicle for later reference.
When the user returns to the area where the vehicle was left, the user then must open the application and request the application to find the vehicle. This is typically performed by placing a marker on a map or radar screen indicating the vehicle and typically another marker on the same screen indicating the location of the user.
One of the major weaknesses of these applications involves the user having to input the original location of the vehicle. Users typically forget this step and then the application is unable provide value. In addition, the act of marking the location is often considered a hassle and additional work for the user.
Thus, a heretofore unaddressed need exists in the industry to address the aforementioned deficiencies and inadequacies
In a first embodiment there is provided a system and method for contextual awareness of a portable device, including but not limited to a smartphone, watch, laptop, etc. The present system and method increases the accuracy and reliability of contextual awareness in movement related situations and particularly for determining mode of transport.
In a second embodiment, there is provided a system and method to determine additional context of a motor vehicle. This context can be used to generate intelligent alerts along a variety of attributes, including but not limited to alerts during times of interest, near points of interest, and in situations of interest, for a number of purposes, including but not limited to traffic data, marketing data, etc.
In a third embodiment, there is provided a system and method for automatically determining the location of a vehicle without user input.
The following highlights included state transitions for a device related to a vehicle as shown in
In one embodiment of sensor optimization for power conservation the sampling rate of the sensor can be modulated based on context, as shown in
In another embodiment of sensor optimization for power conservation it is possible to leverage the accelerometer, a relatively low power consumption sensor, to identify traffic events and then using GPS, a relatively high power consumption sensor, to gain additional context when needed.
As shown in
In the preferred embodiment, it is possible to simulate every motion of a device in 3-d space measuring acceleration and orientation data, i.e. six degrees of freedom. Based on a digital simulation, it is possible to associate movements with context. Based on the contextual understanding of the object in motion, it is possible to change the state of the object as well as communicate with an external system.
Embodiments which uses acceleration and orientation data to simulate motion digitally, include contextual awareness for smartphones, such as a smartphone is with a person who is inside a motorized vehicle, inside a motorized vehicle that is in traffic, or inside a motorized vehicle that is on a highway, walking, running, bicycling.
Additional contextual granularity can also be determined once vehicle context is identified, including: Device Moving inside Vehicle, Device Not Moving inside Vehicle, Vehicle Stopped Context, Vehicle Moving Context, Vehicle Going Forward Context, Vehicle Turning Context (including Left and Right turn), Vehicle Braking Context, and Vehicle Exit Context.
In another embodiment, geographic context can sometimes be inferred without determining exact location. For example, lacking a vehicle stop beyond a given period of time indicates with substantial probability that the vehicle is likely on a freeway or highway.
For example, the following are examples for how acceleration and orientation data may be processed to achieve each of the states described above. These embodiments are intended to be merely exemplary; numerous variations and modifications will be apparent to those skilled in the art.
In one embodiment, when the smart phone has no pitch and no roll yet has acceleration beyond a threshold for a certain period of time, this situation implies that the device is highly likely inside a moving vehicle. In another embodiment, to infer the device is outside the moving vehicle, excessive rotation (i.e. pitch and/or roll) is typically observed for an extended period of time.
This context can be further refined to determine the state Device Moving inside Vehicle because of temporary motion observed during windows where vehicle context is observed. This context can be further refined to determine the state Device Not Moving inside Vehicle because of no rotational motion observed during windows where vehicle context is observed. This context can be further refined to determine the state Vehicle Stopped Context where there is rotation below a threshold once vehicle context has been identified and acceleration below a threshold for a period of time indicating that both the device and associated vehicle are at rest.
This context can be further refined to determine the state Vehicle Moving Context where there is rotation below a threshold once vehicle context has been identified and acceleration above a threshold for a period of time indicating that both the device and associated vehicle are moving.
This context can be further refined to determine Going Forward Context where there is rotation below a threshold once vehicle context has been identified and observing direction of motion vector from stop state to start state. This direction is the going forward direction and assuming the phone doesn't change orientation, this angle represents going forward context for the vehicle. If the phone does change orientation, the forward direction can be calculated based on local changes of the orientation of the device.
This context can be further refined to determine Turning Context where there is rotation below a threshold once Going Forward context has been identified and observing direction of motion vector is greater than some threshold and below another threshold, it implies a turning event. This context can be further refined to determine Braking Context where there is rotation below a threshold once Going Forward context has been identified and observing direction of motion vector is greater than some threshold. This context can be further refined to determine Traffic Context where there are successive stop and go events to represent gridlock traffic or as shown in
If the acceleration of the device is approximately the acceleration of gravity, it can be inferred with a high degree of certainty that the vehicle is Idle. If the acceleration is substantially more than or less than the acceleration of gravity (again, without rotation) then this implies the vehicle is Moving. In another embodiment, if the vehicle oscillates back and forth between Idle and Moving states for a period of time, this implies a Traffic event. The greater the times associated with this back and forth Idle and Moving states, the greater the intensity of traffic. In another embodiment, if the vehicle is moving for a substantial period of time, it can be inferred that the vehicle is no longer in traffic.
In another embodiment, the system would distinguish between different vehicles. Vehicles have different acceleration profiles based on the engine. In particular, it is possible to take the Fast Fourier Transform of a vehicle's acceleration to observe the frequency of the engine.
By performing this method upon entering a vehicle, it is possible to uniquely identify one vehicle from another. In addition, a user typically enters an owned vehicle more often than a vehicle owned by another person. As such, after identifying multiple vehicles, it is possible to identify the particular vehicle owned by the user. It should be noted that the sampling rate for identifying a vehicle will likely need to be greater than determining vehicle exit. As such, another embodiment temporarily increases the sampling rate of the accelerometer but then decreases it to optimize for power consumption.
Assuming the device is inside the vehicle, then in another embodiment excessive rotation for a period of time implies the device has left the vehicle. In another embodiment, determining the user is walking, running, or bicycling implies the device has left the vehicle.
In another embodiment, if the device is left inside the vehicle, then the excessive rotation associated with it leaving that environment will not take place. In such a situation, the device will reflect an idle vehicle for an excessive period of time. Beyond some reasonable threshold for no movement, then in another embodiment it can be sufficiently inferred that the vehicle is in fact parked and not in an idle or traffic event.
There are numerous methods to determine context of a vehicle using just location data. The greater the accuracy and frequency of the data observed by the local system, the greater the resolution of the state. In one embodiment, it is possible to use location data to calculate the speed of the vehicle.
In one embodiment, the following scenarios highlight various contextual states: 1) Speed equal to 0 mph implies a vehicle state of Stopped. 2) Speed not equal to 0 implies a vehicle state of Moving. 3) Speed fluctuates between Stopped and Moving vehicle states implies Traffic. The number of Stopped and Moving events and the duration of each imply intensity of traffic. 4) Speed decreases implies a vehicle state of Braking. 5) Speed increases implies a vehicle state of Accelerating. 6) Series of Braking events implies Traffic. The number of Braking events implies the intensity of traffic. 7) Speed equals 0 mph for an extended period of time implies a vehicle state of Parked.
In another embodiment, the local system can compare the speed data to local map data to infer if these speeds are appropriate for the given location. If a slower speed than what is typically expected for the given road segment on which it is being calculated is identified, it can be inferred that the vehicle is in a traffic state and uploaded this alert to the central system for distribution. Depending on the difference between expected and actual speeds, the degree of the traffic state can also be computed and communicated.
In another embodiment, the local system applies its location to a map to determine its geographic context, including but not limited to a Rural area, Urban area, Highway, Surface Road, Parking lot, Arterial Road, Mall, Restaurant, School, Office Building, etc.
In another embodiment, the vehicle powers the local system, including but not limited to many popular personal navigation devices (PNDs) and quite frequently mobile devices that the user plugs in. When the vehicle is powered down this state change can also be observed. In this way, the local system can distinguish that the vehicle is parked rather than in an idle state by inferring a charging state to imply not parked and a not charged state to imply parked.
In another embodiment, if the onboard computer of the vehicle is powered down because the vehicle is turned off then the onboard computer system can distinguish the vehicle is parked rather than in an idle state.
In a preferred embodiment, an application would automatically determine the user is inside a vehicle. Upon exiting the vehicle, the application would automatically observe the user's location and mark it, as shown in
In one embodiment, the application would be a smartphone application and would continuously run in the background of the phone determining the user's status of whether the user is inside a vehicle. Upon determining the user is inside a vehicle, the application would wait until the user has exited to mark the location of the vehicle. This application's graphical user interface would be brought to the foreground whenever the user would access the application to observe the location in a graphical manner.
In another embodiment, the application would run on a wearable device, such as a watch.
In another embodiment, the application would run in the onboard computer system of the vehicle and update a website that could be accessed by the user.
In another embodiment, the application would run on a device wirelessly connected to the vehicle.
In another embodiment, the application would be running on a portable device, including but not limited to a smartphone, a watch capable of running applications, a tablet computer, and a personal laptop.
In another embodiment, the application would be running on a device physically connected to the vehicle, such as a smartphone or a personal navigation device. In another embodiment, the local system would attempt to mark the location of the vehicle but fail to obtain the position via a GPS fix. Such a failure is indicative of when there is an overhead obstruction. In this embodiment, the local system prompts the user to ask for user input to enter information manually marking the location of the vehicle. This feature could be enabled/disabled by the user.
In one embodiment, the central system and local system may initiate communication with each other. In another embodiment, the local system may only initiate communication with the central system. In another embodiment, the central system may only initiate communication with the local system.
In one embodiment, the local system obtains updates from the central system. In another embodiment, the central system obtains updates from the local system.
In one embodiment, an update comprises data including but not limited to traffic alerts, geography profiles, and update frequency. In another embodiment, updates are compressed to optimize bandwidth use.
In another embodiment, the central system may dynamically command the local system to report back speeds with high frequency during the times surrounding a particular event or recurring events. For example, a municipality might want to observe the traffic data associated around the geographic area with special events such as baseball games, football games, etc. to allow municipalities to observe traffic patterns and test alternative configurations. For example, a municipality may change a two-way road to a one-way road and then observe its impact in the real world.
In another embodiment, the local system determines a vehicle is moving and downloads from the central system the real-time traffic profile for the geographic region to facilitate intelligent traffic recommendations to the user and to the central system. In another embodiment, the local system may determine a traffic event but not issue a message to the central system because the central system is already aware of this congestion. In another embodiment, the local system may determine a traffic event but not issue a message to the user because the traffic is historically typical for that specific road segment at that time of that particular day of the week.
In another embodiment, the local system determines the vehicle is moving and downloads a communication frequency profile or update frequency from the central system for that particular vehicle because of the time and geographic region. For example, a vehicle traveling at 2am in a rural area may not be instructed by the central system to issue traffic alerts because that area is not frequently traveled and the data will likely become stale before it adds value to another user.
In another embodiment, the local system downloads a geographic traffic definition profile that informs the local system of the unique signatures that comprise a relevant traffic event. For example, certain municipalities have traffic lights lasting 35 seconds while other municipalities have traffic lights lasting 45 seconds. To prevent the system from erroneously sending traffic alerts, the definition of relevant traffic events will change. The geographic traffic definition profile would include but not be limited to definitions of patterns that signify traffic events.
In another embodiment, the local system downloads a points-of-interest definition profile that informs the local system to upload data whenever the vehicle appears within close proximity of a point of interest.
In another embodiment, alerts regarding traffic conditions and road conditions may then be pushed from the central server to vehicles communicating with the central server, wherein the vehicles have communication devices therein. A user of the vehicle may then view the alerts.
One of the leaders in generating traffic data today is Google. Their traffic system crowdsources traffic data from cell phones using both cell phone tower triangulation data and GPS data. In particular, their GPS data is sourced from users using Google Maps.
Google's crowdsourcing mechanism while industry leading suffers from a number of weaknesses. For starters, users must manually start the Google Maps application. The majority of Google Maps users only initiate this application when driving to unknown destinations. The times a driver drives to an unknown destination is a small fraction of the time a driver is inside a vehicle. As a result, this weakness prevents Google from sourcing data from a large number of users that have their application installed.
Another weakness of the Google Maps application is that it is power intensive. When users initiate the application, it continuously runs GPS for navigation purposes. While useful at times, leaving GPS on and constantly polling for updates is power intensive and quickly drains a limited power supply. As a result, drivers choose to not leave this application running in the backgrounds of their phones.
Another weakness of the Google Maps application for crowdsourcing traffic data is that it has an associated graphical user interface that downloads map data and other types of data. For traffic crowdsourcing, this is not a necessary feature at all times and it unnecessarily consumes battery power. These weaknesses present an opportunity to develop a more effective traffic crowdsourcing platform.
In one embodiment, the local system automatically turns on upon determining vehicle context to generate traffic data as shown in
In another embodiment, the local system prompts the user to allow the service to run starting at boot time of the device. In another embodiment, the local system uses less power intensive sensors than GPS to reduce power consumption and collect data, thereby improving the user experience, and uses GPS only at key events as shown in
In another embodiment, the local system initiates certain navigation features such as routing and alerts based on pertinence to the user. The local system may also query the user for inputs such as quality of routes.
In another embodiment, the local system only uploads data to the central system if it is an adverse condition. In another embodiment, the local system only uploads data to the central system if it conflicts with information sent from the central system.
In another embodiment, local system encrypts uploads to the central system, and vice versa. In another embodiment, the local system compresses uploads to the central system, and vice versa, to reduce bandwidth usage.
In another embodiment, the local system sends periodic updates to the central system once vehicle context is determined. A distinguishing feature of this embodiment from the Google Maps application is that the user must manually activate Google Maps. In this embodiment, the service runs automatically and after determining vehicle context, automatically updates the central system with traffic flow data periodically without user intervention, including but not limited to GPS speeds and locations.
In another embodiment, the local system sends contextual updates to the central system once vehicle context is determined. For example, relevant contextual situations include but are not limited to each time a user makes a phone call, each time the vehicle stops, each time the user moves the phone beyond or below a rotational threshold, each time the device accelerates beyond or below a threshold, the each time the user approaches a point of interest, each time the device changes state, including but not limited to charging, not charging, phone call, display on, display off, etc., each time the device determines vehicle entry, each time the device determines vehicle exit, each time the user interacts with the device, each time the device approaches a point of interest, each time a time of interest is identified, etc.
Although it is possible to have all sensors active and operating at the highest sampling rate at the same time, doing so is extremely inefficient use of a scarce power resource. A system and method to reduce power consumption is to intelligently use sensors based on context to optimize for power usage.
In one embodiment, the algorithm for determining the device is inside a car runs for a short period of time and then shuts off for another period of time. For example, the algorithm for vehicle determination may occur for 10 seconds. If the algorithm determines the device is outside a car, it shuts off all sensors, enters a “sleep” state, and remains asleep for 5 minutes. After 5 minutes, the system “wakes up,” activates the necessary sensors again for 10 seconds, and then determines if the device is now inside a vehicle. This process repeats until the algorithm determines the device is inside a vehicle. Upon determining vehicle context, it continues to remain active and activates other algorithms to determine if the vehicle is inside traffic.
In another embodiment, the system reduces the sampling rate of a particular sensor in one context and then increases the sampling rate in another context, when greater frequency of samples is required. For example, the local system may sample the accelerometer at the lowest frequency when the system is attempting to determine that the device is inside a vehicle, but once it determines this state, it increases the sampling rate of the accelerometer to more accurately identify more granular states, including but not limited to the vehicle braking, turning, stopped, and moving. The local system may optionally increase the sampling rate even more to calculate even more granular data, such as speed from the local system.
In another embodiment, the system activates one type of sensor that consumes less power and then after determining an appropriate context activates another sensor that consumes greater power for additional visibility into the context. For example, when the device is inside a vehicle, the system may only run the accelerometer until situations that indicate traffic situations arise. This may be, for example, a series of braking events within a certain time frame. In such a situation, the system would then activate the more power hungry GPS sensor to obtain speed information and upload this information to the central system.
In another embodiment, the system determines a charging state and then activates all sensors since the power constraint has become significantly relaxed. For example, users often power their phones inside vehicles. In such a case, the power constraint is significantly relaxed since energy is now temporarily abundant. In another example, the system may be a physically connected personal navigation device (PND) that remains connected to the vehicle. In these embodiments, the system may or may not keep the same reporting structure depending on whether bandwidth is still a limited resource and remains a consideration.
When performing local computation to determine context, it is possible to intelligently activate sensors to help correlate data and reduce noise in traffic computation purposes. The general concept is to use sensor data from one sensor to confirm or deny the calculated state by another sensor.
In one embodiment, the local system determines a device is inside a vehicle using acceleration data, and then uses GPS to confirm this by calculating the speed of the device and comparing it to local map data.
In another embodiment, the local system determines the vehicle is turning using acceleration, and then activates the compass to confirm the change of state using this secondary sensor.
In another embodiment, the local system intermittently uses speed calculated from GPS to confirm the speed originally calculated from the less power hungry Accelerometer and Gyroscope.
In one embodiment, the same algorithms will be used for identifying context on multiple devices. In another embodiment, different algorithms will be used for identifying context on multiple devices. Different devices, even of the same model number, sometimes exhibit different sensing properties. In particular, the degree of noisiness or accuracy of the sensor can vary. To accommodate for this difference, different algorithms must be developed. These algorithms may be based on the noisiness of the device but may also be based on other factors, such as sensor capabilities. To determine the noisiness of the device, the device is calibrated. In this and other embodiments, the algorithms could also be implemented within the sensors themselves and communicate with each other as necessary.
In another embodiment, the device is calibrated based on the model number. This implies a variety of factors under consideration for the calibration, including but not limited to sensor sampling rates, capabilities of the sensors, and even the existence of the sensors themselves. For example, one device may have a gyroscope while another may only have an accelerometer.
In another embodiment, the device is calibrated by listening to the noise during a period of likely known behavior. For example, at 4am in the morning, it is likely the device is still because the owner is sleeping. During this time, the sensors may be activated and the noise may be sampled to enable calibration.
In another embodiment, thresholds can be applied to the sensor data to ensure the device is being calibrated to noise and not movements associated with user interaction. For example, acceleration beyond a threshold or rotation beyond a threshold implies the device is moving due to user movements and not noise alone.
In another embodiment, the state of the device is observed to ensure the user is not interacting with the device. For example, if the user is engaging the touch screen, the device can determine that and share that state with the algorithm to inform the calibration algorithm.
In another embodiment, different thresholds will be used for identifying context on multiple devices. Different devices, even of the same model number, sometimes exhibit different sensing properties. In particular, the degree of noisiness or accuracy of the sensor can vary. To accommodate for this difference, different thresholds must be developed. These thresholds may be based on the noisiness of the device but may also be based on other factors, such as sensor capabilities. To determine the noisiness of the device, the device is calibrated.
Typically, users choose to not use navigation features for known destinations. Turn by turn directions for known routes are typically considered a distraction and an annoyance for many users. As a result, to the extent traffic data is available for these routes, the user would fail to receive these alerts because traditional navigation services would never be initiated.
One embodiment is for the local system to infer vehicle context and then develop a local database of the typical destinations and origins of the user. This method of observing and recording user behavior automatically is referred to as machine learning. Average drivers typically drive to one of ten known destinations, including but not limited to Home, Work, Gym, Daycare, Friend's House, Family's House, Grocery Store, Mall, Church, etc.
After observing the driver's driving patterns, in one embodiment the local system prompts the user to tag data. Tagging implies adding Meta information to the data to make it more user friendly. For example, the system may ask the user to enter a name for the current destination. Alternatively, based on a variety of attributes, including but not limited to time of day, users movement patterns, origin, etc. the local system may intelligently ask the user to associate the location. For example, the local system may ask the user, “Are you at home?” when the driver first starts driving on Monday morning. Similarly, when the driver first approaches the end destination on Monday morning, the local system may ask the user “Are you at work?” or “How was your drive?” On Sunday morning however, the local system may ask the user “Are you going to Church?” These are sample embodiments and it should be noted that a person of ordinary skill in the art will recognize this as an artificial intelligence system capable of learning the user's driving patterns and preferences.
It should be noted that one benefit of this embodiment is that after a database of typical origins and destinations is developed by the local system, the user need not enter the end location. Although additional information is valued for the sake of accuracy, the act of entering location data by the user is often considered a hassle and additional work. The local system can simply observe the driving behavior of the user and infer end destinations.
In so doing, another embodiment is to provide automated traffic alerts to the user without the user entering an end destination. After machine learning driving patterns of individuals on a daily basis, the local system can provide intelligent traffic alerts when it appears the user is traveling along one of these well-known routes. For example, a driver typically travels between their home and where they work. When this driver enters into the car and begins driving, the system can identify based on the initial driving behavior that the user is travelling along a known route to work and notify the driver in advance if the typical route has abnormal congestion for that time of day. Attributes that may be taken under consideration to identify the specific route include but are not limited to time of day, beginning location, end location (if entered by the user), and specific turns.
In another embodiment, the local system may not only provide traffic alerts along known routes but may also provide guidance information for detours to avoid the congestion. In this way, the local system demonstrates understanding that the user knows how to get to work, but may need guidance to avoid abnormal congestion. This feature is valuable because typical navigation systems provide guidance along entire routes, often annoying users. In these situations, users sometimes turn off the navigation systems and prevent them from providing additional advice. However, in this embodiment, the automated navigation system intelligently shares relevant guidance information when it is pertinent based on context, thereby improving the user experience.
Local contextual determination allows a navigation system to provide customized navigation recommendations to the user from both, data derived from the masses as well as data derived from the specific user. Understanding what the masses are doing in real time allows navigation systems to make recommendations based on this knowledge to individuals, without these users specifying end destinations to the system.
In one embodiment, traffic alerts are issued to the user based on data from the masses, without the user entering an end destination. To enable this embodiment, the traffic system would apply a hidden Markov Model (HMM) that highlights the probabilities of moving from one road segment to another road segment. Traffic data from the masses is collected, either periodically, each time the user drives, or with some frequency associated with various attributes, including but not limited to time of interest, point of interest, and situational context, to develop specific probabilities of moving from one state to another state. When a user is moving along a known road segment, a traffic alert would be issued if there is high probability that the user would likely turn onto another road segment, where there is excessive traffic.
In another embodiment, the local system intelligently notifies a user of excessive traffic near a typical destination. For example, a large percentage of users are exiting a highway to go to a mall. When the user approaches that same exit and indicates a potential turn by demonstrating a turn into the exit lane, then it is possible to make recommendations to that user to drive to the appropriate entrance of the mall that has less congestion.
In another embodiment, the user chooses to turn off advanced navigation features. In another embodiment, the user chooses to release the local database of origins and destinations to the central system. In another embodiment, the user chooses to prevent the local system from sharing its local database of origins and destinations with the central system.
In one embodiment, the architecture used to implement the system can be configured in multiple network topologies including, but not limited to, star, bus, or ring configurations. Also, the system can be broadly categorized as belonging to a particular architecture including, but not limited to, peer-to-peer or client/server architectures. The network can additionally be classified by the geographical location of the communication devices and the types thereof. For example, if the network connects a number of computer systems or servers located in relatively close proximity to each other, such as within a building, the network is referred to as a local-area network (LAN). If the computer systems are located farther apart, the network is generally referred to as a wide-area network (WAN), such as the Internet. If the computer systems are located within a limited geographical area, such as a university campus or military establishment, the network is referred to as a campus-area network (CAN). Similarly, if the computer systems are connected together within a city or town, the network is referred to as a metropolitan-area network (MAN).
In another embodiment, these various communications could take place through a rich set of devices, including but not limited to: PDAs, smarthpones, phones, pagers, laptop computers, and desktops. In addition, the system described herein could be facilitated by email, phone, and other communication devices. It need not be limited to an online presence, as a human component may also facilitate various parts of the system.
In another embodiment, association between a communication device and a vehicle may be provided by the communication device being located within the vehicle, on the vehicle, connected to the vehicle, via, for example, but not limited to, an on board diagnostics (OBD), or OBDII vehicle connection, or other manner of being connected to the vehicle, or provided through any other manner of communication between the vehicle and the communication device. As an example, another manner of communication between the vehicle and the communication device may be where the communication device does not communicate with the vehicle, but instead, is capable of detecting actions taken or not taken by the vehicle. In such an embodiment, the communication device may be portable.
In another embodiment of the network each of the communication devices may communicate with a central server. The communication devices may communicate with the central server via use of one or more communication protocol provided by a transmission means, which is known to one having ordinary skill in the art. As a non-limiting example, the communication device may communicate with the central server via the Internet, through wireless communication, mobile telephone networks, local wireless networks (e.g., WiFi, ZigBee), or through a different communication means. It should be noted that, within the network, communication may be from communication devices to the central server, or from the central server to one or more of the communication devices. In addition, communication may be from communication device to communication device.
The central server is also capable of accumulating data received from multiple communication devices within the network, aggregating conditions regarding vehicle behavior, and transmitting alerts to communication devices.
Functionality of the communication device can be implemented in software, firmware, hardware, or a combination thereof. In an embodiment, a portion of the communication device is implemented in software, as an executable program, and is executed by a special or general-purpose digital computer, such as a personal computer, personal data assistant, smart phone, workstation, minicomputer, or mainframe computer.
In another embodiment, in terms of hardware architecture, the communication device includes a processor, memory, storage device, and one or more input and/or output (I/O) devices (or peripherals) that are communicatively coupled via a local interface. The local interface can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.
In another embodiment, the processor is a hardware device for executing software, particularly that stored in the memory. The processor can be any custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the communication device, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or generally any device for executing software instructions.
In another embodiment, the memory can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.). Moreover, the memory may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory can have a distributed architecture, where various components are situated remote from one another, but can be accessed by the processor.
In another embodiment, the software in the memory may include one or more separate programs, each of which contains an ordered listing of executable instructions for implementing logical functions of the communication device. The software in the memory defines the functionality of the communication device. In addition, although not required, it is possible for the memory to contain an operating system (O/S). The operating system essentially controls the execution of computer programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services.
In another embodiment, the communication device may be provided by a source program, executable program (object code), script, or any other entity containing a set of instructions to be performed. When a source program, then the program needs to be translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory, so as to operate properly in connection with the O/S. Furthermore, the communication device can be written as (a) an object oriented programming language, which has classes of data and methods, or (b) a procedure programming language, which has routines, subroutines, and/or functions.
In another embodiment, the I/O devices may include input devices, for example but not limited to, a touch screen, a keyboard, mouse, scanner, microphone, or other input device. Furthermore, the I/O devices may also include output devices, for example but not limited to, a display, or other output devices. The I/O devices may further include devices that communicate via both inputs and outputs, for instance but not limited to, a modulator/demodulator (modem; for accessing another device, system, or network), a radio frequency (RF), wireless, or other transceiver, a telephonic interface, a bridge, a router, or other devices that function both as an input and an output. I/O devices are used to transmit data from the vehicle to the central server.
In another embodiment, when the communication device is in operation, the processor is configured to execute the software stored within the memory, to communicate data to and from the memory, and to generally control operations of the communications device pursuant to the software. The software and the O/S, in whole or in part, but typically the latter, are read by the processor, perhaps buffered within the processor, and then executed.
In another embodiment, when the communication device is implemented in software, it should be noted that the communication device can be stored on any computer readable medium for use by or in connection with any computer related system or method. In the context of this document, a computer readable medium is an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer related system or method. The communication device can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “computer-readable medium” can be any means that can store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
In another embodiment, the computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, sensor, apparatus, device, or propagation medium. More specific examples (a nonexhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM) (electronic), a read-only memory (ROM) (electronic), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory) (electronic), an optical fiber (optical), and a portable compact disc read-only memory (CDROM) (optical). Note that the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
In another embodiment, the storage device of the communication device may be one of many different types of storage device, including a stationary storage device or portable storage device. As an example, the storage device may be a magnetic tape, disk, flash memory, volatile memory, or a different storage device. In addition, the storage device may be a secure digital memory card or any other removable storage device 30.
In another embodiment, the communication device may also contain an accelerometer for sensing orientation of the communication device. The accelerometer is capable of detecting acceleration and deceleration of a vehicle in which the communication device is positioned. It should be noted that the accelerometer may instead be an inertial measurement unit (IMU) or the equivalent, providing information regarding orientation of the communication device.
In another embodiment, the communication device may also contain a gyroscope for sensing orientation of the communication device. The gyroscope is capable of detecting rotation movements of a vehicle in which the communication device is positioned.
In another embodiment, the communication device may also contain a communication port. The communication port allows communication between the communication device and one of many different devices. As an example, the communication port is capable of allowing the communication device to communicate with the OBD or OBDII connection port of a vehicle. As a result, the communication port is capable of communication via one or more of many different communication protocols. It should be noted that in accordance with the above-described embodiment, a different means of communication is used for communication with the OBD or OBDII than for communication with the central server. While this is the case, one having ordinary skill in the art will appreciate that in accordance with an alternative embodiment, a similar means of communication may be used for both communication with a vehicle and with the central server.
In another embodiment, the communication device uses Global Positioning System (GPS) data. As a result, the communication device may either have a GPS device located therein, communicate with the GPS for a vehicle to which the communication device is connected, or communicate with another device that supplies GPS information, via the communication port. It should be noted that while the present description provides for use of GPS data, the embodiment is not intended to be limited to the use of GPS data. Instead, a different category of positioning data may be used.
In one embodiment, the architecture used to implement the system can be configured in multiple network topologies including, but not limited to, star, bus, or ring configurations. Also, the system can be broadly categorized as belonging to a particular architecture including, but not limited to, peer-to-peer or client/server architectures. The network can additionally be classified by the geographical location of the communication devices and the types thereof. For example, if the network connects a number of computer systems or servers located in relatively close proximity to each other, such as within a building, the network is referred to as a local-area network (LAN). If the computer systems are located farther apart, the network is generally referred to as a wide-area network (WAN), such as the Internet. If the computer systems are located within a limited geographical area, such as a university campus or military establishment, the network is referred to as a campus-area network (CAN). Similarly, if the computer systems are connected together within a city or town, the network is referred to as a metropolitan-area network (MAN).
In another embodiment, these various communications could take place through a rich set of devices, including but not limited to: PDAs, smarthpones, phones, pagers, laptop computers, and desktops. In addition, the system described herein could be facilitated by email, phone, and other communication devices. It need not be limited to an online presence, as a human component may also facilitate various parts of the system.
In a preferred embodiment, the local system intelligently determines traffic events and other events of interest and selectively uploads this data to a central system, thereby reducing the resource requirements of real-time traffic data in terms of bandwidth, power consumption, and privacy intrusiveness through a joint local and central computation based approach. The local system and the central system dynamically coordinate attributes of the total system through two-way communication.
In one embodiment, the local system comprises a portable communication device, including but not limited to a smartphone, a feature phone, a watch capable of connecting to the Internet, a tablet computer, and a personal laptop. In another embodiment, the local system comprises the onboard computer system of the vehicle. In another embodiment, the local system comprises a device wirelessly connected to the vehicle. In another embodiment, the local system comprises a device physically connected to the vehicle, including but not limited to a smartphone and a personal navigation device.
Sample claims for a device not previously inside a vehicle but one that determines it is inside a vehicle and then performs computation locally as follows. What is claimed is a communication device for providing road conditions and monitoring congestion, comprising: a memory; and a processor configured by the memory to perform the steps of: determining if the device is inside a vehicle; if the device is inside the vehicle extracting key current vehicle and/or traffic insight; determining if the extracted key current vehicle and/or traffic insight represents a departure from expected vehicle and/or traffic insight; if the extracted key current vehicle and/or traffic insight is a departure from expected vehicle and/or traffic insight, using the key current vehicle and/or traffic insight to create a smart message, wherein the smart message contains the extracted key current vehicle and/or traffic insight; and if the extracted key current vehicle and/or traffic insight is a confirmation of expected vehicle and/or traffic insight, creating a confirmation message that does not contain the extracted key current vehicle and/or traffic insight, yet confirms similarity between the extracted key current vehicle and/or traffic insight and the expected vehicle and/or traffic insight as shown in
In another embodiment, this communication device is physically connected to the current vehicle, but not electronically connected to the current vehicle.
In another embodiment, this communication device is in communication with the current vehicle through use of an on board diagnostics (OBD) or OBDII vehicle connection.
In another embodiment, within this communication device, the expected vehicle and/or traffic insight is previously extracted key current vehicle and/or traffic insight from the current vehicle.
In another embodiment, within this communication device, wherein the expected vehicle and/or traffic insight is prior extracted key vehicle and/or traffic insight from other vehicles that were driving on a path that is the same as a path currently being driven by the current vehicle.
In another embodiment, within this communication device, wherein a departure from expected vehicle and/or traffic insight is wherein extracted key current vehicle and/or traffic insight is outside of a predefined range in values from the expected vehicle and/or traffic insight.
In another embodiment, within this communication device, wherein confirmation of expected vehicle I and/or traffic insight is wherein extracted key current vehicle and/or traffic insight is within a predefined range in values from the expected vehicle and/or traffic insight.
In another embodiment, within this communication device, wherein the communication device transmits the smart message when extracted key current vehicle and/or traffic insight is a departure from expected vehicle and/or traffic insight.
A system for providing road conditions and monitoring congestion, comprising: at least one communication device, wherein the communication device further comprises: a memory; and a processor configured by the memory to perform the steps of: determining if the device is inside a vehicle; if the device is inside the vehicle extracting key current vehicle and/or traffic insight; determining if the extracted key current vehicle and/or traffic insight represents a departure from expected vehicle and/or traffic insight; if the extracted key current vehicle and/or traffic insight is a departure from expected vehicle and/or traffic insight, using the key current vehicle and/or traffic insight to create a smart message, wherein the smart message contains the extracted key current vehicle and/or traffic insight; and if the extracted key current vehicle and/or traffic insight is a confirmation of expected vehicle and/or traffic insight, creating a confirmation message that does not contain the extracted key current vehicle and/or traffic insight, yet confirms similarity between the extracted key current vehicle and/or traffic insight and the expected vehicle and/or traffic insight; and a server for communicating with the communication device to receive the smart message and/or the confirmation message.
In another embodiment, for this system, wherein the server aggregates traffic conditions and/or road conditions from multiple communication devices, including the at least one communication device.
In another embodiment, for this system, wherein the server transmits alerts regarding the aggregated traffic conditions and/or road conditions to at least one of the multiple communication devices.
In another embodiment, for this system, wherein the communication device is physically connected to a vehicle, but not electronically connected to the vehicle.
In another embodiment, for this system, wherein the communication device is in communication with a vehicle through use of an on board diagnostics (OBD) or OBDn vehicle connection.
In another embodiment, for this system, wherein the expected vehicle and/or traffic insight is previously extracted key current vehicle and/or traffic insight from the current vehicle.
In another embodiment, for this system, wherein the expected vehicle and/or traffic insight is prior extracted key vehicle and/or traffic insight from other vehicles that were driving on a path that is the same as a path currently being driven by the current vehicle.
In another embodiment, for this system, wherein a departure from expected vehicle and/or traffic insight is wherein extracted key current vehicle and/or traffic insight is outside of a predefined range in values from the expected vehicle and/or traffic insight.
In another embodiment, for this system, wherein confirmation of expected vehicle and/or traffic insight is wherein extracted key current vehicle and/or traffic insight is within a predefined range in values from the expected vehicle and/or traffic insight.
In another embodiment, for this system, wherein the communication device transmits the smart message when extracted key current vehicle and/or traffic insight is a departure from expected vehicle and/or traffic insight.
A portable communication device for providing road conditions and monitoring traffic congestion for a user, wherein the portable communication device comprises: a device for determining the device is inside a vehicle, sensing orientation of the portable communication device, including acceleration and/or deceleration of the vehicle if the communication device is located inside a vehicle; a memory; and a processor configured by the memory to perform the step of calculating space time series data to determine traffic congestion characteristics, wherein the step of calculating space time series data includes at least one step selected from the group consisting of determining position of the portable communication device as a function of time, determining velocity of the portable communication device as a function of time, and determining acceleration of the portable communication device as a function of time.
In another embodiment, for this portable communication device, wherein the step of calculating space time series data further comprises reviewing a velocity profile to determine if the vehicle had several periods of no motion with short distances traveled between stops. Wherein the portable communication device further comprises a global positioning system device for determining a current location of the vehicle, wherein the processor is further configured by the memory to perform the step of using the current location of the vehicle to identify a traffic intersection associated with the several periods of no motion with short distances traveled between the stops.
In another embodiment, for this portable communication device, wherein the step of calculating space time series data further comprises calculating fuel consumption as a function of velocity and acceleration. Wherein fuel consumption is calculated as a function of velocity and acceleration by the equation.
In another embodiment, for this portable communication device, wherein the step of calculating space time series data further comprises use of a time interval of the absolute value of exhilaration to determine fuel consumption. In another embodiment, for this portable communication device, wherein the step of calculating space time series data further comprises use of kinematic metrics to determine fuel consumption. In another embodiment, for this portable communication device, wherein the processor is further configured by the memory to perform the steps of receiving current traffic conditions or previous traffic conditions and comparing the stored previous traffic conditions to the determined traffic congestion characteristics.
In another embodiment, for this portable communication device, further comprising a storage device for storing previous traffic conditions therein, wherein the processor is further configured by the memory to perform the step of comparing the stored previous traffic conditions to the determined traffic congestion characteristics.
In another embodiment, for this portable communication device, wherein the communication device is physically connected to the current vehicle, but not electronically connected to the current vehicle. In another embodiment, for this portable communication device, wherein the communication device is in communication with the current vehicle through use of an on board diagnostics (OBD) or OBDII vehicle connection. In another embodiment, for this portable communication device, wherein the communication device further comprises a storage device and wherein the storage device has stored therein at least one previous traffic condition report. In another embodiment, for this portable communication device, wherein the device for sensing orientation is an accelerometer.
In another embodiment, for this portable communication device, wherein the step of calculating space time series data is used by the portable communication device to determine whether the vehicle is in traffic, wherein the step of calculating space time series data further comprises determining a series of periods of no acceleration of the vehicle with short bursts of acceleration/deceleration of the vehicle in-between and determining that the stops are not in a regular cycle.
In another embodiment, for this portable communication device, further comprising a transceiver for transmitting the determined traffic congestion characteristics and receiving data. One embodiment contains a local device traffic and application system. In this system, the local device determines whether the device is inside a vehicle or not. Associated with this system of determining vehicle identification are unique calibration, power optimization, and bandwidth optimization algorithms. This is highlighted as Module I in
There is also the notion of dumb messages, which implies sending periodic messages to an external system independent of local context. Module III in
There are numerous methods of calculating states and determining the appropriate time when one state has changed to another. In one embodiment, the sensor data is filtered to remove noise. Method for filtering include and are not limited to applying low-pass filters, high-pass filters, Kalman filters, and other types of filters as would be understood by those reasonably skilled in the art.
In another embodiment, the current state is associated with a confidence level. For example, the likelihood that the user is inside a vehicle could be determined with 95% confidence based on the sensor data. One method for determining the appropriate time to switch states includes observing the confidence level to be below a threshold.
In another embodiment, a state machine is implemented using the sensor data to determine current context and the appropriate time to switch contexts. Calculating the speed of a vehicle has important implications for traffic data. In one embodiment, after vehicle context has been identified by a device, the acceleration of the vehicle, obtained through the device, is integrated. Periods of standard deviation below a threshold correlate to periods of zero speed. If available, a gyroscope may be used to determine rotation of the device below a threshold which corresponds to rotation of the vehicle. Rotation beyond a threshold can be associated with device movement not related to the vehicle.
In another embodiment, after vehicle context has been identified by a device, situations of interest are identified using a variety of onboard methods, such as accelerometer fluctuations, and then the speed of the vehicle is determined using GPS.
In another embodiment, after vehicle context has been identified by a device, situations of interest are identified using a variety of onboard methods, such as accelerometer fluctuations, and then the speed of the vehicle is determined using a combination of Wifi and GPS signals.
In another embodiment, after vehicle context has been identified by a device, situations of interest are identified using a variety of onboard methods, such as accelerometer fluctuations, and then the speed of the vehicle is determined using a combination of Cell tower data, Wifi and GPS signals.
What is claimed is a method of determining whether a smartphone is in a motor vehicle, using processes running within the smartphone, wherein the processes comprise: in a first process, using accelerometer outputs in the smartphone to store, repeatedly over a time interval, acceleration vector data; in a second computer process, processing the vector data to determine whether some acceleration is present as to be indicative of motion of the smartphone, and further, whether the extent of rotation of the smartphone is sufficiently small as to make it likely that the smartphone is within a motor vehicle.
A method of determining whether a smartphone is in a motor vehicle, using processes running within the smartphone, wherein the processes comprise: in a first process, using accelerometer outputs in the smartphone to store, repeatedly over a time interval, acceleration vector data and using gyroscope outputs in the smartphone to store, repeatedly over a time interval, rotation data; in a second computer process, processing the vector data to determine whether some acceleration is present as to be indicative of motion of the smartphone, and processing the rotation data to determine whether the extent of rotation of the smartphone is sufficiently small as to make it likely that the smartphone is within a motor vehicle.
A method of determining motion related context of a device, using processes running within the device, wherein the processes comprise: in a first process, using accelerometer outputs in a device to store, repeatedly over a time interval, acceleration vector data; in a second computer process, processing the vector data to determine whether acceleration is present as to be indicative of motion of the device, and further, whether the extent of rotation of the device is present as to be indicative of rotation of the device to make it likely that the device is of particular context.
A method of determining whether a smartphone is outside a motor vehicle, using processes running within the smartphone, wherein the processes comprise: in a first process, using accelerometer outputs in the smartphone to store, repeatedly over a time interval, acceleration vector data; in a second computer process, processing the vector data to determine whether sufficient acceleration is present as to be indicative of motion of the smartphone, and further, whether the extent of rotation of the smartphone is sufficiently large as to make it likely that the smartphone is outside a motor vehicle.
In one embodiment, when the smart phone has no pitch and no roll yet has acceleration beyond a threshold for a certain period of time, this situation implies that the device is highly likely inside a moving vehicle. In another embodiment, to infer the device is outside the moving vehicle, excessive rotation (i.e. pitch and/or roll) is typically observed for an extended period of time.
Assuming the device is inside the vehicle then in another embodiment the magnitude of the acceleration may be observed to infer the context of the vehicle. If the acceleration of the device is approximately the acceleration of gravity, it can be inferred with a high degree of certainty that the vehicle is Idle. If the acceleration is substantially more than or less than the acceleration of gravity (again, without rotation) then this implies the vehicle is Moving. In another embodiment, if the vehicle oscillates back and forth between Idle and Moving states for a period of time, this implies a Traffic event. The greater the times associated with this back and forth Idle and Moving states, the greater the intensity of traffic. In another embodiment, if the vehicle is moving for a substantial period of time, it can be inferred that the vehicle is no longer in traffic.
Historically, the quality of traffic data has been substantially delayed and at times highly inaccurate. The technologies used to generate this traffic data have been inefficient or too expensive in terms of cost, battery power, or privacy intrusion. In addition, where traffic data is available, the quality of the information currently tends to be average speeds along a road segment, instead of speeds associated with specific vehicles. As a result, the quality of traffic data generated has suffered substantially along with the quality of the recommendations associated with it.
While tremendous research has been done solely to use location data (i.e. GPS data, WPS data, Cell Phone Tower data, etc.) for traffic computation purposes, most approaches to date have been to use this information in a centralized location, such as a traffic server hosted in the Internet cloud.
There are numerous reasons for why this has been the case. A requirement of a traffic data system is that traffic data must be real-time, so delaying uploads to facilitate bundling and compression over data over extended periods of time to optimize bandwidth and power consumption, such as sending updates daily, poses a challenge since the data loses value over time. It is for this reason that the data is currently being sent with such high frequency.
Central computation of traffic data has numerous problems. Sending location data with vehicles to a centralized location for traffic computation purposes requires significant bandwidth due to the vast amount of data that must be processed. Sending the data itself requires a device that consumes significant power from the frequent uploads. Finally, allowing the location data to be aggregated in a central location is a privacy risk for the associated users, since the routes taken by vehicles could be tracked if the data is not correctly anonymized. As an example, Google users this approach and crowdsources traffic data by having phones report the speed of the device.
Power is a major resource constraint. In situations where the communication device is not being charged by the vehicle, the device must rely on a battery, which is a limited power source. Mobile devices, like smartphones, cannot continuously report locations and speeds using GPS since obtaining GPS fixes are power intensive. In addition, calculating speed and sending data is power hungry and dramatically reduces the battery's lifetime. In addition, the bandwidth requirements of a solution that continuously reports data is prohibitive, since bandwidth is a limited resource.
It should be noted that the current quality of traffic alert notifications and navigation recommendations is associated with navigation devices rerouting users along routes with known end destinations. In particular, this mostly occurs on highways, since the quality of traffic data on surface roads is poor such that recommendations along surface roads are not typically provided.
Embodiments provide a system and method to determine local context of a vehicle. This context can be used to generate intelligent alerts along a variety of attributes, including but not limited to alerts during times of interest, near points of interest, and in situations of interest, for a number of purposes, including but not limited to traffic data, marketing data, etc.
In a preferred embodiment, the local system intelligently determines traffic events and other events of interest and selectively uploads this data to a central system, thereby reducing the resource requirements of real-time traffic data in terms of bandwidth, power consumption, and privacy intrusiveness through a joint local and central computation based approach. The local system and the central system dynamically coordinate attributes of the total system through two-way communication.
In one embodiment, the local system comprises a portable communication device, including but not limited to a smartphone, a feature phone, a watch capable of connecting to the Internet, a tablet computer, and a personal laptop. In another embodiment, the local system comprises the onboard computer system of the vehicle. In another embodiment, the local system comprises a device wirelessly connected to the vehicle. In another embodiment, the local system comprises a device physically connected to the vehicle, including but not limited to a smartphone and a personal navigation device.
There are numerous methods to determine context of a vehicle using just location data. The greater the accuracy and frequency of the data observed by the local system, the greater the resolution of the state. Using location data it is possible to calculate the speed of the vehicle.
In one embodiment, the following scenarios highlight various contextual states: Speed equal to 0 mph implies a vehicle state of Stopped, Speed not equal to 0 implies a vehicle state of Moving, Speed fluctuates between Stopped and Moving vehicle states implies Traffic. The number of Stopped and Moving events and the duration of each imply intensity of traffic. Speed decreases implies a vehicle state of Braking Speed increases implies a vehicle state of Accelerating. Series of Braking events implies Traffic. The number of Braking events implies the intensity of traffic. Speed equals 0 mph for an extended period of time implies a vehicle state of Parked.
In another embodiment, the local system can compare the speed data to local map data to infer if these speeds are appropriate for the given location. If a slower speed than what is typically expected for the given road segment on which it is being calculated is identified, it can be inferred that the vehicle is in a traffic state and uploaded this alert to the central system for distribution. Depending on the difference between expected and actual speeds, the degree of the traffic state can also be computed and communicated. In another embodiment, the local system applies its location to a map to determine its geographic context, including but not limited to a Rural area, Urban area, Highway, Surface Road, Parking lot, Arterial Road, Mall, Restaurant, School, Office Building, etc.
In one embodiment, the central system and local system may initiate communication with each other. In another embodiment, the local system may only initiate communication with the central system. In another embodiment, the central system may only initiate communication with the local system. In one embodiment, the local system obtains updates from the central system. In another embodiment, the central system obtains updates from the local system. In one embodiment, an update comprises data including but not limited to traffic alerts, geography profiles, and update frequency. In another embodiment, updates are compressed to optimize bandwidth use.
In another embodiment, the central system may dynamically command the local system to report back speeds with high frequency during the times surrounding a particular event or recurring events. For example, a municipality might want to observe the traffic data associated around the geographic area with special events such as baseball games, football games, etc. to allow municipalities to observe traffic patterns and test alternative configurations. For example, a municipality may change a two-way road to a one-way road and then observe its impact in the real world.
In another embodiment, the local system determines a vehicle is moving and downloads from the central system the real-time traffic profile for the geographic region to facilitate intelligent traffic recommendations to the user and to the central system. In another embodiment, the local system may determine a traffic event but not issue a message to the central system because the central system is already aware of this congestion. In another embodiment, the local system may determine a traffic event but not issue a message to the user because the traffic is historically typical for that specific road segment at that time of that particular day of the week.
In another embodiment, the local system determines the vehicle is moving and downloads a communication frequency profile or update frequency from the central system for that particular vehicle because of the time and geographic region. For example, a vehicle traveling at 2 am in a rural area may not be instructed by the central system to issue traffic alerts because that area is not frequently traveled and the data will likely become stale before it adds value to another user.
In another embodiment, the local system downloads a geographic traffic definition profile that informs the local system of the unique signatures that comprise a relevant traffic event. For example, certain municipalities have traffic lights lasting 35 seconds while other municipalities have traffic lights lasting 45 seconds. To prevent the system from erroneously sending traffic alerts, the definition of relevant traffic events will change. The geographic traffic definition profile would include but not be limited to definitions of patterns that signify traffic events.
In another embodiment, the local system downloads a points-of-interest definition profile that informs the local system to upload data whenever the vehicle appears within close proximity of a point of interest. In another embodiment, alerts regarding traffic conditions and road conditions may then be pushed from the central server to vehicles communicating with the central server, wherein the vehicles have communication devices therein. The alerts may then be viewed by a user of the vehicle.
Although it is possible to have all sensors active and operating at the highest sampling rate at the same time, doing so is extremely inefficient use of a scarce power resource. A system and method to reduce power consumption is to intelligently use sensors based on context to optimize for power usage.
In one embodiment, the algorithm for determining the device is inside a car runs for a short period of time and then shuts off for another period of time. For example, the algorithm for vehicle determination may occur for 10 seconds. If the algorithm determines the device is outside a car, it shuts off all sensors, enters a “sleep” state, and remains asleep for 5 minutes. After 5 minutes, the system “wakes up,” activates the necessary sensors again for 10 seconds, and then determines if the device is now inside a vehicle. This process repeats until the algorithm determines the device is inside a vehicle. Upon determining vehicle context, it continues to remain active and activates other algorithms to determine if the vehicle is inside traffic.
In another embodiment, the system reduces the sampling rate of a particular sensor in one context and then increases the sampling rate in another context, when greater frequency of samples is required. For example, the local system may sample the accelerometer at the lowest frequency when the system is attempting to determine that the device is inside a vehicle, but once it determines this state, it increases the sampling rate of the accelerometer to more accurately identify more granular states, including but not limited to the vehicle braking, turning, stopped, and moving. The local system may optionally increase the sampling rate even more to calculate even more granular data, such as speed from the local system.
In another embodiment, the system activates one type of sensor that consumes less power and then after determining an appropriate context activates another sensor that consumes greater power for additional visibility into the context. For example, when the device is inside a vehicle, the system may only run the accelerometer until situations that indicate traffic situations arise. This may be, for example, a series of braking events within a certain time frame. In such a situation, the system would then activate the more power hungry GPS sensor to obtain speed information and upload this information to the central system.
In another embodiment, the system determines a charging state and then activates all sensors since the power constraint has become significantly relaxed. For example, users often power their phones inside vehicles. In such a case, the power constraint is significantly relaxed since energy is now temporarily abundant. In another example, the system may be a physically connected personal navigation device (PND) that remains connected to the vehicle. In these embodiments, the system may or may not keep the same reporting structure depending on whether bandwidth is still a limited resource and remains a consideration.
When performing local computation to determine context, it is possible to intelligently activate sensors to help correlate data and reduce noise in traffic computation purposes. The general concept is to use sensor data from one sensor to confirm or deny the calculated state by another sensor.
In one embodiment, the local system determines a device is inside a vehicle using acceleration data, and then uses GPS to confirm this by calculating the speed of the device and comparing it to local map data. In another embodiment, the local system determines the vehicle is turning using acceleration, and then activates the compass to confirm the change of state using this secondary sensor. In another embodiment, the local system intermittently uses speed calculated from GPS to confirm the speed originally calculated from the less power hungry Accelerometer and Gyroscope.
Advanced Navigation Features include Intelligent Routing and Intelligent Traffic Alerts to User. Typically, users choose to not use navigation features for known destinations. Turn by turn directions for known routes are typically considered a distraction and an annoyance for many users. As a result, to the extent traffic data is available for these routes, the user would fail to receive these alerts because traditional navigation services would never be initiated.
One embodiment is for the local system to infer vehicle context and then develop a local database of the typical destinations and origins of the user. This method of observing and recording user behavior automatically is referred to as machine learning. Average drivers typically drive to one of ten known destinations, including but not limited to Home, Work, Gym, Daycare, Friend's House, Family's House, Grocery Store, Mall, Church, etc.
After observing the driver's driving patterns, in one embodiment the local system prompts the user to tag the destination. Tagging implies adding Meta information to the data to make it more user friendly. For example, the system may ask the user to enter a name for the current destination. Alternatively, based on a variety of attributes, including but not limited to time of day, users movement patterns, origin, etc. the local system may intelligently ask the user to associate the location. For example, the local system may ask the user, “Are you at home?” when the driver first starts driving on Monday morning. Similarly, when the driver first approaches the end destination on Monday morning, the local system may ask the user “Are you at work?” On Sunday morning however, the local system may ask the user “Are you going to Church?” These are sample embodiments and it should be noted that a person of ordinary skill in the art will recognize this as an artificial intelligence system capable of learning the user's driving patterns.
It should be noted that one benefit of this embodiment is that after a database of typical origins and destinations is developed by the local system, the user need not enter the end location. Although additional information is valued for the sake of accuracy, the act of entering location data by the user is often considered a hassle and additional work. The local system can simply observe the driving behavior of the user and infer end destinations.
In so doing, another embodiment is to provide automated traffic alerts to the user without the user entering an end destination. After machine learn driving patterns of individuals on a daily basis, the local system can provide intelligent traffic alerts when it appears the user is traveling along one of these well-known routes. For example, a driver typically travels between their home and where they work. When this driver enters into the car and begins driving, the system can identify based on the initial driving behavior that the user is travelling along a known route to work and notify the driver in advance if the typical route has abnormal congestion for that time of day. Attributes that may be taken under consideration to identify the specific route include but are not limited to time of day, beginning location, end location (if entered by the user), and specific turns.
In another embodiment, the local system may not only provide traffic alerts along known routes but may also provide guidance information for detours to avoid the congestion. In this way, the local system demonstrates understanding that the user knows how to get to work, but may need guidance to avoid abnormal congestion. This feature is valuable because typical navigation systems provide guidance along entire routes, often annoying users. In these situations, users sometimes turn off the navigation systems and prevent them from providing additional advice. However, in this embodiment, the automated navigation system intelligently shares relevant guidance information when it is pertinent based on context, thereby improving the user experience.
Local contextual determination allows a navigation system to provide customized navigation recommendations to the user from both, data derived from the masses as well as data derived from the specific user. Understanding what the masses are doing in real time allows navigation systems to make recommendations based on this knowledge to individuals, without these users specifying end destinations to the system.
In one embodiment, traffic alerts are issued to the user based on data from the masses, without the user entering an end destination. To enable this embodiment, the traffic system would apply a hidden Markov Model (HMM) as shown in
In another embodiment, the local system intelligently notifies a user of excessive traffic near a typical destination. For example, a large percentage of users are exiting a highway to go to a mall. When the user approaches that same exit and indicates a potential turn by demonstrating a turn into the exit lane, then it is possible to make recommendations to that user to drive to the appropriate entrance of the mall that has less congestion.
In another embodiment, the user chooses to turn off advanced navigation features. In another embodiment, the user chooses to release the local database of origins and destinations to the central system. In another embodiment, the user chooses to prevent the local system from sharing its local database of origins and destinations with the central system.
In another embodiment, the vehicle powers the local system, including but not limited to many popular personal navigation devices (PNDs) and quite frequently mobile devices that the user plugs in. When the vehicle is powered down this state change can also be observed. In this way, the local system can distinguish that the vehicle is parked rather than in an idle state by inferring a charging state to imply not parked and a not charged state to imply parked.
In another embodiment, if the onboard computer of the vehicle is powered down because the vehicle is turned off then the onboard computer system can distinguish the vehicle is parked rather than in an idle state.
One of the leaders in generating traffic data today is Google. Their traffic system crowdsources traffic data from cell phones using both cell phone tower triangulation data and GPS data. In particular, their GPS data is sourced from users using Google Maps.
Google's crowdsourcing mechanism while industry leading suffers from a number of weaknesses. For starters, users must manually start the Google Maps application. The majority of Google Maps users only initiate this application when driving to unknown destinations. The times a driver drives to an unknown destination is a small fraction of the time a driver is inside a vehicle. As a result, this weakness prevents Google from sourcing data from a large number of users that have their application installed.
Another weakness of the Google Maps application is that it is power intensive. When users initiate the application, it continuously runs GPS for navigation purposes. While useful at times, leaving GPS on and constantly polling for updates is power intensive and quickly drains a limited power supply. As a result, drivers choose to not leave this application running in the backgrounds of their phones.
Another weakness of the Google Maps application for crowdsourcing traffic data is that it has an associated graphical user interface that downloads map data and other types of data. For traffic crowdsourcing, this is not a necessary feature at all times and it unnecessarily consumes battery power. These weaknesses present an opportunity to develop a more effective traffic crowdsourcing platform.
In one embodiment, the local system automatically turns on upon determining vehicle context to generate traffic data. By automatically activating its traffic recognition algorithm, the local system saves the user from having to start the application to contribute to the overall system.
In another embodiment, the local system's traffic generating service may run in the background of the device, thereby not interfering with the user's experience. In another embodiment, the local system prompts the user and determines if the user will allow this service to continue running in the background of the user's device for situations in the future when the user is inside a vehicle. In another embodiment, the local system prompts the user to allow the service to run starting at boot time of the device. In another embodiment, the local system uses less power intensive sensors than GPS to reduce power consumption and collect data, thereby improving the user experience, and uses GPS only at key events. In so doing, it generates smart data collection techniques.
In another embodiment, the local system initiates certain navigation features such as routing and alerts based on pertinence to the user. In another embodiment, the local system only uploads data to the central system if it is an adverse condition. In another embodiment, the local system only uploads data to the central system if it conflicts with information sent from the central system.
In another embodiment, local system encrypts uploads to the central system, and vice versa. In another embodiment, the local system compresses uploads to the central system, and vice versa, to reduce bandwidth usage.
In another embodiment, the local system sends periodic updates to the central system once vehicle context is determined. A distinguishing feature of this embodiment from the Google Maps application is that the user must manually activate Google Maps. In this embodiment, the service runs automatically and after determining vehicle context, automatically updates the central system with traffic flow data periodically without user intervention.
In another embodiment, the local system sends contextual updates to the central system once vehicle context is determined. For example, relevant contextual situations include but are not limited to each time a user makes a phone call, each time the vehicle stops, each time the user moves the phone beyond or below a rotational threshold, each time the device accelerates beyond or below a threshold, the each time the user approaches a point of interest, each time the device changes state, including but not limited to charging, not charging, phone call, display on, display off, etc., each time the device determines vehicle entry, each time the device determines vehicle exit, each time the user interacts with the device, each time the device approaches a point of interest, each time a time of interest is identified, etc.
In one embodiment, the same algorithms will be used for identifying context on multiple devices. In another embodiment, different algorithms will be used for identifying context on multiple devices. Different devices, even of the same model number, sometimes exhibit different sensing properties. In particular, the degree of noisiness or accuracy of the sensor can vary. To accommodate for this difference, different algorithms must be developed. These algorithms may be based on the noisiness of the device but may also be based on other factors, such as sensor capabilities. To determine the noisiness of the device, the device is calibrated.
In another embodiment, the device is calibrated based on the model number. This implies a variety of factors under consideration for the calibration, including but not limited to sensor sampling rates, capabilities of the sensors, and even the existence of the sensors themselves. For example, one device may have a gyroscope while another may only have an accelerometer.
In another embodiment, the device is calibrated by listening to the noise during a period of likely known behavior. For example, at 4am in the morning, it is likely the device is still because the owner is sleeping. During this time, the sensors may be activated and the noise may be sampled to enable calibration.
In another embodiment, thresholds can be applied to the sensor data to ensure the device is being calibrated to noise and not movements associated with user interaction. For example, acceleration beyond a threshold or rotation beyond a threshold imply the device is moving due to user movements and not noise alone.
In another embodiment, the state of the device is observed to ensure the user is not interacting with the device. For example, if the touch screen is being engaged by the user, the device can determine that and share that state with the algorithm to inform the calibration algorithm.
Another embodiment relates to a system and method for determining context of an object in motion, and more particularly is related to the use of a portable communication device for determining motion related context.
Application publishers that develop mobile applications for smart phones generally attempt to determine contextual awareness to facilitate application development. As shown in
Determining mode of transportation is of particular value for a wide array of mobile applications. Some valuable applications that would benefit from accurate context awareness include the ability to change the state of phone to prevent unfavorable user behavior like texting while driving. Another sample application would determine vehicle congestion and communicate that information to a traffic server. Yet another sample application would change the state of the phone into “driver mode” that would make certain functions more easily accessible while others could be less easily accessible.
To date, most methods for determining vehicle awareness have been through the use of location technologies such as GPS, A-GPS, Wife Position System (WPS), and Cell Phone Tower triangulation. Historically, the general approach has been to calculate periodically the location of a phone and then determine the speed of movement. Assuming fairly high speed, one can infer the device has been inside a moving vehicle.
Using location based technologies to determine vehicle context has numerous weaknesses. Accuracy and frequency of location information has bandwidth, power, and privacy implications. In addition, certain vehicle situations such as a vehicle inside a city become substantially more difficult to identify since the vehicle does not travel very far within a fixed amount of time. Finally, the delay in determining context has substantial implications for applications that desire to implement solutions based on this knowledge.
Others have attempted to determine mode of transportation through pure acceleration analysis. Smartphones have accelerometers that sense acceleration movements, and based on their movements, context can often be determined. Typical approaches using this information leverage the magnitude of acceleration and patterns of movement embedded in the data to determine the context. The greatest challenge with only using acceleration of a device is that modes can often be mistaken. For example, bicycling can be mistaken for being inside a motorized vehicle, running can easily be mistaken for walking, etc.
Embodiments provide a system and method for contextual awareness of a portable device, including but not limited to a smartphone, watch, laptop, etc. The present system and method increases the accuracy and reliability of contextual awareness in movement related situations and particularly for determining mode of transport.
Using acceleration and orientation data (six degrees of freedom), it is possible to simulate every motion of an object in 3-d space. Based on a digital simulation, it is possible to associate movements with context. Based on the contextual understanding of the object in motion, it is possible to change the state of the object as well as communicate with an external system.
Orientation data may be calculated from acceleration data if the acceleration data is provided from an accelerometer that provides data in 3 dimensions and the acceleration reflects the components of gravity in each dimension. Alternatively, orientation data may be obtained directly through the use of a sensor other than an accelerometer, such as a gyroscope.
The type of context one attempts to compute influences the quality of orientation data required and whether the source can be an accelerometer or a gyroscope. Regardless of the source, error correction and noise reduction techniques must typically be applied to sensor information to effectively infer context.
Acceleration data in combination with orientation data can be used to determine context with much greater accuracy than simply acceleration information alone. Depending on orientation and acceleration, it is possible to determine highly accurate context associated with device motion. In addition to leveraging this information, the timeframe over which this data is captured and analyzed also influences the type of context that can be computed, which varies from micro-context movements such as device in hand, device resting, device next to ear, to macro-context movements such as device is with a walking person, device is with a running person, device is inside a car, etc. This approach provides a method of contextual awareness that for some situations, like whether the phone is inside a moving vehicle, is accurate up to 99% of the time.
Accurately determining context provides tremendous value. For example, in one embodiment the state of the device may change such that it limits or enables certain functionality. Examples of this embodiment include preventing texting while the device is inside a moving vehicle, preventing an incoming call while the device is inside a moving vehicle, and enabling “car mode” which facilitates certain features to be more easily accessed (i.e. navigation).
Another benefit of determining context is communicating with another system the change in state. For example, in one embodiment a device could upload fitness statistics to an external system, and this information could be used on the micro level, specifically to inform a single individual, or on the macro level to provide general statistics about movements about a specific population.
There are many other benefits that could be envisioned based on understanding context of a device. Embodiments, which uses acceleration and orientation data to simulate motion digitally, include contextual awareness for smartphones, such as a smartphone is with a person who is: Holding the phone, Walking, Running, Bicycling, Dancing, Walking up stairs, Inside a motorized vehicle, Inside a motorized vehicle that is in traffic, Inside a motorized vehicle that is on a highway, Etc.
Using both acceleration and orientation, it is possible to create state diagrams that indicate many contextual situations associated with movement because the accuracy is of sufficient quality that the change in states can be reliably determined.
The above state determination of whether a device is inside a vehicle is performed using acceleration and rotation data. In one embodiment, when the smart phone has no pitch and no roll yet has acceleration beyond a threshold for a certain period of time, this situation implies that the device is highly likely inside a moving vehicle. In another embodiment, to infer the device is outside the moving vehicle, excessive rotation (i.e. pitch and/or roll) is typically observed for an extended period of time.
Assuming the device is inside the vehicle (determined using the method above), then in another embodiment the magnitude of the acceleration may be observed to infer the context of the vehicle. If the acceleration of the device is approximately the acceleration of gravity, it can be inferred with a high degree of certainty that the vehicle is Idle. If the acceleration is substantially more than or less than the acceleration of gravity (again, without rotation) then this implies the vehicle is Moving.
In another embodiment, if the vehicle oscillates back and forth between Idle and Moving states for a period of time, this implies a Traffic event. The greater the times associated with this back and forth Idle and Moving states, the greater the intensity of traffic. In another embodiment, if the vehicle is moving for a substantial period of time, it can be inferred that the vehicle is no longer in traffic.
In another embodiment, if the device is left inside the vehicle, then the excessive rotation associated with it leaving that environment will not take place. In such a situation, the device will reflect an idle vehicle for an excessive period of time. Beyond some reasonable threshold for no movement, then in another embodiment it can be sufficiently inferred that the vehicle is in fact parked and not in an idle or traffic event.
As additional embodiments, the following approaches are methods for computing a Vehicle Stopped Context, Vehicle Moving Context, Vehicle Going Forward Context, Vehicle Turning Context, and Vehicle Braking Context, once the device has been determined to be inside a vehicle.
In another embodiment, geographic context can sometimes be determined without determining exact location. For example, lack of a vehicle stop event for a given period of time indicates with substantial probability that the vehicle is likely on a freeway or highway. There are multiple methods for computing acceleration data. It should be noted though that one of the major benefits of using the magnitude of acceleration for determining state is that state can be determined independent of rotation associated with the surrounding environment. An example of this is the method for determining an Idle state for a vehicle, described previously.
To accommodate for noise associated with acceleration from a smartphone when integrating acceleration to obtain velocity of a vehicle, it is possible to “zero” out small changes in acceleration. In particular, if the acceleration is relatively constant for a period of time, a real world observation can be applied that such an event does not occur unless the device is still. As such, relatively constant acceleration for a period of time implies zero movement or a speed of zero for the associated vehicle. This same method can be used to also determine the speed of the phone, which implies the speed of the surrounding context in which the phone is moving.
Depending on the noise in the environment, the predefined thresholds for a zero may not be appropriate. In another embodiment, it may be necessary to have adaptive zeros. Adaptive zeros imply relatively constant acceleration within some acceptable bandwidth. If a predefined zero is not being observed for some reasonable timeframe, the system may review previous data and determine that a more generous threshold needs to be determined. In another embodiment, this method can also be referred to as calibration in a new environment.
Simply put, calculation for speed may not occur until the device has calibrated itself to the current environment. So in another embodiment, the algorithm here is first determine the context as being inside a vehicle, then calibrate the zero threshold based on a small but relatively constant acceleration, and then begin computing speed whenever the acceleration changes more than the allowed threshold.
In one embodiment, it is possible to infer the speed of the vehicle using a combination of acceleration and orientation information. To do this accurately, the orientation data must be of sufficient quality, so as to accommodate for rotation associated with hills and turns. If the device has a gyroscope, the noise of the gyroscope must be removed. One method for removing noise from a gyroscope is to ignore rotation below a given threshold. In such a way, when the device is actually rotated, the majority of the rotation is captured but the noise associated with rotation of the gyroscope's movements is removed.
In one embodiment, the same thresholds will be used for identifying context on multiple devices as a method for supporting multiple devices. In another embodiment, different thresholds will be used for identifying context on multiple devices. Different devices, even of the same model number, sometimes exhibit different sensing properties. In particular, the degree of noisiness or accuracy of the sensor can vary. To accommodate for this difference, different thresholds must be developed. These thresholds may be based on the noisiness of the device but may also be based on other factors, such as sensor capabilities. To determine the noisiness of the device, the device is calibrated.
In another embodiment, the device is calibrated based on the model number. This implies a variety of factors under consideration for the calibration, including but not limited to sensor sampling rates, capabilities of the sensors, and even the existence of the sensors themselves. For example, one device may have a gyroscope while another may only have an accelerometer.
In another embodiment, the device is calibrated by listening to the noise during a period of likely known behavior. For example, at 4am in the morning, it is likely the device is still because the owner is sleeping. During this time, the sensors may be activated and the noise may be sampled to enable calibration.
In another embodiment, thresholds can be applied to the sensor data to ensure the device is being calibrated to noise and not movements associated with user interaction. For example, acceleration beyond a threshold or rotation beyond a threshold imply the device is moving due to user movements and not noise alone.
In another embodiment, the state of the device is observed to ensure the user is not interacting with the device. For example, if the touch screen is being engaged by the user, the device can determine that and share that state with the algorithm to inform the calibration algorithm.
Exemplary embodiments include a system and method for intelligently determining the location of a vehicle. Numerous applications charge for helping users determine the location of a user's vehicle. The general workflow requires a user to mark the location of the vehicle upon its exit. To do so, the user must open the application, input a request to mark the location, and then the application must obtain a location reference, typically a GPS fix. In so doing, the application has now marked the user's vehicle for later reference.
When the user returns to the area where the vehicle was left, the user then must open the application and request the application to find the vehicle. This is typically performed by placing a marker on a map or radar screen indicating the vehicle and typically another marker on the same screen indicating the location of the user.
One of the major weaknesses of these applications involves the user having to input the original location of the vehicle. Users typically forget this step and then the application is unable provide value. In addition, the act of marking the location is often considered a hassle and additional work for the user.
Exemplary embodiments include a system and method for intelligently determining the location of a vehicle without the user inputting its location.
In a preferred embodiment, an application would automatically determine the user is inside a vehicle. Upon exiting the vehicle, the application would automatically observe the user's location and mark it. The user, upon a desire to find the location of the vehicle, could then access the application and request the location of the vehicle.
In one embodiment, the application would be a smartphone application would continuously run in the background of the phone determining the user's status of whether the user the is inside a vehicle. Upon determining the user is inside a vehicle, the application would wait until the user has exited to mark the location of the vehicle. This application's graphical user interface would be brought to the foreground whenever the user would access the application to observe the location in a graphical manner.
In another embodiment, the application would run on a wearable device, such as a watch. In another embodiment, the application would run in the onboard computer system of the vehicle and update a website that could be accessed by the user. In another embodiment, the application would run on a device wirelessly connected to the vehicle. In another embodiment, the application would be running on a portable device, including but not limited to a smartphone, a watch capable of running applications, a tablet computer, and a personal laptop. In another embodiment, the application would be running on a device physically connected to the vehicle, such as a smartphone or a personal navigation device.
Automatically determining vehicle entry and vehicle exit, precursors for determining when the application should mark the location of the vehicle, are components to the application.
Determining the appropriate method to observe car exit and obtain a location fix depends on the device on which the application is running In one embodiment, the application would run on the onboard computer system of a vehicle. In this application, the system would mark the location of the vehicle each time the vehicle is turned off. The application would then update an external system, such as a website server, so that this information could be later accessed by the user.
In one embodiment, the application would run on a smartphone. Using a smartphone, the application could determine vehicle context using a variety of methods including but not limited to using only acceleration data, acceleration data and rotation data, and acceleration data combined with rotation data combined with location data. Using only acceleration data, it is possible to observe the data directly and perform a Fast Fourier Transform on the data to identify frequencies that represent vehicle mode of transport. Using both acceleration data and rotation data, it is possible to observe acceleration magnitude above a threshold and rotation below a threshold to infer vehicle context. Using only location data, it is possible to infer vehicle context by the speed of the device. If the device is moving above a threshold, vehicle context can be inferred. Periods of continued vehicle context interposed with short periods where the device is moving below a threshold can be determined to be a continuous vehicle journey. Extended periods below a threshold can imply vehicle exit. Combinations of these methods can also be used to increase accuracy of the vehicle entry and vehicle exit. Vehicle exit could also be inferred by observing a walking behavior or a running behavior.
In another embodiment, the device is physically connected to the vehicle. Examples of this embodiment include but are not limited to a smartphone being charged by the vehicle and a personal navigation device being charged by the vehicle. In this embodiment, one method for determining vehicle entry and vehicle exit is the power state of the vehicle. If the vehicle is running (i.e. in a turned on state), it can be implied that the user is driving the vehicle and has entered the vehicle. If the vehicle is not running (i.e. powered off), it can be inferred that the user is no longer driving the vehicle and has exited.
In another embodiment, the device is wirelessly connected to the vehicle. In this example, one method for determining vehicle entry and vehicle exit is the state of the wireless connection. A connected wireless state would imply the user has entered the vehicle. A disconnected wireless state would imply the user has exited the vehicle.
In another embodiment, combinations of the previous embodiments would be used to more accurately determine vehicle entry and vehicle exit. It another embodiment, this system would be used to issue traffic alerts while the application is running and vehicle context has been confirmed.
The following embodiments are new subject matter relative to the previous filings. In another embodiment, the contextually aware mobile device is an auditory device that fits on or inside the ear. The auditory device may have a rather large visual cue, such as in the form of a light, that indicates usage. Although the user of the device may not be able to see the visual cue, this capability could be enabled for guests approaching the user wearing the device. These guests would be able to identify the user's availability based on the visual cue. By way of example, in one embodiment the visual cue could be a light that changes color based on activity. For example, the light could be green if the user is free to speak, red if the user is on a phone call, and blue if the user is listening to music, and off (disabled) if the device is off. Furthermore, the user could manually set the color of this light so as to automatically communicate externally their state, such as them being busy or free.
In another embodiment, the user could use this auditory device to connect with other devices, such as their watch or their smartphone. Using physical interaction by way of direct touching or motions, the auditory device could change state of itself or any connected devices. For example in one embodiment, the user could activate a third party application such as Apple's Siri by pressing a button or touching the side of the auditory device. In another embodiment, the user could gesture using hand motion leveraging motion sensors and proximity sensors to activate the third party application. In another embodiment, the user could use physical head motion, such as moving one's head up and down to initiate and interact with an application, and leverage motion sensors within the device including but not limited to accelerometers and gyroscopes. Furthermore the speed and timing of the motions could also be leveraged to enact specific behavior at times of interest. For example in another embodiment concerning a music application, turning one's head slowly to the left could decrease the music volume but turning one's head quickly to the left (i.e. jerk) could cause the previous song to be played.
In another embodiment, the auditory device keeps the microphone active at all times. There are many benefits of keeping the microphone sensor active. For example, the user may activate applications on the auditory device or devices connected to the auditory device, like a watch or smartphone. Furthermore the auditory device may itself be a screenless smartphone or it could also be an auditory smartphone, i.e. a smartphone that is entirely encapsulated through an in-ear device. In another embodiment, the auditory device reduces the volume of a music application automatically when there is external noise, such as from a guest speaking.
In another embodiment, the local system downloads software in advance of its scheduled release date but does not allow it to be either installed or installed but not activated until the official date and launch time. A number of systems have been known for slowing down during major releases excessive user traffic. By allowing users to download software in advance, but not activate the software, the integrity of the marketing campaign is maintained. Furthermore, automatic push downloads can further ration installations so there is not excessive traffic during any particular period of time.
In another embodiment, the central system activates the local system's sensors for identifying real time data. For example, the Find My Phone application currently allows users to activate the GPS of the device in the local system to determine its whereabouts. However, this application could be further enabled to activate the camera so the user in the central location can see from the perspective of the device in the local system. Furthermore, the microphone and speaker could also be activated so as to make the device in the local system easier to find or to allow the user in the central system to communicate with the user in the local system who may have the device.
Claims
1. A method for enabling an automatic checkout of a user associated with a mobile device comprising:
- monitoring the mobile device for a first time interval during a first period of time;
- calculating one or more characteristics of the mobile device during the first time interval;
- comparing the one or more characteristics to a first set of threshold values;
- determining a state of the mobile device based on the comparison of the one or more characteristics to the first set of threshold values, wherein the state of the mobile device comprises an at-destination state and an in-transit state; and
- based on determining a change of the state of the mobile device from the at-destination state to the in-transit state, performing an automatic checkout.
2. The method of claim 1, wherein the one or more characteristics of the mobile device include acceleration.
3. The method of claim 1, wherein the one or more characteristics of the mobile device include rotation.
4. The method of claim 1, wherein the one or more characteristics of the mobile device include location.
5. The method of claim 1, wherein performing the automatic checkout includes the location of the mobile device.
6. The method of claim 1, wherein performing the automatic checkout includes updating another system.
7. The method of claim 1, wherein performing the automatic checkout includes asking for user approval.
8. The method of claim 1, wherein prior to performing the automatic checkout a check-in is performed.
9. The method of 8, wherein performing the automatic checkout includes calculating a time period between the check-in and the automatic checkout.
10. A method for enabling an automatic check-in of a user associated with a mobile device comprising:
- monitoring an acceleration of the mobile device for a first time interval during a first period of time;
- calculating one or more characteristics of the acceleration of the mobile device during the first time interval;
- comparing the one or more characteristics of the acceleration to a first set of threshold values;
- determining a state of the mobile device based on the comparison of the one or more characteristics of the acceleration to the first set of threshold values, wherein the state of the mobile device comprises an at-destination state if the acceleration is less than a first minimum threshold and an in-transit state if the acceleration is more than the first minimum threshold; and
- based on determining a change of the state of the mobile device from the in-transit state to the at-destination state, performing an automatic check-in.
11. The method of claim 10, further comprising calculating one or more characteristics of the rotation of the mobile device.
12. The method of claim 10, further comprising calculating one or more characteristics of the location of the mobile device.
13. The method of claim 10, wherein performing the automatic check-in includes the location of the mobile device.
14. The method of claim 10, wherein performing the automatic check-in includes updating another system.
15. The method of claim 10, wherein performing the automatic check-in includes asking for user approval.
16. A method for automatically determining a user associated with a mobile device has exited a vehicle comprising:
- monitoring the mobile device for a first time interval during a first period of time;
- calculating one or more characteristics of the mobile device during the first time interval;
- comparing the one or more characteristics to a first set of threshold values;
- determining a state of the mobile device based on the comparison of the one or more characteristics to the first set of threshold values, wherein the state of the mobile device comprises an in-vehicle state, a non-vehicle state; and
- based on determining a change of the state of the mobile device from the in-vehicle state to the non-vehicle state, calculating the user associated with the mobile device has exited the vehicle.
17. The method of claim 16, wherein the one or more characteristics of the mobile device include motion.
18. The method of claim 16, wherein the one or more characteristics of the mobile device include location.
19. The method of claim 16, wherein the one or more characteristics of the mobile device include Bluetooth connectivity.
20. The method of claim 16, wherein the one or more characteristics of the mobile device include sound.
Type: Application
Filed: Sep 23, 2013
Publication Date: Mar 26, 2015
Inventor: Amit Goyal (Stone Mountain, GA)
Application Number: 14/034,370
International Classification: H04W 12/08 (20060101); H04W 4/02 (20060101);
