Mobile Image Capture System
A mobile image capture system, a system comprising: a sensing unit for attaching to a vehicle, the sensing unit having a camera constructed and arranged to view a participant on the vehicle, the camera capturing at least one image; and processing electronics for storing data representing the captured at least one image or for relaying data representing the captured at least one image to a computer or a network.
This application is a continuation of U.S. patent application Ser. No. 14/281,230 filed on May 19, 2014, to be issued as U.S. Pat. No. 10,147,265 on Dec. 4, 2018, which is a continuation of U.S. patent application Ser. No. 12/874,942 filed Sep. 2, 2010, now U.S. Pat. No. 8,731,865 issued May 20, 2014, which is a continuation of U.S. patent application Ser. No. 11/646,799 filed Dec. 28, 2006, now U.S. Pat. No. 7,840,378 issued Nov. 23, 2010, which is a division of U.S. patent application Ser. No. 09/607,678 filed Jun. 30, 2000, now U.S. Pat. No. 7,739,076 issued Jun. 15, 2010, which claims the benefit of U.S. Provisional Patent Application No. 60/201,544 filed May 3, 2000 and claims the benefit of U.S. Provisional Patent Application No. 60/141,794 filed Jun. 30, 1999, each of which are expressly incorporated by reference herein in its entirety.
This application is also related to commonly-owned and co-pending U.S. patent application Ser. No. 14/065,019 filed Oct. 28, 2013, which is a continuation of U.S. patent application Ser. No. 12/781,562 filed May 17, 2010, which is also a continuation of U.S. patent application Ser. No. 09/607,678 filed Jun. 30, 2000, now U.S. Pat. No. 7,739,076.FIELD OF THE INVENTION
This invention relates to sports measurement sensors, event systems, and video systems; more particularly, the invention relates to various sports measurement metrics detected by sensors and relayed to an event system or personal display device and the production and use of video for spectator and/or training purposes.BACKGROUND OF THE INVENTION
Sports participants, whether professional or amateur, as well as spectators desire more information about the performance of an athlete. United States Patent Application, entitled “Apparatus and Methods for Determining Loft Time and Speed,” U.S. Pat. No. 5,636,146, by Peter Flentov, Dennis M. Darcy, and Curtis A. Vock, assigned to PhatRat Technology, Inc., filed on Nov. 21, 1994, issued on Jun. 3, 1997, and incorporated herein by reference provides some systems and methods for quantifying airtime and speed for athletic performance, especially in the sports of skiing and snowboarding.
Patent Cooperation Treaty (PCT) Application, entitled “Sport Monitoring System for Determining Airtime, Speed, Power Absorbed and Other Factors Such as Drop Distance,” PCT Publication No. WO 98/54581, by Curtis A. Vock, Dennis M. Darcy, Andrew Bodkin, Perry Youngs, Adrian Larkin, Steven Finberg, Shawn Burke, and Charles Marshall, assigned to PhatRat Technology, Inc., filed on Jun. 2, 1998, published on Dec. 3, 1998, and incorporated herein by reference provides some additional systems and methods for quantifying athletic performance.
However, athletes and spectators desire new, quantifiable performance metrics, enhanced events systems, and use of visual images. For example, currently photographers can be found on the ski slopes at either the top or the bottom taking pictures, which can be later purchased at the end of the day from the Lodge. Whilst these are usually good quality photographs, they are not action images. Needed are new methods and apparatus to record a users performance from an action point of view as well as for other perspectives, and to distribute these recorded still and video images and video for entertainment and training purposes.SUMMARY OF THE INVENTION
On embodiment of the invention includes a system comprising a sensing unit for attaching to a vehicle and processing electronics. The sensing unit has a camera constructed and arranged to view a participant or the vehicle, with the camera capturing at least one image. The processing electronics stores data representing the captured at least one image or relaying data representing the captured at least one image to a computer or a network.
The appended claims set forth the features of the invention with particularity. The invention, together with its advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
Methods and apparatus are disclosed for detecting and measuring performance characteristics and metrics of participants and vehicles. These performance characteristics and metrics include, but not are limited to, airtime, g-force, spin, rotation, drop distance, acceleration, and video and still images. These vehicles include, but are not limited to a snowboard, ski, skateboard, wakeboard, motorcycle, bicycle, ice skates and rollerblades.
One embodiment provides a camera for providing near real-time images and video footage of a participant's actions on a vehicle. The camera may be located on the participant, the participant's vehicle or other equipment, or from some other observation point. The images recorded by the camera can be downloaded to a recording or other storage device to produce memorabilia (e.g., a CD ROM, or video cassette). If desired, the images can be sent in real-time through an event system and network (e.g., using a radio or other transmitter) to television, the Internet, and to other locations for producing the memorabilia or for providing images to television display devices, such as those located in a ski lodge for entertainment purposes or in a coach's or personal trainer's office for training purposes.
For example, a camera may be attached to a snowboard or user for recording a user's performance. The camera should be easily but securely attached to the user's vehicle or body. Multiple cameras can be used to record multiple views simultaneously, such as a view of the user, a forward and a reverse view. The recorded images can be then be optionally digitally processed, and then recorded onto a compact disc for playback on the user's personal computer.
One embodiment provides a system that monitors and tracks vehicle action for teaching and training purposes. For example, a sensing unit (e.g., airtime sensor, etc.) may be attached to a skateboarder so that real-time and delayed data can be determined in a skateboarding training exercise or event. Further, a sensing unit and/or data unit may include one or more translational and/or rotational accelerometers to provide additional information such as, but not limited to, maximum rotation of the vehicle, rotation of the person relative to the vehicle, flip information, scraping information (e.g., one side of the vehicle relative to the other side of the vehicle), and a time duration that a vehicle is on its side or at an edge of a ramp.
Sensing units typically contain one or more transducers with suitable conditioning, filtering and conversion electronics. They typically also contain a processor, a data logging system and primary and secondary communication channels. Their purpose is to measure and record a parameter or range of parameters for a participant's performance and communicate the results to an event system or personal display device (e.g., watch, pager, cell phone, PDA, etc.). When sensing units are used in an event or resort/park situations, they typically transmit their results to a base station either directly or via a relay. For personal use, sensing units typically either transmit or display their results to a personal display unit integrated into the sensing unit or on a receiving device (e.g., watch, pager, cell phone, PDA, etc.). In one embodiment, the primary communication channel will typically be a one way radio frequency link or direct cable connection, which is used to transmit data to the rest of the system. A secondary bi-directional infrared link may be included, which allows administration and control of the sensing unit and also provides a path for the logged data to be downloaded.
One embodiment provides airtime and other information (e.g., performance metrics) related to Baja racing or other wheeled vehicles, in real-time, if desired, to television, event systems or judging centers, and/or the drivers of these vehicles. An embodiment uses a sensor that mounts to the vehicle in one or more places to monitor the airtime for one or multiple wheels. Various embodiments employ contact closures, stress sensing devices, accelerometers, and/or devices that measure the position of a shock absorber or coil spring for a wheel of the vehicle.
As used herein, computer-readable medium is not limited to memory and storage devices; rather computer-readable medium is an extensible term including other storage and signaling mechanisms including interfaces and devices such as network interface cards and buffers therein, as well as any communications devices and signals received and transmitted, and other current and evolving technologies that a computerized system can interpret, receive, and/or transmit.
Sensing device(s) 135 may include accelerometers, stress sensors, magnetic field sensors, peizo foil sensors, pressure sensors, contact closures, global positioning system (GPS) devices, strain gauges, microphones, clocks, spectra, or any other sensing and/or measurement device. The exact device(s) incorporated into a sensing device 135 will typically correspond to the type of measurement desired. For example, magnetic field sensors and accelerometers, alone or in combination, can be used to measure rotation.
Each sensing unit 130 may contain a data logging data structure in memory 132 or storage devices 133, which will be used to record the performance data generated by a competitor during a run. It typically will have sufficient capacity to hold the data for an entire run. This performance data stored in this data structure can be extracted at the end of each run. One embodiment of this data structure uses a FIFO principle; hence it will be self-maintaining and need not be interrogated should this be found inconvenient or unnecessary.
In the limited cases where data is lost during a competitors run then each sensors can be interrogated immediately on completion of that run. Live data collected by each sensor unit will normally be transmitted in real-time through an event system in order that judging can take place as the action is happening and also so that a live feed of performance information can be provided to TV or other medium, e.g., Internet or radio. Should a sensing unit 130 be unable to communicate through its primary communication channel then the accumulated performance data held by the sensors logging sub-system can be data can be download when the competitor has completed his/her run. This would take place using a secondary communication channel implemented with a different signaling technology. Typically, the primary communication channel with uni-directional (transmit only), the secondary channel will be bi-directional and used for downloading data from the logger sub-system and uploading one time pads.
Should the failure of a sensing unit 130 be more severe then unit can be open and the logging sub-system be downloaded directly. Each unit in the data chain will have the facility to download its data via secondary link using an alternative signaling system. In most case the units will be using radio frequency or RS232/RS485 as their primary medium of communication. In addition, a sensing unit 130 may have the capability to download its data via a secondary data link, such as infrared signaling. This would normally be carried out each time the a run has been completed.
Sensing units 130 typically transmit use a cyclic redundancy checksum (CRC) as part of a message so a relay unit or base station can detect a transmission error. In some embodiments, one or more error correction techniques (e.g., forward error correction) are used, which may allow corrupted data to be automatically corrected. A sensing unit 130 can use bi-directional communication techniques, but typically sensing units 130 only transmit their data in a datagram fashion, so no acknowledgement is received. Therefore, a sensing unit 130 will typically transmit each data packet several times to increase the probability of the message being properly received by an event system.
Many different methods are employed by a sensing unit 130 to determine a performance metric, such as airtime. In one embodiment, the sensor signal is filtered to give a cutoff frequency well below the Nyquist frequency for the sampling rate of 9600 Hz. The signal is typically sampled using an eight-bit analogue to digital converter. The 9600 bytes of information per second are preferably reduced to a more manageable level of 40 bytes per second by a pre-processing algorithm. The absolute difference of the current sample value from the previous sample value is, for example, accumulated for 240 values into a 16-bit number. Due to the high sample rate and the low frequency signal, the difference is always relatively small, and the 16-bit accumulator does not overflow. After 240 sample differences have been accumulated, the sum is divided by four and limited to 255. This value gives a ‘signal activity level’ for the 25 ms period. This technique effectively ignores low frequency signal content and any digital offset component. These values are fed into two Infinite Impulse Response (IIR) digital filters to determine if the vehicle is moving and if the vehicle is in the air.
Certain flags can be used in determining a performance metric. By way of example, the Motion IIR accumulator is 16-bits. The 8-bit signal activity level value is added in, and then the accumulator is reduced by 1/32nd of its current value. If the accumulator level is above a ‘Motion Threshold’, the vehicle is deemed to be in motion. The Air IIR accumulator is 16-bits. The 8-bit signal activity level value is added in, and then the accumulator is reduced by of its current value. If the accumulator level is below the ‘Air Threshold’, the vehicle is deemed to be in the air. A landing thump is flagged when the signal activity level is higher than the ‘Thump Threshold’.
The above flags are monitored and the following algorithm determines if airtime is valid. In one embodiment, the rules for valid airtime are straight forward: the board must be in motion before the airtime starts; the board may be in motion after the airtime ends; a maximum of 5 seconds of airtime is recognized (for a typical event or competition); valid airtime ends with a Thump (i.e., a landing). Pseudo code for one embodiment is illustrated in
Certain embodiment employ certain enhancements, such as to help limit the effect of different signal levels on the algorithm outputs, the output value from the preprocessing can be limited to a certain value before being applied to the IIR filters. This limits the range of the filters, and restricts the effect of large signal inputs.
For certain events and embodiments, multiple sensing units 130 may be attached to participants and their vehicles. These multiple sensing units 130 may measure different performance metrics, or measure one or more of the same metrics as to provide some level of redundancy.
In one embodiment, sensing units 130 transmit a short block of data at relatively long intervals, for the remainder of the time the transmission band is free. By assigning different repeat patterns to each sensing unit 130 and repeating the same data a number of times then data loss due to overlapping messages can be virtually eliminated. In some embodiments, spread spectrum technology is used which typically provides higher reliability and security.
In one embodiment, each sensing unit and data link within an event system will facilitate or make use of encryption techniques to ensure the system cannot be subverted to the advantage of third parties such as competitors or gambling syndicates etc. The performance data in the system may be encrypted. In addition to, or in place of this encryption, Message Authentication Codes (MACs) may be included in the data streams. The MACs will accompany the data at all stages and locations within the event system including logging subsystems. The MACs will be used by a control center within an event system to establish the authenticity of any performance data received. In one embodiment, the performance data generated by a sensor unit within the event system will be grouped into blocks, a MAC will be generated for each block of data using that data. The MAC generation will be carried out by and within the sensor unit producing the data. The MAC will be an encrypted value derived from all the data within the block.
Additionally, in one embodiment, a system of One Time Pads (OTPs) is used to encrypt the Cyclic Redundancy Checksum (CRC) to generate the MAC instead of the processor intensive method common in standard encryption systems. Each byte of data within the data block will be used to generate the CRC for the block in addition a number of randomly selected bytes from the data block will be including in the CRC calculation a second time. This will prevent a third party from deriving the value of the entry of the OTP used to encrypt the CRC then using this information to generate a valid block of data and insert it into network without detection. Each entry in the OTP typically will consist of a pair of random numbers, one of the numbers will be used to select which data item are duplicated in the CRC, the other random number will be used to encrypt. This method allows a high level of data security while imposing a minimal processing burden where resources are at a premium. The OTP consists of a table of random numbers held in both the unit generating the data and the unit receiving the data. The table is unique to these two units and each entry in the table is only ever used once.
The rate at which MACs are included in the data stream, and hence the size of the data blocks, is determined by the amount of non-volatile storage available to hold the OTP and the frequency at which the OTP can be updated. It is not essential that the frequency of MACs is high.
Sensor units 130 may be uploaded with a unique and random OTP in a secure manner prior to each session the field unit might be used in. For this activity a single mobile security broker unit will be used this will generate a full set of OTPs for the entire event system for a session at an event. Each of the control units will be uploaded with a full set of OTPs. Once an OTP is loaded into a field unit and each of the control units it will be erased from the security brokers memory.
In certain embodiments, where radio links are used to transfer data between units, then a suitable transmitter and receiver beam shape will be employed to maximize link reliability. In the case of units in the relay array, a high gain directional antenna will be typically employed with the beam focused within the appropriate section of the event arena. In the case of repeater units, an Omni directional antenna will typically be employed. This embodiment should decrease the probability of a lost transmission even as the participant's orientation varies with respect to the event system.
In one embodiment, an array of m video cameras 221-229, where m is 1 or greater, are placed along the event area 200 or at certain strategic locations (in addition to, or in place of relay units 211-219). Cameras 221-229 communicate with base station 205 via radio signals and/or cable 220 (e.g., using RS 485 protocol). Cameras 221-229 can be used to determine performance metrics, e.g., airtime, etc., by visually inspecting or digital processing the produced images.
The video cameras record events and then relay the events to a base station, which then might forward them to another device, such as a ski lodge video server so people in their rooms or in the lobby or bar can watch the action. In one embodiment, the event system automatically correlates participants having a sensing unit 130 (
Moreover, performance data received from a sensing unit by an event system may be correlated with image data received by the event system. In one embodiment, data received from camera and sensing devices is time-stamped for later correlation and retrieval purposes, and/or marked with data identifying a participant or sensing unit. In one embodiment, the time value associated with at least some of the received performance or image data is adjusted based on a calculated, received, or some predetermined delay value. For example, a sensing unit or camera might add a relative delay time value to data it sends so the event system will be able to determine an “actual” time of occurrence. In this manner, events can be correlated based on a common time reference, such as that of the event system. In another embodiment, the clocks of sensing devices and cameras are routinely synchronized so that they can independently time-stamp data based on a common time reference, which will allow data received from different devices to be correlated.
Sensing unit 360 typically includes a processor 361, memory 362, storage devices 363, one or more magnetic field sensing devices 364, and one or more external interfaces 365 (such as a display or a radio transmitter for communicating with an event system or personal display device). Sensing unit 370 typically includes a microchip PIC with memory 371 (or processor and memory), clock 372, 3-axis magnetic field sensing device 374, optional pitch and roll sensor 376, one or more external interfaces 375 (such as a display or a radio transmitter for communicating with an event system or personal display device), and a battery source 377. The operation of sensing unit 370 is further described by the flow diagrams of
In view of the many possible embodiments to which the principles of our invention may be applied, it will be appreciated that the embodiments and aspects thereof described herein with respect to the drawings/figures are only illustrative and should not be taken as limiting the scope of the invention. To the contrary, the invention as described herein contemplates all such embodiments as may come within the scope of the following claims and equivalents thereof.
1. A mobile data capture system comprising:
- a plurality of sensing units comprising at least a first sensing unit and a second sensing unit, wherein each of the first sensing unit and the second sensing unit comprises at least one sensor that, in operation, generates sensor data based on detected motion of a user while the sensing unit is attached to one of the user or equipment associated with the user;
- a camera that is attached to one of the user or equipment associated with the user and that, in operation, captures a plurality of images of the user while the at least one sensor generates the sensor data; and
- processing electronics that, in operation: receive and store first sensor data from the first sensing unit, second sensor data from the second sensing unit, and image data from the camera; and correlate at least a first image of the plurality of images to a first performance metric and at least a second image of the plurality of images to a second performance metric, wherein the first performance metric is based on the first sensor data and the second performance metric is based on the second sensor data.
2. The mobile data capture system of claim 1, wherein each of the plurality of sensing units communicates via spread spectrum technology.
3. The mobile data capture system of claim 2, wherein each of the plurality of sensing units communicates via a different repeat pattern.
4. The mobile data capture system of claim 1, wherein the first performance metric and the second performance metric are the same metric.
5. The mobile data capture system of claim 1 wherein the first performance metric is airtime.
6. The mobile data capture system of claim 1 wherein the second performance metric is G-force.
7. The mobile data capture system of claim 1 wherein one of the first performance metric or the second performance metric is drop distance.
8. The mobile data capture system of claim 1 wherein:
- each of the first sensing unit and the second sensing unit further comprise an interface for transmitting at least one of the sensor data and the data representing the plurality of images to a computer via a network.
9. The mobile data capture system of claim 8 further comprising:
- an antenna and relay electronics for transferring at least one of the sensor data and the data representing the plurality of images from each of the plurality of sensing units to the to the computer in real-time.
10. The mobile data capture system of claim 1 wherein:
- the processing electronics apply a time stamp to each of the plurality of images based upon a common time reference.
11. A method of correlating image data with performance metrics comprising:
- generating first sensor data using a first sensor unit based on detected motion of a user while the first sensing unit is attached to one of the user or equipment used by the user;
- generating second sensor data using a second sensor unit based on detected motion of the user while the second sensing unit is attached to one of the user or equipment used by the user;
- capturing a first plurality of images of the user using at least one camera during the detected motion of the user and while the at least one camera is attached to one of the user or the equipment used by the user;
- receiving and storing the first sensor data, the second sensor data, and image data representing the first plurality of images by processing electronics;
- determining, using the processing electronics, a first performance metric based on the first sensor data and a second performance metric based on the second sensor data; and
- correlating at least one of the first plurality of images to at least one of the first performance metric or the second performance metric using the processing electronics.
12. The method of claim 11, wherein at least one of the first performance metric and the second performance metric is airtime.
13. The method of claim 11, wherein the at least one of the first performance metric and the second performance metric is G-force.
14. The method of claim 11, wherein at least one of the first performance metric and the second performance metric is drop distance.
15. The method of claim 11, further comprising:
- transmitting at least one of the first sensor data, the second sensor data, and the data representing the first plurality of images to a computer via a network.
16. The method of claim 15, further comprising:
- transferring at least one of the first sensor data, the second sensor data, and the data representing the first plurality of images to the computer in real-time using an antenna and relay electronics.
17. The method of claim 15, further comprising:
- marking, first sensor data, the second sensor data, and image data with information identifying the user.
18. The method of claim 11, further comprising:
- applying a time stamp to each of the first plurality of images based upon a common time reference using the processing electronics.
19. The method of claim 11, further comprising:
- receiving, at a computing device, video captured by an array of cameras, each located along an event area;
- identifying, by the computing device, a user location based on the first sensing unit or the second sensing unit; and
- correlating, by the computing device, video recorded by the array of cameras with the user location.
20. The method of claim 11, wherein the equipment used by the user comprises one of a snowboard, a motor vehicle, or a rollerblade.