Motion Detection Method and System for Training a User in the Performance of a Motion in an Activity

Abstract

A method trains a user in performing a motion for an activity. The method employs an AV-equipped portable computing device positioned so that its camera faces the target. The device's processor executes computer processes that include: receiving and processing an output from the camera to determine an initial position of the target; processing the camera's output to determine a series of coordinates corresponding to positions of the target, assumed after the initial position; processing the series of coordinates, against a set of fault rules, to determine presence of a fault in performing the motion, wherein the fault rules relate to the motion based on target position in a real, physical space; and if a fault is determined, causing notification of the user, so as to train the user to perform the activity in a manner free of the fault.

Description

PRIORITY

This application claims the benefit of U.S. provisional patent application Ser. No. 62/864,114, filed Jun. 20, 2019, and U.S. provisional patent application Ser. No. 62/890,652, filed Aug. 23, 2019. Each of these application is hereby incorporated, in its entirety, by reference.

TECHNICAL FIELD

The present invention relates to computer-implemented methods for training a user in performing a motion in the course of an activity.

BACKGROUND ART

In order for a user to train a motion for an activity, it is useful to understand the position of important points of the user's body or equipment throughout the motion. For example, to develop a good golf swing, a golfer should restrict motion of the head within a boundary until after the ball is struck. Unfortunately, it is difficult to detect these types of movement in a simple, accurate, non-invasive and real-time manner and without having to train the system.

Some prior art systems use a smartphone's camera to determine an initial position of the head of a user, and allow a user to set boundaries around the motion of the head. But these boundaries are simply lines on the smartphone screen and are not determined based on distances in the real, physical space of the user. For example, the location of these boundaries in real, physical space change based on the camera placement relative to the user and the characteristics of the smartphone camera, so the usefulness of these systems in training a motion is severely limited.

Other prior art systems involve a sheet of opaque material supported above a golf ball and having a slot therein through which the golfer views a golf ball while swinging, or a hollow tee through which light shines. These systems require the golfer to adjust his/her setup in order to detect head movement forward of a starting position, obscure the view of the ball during the swing, and, by requiring the golfer to re-observe the golfer's prior action, provide information as to head position only after the swing.

Another prior art system for determining an amount and direction of a golfer's head movement during a swing consists of at least two light baffles. The baffles reflect light into the golfer's eyes when the golfer's head is outside of a preset range. The system requires precise setup and the golfer's visual observation in order to determine head movement. And again, the system does not indicate movement without the golfer looking at the system, so the system cannot monitor the head position throughout the entire swing.

Another prior art system involves head gear worn by a golfer that includes motion/position sensing devices from which data is processed to determine a signal indicative of the golfer's head motion. In this system, the motion is detected by the system in the head gear and therefore is difficult to provide head movement information relative to the ball. In addition, the golfer must train the system with desired and undesirable motions, then compare the stored motions with the current motion.

SUMMARY OF THE EMBODIMENTS

In accordance with one embodiment of the invention, there is provided computer-implemented method of training a user in performing a motion for an activity. The method employing an AV-equipped portable computing device having a processor. The computing device having been positioned so that its camera faces a target. The processor executing computer processes defined by an application. The computer processes include receiving and processing by the processor an output from the camera to determine an initial position of the target. The computer processes also include processing the output from the camera by the processor to determine a series of coordinates corresponding to successive positions of the target, assumed after the initial position in the course of the motion. The computer process further include processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion, wherein the fault rules relate to target position in a real, physical space of the target. The computer processes also include, if a fault is determined to be present, causing by the processor notification of the user via components of the portable computing device, so as to train the user to perform the activity in a manner free of the fault.

Optionally, the fault rules additionally relate to whether the target has moved to a position outside of one of a set of boundaries determined by distances established for motion of the target relative to the target's initial position in the real, physical space. Alternatively or additionally, the fault rules additionally relate to a time of occurrence of a moment when the target has moved to a position outside the boundary relative to a time of occurrence of a defining moment of the motion, and the computer processes further include processing by the processor the series of coordinates to determine (a) the time of occurrence of the defining moment and (b) the time of occurrence of the moment when the target has moved to a position outside the boundary.

Optionally, the fault rules additionally relate to target position in three dimensions of the real, physical space. Alternatively or additionally, the fault rules additionally relate to target orientation in the real, physical space. Optionally, the computing device has a display, and the notification of the user is by visual notification on the display, and the computer processes further include providing, in the display, a link to a resource for addressing the fault. Alternatively or additionally, the link for addressing the fault is determined by user-specified criteria for training instruction based on items selected from the group consisting of identification of an instructor, level of instruction desired, and combinations thereof.

Optionally, the set of fault rules are defined based on input from an individual identified by the user as an instructor. Optionally, processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion includes identifying the presence of a plurality of faults and selecting from the plurality of faults a most salient fault, and wherein the motion is performed in a set of repetitions by the user and the computer processes are executed for each repetition of the set for identifying, separately for each such repetition, the presence of the plurality faults, and the computer processes further include determining by the processor, cumulatively for the set of repetitions, the most salient fault based on of its frequency of occurrence and its severity. Optionally, processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion includes identifying the presence of a plurality of faults and selecting from the plurality of faults a most salient fault, and wherein the motion is performed in a set of repetitions by the user and the computer processes are executed for each repetition of the set for identifying, separately for each such repetition, the presence of the plurality faults, and the computer processes further include determining by the processor, cumulatively for the set of repetitions, the most salient fault based on input from an individual identified by the user as an instructor.

Optionally, the computing device includes a display, and the computer processes further include storing the series of coordinates corresponding to the successive positions of the target assumed after the initial position, and an instant in time when each position was assumed; and computing and causing display of a path of movement of the target, based on the series of coordinates corresponding to the successive positions of the target assumed after the initial position. Alternatively or additionally, the computing device is configured to communicate over a network, and the computer processes further include providing by the processor, to an individual identified by the user as an instructor, a notification that the series of coordinates have been stored at a storage location in the network; retrieving by the processor, in response to input by the identified individual, the series of coordinates from the storage location; and causing display by the processor of a representation of the series of coordinates.

Optionally, the computer processes further include storing the series of coordinates corresponding to the successive positions of the target assumed after the initial position, and an instant in time when each position was assumed; detecting a defining moment in the activity; and on detecting the defining moment, storing the series of coordinates and the instant in time each position was assumed. Alternatively or additionally, the computing device includes a microphone and the defining moment is a ball-strike, and the computer processes further include processing by the processor sound detected by the microphone to detect the ball-strike. Alternatively or additionally, the computer processes further include processing by the processor of the output from the camera to detect the defining moment, and wherein the processing of the output includes computing the orientation of the target.

Optionally, determining the initial and successive positions of the target includes tracking initial and successive positions of another target of the user. Optionally, the computer processes further include storing by the processor a quality of a result of the activity. Alternatively or additionally, the quality of the result of the activity is an indication provided by the user, and the computer processes further include prompting by the processor the user to provide the indication of the quality of the result of the activity and storing the indication. Alternatively or additionally, the series of coordinates constitutes a first series of coordinates, and the computer processes further include displaying a path of movement based on a second series of coordinates corresponding to successive positions of the target in performing a second motion for the activity; and providing a visual comparison of measurements between the first series of coordinates and the second series of coordinates.

In accordance with a further embodiment of the invention, there is provided a computer-implemented method of establishing orientation of an AV-equipped portable computing device, for monitoring a user in performing an activity. The device having an accelerometer and a processor. The processor executing computer processes defined by an application. The computer processes include receiving and processing by the processor an output from the accelerometer to determine current orientation of the portable computing device. The computer processes also include directing by the processor to the user to cause a change in orientation of the computing device in a manner suitable for monitoring the user. The computer processes further include receiving and processing by the processor an output from the camera to determine a position of a target. The computer processes also include directing by the processor positioning of the user to facilitate capture of the motion of the target in the course of the motion.

In accordance with another embodiment of the invention, there is provided a computer-implemented method of training a user in performing a motion for an activity. The method employs an AV-equipped portable computing device having a processor, and the computing device has been positioned so that its camera faces a target. The processor executes computer processes defined by an application. The computer processes include receiving and processing by the processor an output from the camera to determine an initial position of the target. The computer processes also include defining by the processor a set of boundaries determined by distances in a real, physical space of the target, for motion of the target relative to the initial position, based on the activity. The computer processes further include determining by the processor a series of coordinates corresponding to successive positions of the target assumed after the initial position by processing the output from the camera. The computer processes also include processing by the processor the series of coordinates to determine whether one of these positions is located outside of one of the boundaries, and if so, then causing notification of the user via components of the portable computing device, so as to train the user to perform the activity in a manner that causes desired positioning of the target in space.

Optionally, the computing device has a display, and the notification of the user when a position is located outside a boundary is by visual notification on the display. Alternatively or additionally, the computing device has an audio transducer, and the notification of the user when a position is located outside a boundary is by audible notification. Alternatively or additionally, the computer processes further include, in a set of setup steps, prompting the user to initiate a detectable motion and detecting the motion.

Optionally, the computer processes further include storing the series of coordinates corresponding to successive positions of the target assumed after the initial position and an instant in time each position was assumed. Alternatively or additionally, the computer processes further include terminating measurement and storage of the series of coordinates corresponding to successive positions of the target assumed after the initial position and the instant in time each position was assumed on detecting a defining moment. Alternatively or additionally, the computing device includes a microphone and the defining moment is a ball-strike, and the computer processes further include processing sound detected by the microphone to detect the ball-strike. Alternatively or additionally, the computer processes further include processing the output from the camera to detect the defining moment. Alternatively or additionally, the computing device includes a display, and the computer processes further include computing and causing display of a path of movement of the target, based on the series of coordinates corresponding to the successive positions of the target assumed after the initial position. Alternatively or additionally, the computer processes further include indicating, in the display, the set of boundaries, and visually marking therein any boundary that has been traversed by the path. Alternatively or additionally, the computing device includes a display, and the computer processes further include computing and displaying performance statistics relative to the activity.

Alternatively or additionally, the computing device includes components to communicate over a network, and the computing device is coupled over the network to a server system configured to receive and store data from the computing device, and the computer processes further include providing, over the network, for storage by the server system, the series of coordinates corresponding to successive positions of the target assumed after the initial position and the instant in time each position was assumed. Alternatively or additionally, the computer processes further include receiving, from the server system, data processed by the server system relative to the activity providing statistics over one or more users and displaying the statistics. Alternatively or additionally, the computer processes further include storing a series of coordinates corresponding to successive positions of multiple targets assumed after the initial position and an instant in time each position was assumed.

Optionally, the computer processes further include providing hysteresis in resetting of the notification, so that when the notification has been generated as a result of the target moving outside a boundary, the computing device will not generate a subsequent notification unless the target has moved inside a narrower boundary and remained inside that boundary for a predetermined amount of time. Alternatively or additionally, the computer processes further include determining by the processor a series of angles corresponding to orientations of the target by processing the output from the camera. Alternatively or additionally, the computer processes further include storing a quality of a result of the activity. Alternatively or additionally, a quality of the result of the activity is a user's indication of the quality of the result of the activity, and the computer processes further include prompting the user to give an indication of the quality of the result of the activity and storing the user's indication. Alternatively or additionally, the computing device includes a means for determining a quality of the result of the activity, and the computer processes include storing the determined quality. Alternatively or additionally, the computing device includes a means for communicating with a system that can determine a quality of the result of the activity, and the computer processes further include receiving and storing the determined quality. Alternatively or additionally, the computer processes further include computing a model to relate the series of coordinates corresponding to successive positions of the target assumed after the initial position and the instant in time each position was assumed of multiple instances of the activity to the corresponding stored quality of the result.

Optionally, the computer processes further include analyzing the series of coordinates corresponding to successive positions of the target assumed after the initial position and the instant in time each position was assumed to generate a computed performance measure; and notifying the user of the computed performance measure. Alternatively or additionally, the computer processes further include storing an amplitude of the sound detected by the microphone and the instant in time that sound was detected over the course of the activity, analyzing the stored amplitude of the sound detected to generate a computed performance measure, and notifying the user of the computed performance measure.

Optionally, the computing device includes a display, and the computer processes further include computing a fault in performance of the activity, and causing display of the fault. Alternatively or additionally, the computer processes further include providing, in the display, a link to a resource for addressing the fault.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of embodiments will be more readily understood by reference to the following detailed description, taken with reference to the accompanying drawings, in which:

FIG. 1 schematically shows a side view of a system to track a target during an activity in accordance with an embodiment of the present invention;

FIG. 2A depicts device-based and real, physical space coordinate systems used to implement the method of the present invention in some embodiments;

FIG. 2B depicts boundaries in terms of the device-based and real, physical space coordinate systems used to implement the method of the present invention in some embodiments;

FIG. 2C illustrates the use of multiple targets in systems implementing a method in accordance with an embodiment of the present invention;

FIG. 3 is a high-level flowchart describing an algorithm that may be used to implement a method in accordance with an embodiment of the present invention;

FIG. 4 schematically shows a top view of an embodiment of the present invention where the orientation of equipment is detected;

FIG. 5 is a flowchart describing an algorithm that may be used to in the implementation of boundaries in some embodiments of the present invention;

FIG. 6 schematically shows a top view of an embodiment of the present invention where the alert system includes an algorithm for preventing subsequent alerts until specified motion has occurred after a first alert;

FIG. 7 is a flowchart describing an algorithm that may be used to prevent subsequent alerts until specified motion has occurred after a first alert, according to embodiments of the present invention;

FIG. 8 schematically shows a display of stored target coordinates and boundaries according to embodiments of the present invention;

FIG. 9 schematically shows a system that communicates with a network, and, through the network, to computing devices, according to embodiments of the present invention;

FIG. 10 schematically shows the operation of a model that takes stored position data and returns either qualitative or quantitative results in accordance with an embodiment of the present invention;

FIG. 11 schematically shows the operation of a model that takes stored sound amplitude data and returns either qualitative or quantitative results in accordance with an embodiment of the present invention;

FIGS. 12 and 13 are high-level block diagrams depicting functional elements that may be used in some embodiments of the present invention;

FIGS. 14A, 14B and 14C and FIGS. 15A, 15B and 15C are depictions of screens which may be displayed to a user by an embodiment of the present invention;

FIG. 16 is a depiction of another screen which may be displayed to a user by an embodiment of the present invention;

FIG. 17 is a flowchart describing an algorithm for determining a list of faults in an embodiment of the present invention;

FIG. 18 is a flowchart describing an algorithm for choosing a most salient fault in an embodiment of the present invention;

FIG. 19 is a flowchart describing an algorithm for associating an instructor with a user in an embodiment of the present invention.

FIG. 20 is a flowchart describing an algorithm for directing a user to position a computing device correctly in an embodiment of the present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Definitions. As used in this description and the accompanying claims, the following terms shall have the meanings indicated, unless the context otherwise requires:

An “AV-equipped portable computing device” is computer hardware (such as a processor, field-programmable gate array or other electronic combinatorial logic, or similar device) combined with a camera so that the computer hardware can process the output pf the camera.

A “computer process” is the performance of a described function in a computer using computer hardware (such as a processor, field-programmable gate array or other electronic combinatorial logic, or similar device), which may be operating under control of software or firmware or a combination of any of these or operating outside control of any of the foregoing. All or part of the described function may be performed by active or passive electronic components, such as transistors or resistors. In using the term “computer process” we do not necessarily require a schedulable entity, or operation of a computer program or a part thereof, although, in some embodiments, a computer process may be implemented by such a schedulable entity, or operation of a computer program or a part thereof. Furthermore, unless the context otherwise requires, a “process” may be implemented using more than one processor or more than one (single- or multi-processor) computer.

A “set” includes at least one member.

An “activity” is something that a user does and includes: recreational pursuits such a golf; elements of such pursuits, such as a golf swing; types of such elements, such as a putt of a drive; or categories of such elements, such as a level like beginner or pro.

A “defining moment” in an activity is an exact point, in execution of the activity, defined by occurrence of a specific event, such as the start of the motion, the commencement of the downswing, striking of a ball by a club (wherein the ball-strike is the defining moment) or release of a pitched ball from the hand (wherein ball-release is the defining moment).

A “target” is a member selected from a group of consisting of a face or other body part of the user, a detectable pattern placed on the user, a piece of equipment held by the user, and anything detectable as it moves through space, and combinations thereof, embodiments of the present invention train the user to perform an activity in a manner that clauses desired positioning of the target, or targets, in space.

“Device-based coordinates” are a set of numbers used to indicate the position of a point in an image, generated by the camera, wherein the position is expressed in image coordinates and is device-specific.

“Real, physical space of the target” and “physical space of the target” are the dimensions of height, depth, and width within which the target exists and moves.

“Coordinates corresponding to positions of the target” refers to either device-based coordinates or coordinates indicating the position of the target in the real, physical space of the target.

“Quality of the result of the activity” is a qualitative or quantitative measure of the goal of the activity.

“Model” is a mathematical algorithm that takes as input some data and outputs data corresponding to the input.

A “fault” in performing a motion for an activity is a characteristic of the motion, and the presence of the characteristic indicates a deficiency that can be remedied to improve the result of the activity.

Various embodiments of the present invention determine when a target has experienced motion beyond a boundary over the course of an activity. For example, embodiments of the present invention provide a method which detects and alerts a golfer of head movement beyond boundaries defined in the real, physical space of the golfer as the golfer executes a golf swing, which solves the problems and overcomes the drawbacks and deficiencies of prior art golf swing assistance systems. The system of various of these embodiments implementing the method includes an AV-equipped portable computing device that processes the output from its sensors or camera to determine the distances moved by a target in the real, physical space of the user. The method includes determining an initial position of the target and defining a set of boundaries in that space, processing the camera output to determine whether the target is within or outside of those boundaries, and if the target is determined to be outside the boundaries, notifying the user, so as to train the user to perform the activity in a manner that causes desired positioning of the target in space. In a further embodiment, the method has a feature by which it ignores the movement of the target once the swing has completed, such as contact has been made with a ball, a feature that eliminates false alarms, because, after the swing has been completed, the position of the target is no longer important in training. The starting position of the target, deviation from which in the course of an activity creates an alerting condition, may be the first determined position of the target. Alternatively, the system may determine an adaptive starting position of the target. In various embodiments, the method provides an initialization mechanism by deferring calculation of the starting position until the user has indicated they are ready to initiate the activity, for example a golfer has looked away from the ball for a specified period. In various embodiments, the boundaries are user selectable. The alerting mechanism in various embodiments includes hysteresis, so that when the alert system has generated the alert as a result of target movement beyond a boundary, the alert system is configured not to generate a subsequent alert unless the target has moved back within a boundary closer to the starting position and remained there for a predetermined amount of time. In further embodiments, the orientation of a target instead of or in addition to the position of a target may be determined, and the system may alert the user if the orientation is outside of certain angular limits. In further embodiments, the system may determine the location of multiple targets and alert the user if some combination of targets is outside of boundaries, or the orientation of multiple targets and alert the user if some combination of targets are outside angular limits, or alert the user based on combinations of the results of both location and orientation detection. In further embodiments, the system may allow the user to display and analyze a record of their motion during the activity. In further embodiments, the analysis of their motion may include the identification of specific faults or issues that reduce the effectiveness of the motion, and in further embodiments, the system may alert the user to the faults or issues, and in further embodiments, the system may display the faults or issues to the user, and in further embodiments. the display may include links to resources to help the user ameliorate the faults and improve their motion. In further embodiments, the system may allow the user to display and analyze the record of their motions in multiple instances of the activity. In further embodiments, the system may allow the user to apply statistical analysis to the record of their motions, and in further embodiments may allow the user to apply statistical analysis to compare their motions to the motions of other individuals or to the aggregated motions of a set of other individuals. In further embodiments, the system may include connectivity with a computer network. In further embodiments, the method may include determining qualitative or quantitative measures of the results of the activity. In further embodiments, the method may include creating mathematical models to map the record of successive positions to qualitative or quantitative measures of the results of the activity. Details of illustrative embodiments are discussed below.

FIG. 1 depicts the typical setup of an embodiment of this invention, used by a golfer to improve the golfer's golf swing. A smartphone is 13 is placed in a stand 12 facing the golfer 11. The golfer 11 swings the golf club 16 to strike the golf ball 14. The smartphone 13 is an AV-equipped portable computing device and includes a camera and a processor that is programmed to translate the outputs of the camera to determine the position of a target. In this figure the target is the center 15 of the face of the golfer 11. In some embodiments, the device-based coordinates of the image of the target 15 are determined by using the Google ML Kit machine learning processing module, available in the Google Firebase product, to compute coordinates of the bounding box of the face, and the device-based coordinates are taken to be the pixel coordinates of target is the center of the bounding box. The smartphone 13 also includes a microphone, coupled to the processor, that can detect the sound created by the golf club 16 striking the golf ball 14.

We have found it valuable, in embodiments of the present invention to provide a system for alerting a user when the movement of a target exceeds a threshold as measured in distances in the real, physical space of the user. For example, in various embodiments, the user is able to set a threshold for movement in terms of real-world distance, e.g., “set a boundary 100 mm on either side of my initial position.” Fundamental to the method of the present invention, then, is the ability to determine distances in the real, physical space of the user by processing the output of a sensor or camera. Here we discuss two approaches: a simple technique using distance scale factors and a more complex technique using coordinate transformation.

As an example of the simple approach using scale factors, consider the embodiment of FIG. 1, where the target is the center of the face of a user, tracked by a smartphone using image processing algorithms that compute the camera pixel coordinates of the bounding box of the user's face, such as Android smartphones or Apple iPhones using Google Inc.'s machine learning ML Kit. To “set a boundary 100 mm on either side of my initial position”, if the system can determine the scale factor defining how many camera pixels correspond to a distance of 100 mm in the real, physical space of the target, then it can determine thresholds in device coordinates to implement that boundary. Thus, if the system determines that a one pixel difference in camera coordinates corresponds to a 20 mm displacement of the target, then in this case the system would alert the user when pixel coordinates of the image of the target stray more than 5 pixels from the coordinates of the initial position. To determine the scale factor, a single pre-determined value might be adequate. For example, if identical devices are used in each instantiation of the system, and the target is always placed the same distance from the camera, the scale factor can be determined empirically when designing the system. Some embodiments may use camera parameters—field of view, resolution, . . . —and assumptions—distance to user from camera—to determine the scale factors. Other embodiments may have the user hold an object of known length at the location of the target—e.g., a half meter long red rod held at the user's face—to determine the scale factor. Other embodiments may assume an average width of a face—e.g., 100 mm—and use the device coordinates of the bounding box of the detected face to estimate the scale factor.

Some embodiments may instead implement the method of the present invention with explicit transformations between coordinate systems. This approach is particularly useful if the mapping of target position to device-based coordinates is too different from uniformly linear to be useful. For example, if the device-based coordinates are pixels in a camera image, departure from uniform linearity is present if identical horizontal and vertical displacements of the target do not result in equally sized changes in camera pixel coordinates. Similarly, such a departure may be found if identical displacements, when the target image is in different areas of the camera field, do not result in identical changes in camera pixel coordinates. The coordinate transformation approach is also preferred if the embodiment is designed to track a target and set boundaries in three dimensions.

FIG. 2A illustrates the coordinate transformation approach to determining distances in the real, physical space of the user. On the left is shown the device-based space and its coordinate system, and on the right is shown the real, physical space and its coordinate system. If the device is a camera, for example, the device-based coordinates may be the pixel coordinates of the target in the camera image, for example coordinates relative to the x-axis 21 and y-axis 22. In some embodiments of the invention, operating on Android smartphones or Apple iPhones, the Google Inc.'s machine learning ML Kit available may be used to retrieve the bounding rectangle 23, in camera coordinates, of the face of the user, and the camera coordinates of a specific corner or of the center 24 of the rectangle can be used as the device-based coordinates of the target. In other embodiments, image processing capabilities such as those provided by the OpenCV Open Source Computer Vision Library toolkit or other image processing toolkits may be used to determine the device-based coordinates of the target Note that the coordinates provided by the toolkits may need to be transformed to match the conventions used by embodiments for their computations. For example, if the toolkit uses a coordinate system where the y coordinate increases in the downward direction, but the convention in the embodiment is that the y coordinate increases in the upward direction, then y coordinate values should be negated before performing further computations.

The system then translates those device-based coordinates to a set of coordinates in a coordinate system corresponding to the real, physical space of the user. The coordinate set can contain one, two, or three values, corresponding to one-, two- or three-dimensional coordinate systems in the space of the user. For example, if only lateral movement of the target is to be considered, a coordinate set of only one value is appropriate, and the coordinate would reflect the distance of the target from the initial position along a line.

First we consider one-dimensional or two-dimensional coordinate systems in the plane 25 that contains the target 26, where the plane 25 is perpendicular to the line 29 from the camera on the device 13 to the target 26 and the camera on device 13 is oriented along the line 29. Typically the one dimensional coordinate system would correspond to either the horizontal or vertical direction in that plane, corresponding to either the horizontal axis 27 or vertical axis 28, and the coordinates on that axis would be determined from the horizontal or vertical coordinates measured against the device coordinate axes 21 or 22. For the two dimensional coordinate system, in some cases, the physical coordinates along axis 27 may be determined solely from the device-based coordinate along axis 21, and coordinates along 28 solely from those measured along 22. That simple approach holds only if the device sensor's horizontal and vertical directions match those of the user's physical space, so that a movement of a target along only the axis 27 in physical space results in a displacement of the image of the target along only the axis 21 in device-based coordinates, and similarly movement along only 28 results in displacement only along 22. If, on the other hand, there is a skew in the setup—for example when the device is rotated relative to the real horizontal, or the camera in the device is not pointed directly at the target, or, in the one dimensional case, the line we're interested in is not horizontal or vertical in the target's physical space—someone skilled in the art will be able to apply the appropriate transformation to the device-based coordinates before proceeding.

To perform the translation from device-based coordinates to coordinates in real, physical space, the system must determine the scaling of displacements in device coordinates to displacements in physical space coordinates. For example, a movement of the target 26 by 20 mm along the axis 27 might correspond to a displacement of the image of the target by ten pixels along axis 21 in the device coordinate system. Embodiments may take various approaches to determining the scale factor. For some applications, a single pre-determined value might be adequate. For example, if identical devices are used in each instantiation of the system, and the target is always placed the same distance from the camera, the scale factor could be determined empirically when designing the system. Some embodiments may use camera parameters—field of view, resolution, . . . —and assumptions—distance to user from camera—to determine the scale factors. Other embodiments may have the user hold an object of known length at the location of the target—e.g. a half meter long red rod held at the user's face—to determine the scale factor. Other embodiments may assume an average width of a face—e.g. 100 mm—and use the device coordinates of the bounding box 23 of the detected face to estimate the scale factor. The units of the scale factor depend on the measurements used in real, physical space and the units in the device space; in the above example it is millimeters per pixel.

To perform the translation, the system must also determine the device-based coordinates of the origin of the physical space coordinates. Some embodiments might simply use the center or a particular corner of the camera frame as the device-based coordinates of the origin. Some embodiments might use a set of device-based coordinates to determine an initial position of the target in device coordinates—for example, by taking an average of a sequence of coordinates gathered over a certain time interval—and use those coordinates to correspond to the origin.

For example, suppose the device-based coordinates of the origin are (x0, y0), the scale factor is S, and the horizontal and vertical directions of the device correspond to the horizontal and vertical directions of the physical space. Then the physical space coordinates of the target, when it is detected at device coordinates x and y, are X=S (x−x0) and Y=S (y−y0).

The translation from real, physical coordinates to device-based coordinates is the inverse of the transformation from device-based coordinates to real, physical coordinates. For example, in the previous example, physical coordinates X, Y will map to device-based coordinates x=X/S+x0, and y=Y/S+y0.

In embodiments where the scale factor is not uniform in all directions, someone skilled in the art will be able to apply the correct transformations, e.g. determine two scale factors Sx and Sy and use a transformation that takes device (x, y) to physical (Sx (x−x0), Sy (y−y0)). In embodiments where the scale factor varies with distance from the center of the device sensor, and the variation is so great it would adversely impact the usefulness of the system, then those factors Sx and Sy will depend on x and y, and someone skilled in the art will be able to apply the appropriate nonlinear transformations and nonlinear inverse transformations. Similarly, when the horizontal axis of the device used to track location is perfectly aligned with the horizontal direction in physical space, note that the vertical axis 28 may not align with the vertical direction in physical space—e.g. if the line 29 from device to target is not horizontal—and in some applications this discrepancy may have a large enough effect to necessitate adjustments to the transformation. For example, one such adjustment that might be sufficient in some embodiments would be to replace the vertical coordinate Y with the value Y/cos(theta) when Y is positive and with Y*cos(theta) when Y is negative, where theta is the angle from horizontal of the line 29.

Embodiments may also be designed to track targets in three-dimensional coordinate systems in physical space. The method begins as above with translating two-dimensional device-based coordinates along axes 21 and 22 to a two-dimensional coordinate system in physical space along axes 27 and 28, where those axes span the plane 25 containing the target 26 and perpendicular to the line 29 from the device sensor 13 to the target 26. Then a third coordinate may be added based on the length of line 29. One way to determine changes in the length of line 29 is to measure the change in length of objects in the device-based coordinates as they move closer and further from the device, and assign the third coordinate based on that change. For example, if the output of the camera is used to determine the bounding box of a face, the width or height of that face can be used. If the target is circular, its radius can be used. Suppose the face or target is initially positioned a distance D from the device and its length in device coordinates is L0. That distance D will be considered the origin of the Z-axis. When the target or face approaches the device, growing larger to length L in device coordinates, its Z coordinate in physical space can be set to +/−D*(L−L0)/L0, following the rules of perspective, where the plus or minus is chosen based on the physical directions of the X and Y axes and the requirement that the physical coordinate system be right-handed or left-handed. For example, if a right-handed three-dimensional physical coordinate system is desired, and as the target moves to the right its device x coordinate increases, and as the target moves up its device y coordinate increases, then positive movement in Z is when the target approaches the camera, so the plus version Z=D*(L−L0)/L should be used.

FIG. 2B depicts horizontal and boundaries that may be used in the implementation of the method of the present invention. Element 2b02 are boundaries set by using distances in the real, physical space of the user, for example as horizontal and vertical displacements from an initial position of the target. Element 2b01 is the representation of those boundaries in the device-based coordinate system. Determination of whether the target has crossed a boundary and whether or not the target is within or outside the boundaries may be accomplished in either coordinate system.

FIG. 2C is an example of multiple targets whose successive positions may be determined in some embodiments of the present invention. In this example the positions of two elements—the head 2c02 of the golfer 2c01 and the knee 2c04 of the golfer 2c01—are tracked. The golfer is alerted when either the head 2c02 moves outside the boundaries depicted as rectangle 2c03, or the knee 2c04 moves outside the boundaries depicted as rectangle 2c05. In some embodiments the alert may only occur when all targets are outside boundaries. If there are more than two targets, the system may generate various alerts for various sets of combinations of targets crossing boundaries. Determining the position of multiple targets simultaneously may be accomplished by someone skilled in the art by using software modules such as Google's BodyPix TensorFlow machine-learning body-part segmentation model, which can be used to label body parts in images of a body.

In some embodiments, it may be desirable to compute the coordinates of one target from the coordinates of another target. For example, consider an embodiment where a target detection module provides the coordinates of the bounding box and the Euler angles of the rotation of the face, and a goal of the embodiment is to track the position of the base of the neck. The embodiment may approximate the coordinates of the base of the neck by taking the coordinates of the center of the bounding box and translating them a certain distance along a direction determined by applying the Euler angle measuring the rotation of the face in the plane of the image, where that direction is vertically downward when there is no rotation. For example, Google Inc.'s ML Kit analyzes camera images and provides values that can be used to describe the location of a face in device-based coordinates: the bounding box of a face with four values—xleft, ytop, xright, ybottom, where the x and y values are displacements along the device-based x and y axes of FIG. 2B—and a z-axis Euler angle ez indicating rotation of face in the xy plane. As described above in FIG. 2A, some embodiments may transform the values provided by the face detection module to match the convention used in their computations, so here we assume that the x coordinates increase with displacements to the right on the x-axis, that the y coordinates increase with displacements upward along the y-axis, and the Euler angle ez is provided in degrees and is positive for a counter-clockwise rotation of the face, e.g. the user's head tilts to their left.

Take, for example, the coordinates of the center of the face to be xface=(xleft+xright)/2 and yface=(ytop+ybottom)/2. Suppose the base of the neck is determined to be a distance of 90% of the height of the face below the center of the face. Then the base of the neck is determined to be a distance d=0.9*(ybottom−ytop) along the direction ez, so its coordinates will be xneck=xface+d*cos(ez−90) and yneck=yface+d*sin(ez−90).

This value to be used in computing the distance from the center of the face to the base of the neck, in different embodiments, can be determined empirically for a single user by examining a photograph of the user, or across a population of users by examining many photographs and computing an average value, or by an initialization process where a target is held by the user at the base of their neck and the computing device compares the location of the center of the face returned by a face detection module with the location of the target as returned by a suitable image processing module.

FIG. 3 is a high-level flowchart illustrating an embodiment of the invention. The flowchart describes the algorithm as a synchronous series of steps, but in various embodiments the steps may be implemented in software using many alternative approaches, such as an event-driven programming or interrupt-driven programming. Note also that at any time in the execution of the steps of the flowchart of FIG. 3, the system might detect that the user has stopped proceeding with the method and the system will return to step 31, for example the method uses face detection by a camera to locate the target and the user's face has disappeared from view of the camera. Further, the user might terminate the method, for example by interacting with the device. In FIG. 3, the boxes 35 and 37 with double-lined walls refer to flowcharts appearing in later figures.

Step 393 represents the beginning of the algorithm. At step 31 the system indicates the user should get ready for the activity, and the system remains at step 32 until the user is ready to proceed. The indication at step 31 may be an audible prompt or a visual prompt, or an embodiment may make no prompt at all and simply wait for the user to be ready. The readiness test could involve the user interacting with the device, making an audible or visual response, or moving the target in a manner detectable by the system. The readiness prompt and test may also encompass several cycles of prompt and response. For example, consider a system where the target is the center of a golfer's face, determined by applying image processing algorithms to the output of a smartphone camera. The system might first prompt the user to look away from the device, then detect that they have. Detecting that the user has looked away may be accomplished by the image processing algorithm failing to return a target position within a specified time interval—e.g. 2 seconds—or, for image processing algorithms that can detect the rotation of the target, such as the face detection algorithm provided by the Google ML Kit machine learning processing module, by the algorithm determining the target is sufficiently rotated—say 40 degrees—for a specified time interval. Then the system prompts the user to face the device, and the system determines readiness when it detects the target is in view.

Once the user is ready, at step 33 the system determines the device-based coordinates corresponding to a fiducial position of the target in space. Some embodiments might use the first set of device-based coordinates computed after the user is ready, and others might use a sequence of device-based coordinates of the target, for example by taking an average of a set of coordinates gathered over a certain time interval. At step 34 the system determines the distance scale factor described above, or the translations between device-based coordinates and coordinates in the real, physical space of the user discussed in connection with FIG. 2A.

Suppose the method is used to train a golfer's golf swing and step 33 is used to obtain the initial position of the center of the golfer's face, applying image processing algorithms to the output of a smartphone camera. Having passed the test at step 32, the user is ready and facing the camera. The location of the center of the face in camera coordinates is computed until the coordinates demonstrate that the user has stopped moving, e.g., when the next x coordinate is close to the moving average of the prior six x coordinates. These device coordinates may then be used in step 34 to set the origin of the coordinates in the physical space of the target. Additionally, during these steps a heuristic may be applied to set the scaling factors needed, e.g. by measuring the extent of the x coordinates in the bounding box of the face detected and assuming that extent corresponds to 150 mm. The origin and scale factors need not stay constant in some embodiments and may be altered as the target moves during the next steps of FIG. 3, for example if two-dimensional coordinates are used and the system detects movement towards and away from the device.

At step 35 the system determines the boundaries that will be used to determine if the target is in-bounds or out-of-bounds at step 37. The boundaries desired by the user are applied to the fiducial position in real, physical space whose coordinates were determined at step 33 to determine boundaries in real, physical space. For example, the system may allow the user to specify thresholds, for example the user might want to be alerted when the target moves more than 50 mm upward or downward, 75 mm forward or 25 mm backward. In some embodiments, the boundaries may be set based on variables such as the skill of the player, the type of sport, and the intent of the swing. For example, the boundaries may be placed further from the fiducial position for a beginning golfer than for an advanced golfer, or closer to the fiducial position if a golfer is putting rather than driving the ball. The placement of the boundaries may also depend on other factors, such as whether the user is left-handed or right-handed. The system may provide for the user to specify any of these factors, for example through a user interface on a smartphone.

In some embodiments of the present invention, the orientation of a target instead of, or in addition to, the position of a target may be determined. In FIG. 4, the system of FIG. 1 is configured so that the target is a putter, a piece of equipment employed by the user in an activity, and the system detects the orientation of the putter using image processing algorithms, such as those provided by the OpenCV Open Source Computer Vision Library. In this example, the system is used to determine whether or not the putter face 45 of the putter 46 used by a golfer is oriented parallel to line 43. Line 43 is perpendicular to the line 42, which is parallel to the path from the ball 44 to the golfer's target hole. If the putter face 45 is oriented perpendicular to the line 42 when the golfer strikes the ball, the strike is known to golfers as “square”. The device 41 is positioned so the camera has a view of the golf ball 44. To aid in determining the orientation of the putter face 45 relative to the direction of line 42, embodiments may require that the device 41 be positioned so the camera is pointing perpendicular to the line 42, or an object that can be detected by the device's image processing algorithms may be placed in the field of view, such as an alignment rod lying along the direction 42 near the golf ball 44, or the image processing software might determine the direction of line 42 by tracking the position of the putter face 45 as the golfer swings. Using the same techniques for determining the moment of impact as discussed below in regard to step 39 of FIG. 3, the system can alert the user as to whether the equipment was oriented as desired at and around the moment of impact within specified angular limits, such as a within a pre-determined deviation from perpendicular.

In some embodiments, the system may determine the orientation of multiple targets, or may determine the position of some targets, the orientation of other targets, and both the position and orientation of other targets in applying the method of the present invention, and the alert system may generate various alerts depending on combinations of positions and orientations that are within and outside of specified position or orientation boundaries.

In some embodiments, the boundaries may be expressed as minimum and maximum values of each coordinate in physical space. FIG. 5 shows a flowchart for selecting values. The computations shown in steps 54, 55, 56 or 57 of FIG. 5 are for an embodiment where the target is tracked in only one dimension. The position value is the coordinate along that one dimension in either the device-based or the real, physical coordinate space. The threshold values SWING_F, . . . are in the same coordinate space. If using real, physical space, those thresholds may be specified as measurements, e.g. 100 mm. If using device-based coordinates, the thresholds may be determined by determining the change in device-based coordinates when a target is moved by a specified amount, for example by dividing by the distance scale factor or the scale factor S described above.

The algorithm depicted in FIG. 5 applies to an embodiment as in FIG. 1 that is used to train a golfer in their swing. In this depiction, the position value decreases as the target moves in the same direction as the desired flight of the ball. So for a right-handed golfer, the position value increases as the golfer moves to their right, and for a left-handed golfer the position value increases as the golfer moves to their left. The start of the algorithm is represented by 51. At 52 the fiducial position is computed. As described for step 33 of FIG. 3, some embodiments may use the first set of device-based coordinates computed after the user is ready, and others may use a sequence of device-based coordinates of the target, for example by taking an average of a set of coordinates gathered over a certain time interval. At step 53 the user indicates their skill level and the swing they would like to practice, for example by using a user interface on a smartphone used to implement the system, and only one of steps 54, 55, 56 and 57 is chosen, corresponding to the choices that the user has configured. FIG. 5 shows by way of example three types of swing—drive, pitch and putt—and for drive, a normal and pro skill level. If the user has configured the device for normal skill level and drive practice swings, then at step 54 of FIG. 5, minimum and maximum limits for the position value are set, in this case the minimum limit MIN_LIMIT variable is set to the fiducial position value minus the parameter DRIVE_F, and the maximum limit variable MAX_LIMIT is set to the fiducial position value plus the parameter DRIVE_B. DRIVE_F and DRIVE_B and the other parameters in FIG. 5—PRO_B, PRO_F, PITCH_B and PUTT_F—are determined by the designer of the embodiment. When the thresholds are not symmetric, the system may have to adjust the formula based on whether the user is left- or right-handed. If the user configured drive practice and “pro” skill level, thresholds PRO_F and PRO_B are used, which are numerically smaller than DRIVE_F and DRIVE_B, yielding boundaries that are closer to the fiducial position, meeting the needs of a more advanced practitioner. If the user configured “Pitch” practice, then at step 56 of FIG. 5 only a maximum position value limit is set, and there will be no minimum position value used, so that “pitch” practice mode can be used to alert the user only if the user moves away from the target, for example to remedy a common issue with a golf pitch swing where the golfer sways backwards during the swing. If the user configured “Putt” practice, then only a minimum position value limit is set and no maximum limit is set, so that the user will only be alerted if the user moves towards the target during the swing. At step 58 the algorithm is complete.

The target's position is continually determined at step 36 of FIG. 3. The coordinates, either device-based or physical, can be stored by the system for later display and analysis, for example in device memory or at a server on the Internet. The time that the position was recorded may also be stored.

At step 37, the position is compared against the boundaries determined in 35. In some embodiments, rather than minimum or maximum of coordinate values as described in FIG. 5, the boundary condition tested at step 37 of FIG. 3 might be a function of the coordinates, e.g., implementing a circular boundary by requiring that the Euclidean distance from the fiducial position is less than a value. In some embodiments, the test at step 37 will be done in device-based coordinates, and in other embodiments in the real, physical coordinates of the space about the user. For embodiments using the translation approach of FIG. 2a, because the method of the present invention includes translations from physical coordinates to device-based coordinates, some embodiments might translate the boundaries to device-based coordinates and perform the in- and out-of-bounds determination in device-based coordinates. For embodiments using a distance scale factor, the scale factor can be used to define thresholds in device-based coordinates to be used in the test at step 37.

If the test at step 37 shows the target is outside the boundaries, the user can be alerted at step 38. In some embodiments the fault is stored step 38 for later display or analysis. The alert can be a sensory alert including alerts communicated visually, audibly, or through touch, e.g., lights, sounds, and/or vibrations.

Referring to the one-dimensional case depicted in FIG. 5, the test at step 37 of FIG. 3 may be a simple comparison of the position value with the limits MIN_LIMIT or MAX_LIMIT or both as set at steps 54, 55, 56 or 57 of FIG. 5. Or, the test at step 37 may be the more complex algorithm illustrated in FIG. 6 and described in FIG. 7. The goal of this enhancement is to prevent subsequent alerts until a specified motion has occurred; the algorithm prevents the control unit from signaling repeatedly as, for example, a user's head moves quickly back and forth across a location threshold. FIGS. 6 and 7 illustrate the algorithm as applied to a right-handed golfer, where movement of the golfer to the left reflects a decrease in the position value. In FIG. 6, when the target 61 initially moves across location threshold 62 from right to left, the system will generate an alert. If the user then moves back to the right across 62 but not as far as 63, then again to the left across 62, an embodiment with this enhancement will not generate another alert. Or, if the user does move back across 63 but immediately moves to the left again across 62, there will be no alert if this enhancement is implemented. Only if the user moves to the right of 63 and remains to the right of 63 for at least a specified time interval will a subsequent crossing of 62 cause an alert. Similarly, after moving from left to right across location threshold 64 and generating an alert, the user must move right to left across 65 and remain to the left of 65 for a specified interval of time in order to enable the generation of alerts.

The flowchart in FIG. 7 applies to an embodiment implementing the procedure shown in FIG. 5. There are two variables, HYST_TEST and HYST_CROSS_TIME, used to track the state of the algorithm. HYST_TEST is used to indicate the new threshold 63 or 65 of FIG. 6 that must be crossed in order for subsequent alerts to be generated, and HYST_CROSS_TIME stores the most recent time that threshold 63 or 65 of FIG. 6 was crossed in the correct direction. When this algorithm is included in an embodiment using the approach of the flowchart in FIG. 3, HYST_TEST will be cleared at step 31.

Step 71 represents the start of the algorithm depicted by the flowchart of FIG. 7. The current state of HYST_TEST is checked at step and, if it has not been set, at step 73 the position value is compared to the position value limits set as per the flowchart in FIG. 5. The comparison uses the MIN_LIMIT and/or MAX_LIMIT variables. If both MIN_LIMIT and MAX_LIMIT have been computed following the flowchart in FIG. 5, then the position value is within limits if it is between them; if only MIN_LIMIT has been computed then the position value is within limits if the position value exceeds MIN_LIMIT; if only MAX_LIMIT has been computed, then the position value is within limits if the position value is less than MAX_LIMIT. If the position value is within limits, the routine finishes at step 714, returning an indication to step 37 of FIG. 3 that the position value is within thresholds. If the position value is outside the thresholds, then at step 74 or 75 HYST_TEST is set appropriately; for example if the user has crossed location threshold 62 of FIG. 6, then at step 74 of FIG. 7 HYST_TEST will be set to test if position values are to the right of location threshold 63 of FIG. 6 by setting HYST_TEST to test that a position value is greater than MIN_LIMIT plus the parameter HYST_F. The parameters HYST_F and HYST_B of FIG. 7 are determined by the designer of the embodiment to implement the correct amount of hysteresis in the embodiment, for example increasing HYST_F means the user has to move back further to re-enable the alert. At 77 of FIG. 7, step 37 of FIG. 3 receives an indication that threshold 62 or 64 of FIG. 6 has been crossed and the controller can alert the user at step 38.

If HYST_TEST is found to be set at step 72 of FIG. 7, then at step 78 HYST_TEST is applied to the position value. If the position value fails to meet HYST_TEST—e.g. HYST_TEST is testing for the position value to be to the right of threshold 63 of FIG. 6 and the position value is to the left of it—then at step 79 of FIG. 7 the HSYT_CROSS_TIME variable is cleared. Because this algorithm is designed not to alert the user until certain criteria are met, at step 714 the indication passed to step 37 of FIG. 3 is that the position value is within thresholds, with the result that the algorithm proceeds to step 39 of FIG. 3 and no alert is generated. Note that this indication is made at step 714 regardless of the position value; even if the position value indicates a position to the left of 62 or the right of 64 in FIG. 6, no alert will be generated. Some embodiments may instead implement an algorithm where the user is indeed alerted that they have gone back so far as to have crossed the opposite location threshold, for example they cross threshold 62 from right to left then immediately cross threshold 64 from left to right.

If the position value passes HYST_TEST at step 78 of FIG. 7, then if HYST_CROSS_TIME is not set, HYST_CROSS_TIME is set to the current time. The effect of the manipulations of HYST_CROSS_TIME at steps 76, 79, 710 and 711 is to keep HYST_CROSS_TIME set to the earliest time the position value agrees with HYST_TEST with no subsequent position values that fail to agree with HYST_TEST. At step 712 the current time is compared to HYST_CROSS_TIME, and if the desired amount of time has elapsed—e.g. the user has stayed to the right of 62 of FIG. 6 for long enough—then HYST_TEST is cleared at step 713 of FIG. 7 so that subsequent position values might again trigger alerts. The time interval used in the test at step 712 is determined by the designer of the embodiment to implement the correct amount of hysteresis in the embodiment, for example increasing the required time interval will mean the user has to stay to the right of 62 for a longer time to re-enable the alert.

At step 39 of FIG. 3 the system determines if an appropriate defining moment has occurred. The defining moment in the golfer/smartphone embodiment of FIG. 1 is the moment that the golf club 16 strikes the ball 14, detected by a microphone in the smartphone 13. Other embodiments, depending on the nature of the activity and the target, may detect the appropriate defining moment through sound detection, or detecting the target moving past certain boundaries, or by reaching the end of a time interval. Other embodiments may determine the defining moment for step 39 by detecting an action taken by the user that indicates they have completed the motion, for example the user turning their head or waving their hand may cause these embodiments to determine the appropriate defining moment to have occurred some amount of time prior to the user's signal. For example, in a golf swing, the system may determine the defining moment by detecting the sound of the user's golf club striking a ball. If the user is performing the golf swing without striking a ball, the system may determine the appropriate defining moment—the moment that the club would have struck a ball—algorithmically using data captured from the camera's output, such as detecting the moment that the user has turned their head away from the location of an imaginary ball toward the target at the end of their swing and assuming that the defining moment occurred some time interval before the detection that the user has turned away; the time interval to apply may be a constant determined by the designer of the system, a constant dependent on the club the user is swinging, may be derived from analysis of other moments in the user's swing, may be derived from analysis of other of the user's swings with or without a ball present, may be derived from analysis of swings of other users, or may be derived from some combination of the above. A similar algorithm can be used in golf strokes where the impact of the club on the ball is too quiet to produce a detectable sound. In some embodiments, detection of this defining moment may involve the use of other sensors accessible to the embodiment, such as LIDAR sensors or proximity sensors or photoelectric detectors. For example, in a golf swing, such sensors may detect the moment that the golf ball is no longer present at its initial location, or, when training without a ball, may detect the moment that the head of the golf club passes the location where the ball would have been located. In some embodiments the sensors may be physically integrated with the computing device, for example a LIDAR sensor within a smartphone, and in others the sensors may be external devices, for example a photoelectric detector communicating with a smartphone via Bluetooth. In some embodiments the detection of the defining moment may occur asynchronously and not at the point in the flowchart suggested by the placement of step 39, for example in embodiments built with event-driven or interrupt-driven programming techniques.

If the defining moment has occurred, the system may store indicators relevant to the activity at step 391 of FIG. 3, e.g. how many times boundaries were crossed and whether or not the target was within the boundaries at the defining moment. The boundaries and other information—data to identify the user, date and location of the activity, weather conditions—may be stored as well. If device-based coordinates are stored at step 36, then the information needed to translate between the coordinate systems—for example the origin and scaling factor described above—may be stored as well.

Some embodiments may, upon detecting the defining moment and storing the performance, display the results to the user at step 392. As described below at FIGS. 8 and 15, the display may be an image or animation of the motion. Some embodiments may perform further analysis of the motion at step 392, for example identifying faults that reduce the effectiveness of the motion, and the display may include descriptions of the fault and links to resources to help the user ameliorate the fault and improve their motion, as described below at FIGS. 15 and 16. FIG. 17 is an illustration of one algorithm that may be used at step 392 to identify faults.

Some embodiments may repeat the algorithm of FIG. 3. In some embodiments this may occur after displaying results to the user at step 392 for a certain amount of time, for example 5 seconds. Some embodiments may repeat the algorithm immediately upon storing the results at step 391. Some embodiments may repeat the algorithm a certain number of times, for example ten times, and then display cumulative results to the user.

FIG. 8 is an example of an image that may be displayed by an embodiment of the present invention. In the example, the image contains a display of stored target coordinates of the target 82 representing the path of motion during the activity. Also displayed are the boundaries 83 within which the target should remain during the activity. Markers 81, 84 and 86 label defining moments of the path of motion. 84 indicates the coordinates at the start of the activity, for example, the coordinates of the fiducial position determined at step 33 of FIG. 3; 84 marks the initial position of the target from which the boundaries 83 are set for this particular activity. If the activity is a golf swing, 84 is commonly referred to as the address position, and 81 indicates the location of the target at the moment of impact of the golf club striking the ball, as described above for step 39 of FIG. 3. 86 is an example of an intermediate position that may be useful to display in an activity. For example, if the activity is a golf swing, the moment the user has completed the backswing, commonly referred to as the top of the swing, may be used to differentiate between the backswing and the downswing. This differentiation may be useful in display of the swing, analysis of the swing, and determination of faults with the swing. The defining moment represented by 86 may be determined by detecting the position of the golf club staff or the club head, by analysis of the golfer's head's position, rotation or tilt, by analysis of the golfer's body motion, or by a combination thereof. If applicable for the given activity, the image may also contain the desired direction of travel 85 of an object, such as a golf ball if the activity is a golf swing. The image could also indicate graphically whether an individual boundary was crossed before the defining moment of step 39 of FIG. 39, for instance the boundary could be displayed in the color red. Similarly, 81 could be displayed in a color to indicate whether or not the target was within the boundaries at that defining moment.

In some embodiments the image may be augmented by other information gathered during the activity. For example, in displaying the path of motion of a golfer's head during a golf swing, it may be useful to indicate the distance of the head from the camera, the rotation of the head to the left or right, or the tilt of the head towards the golfer's right or left shoulder, during the swing. This indication can be accomplished, for example, by altering the color or thickness used to draw the line 82 to indicate the value of such other information at the moment in time corresponding to the stored coordinates being drawn. Or an embodiment may add other graphical elements superimposed on or near some points of the line 82, for example displaying the value in degrees of the tilt of the head from vertical at the moment of time corresponding to the coordinates of the points.

Since the stored target coordinates can be ordered by time relative to the time the initial position was determined, the image displayed could display a replay of the motion of the target by moving a display object along the path. This type of animation is well known to those skilled in the art. The animation of the motion of the target could also indicate when the when the target moved outside of a boundary by turning the color of the boundary red until the target moves back within the boundaries. Animation may also be used to indicate other information gathered during the activity, for example in the display of the motion of a golfer's head during a golf swing, an embodiment may include a rectangle in the display and alter the orientation of the rectangle to reflect the tilt of a golfer's head at corresponding moments of the activity. If the activity is a golf swing, the image could also include the amount of time elapsed from the moment when the initial position was determined until the defining moment representing the impact of the club with the ball, as described at step 39 of FIG. 3.

In FIG. 9, the system of FIG. 1 is extended by connecting the computing device 13 to a computer network, for example the Internet. Computing device 13 can forward results to resources in the network 904, including a server system 905 that stores such data, and, for example, makes such data available to a web browser process running in a desktop computer 903 of the user or, optionally of a community of users. The data may include the stored coordinates of step 36 of FIG. 3, or the performance faults determined at FIG. 17 below, or images or videos captured by the camera as the user performs the activity. Additionally, such data can be retrieved and analyzed by other devices with access to that resource, for example an instructor's smartphone 902 or desktop computer 901. The retrieval can occur asynchronously to the performance of the activity. For example an instructor may review a student's data several days after the student's practice session. Or the retrieval may occur in real-time, where the instructor receives the data as the student performs the activity, enabling the instructor to give immediate feedback to the student. The retrieval by the instructor's smartphone 902 can occur automatically when new data has been stored, for example by processes running on the smartphone 902 that continually poll server 905 to check for new data, or processes running on server 905 that communicate to smartphone 902 that new data is available. Training may be further enhanced in some embodiments by augmenting the real-time data retrieval on the smartphone 902 with simultaneous audio or video communication with the student, for example, by holding a phone conversation with the student while the student performs the activity, or by using real-time communication facilities available in the embodiment of the present invention, for example, the FaceTime application available on the Apple Inc. iPhone or the facility provided to smartphone applications by the WebRTC open source project.

In addition to visualization, embodiments can provide query, sorting and statistical functionality that may be applied to the data collected by the device. For example, consider the embodiment of FIG. 1, where a golfer uses the system to train a golf swing. The user may choose to review the data collected by the system in many ways. The golfer may want to see how many times their head or part of the body stayed within the boundaries for swings of a particular type of club each of the preceding sessions, for example, to evaluate their progress in keeping their head steady. Or the golfer may want to see his progress in reducing the occurrence of performance faults detected by the system. Statistical analysis may also be applied to compare a user's data with the data of other individuals or with aggregate data of many individuals. Some embodiments may store statistical analyses for later retrieval to increase the responsiveness and usefulness of the system, for example the files 906 of the performance statistics of individual users or the file 907 of the average performance of the activity aggregated across many individuals.

Suppose an embodiment also stores an indicator of the result of the swing. The indicator can be qualitative—for example an embodiment could ask the user if they felt the swing was good or bad—or quantitative, for example, data on the velocity or path of the golf ball after the strike or the distance the golf ball traveled, perhaps gathered by integration of the device with a launch monitor such as the TrackMan. The user could then compare the visualizations of their good swings with their bad swings to help in their training.

More complex analytical techniques may be also be applied to the data. For example, FIG. 10 depicts an embodiment where an indicator of the result of a golf swing is stored in addition to a series of timestamps and target positions. Using machine learning algorithms, models 1003 or 1004 can be computed that will predict the qualitative result—good or bad 1005—or the quantitative result—expected distance 1006 the golf ball will travel—given the series of timestamps and target positions 1001 or 1002. The model can be built for a single user based on their data, or many users' data can be aggregated to build a more general model. Some embodiments may be capable of building and using such models using their processor, others may rely on a server the device connects with over a network.

In FIG. 11, suppose an embodiment includes a microphone, as described in FIG. 1 to detect the defining moment of the golf club striking the golf ball. The device could record the sound amplitude over time 1101 and 1102 as the club strikes the ball. Rather than, or in addition to, applying machine learning to predict the result of a swing from the series of target positions 1001 and 1002, the system could apply machine learning to compute models 1103 or 1104 to yield qualitative 1105 or quantitative 1106 results from such sound recordings 1101 and 1102. The model mapping sound recordings to swing results may be more universal than models mapping head movement to swing results—predictive of swing results of many users, not just those of the user whose data is used to build the model—so in such embodiments the user could be informed, just after the swing, as to how good the swing was, without having to train a model with their own data.

FIG. 12 is a block diagram of the functional units that may be used to accomplish the method of the present invention in some embodiments. The blocks in the diagram are high-level concepts and may refer to hardware components, to software created specifically for the embodiment, to generic software libraries, or to combinations of those. The process controller 1201 controls the execution of the software by the processor, for example the computing hardware in the smartphone 13. The target detection unit 1202 and the target orientation detection unit 1203 each process the output of the camera 1204 to determine the location and orientation of the target. Embodiments may include only the target detection unit 1202, only the orientation detection unit 1203, or both, depending on the intended use of the embodiment. The system to detect the defining moment for step 39 of FIG. 3 is illustrated by block 1205. The system for detecting that defining moment may involve detecting sound with a microphone, processing images from a camera, processing the output of sensors integrated with the computing device, receiving signals from sensors external to the computing device, or combinations thereof. The alert system 1206 is used to alert the user when the target is outside of the boundaries. Some embodiments may use local data storage functionality 1207 or network communication functionality 1208 to store coordinates of a motion and other data.

FIG. 13 is a block diagram of the functional units that may be used to accomplish the method of the present invention in embodiments where the computing device is a smartphone, such as an Android phone or an Apple iPhone. The process controller 1201 may be realized in a smartphone app 1301 running in the smartphone's processor. Target detection 1202 and target orientation detection 1203 may be accomplished by processing of the output of the camera 1204 by the Google MLKit face detection module 1309 and the OpenCV Open Source Computer Vision Library toolkit 1310 respectively, or other similar software. Defining moment detection 1205 may be accomplished by monitoring the output of the smartphone microphone 1311 or by processing the output of the camera 1204. Google's Flutter portable UI toolkit may be used to provide a user interface 1307 to interact with the user. The user may control the app, receive instructions from the app, and view results through interaction with the smartphone touchscreen 1308 mediated by the user interface 1307. The app may further interact with the user by using the user interface 1307 to send audio prompts to a text to speech module 1313, which may then make the prompt audible through the smartphone speaker 1313. The alert system 1206 may provide the user with audio notifications through the user interface 1307 to the smartphone speaker 1312, and visual notifications through the user interface 1307 to the smartphone touchscreen 1308. In some embodiments, local data storage 1207 or network-based storage mediated through network communication functionality 1208, as well as other desirable features, such as user authentication and real-time data sharing between instructors and students, may be provided by Google Inc.'s Firebase product 1314, available for Google Inc.'s Android smartphone and Apple Inc.'s iPhone.

FIGS. 14A, 14B and 14C are examples of a series of images of screens which may be displayed to a user by an embodiment of the present invention. In FIG. 14A, 1401 is an image of a screen indicating to the user that the embodiment is capturing the initial position of the target. This screen may be preceded by a series of screens and voice prompts instructing the user to set up and perform actions in order to indicate to the embodiment that the user is ready. In FIG. 14B, 1402 is an example of an image of a screen that may be presented to the user to indicate that the user should commence the activity because the embodiment has obtained the initial position of the target. 1402 also displays a set of boundaries, the positioning of said boundaries determined by a set of selectable options chosen by the user, in this example the type of golf club the user is using and the skill level of the user. 1402 also includes a graphical representation of the position of the target as it moves in real time, in this example, a bullseye graphic.

In FIG. 14C, 1403 is an example of an image of a screen that may be presented to the user by the embodiment if the user causes the target to move outside of the boundaries in real physical space. In the example, the user may have moved too far forward relative to a desired direction of travel such that the embodiment has alerted the user of this movement of the target outside of the boundaries visually perhaps by a text message displayed on the screen and a change in color of the specific boundary representing the boundary in real space of which the target has moved outside. In an embodiment of the invention, this occurrence may also be accompanied by an auditory alert, such as a beep sound.

FIGS. 15A, 15B and 15C are examples of a series of images of screens which may be displayed to a user by an embodiment of the present invention. In FIG. 15A, 1501 is an image of a screen that depicts the description of the embodiment of FIG. 8. In the example, the image contains a display 1504 representing the path of motion during the activity determined by the stored target coordinates of the target. Also displayed are the boundaries 1505 within which the target should remain during the activity. 1506 is the initial position of the target from which the boundaries 1505 are set for this particular activity. 1507 indicates the location of the target at the defining moment described at step 39 of FIG. 3. E.g. for a golf swing, 1507 may indicate the location of the target at the moment the golf club strikes the ball. If applicable for the given activity, the image also contains the desired direction of travel 1508 of an object, such as a golf ball if the activity is a golf swing. The image could also indicate graphically whether an individual boundary was crossed before the defining moment of step 39 of FIG. 3, for instance the boundary could be displayed in the color red. Similarly, 1507 could be displayed in a color to indicate whether or not the target was within the boundaries at that defining moment.

In FIG. 15B, 1502 is an example of an image of a screen displayed to a user in an embodiment of the invention that indicates that the target moved outside of the boundaries 1505 during the activity and remained outside the boundaries at the defining moment of step 39 of FIG. 3. In this example, the screen displays a text message 1513 indicating a specific fault that has occurred during the activity and a description of the fault 1514. If more than one fault was detected during the activity, as in this example of FIG. 15, the message 1513 includes an indication of more faults, e.g., a plus sign and then the additional number of faults detected. In some embodiments, the boundary that was crossed during the activity 1509 is shown in red. In this example, the screen offers the user various options to evoke, such as 1510, which may cause the displaying of a replay of the motion of the target by moving a display object along the path as described above. Another option offered to the user may include the ability to display more analysis 1515, such as more faults detected during the activity, and a link to a method of fixing the fault or issue. The link to a fix may open a browser window and display a list of videos created by various professionals or by the instructor associated with the user as described below for FIG. 19. Another option offered to the user may be sharing an image of the screen—or the data used to create the screen—with another person by displaying a button with a share icon 1512. Another option, indicated by the button 1511, may be to compare the path 1504 of the motion of the target to another path, for example a path chosen by the user to be the basis of comparison, or a path generated by the motion of a person known to be highly skilled in the activity, such as a professional golfer.

In FIG. 15C, 1503 is an example of an image of a screen displayed to a user in an embodiment of the invention that may be displayed if the user taps the button 1511. In this example, the screen displays the path 1504 of the motion of the target during the user's activity and also the path 1516 of the motion of the target during the activity of a person known to be highly skilled in the activity, such as a professional golfer. Alternatively, the path displayed may be a path chosen by the user from among their own stored paths to be the basis of comparison. The specific path of the motion of the target displayed for comparison to that of the user's may be matched to the type of equipment used by the user in the activity in order to make the comparison more useful. In this example, the user's motion is compared to that of highly skilled person in an activity in which both used a specific type of golf club, a wedge. In the example, the embodiment may offer the user the ability to display an animated replay of the path 1504 of the target during the user's or the highly skilled person's activity. The display may also include a measure of the similarity between the two paths, for example, in golf, the differences in the ratios of the duration of backswings to downswings, to aid the user in evaluating their activity.

FIG. 16 is a depiction of another screen which may be displayed to a user by an embodiment of the present invention. In embodiments, this screen 1601 may be displayed if the user taps the button 1515 shown in FIG. 15B. In the example in FIG. 16, the image displays a popup window 1602 which contains more analysis and information pertaining to faults detected during the activity. In this example, 1603 details the primary fault and provides a description useful to the user, and button 1604 provides a link to a method of fixing the fault or issue. When the user taps the button 1604, the device may play a video, or may open an Internet browser window and display a list of videos created by various professionals or by the instructor associated with the user as described below for FIG. 19. If other faults were detected during the activity, at 1605 a further list of these faults may be included.

FIG. 17 is a flowchart illustrating an algorithm used in an embodiment of the present invention to determine faults with the user's motion. Embodiments may detect a variety of faults in analyzing a user's motion. For example, lateral and vertical movements beyond boundaries, forward and backward movement, body sways, head tilts and head rotations may be detected by some embodiments. In analyzing a motion for faults, specific critical moments may also be identified. For example, in a golf swing, the moment of maximum lateral movement away from the target of the swing can be identified as the moment separating the backswing from the downswing, and the analysis can identify faults only relevant to the backswing or the downswing. In some embodiments, the rotation or tilt of the head or other body part may be used in the analysis to identify other faults.

Step 1710 represents the beginning of the algorithm. At step 1701, the processor retrieves the data relevant to the activity; in some embodiments the data may be retrieved from storage associated with the processor, for example, the non-volatile memory of a smartphone, and in some embodiments, the data may be stored in external storage and retrieved over a network, for example, requesting the data from a remote database using the functionality provided by Google Inc.'s Cloud Firestore product in the Firebase offering. At step 1702, various factors of the activity are computed. For example, if the activity is a golf swing, relevant factors may include: the number of times each virtual boundary was crossed; the moment of the start of the swing, the top of the swing, and impact with the ball; and the ratio of the time interval from the start of the swing to the top of the swing to the time interval from the top of the swing to impact with the ball, called the rhythm of the swing. Steps 1703, 1707 and 1708 represent the algorithm looping through a set of faults. At step 1704, each fault is individually considered. In the example of FIG. 17, the fault under consideration at step 1704 is defined as “Too forward at impact”. The set of rules for this fault are: the golf club being swung is “drive”, “iron” or “wedge”; the front boundary must have been crossed; the cross of the front boundary must have occurred after the top of the swing; and the cross of the front boundary must have occurred before impact. At step 1705, the swing is tested against the set of rules for this fault, and if the swing meets the conditions of each rule, then, at step 1706, this fault is added to the set of faults for this activity (swing). The algorithm completes at step 1709.

Embodiments may use many types of rules for determining a fault. The above example fault has a set of rules that include considering the type of equipment used (golf club in this example), considering whether or not a particular boundary has been crossed, and considering the moment that a particular boundary has been crossed relative to other defined moments in the swing. To ease development and implementation of the rules processing, note that some example rules may be thought of as conditions with parameters, such as which types of equipment may qualify (“drive”, “iron” and “wedge” in FIG. 17), which boundary must be crossed (“front”), how far from the fiducial position the boundary is located, and the order and timing of moments of interest (“crossing before impact”). The parameters may vary based on other elements of the activity or the training, such as modifying the locations of the boundaries based on the equipment being used or the skill level of the user. The conditions and parameters of the rules may be determined by analyzing the activity, such as a professional golf instructor determining that crossing the front boundary between the top of the swing and impact will reduce the efficacy of the golf swing because of the resulting orientation of the golf club at impact—or by collecting empirical data relative to the activity, such as measuring the average distance the golf ball travels when a golfer does and does not cross the front boundary between the top of the swing and impact. Rules may be implemented that consider the orientation of the user or the distance of the user from the computing device at particular moments in the activity, or minimum and maximum values of such orientation or distance throughout the activity. In some embodiments, someone skilled in the art will be able to revise existing rules and develop and implement additional rules.

The faults detected by the algorithm illustrated in FIG. 17 may be used in various ways by embodiments of the system. For example, the faults may be displayed to the user as shown at 1502 or 1603. In some embodiments, the faults may have associated severity indices, for example, a value between 1 and 10, where 1 represents that the fault has no little effect on performance and 10 represents that the fault has maximal impact. Then, the system of the embodiment may display the fault in the set of faults for this swing with the highest severity.

A user may elect to repeatedly perform an activity during a continuous period of time to aid in training. For example, in training to improve a golf swing, a user may perform ten golf swings—e.g., by having the system perform the algorithm of FIG. 3 ten times—during a single session and then review their performance.

To aid in training, an embodiment of the present invention may indicate to the user a fault with each performance of the activity. Some embodiments may further improve training by summarizing the faults across all the activities in a session. Some embodiments may summarize faults for subsets of the activities in a session. For example, in training to improve a golf swing, an embodiment may summarize faults according to the club used to perform the activity, e.g., summarizing faults for a driver and faults for an iron and faults for a putter. Some embodiments may further improve training by retaining the most salient fault or faults from a training session and displaying it to the user at their next training session, as a fault or faults to focus on eliminating from their performance of the activity. Some embodiments may then provide an indication to the user of how well they have progressed in eliminating the fault or faults from their performance of the activity.

FIG. 18 illustrates an algorithm that may be used in some embodiments to determine the most salient fault affecting a user's performance across a set of activities. Consider a training session, where the activity is repeated several times. Step 1805 represents the beginning of the algorithm. At step 1801, each activity in the session is applied to the algorithm depicted in FIG. 17 to create a list of faults for that activity. At 1802, the individual fault lists corresponding to each activity are combined to create a list of faults, together with the number of times each fault occurred across all the activities in the session. At step 1803, each fault in the combined list is assigned a weighted severity, for example, by multiplying the number of occurrences of the fault across all the activities in the session by the fault's severity index. At step 1804, the faults are sorted by their weighted severity. This sorted list can be presented to the user to inform them of the most important factors that affected their performance in the session. This sorted list can further be used to suggest a focus or multiple foci for future sessions, for example, to improve a user's training by suggesting the user try to alter his motion to eliminate the most severe fault, and some embodiments may provide feedback to the user as to the user's progress in addressing the focus or multiple foci over multiple performances of the activity.

It may be advantageous for a user to associate with an instructor to aid the user in training. An embodiment of the present invention allows the instructor, thus associated with the user, to view a display of the stored coordinates of the performance of the activity. An embodiment may allow the instructor to view the faults that were detected in the user's performance of the activity. The viewing is provided in real time, as the coordinates are stored in a server system, although in other embodiments the viewing is provided at a later time.

It may be further advantageous for the associated instructor to direct the training for the user. An embodiment of the present invention allows the instructor to provide his own fixes for faults detected by the system. An embodiment of the present invention may allow the instructor to enable and disable the detection of particular faults for the user. An embodiment of the present invention also allows the instructor to specify which fault the user should focus on eliminating from the user's performance of the activity.

Some embodiments of the present invention allow an instructor to add additional faults to be detected, or change the algorithm used to detect particular faults. For example, if an embodiment of the present invention includes a smartphone app 1301 of FIG. 13, the smartphone app 1301 may be a version of the smartphone app developed specifically for the instructor, where the source code of the smartphone app has been revised for the instructor. Adding a fault can be accomplished by adding to the set of faults considered in the iteration at 1708 of FIG. 17, and the algorithm used to detect a particular fault may be changed by altering the tests used at 1704.

Some embodiments of the system allow a user to associate with an instructor to aid in the user's training, as depicted in the method of FIG. 19. Step 1905 represents the beginning of the algorithm. At step 1901, a user enters an identifier (e.g., ProID) for the instructor, which enables the user to access some data stored in the system by the instructor and enables the instructor to access some data stored in the system by the user. At step 1902, the identifier for the instructor is used to retrieve content provided by the instructor. For example, the instructor can provide the links to be used at 1604 for training the user to overcome a fault. In some embodiments the links may be videos recorded by the instructor relevant to the fault. At step 1903, the instructor is able to access the user's motion data, for example, to view the images of the user's motion as depicted in FIG. 8, or the analysis of the user's motion as depicted at 1502, either on demand or in real-time, as described in connection with FIG. 9. At step 1904, the instructor is able to remove faults for consideration in analyzing the user's motion in FIG. 17; add new faults to be considered; specify the severity of faults to be applied at step 1803 of FIG. 18; or specify the most important fault that the user should address in the user's training, in some embodiments overriding the results of the algorithm depicted in FIG. 18.

Some embodiments implement the method of FIG. 19 with the use of the computer network and server of FIG. 9 and the network communications functionality 1207 of FIG. 12. Enabled by any of the variety of mechanisms well known to someone skilled the art, an instructor stores information about their preferred video fixes in the server 905. For example, using Google Inc.'s Firebase product 1314, the smartphone application 1301 can enable the instructor to store Internet URLs as text strings in the Firebase Cloud Firestore database, or such URLs can be entered by the instructor or other authorized users using the browser-based interface provided by Google Inc. to the Cloud Firestore database. In such embodiments, at step 1901, the user enters the ID of their instructor (e.g., ProID) via the user interface 1307 and the smartphone app 1301 communicates with the server 905 via network communications facility 1207, storing the instructor ID with other information stored for the user in the server 905. At step 1902, the smartphone app requests from the server 905 information about the video fixes that have been associated with the instructor. At step 1903, the instructor queries the server 905 for information about users who have stored his instructor ID in this manner, selects a user, and accesses data stored by that user. At step 1904, the instructor may store data at the server 905 specifying faults that the user should focus on eliminating, and the user's smartphone app 1301 communicates with server 905 and retrieves the faults stored by the instructor and may indicate the progress that the user makes towards eliminating those faults.

In embodiments of the present invention as depicted in FIG. 1, it may be desirable to direct the user to position themselves in two phases, as depicted in the flowchart of FIG. 20. First, the user places the smartphone 13 and is directed to orient it properly, and then the user, now free to move away from the smartphone 13, is directed to position his own body. In the setup depicted in FIG. 1, in the absence of this two-phase approach, the user may have to repeatedly move the phone, step back into the user's golf address position, find that the setup is incorrect, step forward and move the phone, and so on.

Step 2010 represents the beginning of the algorithm. At step 2001 of FIG. 20, the user places the smartphone 13 in the appropriate position for the activity. For example, if the activity is golf, the smartphone 13 should be placed on the ground two feet from the location of the golf ball, represented in FIG. 1 as the distance from 13 to 14. At step 2002, the embodiment determines the angle of the smartphone from the vertical direction. For example, if the smartphone 13 has an accelerometer that returns the three-dimensional vector (x, y, z) corresponding to the strength and direction of the field of Earth's gravity, and if the y-coordinate of that vector represents the strength and direction relative to the vertical axis of the smartphone and increases towards the top of the phone, and the z-coordinate represents displacement perpendicular to the face of the smartphone and increases in the direction from the back to the front of the phone, then the angle to be measured is the arctangent of the z-coordinate divided by the y-coordinate of the orientation vector when the y-coordinate is non-zero, and the angle is 90 degrees when the y-coordinate is zero.

At step 2003, the determined angle is tested to see if the smartphone is oriented in a position so that the user is visible to the smartphone camera. For example, if the golfer is to be positioned four feet from the camera—e.g., the distance from 13 to the feet of the golfer 11 in FIG. 1 is 4 feet—the angle may be required to be between 27 degrees and 37 degrees. The angular range may be determined empirically by positioning golfers in front of various cameras, or may be determined from the geometry of the setup and the field of view of the smartphone camera. If the angle is not in the desired range, then at step 2004, if the angle is less than 27 degrees, then the user is instructed to tilt the phone further backward—tilting it so the phone is less vertical—or if the angle is greater than 37 degrees, then the user is instructed to tilt the phone forward—more vertical—and then the algorithm returns to step 2002.

At step 2005, the phone has been determined to be positioned correctly and the user can move away from the phone. At step 2006, the target detection capability of smartphone 13—e.g., 1202 of FIG. 12 or the Google ML Kit face detection module 1309 of FIG. 13—is used to attempt to detect the face of the user. Note that there are issues that may make it impossible to detect the face. For example, the user may be unusually tall or unusually short, or the field of view of the smartphone camera may be unusually narrow, so that the test for the appropriate angle from the vertical, at step 2003, is incorrect. Or the module 1202 fails to detect a face due to lighting conditions or hardware failure. An embodiment must be prepared to abandon the algorithm of FIG. 20 if no face is detected and may provide other functionality to aid the user in diagnosing these issues.

At step 2007, the location of the target is tested to see if it is in the desired position in the camera image. For example, in an embodiment that uses the face of the user as a target and is designed to train the user in a golf swing, it is desirable that the face be near the top of the camera image so that the golfers' body can be photographed in embodiments that store images of the golfer's swing, or so that other targets, such as the position of the golf club, can also be detected in the course of the swing. But the face should not be too near the top of the image, because the golfer's face may move vertically during the course of the swing. For example, in some embodiments with an image height of 320 pixels, it may be desirable that the top of the face appear lower than 5 pixels from the top of the image, and higher than 100 pixels from the top of the image. As with the angular limits in the tilt of the smartphone, these image position limits can be set empirically or by analysis of the geometry.

If the target is not found within the desired limits in the image, then the user is told, at step 2008, to either move back from or forward towards the smartphone 103. If the location of the target is too close to the top of the image, the user is told to move back; due to the tilt of the smartphone, moving back from the smartphone will cause the location of the target to descend further from the top of the image. Similarly, if the location of the target in the image is too far from the top of the image, the user may be instructed to move forward towards the smartphone 13. The method of FIG. 20 ends at step 2009.

Embodiments of the system may be used to train the motion of a wide variety of activities. The system may be used to train motions from a set of sports activities requiring a swing, e.g., golf, hockey, baseball, softball, tennis, badminton, ping pong, volleyball, racquetball, martial arts, cricket, bowling, or lacrosse. In addition, the system may be used to train the motion for a wide variety of activities ranging from medical surgical procedures, such as a hand movement, to the correct method to lift boxes in a warehouse, employing the leg muscles while avoiding straining one's back. In order to train a given motion, the system may track targets including body parts such as the face, head, knee or hip, or equipment such as clubs, rackets, bats and balls, by incorporating image processing algorithms capable of detecting the target.

The present invention may be embodied in many different forms, including, but in no way limited to, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, integrated circuitry (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof.

Computer program logic implementing all or part of the functionality previously described herein may be embodied in various forms, including, but in no way limited to, a source code form, a computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, networker, or locator.) Source code may include a series of computer program instructions implemented in any of various programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, C, C++, JAVA, Dart, Swift, JavaScript, or HTML) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code may be converted (e.g., via a translator, assembler, or compiler) into a computer executable form.

The computer program may be fixed in any form (e.g., source code form, computer executable form, or an intermediate form) either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card), or other memory device. The computer program may be fixed in any form in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies, networking technologies, and internetworking technologies. The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software or a magnetic tape), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).

Hardware logic (including programmable logic for use with a programmable logic device) implementing all or part of the functionality previously described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using various tools, such as Computer Aided Design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL).

While the invention has been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended clauses. While some of these embodiments have been described in the claims by process steps, an apparatus comprising a computer with associated display capable of executing the process steps in the clams below is also included in the present invention. Likewise, a computer program product including computer executable instructions for executing the process steps in the claims below and stored on a computer readable medium is included within the present invention.

The embodiments of the invention described above are intended to be merely exemplary; numerous variations and modifications will be apparent to those skilled in the art. All such variations and modifications are intended to be within the scope of the present invention as defined in any appended claims.

Claims

1. A computer-implemented method of training a user in performing a motion for an activity, the method employing an AV-equipped portable computing device having a processor, the computing device having been positioned so that its camera faces a target, the processor executing computer processes defined by an application, the computer processes comprising:

receiving and processing by the processor an output from the camera to determine an initial position of the target;

processing the output from the camera by the processor to determine a series of coordinates corresponding to successive positions of the target, assumed after the initial position in the course of the motion;

processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion, wherein the fault rules relate to target position in a real, physical space of the target; and

if a fault is determined to be present, causing by the processor notification of the user via components of the portable computing device, so as to train the user to perform the activity in a manner free of the fault.

2. A method according to claim 1, wherein the fault rules additionally relate to whether the target has moved to a position outside of one of a set of boundaries determined by distances established for motion of the target relative to the target's initial position in the real, physical space.

3. A method according to claim 2, wherein the fault rules additionally relate to a time of occurrence of a moment when the target has moved to a position outside the boundary relative to a time of occurrence of a defining moment of the motion, and the computer processes further comprise:

processing by the processor the series of coordinates to determine (a) the time of occurrence of the defining moment and (b) the time of occurrence of the moment when the target has moved to a position outside the boundary.

4. A method according to claim 1, wherein the fault rules additionally relate to target position in three dimensions of the real, physical space.

5. A method according to claim 4, wherein the fault rules additionally relate to target orientation in the real, physical space.

6. A method according to claim 1, wherein the computing device has a display, and the notification of the user is by visual notification on the display, and the computer processes further comprise:

providing, in the display, a link to a resource for addressing the fault.

7. A method according to claim 6, wherein the link for addressing the fault is determined by user-specified criteria for training instruction based on items selected from the group consisting of identification of an instructor, level of instruction desired, and combinations thereof.

8. A method according to claim 1, wherein the set of fault rules are defined based on input from an individual identified by the user as an instructor.

9. A method according to claim 1, wherein processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion includes identifying the presence of a plurality of faults and selecting from the plurality of faults a most salient fault, and wherein the motion is performed in a set of repetitions by the user and the computer processes are executed for each repetition of the set for identifying, separately for each such repetition, the presence of the plurality faults, and the computer processes further comprise:

determining by the processor, cumulatively for the set of repetitions, the most salient fault based on of its frequency of occurrence and its severity.

10. A method according to claim 1, wherein processing by the processor the series of coordinates, against a set of fault rules, to determine presence of a fault in the performing of the motion includes identifying the presence of a plurality of faults and selecting from the plurality of faults a most salient fault, and wherein the motion is performed in a set of repetitions by the user and the computer processes are executed for each repetition of the set for identifying, separately for each such repetition, the presence of the plurality faults, and the computer processes further comprise:

determining by the processor, cumulatively for the set of repetitions, the most salient fault based on input from an individual identified by the user as an instructor.

11. A method according to claim 1, wherein the computing device includes a display, and the computer processes further comprise:

storing the series of coordinates corresponding to the successive positions of the target assumed after the initial position, and an instant in time when each position was assumed; and

computing and causing display of a path of movement of the target, based on the series of coordinates corresponding to the successive positions of the target assumed after the initial position.

12. A method according to claim 11, wherein the computing device is configured to communicate over a network, and the computer processes further comprise:

providing by the processor, to an individual identified by the user as an instructor, a notification that the series of coordinates have been stored at a storage location in the network;

retrieving by the processor, in response to input by the identified individual, the series of coordinates from the storage location; and

causing display by the processor of a representation of the series of coordinates.

13. A method according to claim 1, wherein the computer processes further comprise:

storing the series of coordinates corresponding to the successive positions of the target assumed after the initial position, and an instant in time when each position was assumed;

detecting a defining moment in the activity; and

on detecting the defining moment, storing the series of coordinates and the instant in time each position was assumed.

14. A method according to claim 13, wherein the computing device includes a microphone and the defining moment is a ball-strike, and the computer processes further comprise:

processing by the processor sound detected by the microphone to detect the ball-strike.

15. A method according to claim 13, wherein the computer processes further comprise:

processing by the processor of the output from the camera to detect the defining moment, and wherein the processing of the output includes computing the orientation of the target.

16. A method according to claim 1, wherein determining the initial and successive positions of the target includes tracking initial and successive positions of another target of the user.

17. A method according to claim 1, wherein the computer processes further comprise:

storing by the processor a quality of a result of the activity.

18. A method according to claim 17, wherein the quality of the result of the activity is an indication provided by the user, and the computer processes further comprise:

prompting by the processor the user to provide the indication of the quality of the result of the activity and storing the indication.

19. A method according to claim 11, wherein the series of coordinates constitutes a first series of coordinates, and the computer processes further comprise:

displaying a path of movement based on a second series of coordinates corresponding to successive positions of the target in performing a second motion for the activity; and

providing a visual comparison of measurements between the first series of coordinates and the second series of coordinates.

20. A computer-implemented method of establishing orientation of an AV-equipped portable computing device, for monitoring a user in performing an activity, the device having an accelerometer and a processor, the processor executing computer processes defined by an application, the computer processes comprising:

receiving and processing by the processor an output from the accelerometer to determine current orientation of the portable computing device;

directing by the processor to the user to cause a change in orientation of the computing device in a manner suitable for monitoring the user;

receiving and processing by the processor an output from the camera to determine a position of a target; and

directing by the processor positioning of the user to facilitate capture of the motion of the target in the course of the motion.