SYSTEMS AND METHODS FOR DATA-DRIVEN MOVEMENT SKILL TRAINING

Abstract

A data-driven movement skill training system is disclosed. The system uses movement skill assessment and diagnostics at distinct levels of the human movement system hierarchy to specify training goals for a user. The system may provide various augmentations that are synthesized to help the user pursue the training goals. The system may include features to track and/or manage training or learning processes.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent Application No. 62/529,412, filed Jul. 6, 2017, entitled “Systems and Methods for Data Driven Movement Skill Training,” which is hereby incorporated by reference in its entirety.

FIELD

Disclosed are devices, systems, and methods for movement skill training.

BACKGROUND

Humans rely on motion skills to perform daily tasks ranging from actions essential to our autonomy to more specialized domains requiring highly refined motion skills. Professional athletes, musicians, surgeons, and even elite amateurs require thousands of hours of systematical and focused training, as well as continued training to maintain high skill levels. Even simple daily acts involve complex coordination of a range of processes, from sensing and motor-control to perception and cognition. Learning, maintaining, and rehabilitating movement skills are valuable, but at the same time, complex and challenging tasks. Acquiring and maintaining specialized movement skills takes time. Progress of movement skills does not develop linearly with training time. Rather, skills progress following a power law with measurements suggesting that some skills continue to improve for over 100,000 trials.

Different factors account for this slow progress in skills. Movement performance relies on a broad range of functions (e.g., sensory, perceptual, planning, cognition). Many movement skills within the category of complex movement are unnatural and therefore require adaptation of innate movement skills to accommodate the specific task requirements. Complex movement also involve the coordination of large numbers of muscles and body segments. They may take place over short time-frames, with critical phases spanning 10th to 100th of a few millisecond. They often need to be adapted during performance and synchronized with external events or elements. Movements are typically learned by trial and error, mostly by using some outcomes as feedback for corrections. Due to these complexities, the specific details regarding movement organization are stored in procedural memory and therefore are only known implicitly. Explicit knowledge surrounding movement details are typically not used during practice and execution. The fact that complex movements often unfold quickly and involve many dimensions make them hard or impossible to fully be perceived let alone comprehended. For example, just the path of a piece of equipment, such as a tennis racket, already involves three translational and rotational variables (e.g., six degrees of freedom) with their additional kinematic (speeds and angular rates) as well as dynamic (accelerations) characteristics.

Movement complexity grows dramatically when the various body segments and musculoskeletal and neuro-motor constraints are included. To make matters even more complex, these variables are constrained by the dynamics, which constrain their spatial and temporal evolution. Finally, there are very few feedback stimuli, or signals, available to a user during the training process. As a result, for most people who don't have access to coaching, movement skill relies on self-observation and tedious repetition. In many domains, proficiency cannot be achieved without the assistance of an expert coach or trainer.

Challenges also exist in characterizing and assessing movement. First, human movement is variable. Each repeated trial of the same task results in a slightly different execution. Second, technique is idiosyncratic. Individuals with the same general level of ability have a different approach and style. Third, movement is fast. Often an action unfolds within a fraction of a second, with relevant details only spanning a few milliseconds. Fourth, movement is complex. It often focuses on the control of an end effector, such as a tool (e.g., surgical instrument) or piece of equipment (e.g., tennis racket, baseball bat, golf club), which need to be controlled in three dimensional workspaces. The execution of such movements, requires controlling the various limbs, joints, and muscles, which add many more additional degrees of freedom.

Moreover, because the coordinated motion patterns are typically too complex and execute too quickly to be perceived and processed consciously, it is usually difficult to make training interventions in real time. Athletes or operators usually do not have a sufficiently explicit awareness of the details of their motion execution. These characteristics explain why it is difficult to improve skills once basic motion patterns are acquired. External feedback from a trainer or coach become necessary in order to improve.

Movement skills also depend on a perceptual understanding of the external task elements. These characteristics are much harder to assess from observations of the movement performance. They manifest indirectly in the performance. A good instructor will call attention to important perceptual cues and how these can be used to inform the movement response characteristics.

Finally, one requirement for effective training is to account for individual differences in body type, skill level, health, etc. Such characteristics are much harder to take into account during training. This can be particularly critical for rehabilitation or when working with injured or aging athletes. A training approach should also leverage the properties and natural learning principles and processes of skill development.

Popular wearable and embedded devices currently primarily focus on the identification and tracking of activity (e.g., FITBIT® activity tracker (available from Fitbit, Inc.) or JAWBONE® fitness tracker (available from Aliphcom doing business as Jawbone)). Popular examples of fitness trackers include devices for counting steps and tracking distance covered. More advanced capabilities can be found in devices that are specialized for a particular sport. Tennis, badminton, and golf represent the largest market segments (see, e.g., BABOLAT PLAY™ (from Babolat, France), ZEPP® tennis swing analyzer (available from Zepp US, Inc.), and the Smart Tennis Sensor (available from Sony)). These products aim to provide a description of players' technical performance. Typical features include tracking the type of actions; reconstructing movements, such as the path of the tennis racket during a stroke; tracking select outcome variables of actions such as the racket head speed, the distribution of impacts on the string bed, and the amount of spin.

The outputs of these assessments are typically provided after a training or play session. The data is presented as summaries of session performance, as well as time. The data is also aggregated to provide statistical trends. The main shortcoming of these products is that the analysis is based on outcome variables (referred to as knowledge of results in the human skill literature) and thus does not provide actionable information that can be leveraged directly for training.

One of the most established frameworks for training is the so-called “deliberate practice.” Ericsson developed this framework after reviewing bodies of evidences concerning the conditions of optimal learning. He found that individualized practice with training tasks, selected by a coach or teacher, with clear goals designed to improve a particular aspect of performance, and immediate and informative feedback was associated with best learning.

Deliberate practice enables one to fully engage in a training activity. This engagement can play a critical role. It has been shown that the regular performance of an activity, without deliberate practice, does not lead to improvements past the competency level (Ericsson, 2007). This phenomenon is in part explained by the fact that as part of the natural learning process, the brain learns to automate a significant part of the performance. The automatization itself limits the ability to make adjustments in technique unless very deliberate efforts are applied to identify weaknesses in performance and set goals to address those weaknesses. This is the reason why people get set in their technique and habits.

Besides immediate feedback, other potentially critical aspects of deliberate practice include having training goals that provide a gradual path toward the refinement of one's skills, and providing opportunities to engage in a form of problem solving.

Skill acquisition follows an incremental process; therefore, most people's skills can be considered at some intermediate level that could be further developed. Each successive iteration along a path to improving skills involves increasingly complex mental representations and their supportive functions such as movement coordination, and perception (Ericsson, 2009). It seems that skill acquisition stabilizes along successive skill levels.

No method currently exists for generating goals from performance measurements. In addition, no algorithms currently exist that enable a data-driven, operationalized training process that automatically determines and updates goals as an agent learns skills for an activity.

SUMMARY

Data-driven movement skill training systems are disclosed herein. The systems may use movement skill assessment and diagnostics at distinct levels of the human movement system hierarchy to specify training goals. The systems may then provide different forms of augmentations synthesized to help pursue the training goals. The system may also include a system to track and/or manage the learning process.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a human augmentation system for movement skill training or rehabilitation according to an embodiment.

FIG. 2 is an illustration of an interaction between a stroke motion and task and environment elements, including ball trajectory relative to a court, the impact of the ball, and its bouncing before the interception with the racket trajectory. FIG. 2 also illustrates the gaze of the player along different points of the ball trajectory and court locations, and shows a ball machine as an apparatus that can be programmed to enable different forms of interactions.

FIG. 3A is an illustration of a general movement trajectory envelope delineating the movement phases that typically arise from biomechanical and neuromotor constraints.

FIG. 3B is an illustration of the finite-state model representation for the system shown in FIG. 3A, where each state represents a movement phase.

FIG. 4 is an illustration of the primary movement unit for six movement activities, along with corresponding phase segments. The figure also highlights primary outcome quantities as vectors (e.g., effect of a racket or club on a ball for tennis or golf, effect of arm coordination on hand placement in rehabilitation, propulsive force generated by a foot strike for running, transversal acceleration used in turning while skiing, and propulsive force generated by the pull phase in swimming).

FIG. 5 is an illustration of the progression across different tennis stroke architectures corresponding to different skill levels. The architecture is shown in terms of its constituent movement phases.

FIG. 6 is an illustration of an overview of the movement processing components that form the basis for data-driven skill assessment, learning process analysis, and larger population analysis.

FIG. 7 is an illustration of different outcome levels for tennis and some of the outcome measurements: 1) stroke technique and racket impact; 2) stroke primary outcomes; 3) shot trajectory and type; 4) shot placement (relative to the court and opponent). FIG. 7 also shows trajectories for two types of strokes e.g., flat (FL) and topspin (TS).

FIG. 8 is an illustration of two players' respective ground impact distributions from tennis shots, describing the discretization of the task environment associated with the ball-environment interactions. The skills at the shot level manifest as different resolutions and precision in the interactions with the task environment. FIG. 8 also shows court landmarks that are relevant to the players' task environment perception and the players' court motion.

FIG. 9 is an illustration of three interception types: 1) on the descent; 2) at the apex; 3) on the rise of the trajectory following the ground impact. FIG. 9 also illustrates the racket string bed associated with the corresponding impact conditions and examples of the return shot outcomes.

FIG. 10 is an illustration of the levels of assessment highlighting the elements and outcomes for tennis and summarizing the assessment and diagnostic components across the different levels.

FIG. 11 is an illustration of the movement acquisition as an evolutionary process during which movement patterns are learned either from scratch or through differentiation of existing patterns.

FIG. 12 is an illustration of movement pattern clusters based on features extracted from movement data.

FIG. 13 is a tree diagram illustrating the evolutionary relationship between movement patterns.

FIG. 14 is an illustration of the state space X (measured or estimated dimensions derived from performance data that are relevant to describe the movement behavior), highlighting the classes associated with the movement patterns, and the mapping associated with the outcome space or other attributes used in the skill assessment. The figure also shows the embedding from V into a subspace W that produces meaningful outcome categories (semantic interpretation).

FIG. 15 is an illustration of a generic outcome-movement pattern map showing the movement patterns in terms of associated outcomes dimensions.

FIG. 16 is an illustration of a generic repertoire map showing the movement pattern classes arranged in terms of outcome dimensions that have been rescaled such as through the embedding g: V->W.

FIG. 17 is an illustration of the skill profile of two subjects A, B resulting from their repertoire of movement patterns or skill elements, highlighting the difference in skill profile that results in an overall skill gap as well as a gap in the repertoire range.

FIG. 18 is an illustration of population subgroups based on skill attributes with level lines associated with a performance or skill objective function.

FIG. 19 is an illustration of distribution for an entire population group described by the group distribution highlighting a member (subject A, described by skill element distribution (e₁, e₂)), and the tiers (low, medium, high, very high) associated with an outcome function for the entire population subgroup (e_1,G, e_2,G).

FIG. 20 is an illustration of distribution of motion patterns produced by subject A described by two features (f₁, f₂), showing the center of the ellipsoid (μ₁, μ₂) and the axes given by the eigenvalues (e₁, e₂), and highlighting the level lines of some outcome tiers (low, medium, high, very high).

FIG. 21 is an illustration of a data-driven closed-loop training system including its primary processes organized according to three primary feedback loops.

FIG. 22 is an illustration of a Human Augmentation System. The system encompasses three primary tiers of augmentation that leverage the human information processing hierarchy: real-time feedback (cue stimuli and activity interactions), intermittent feedback, and visualization and instructions.

FIG. 23 is an illustration of the augmented perception-action loop associated with the feedback cueing system. Low-level signal and cues are emphasized.

FIG. 24 is an illustration of the main components used for feedback augmentation in the augmented human movement system.

FIG. 25 is an illustration of process flow along the training process shows an activity across the stack of processes of an assessment and training loop (e.g., data acquisition and processing, motion model, skill model, training goals, augmentation laws) as a function of time. During each session, activity data is collected and processed.

FIG. 26 is an illustration of a diagnostic system building on the assessment system.

FIG. 27 is an illustration of a diagnostic system, which combines a knowledge representation, observations, and an inference mechanism to produce a diagnostic of the movement performance.

FIG. 28 is an illustration of the factors influencing stroke quality (categorized as observations, uncertain factors, or hypotheses) and their relationships.

FIG. 29 is an illustration of population analysis and player or performer profiling.

FIG. 30 is an illustration of an assessment, diagnostics, and training goals across the skill-model hierarchy, incorporating player profile information.

FIG. 31 is an illustration of an assessment including a) different levels of assessment, b) elements that describe each level, c) criteria and quantities that can be used to determine the skill characteristics at a given level, d) analysis or diagnostics to identify the critical characteristics, e) the drivers and mechanisms used to produce training interventions, and f) the intervention or feedback form that can be used.

FIG. 32 is an illustration of the primary outcome characteristics (i.e., pace and spin) for a player's groundstroke repertoire with overall reference ranges from population analysis (gray background tiles).

FIG. 33 is an illustration of spin envelope for the groundstrokes (solid line) divided into forehand and backhand with reference ranges from population analysis (dashed lines).

FIG. 34 is an illustration of a leaderboard for population analysis based on the global score shown as a percentile rank from highest to lowest computed from the skill profile of a population of players.

FIG. 35 is an illustration of play activity summation over a calendar period showing sets and sessions.

FIG. 36 is an illustration of movement outcome trends for a specific motion pattern class with overall reference ranges from population analysis (gray background tiles). The vertical bands delineate the sets.

FIG. 37 is an illustration of the forward swing phase stroke profile for forehand topspin stroke class.

FIG. 38 is an illustration of selected components of a skill element including outcome, attributes, and other characteristics forming a composite skill element score. Two polygons are superposed to provide a comparison.

FIG. 39 is an illustration of the trends in movement patterns and movement outcomes for an activity session delineated in individual sets.

FIG. 40 is an illustration of the skill profile as a bar graph for the values from a composite score across a repertoire of groundstrokes.

FIG. 41 is an illustration of the acquisition stage for the strokes in the groundstroke repertoire.

FIG. 42 is an illustration of impact timing statistics for a player's groundstroke repertoire with overall reference ranges from population analysis (gray background tiles).

FIG. 43 is an illustration of the integrated perspective on the assessment and diagnostic process organized in terms of the assessment levels (i.e., physical, pattern, task, and competitive).

FIG. 44A is an illustration of a skill status screen showing the skill elements arranged according to their acquisition stage (Patterns to Form, Patterns to Consolidate, and Patterns to Optimize).

FIG. 44B is an illustration of a skill status screen showing how training activity over several training sessions (Set 1-3) lead to a change in the skill status of skill elements.

FIG. 45A is an illustration of a training list showing selected training elements e_i and their associated training goals g^k_i expressed in terms of attributes a_i for an epoch k. The list is indexed according to relevant criteria, such as user preference or importance of the element to the activity or to the skill acquisition process.

FIG. 45B is an illustration of a training schedule. The training session is subdivided into sets (Set 1 . . . Set N). Each set focuses on one or more training element e_i (e.g., grouped into one set with related aspects, e.g., forehand and backhand top spin) and its associated goal selected from the skill elements in the skill status.

FIG. 46 is an illustration of a state machine showing the active training element and the criteria for the issuance of notifications to the performer.

FIG. 47 is an illustration of a trend plot displaying the progress along training goals (g₁, g₂, and g₃) over a specified time range shown here as seven sessions.

FIG. 48 is an illustration of a learning curve associated with the data driven training process. The learning curve shows the incremental improvement in some relevant attribute a_i of a skill element e_i over the training activity (sets and sessions).

FIG. 49 is a flow diagram illustrating a data-driven training process according to an embodiment.

FIG. 50 is a flow diagram illustrating the movement modeling processes of FIG. 49.

FIG. 51 is a flow diagram illustrating the processes of the skill modeling and assessment of FIG. 49.

FIG. 52A is a flow diagram illustrating the skill assessment processes of FIG. 49.

FIG. 52B is a flow diagram illustrating the skill status process of FIG. 49.

FIG. 53 is a flow diagram illustrating the training goals and feedback synthesis processes of FIG. 49.

FIG. 54A is a flow diagram illustrating the training goal computation process accounting for skill status of FIG. 49.

FIG. 54B is a flow diagram illustrating the feedback synthesis processes of FIG. 49.

FIG. 55A is a flow diagram illustrating the instructions synthesis of FIG. 49.

FIG. 55B is a flow diagram illustrating the feedback and cueing laws synthesis processes of FIG. 49.

FIG. 56 is a flow diagram illustrating the activity management and monitoring processes of FIG. 49.

FIG. 57A is a flow diagram illustrating the system configuration processes of FIG. 49.

FIG. 57B is a flow diagram illustrating the activity monitoring process including the notification and user input of FIG. 49.

FIG. 58 is illustration of a temporal structure and organization of a typical activity session.

ELEMENTS OF MOVEMENT SKILL ACQUISITION

This section briefly reviews central elements of movement skill acquisition and their execution. These elements highlight the challenges involved with motor skill learning and how technology can be used to augment the skill acquisition process. In particular: how to generate knowledge about movement skills and the associated learning process, how this knowledge can help determine which quantities to use to track the movement learning process, which information to feedback to the performer to help their learning, in which form to communicate this information, and at which time during the movement performance.

Motor Movement Skills Overview

Skill-based movement behaviors are usually fast, coordinated, multi-dimensional movements. Delays in human's signal transmission and processing limit the role of real-time feedback. Therefore, the biological movement control system has to rely extensively on “open-loop” control, meaning that trajectories are implemented from pre-programmed profiles, which are stored in procedural memory and therefore are largely unconscious. The general motor program (GMP) explains how complex movements are programmed. GMP describe the generalized rules that generate spatial and temporal muscle patterns to produce a movement for the collection of movement patterns in the repertoire. These programs are generalized in the sense that GMP encompass the mechanisms needed for adaptation to conditions within a given movement pattern class.

Complex movements frequently involve a sequence of distinct movement phases. Therefore, motor programs encompass mechanisms to support the ordering and timing of these elements in a sequence that forms a movement pattern. The movement phases are usually formed to support various functional characteristics, such as biomechanical constraints, task structure and various sensory interactions with the environment. Movement segments can be conceptualized as a movement directed towards a sub-goal, each with its particular biomechanical and sensory-motor constraints. This structure allows to breakdown complex movements into simpler movement elements. It can also help in the acquisition of complex movement skills, and support the flexibility and adaptability needed to operate in dynamic and uncertain environments.

The human bandwidth limitation for closed-loop feedback involving perceptual motor control is somewhere between 0.5 and 2 Hz, depending on the task. Above that bandwidth, intermittent closed-loop control can be used. Movement phases typically represent open-loop segments. Corrections can be implemented at specific phase transition. These phase transitions are also associated with functional features, such as when specific elements of information are available. For example, in a tennis stroke, an advanced player already has an idea of the intended outcome and anticipates the conditions of the oncoming ball, at the initiation of the stroke. At the end of the backswing phase, and before the initiation of the forward swing, the player makes adjustments based on the up-to-date information available from the oncoming ball trajectory.

As will be appreciated by those skilled in the art, movement skills often involve extensive interactions with the task and environment elements. For example, in tennis these interactions include producing the desired outcome in the task and dealing with the range of impact conditions. See FIGS. 1 and 2 which illustrate the interactions of a tennis player's racket with a delineation of the racket stroke trajectory, and FIG. 9 which illustrates the interception conditions that the performer has to accommodate to best control the ball trajectory. The perceptual system usually provides cues that are used to select the type of motion pattern from the repertoire of learned movement patterns. Signals from the sensory or perceptual system are used to modulate specific aspects of the pattern, such as the timing of the stroke phases based on a tennis ball's perceived speed. Training movement skills, therefore, involves acquiring a comprehensive set of mechanisms. Movements are not simply programs to steer body segments; they encompass numerous mechanisms and capabilities to support the interactions and adaptation to conditions. Therefore, skill acquisition also includes learning how to extract relevant signs or cues from the task environment, and developing plans for sequencing individual movement patterns. The basic motor learning concepts are introduced next.

Motor skills require integration of both sensory information and motor responses to attain a particular goal. Goal-directed, deliberate, instrumental, or intentional movements are movements characterized by forethought with reference to the consequences they produce. The outcome to be obtained is clear to the performer and determines how they organize their movement pattern. Such deliberate movements contrast reflexes or fixed action patterns. Motor skills are categorized on a continuum defined by the dynamics of the task and environment condition. On one end of the continuum are the open skills, which take place in temporally and spatially changing conditions; on the other end are the closed skills, which take place under fixed, unchanging environmental conditions.

In open skills, a new movement formed to respond to a new aspect of the task environment may either originate as a variation of an existing pattern, or as a new movement that is formed as a unique new pattern (see FIG. 11), albeit the new pattern may be reusing components of the original pattern. Therefore, in open skills, the user develops a repertoire of movement patterns that match the range of environmental conditions and task requirements. On the other hand, in closed skills, as the user learns to master the task, the movement performance converges over time to a fixed movement pattern that optimizes the outcome in relationship to the task requirement. As described herein, the term “user” may refer to a user of the data-driven training system, an agent using the system, a subject to whom the system is applied, or a combination thereof.

The movement segments that compose most complex movements result from how the subject exploits the large number of degrees of freedom (DOF). The high DOF in human motion result in redundant movement solutions. For example, racket swinging can be achieved through various combinations of joint motions such as wrist, elbow, shoulder, hips, etc. Each DOF has its own specific displacement range as well as other constraints such as speed or torque. Different executions of the same general movement will cause saturations at different stages of the overall trajectory and will result in a different sequence of movement phases.

Furthermore, human subjects mostly learn through practice; they essentially discover how to best exploit the rich movement space to accomplish the desired outcome. As a result, complex movement skill acquisition, and more specifically the development of movement architecture, proceeds through stages, with each stage making an increasing use of the available degrees of freedom (see FIG. 5).

Typically, a deliberate movement is needed to produce a particular outcome or change in the environment. Many skilled movements involve the control of an end effector such as the hand, foot, or a piece of equipment or instrument. Another class of skilled motions are characterized by controlling the dynamics of interactions with an environment such as in skiing or surfing. These interaction behaviors involve the performance of particular maneuvers to allow deliberate control of motion. Examples of maneuvers include different turning techniques (stem, parallel, carve) as well as other maneuvers such as rapid stopping, jumping, etc. These maneuvers are movement units that can be used to interact with the environment under different conditions or purposes. Movement skill acquisition can be defined as the process used by an individual to best change or maintain either their own state, or the state of objects, in space.

These end effector motions encompass a variety of different movement behaviors including reaching motions, such as those used to grab an object or touch something, or interception and throwing or hitting motions. All of these motions guide the end effector along a path to a particular location in space. Most of the reaching motions involve stationary end conditions. Interception and hitting involve more dynamic end conditions. Most skillful end-effector motions involve the precise control of its state at various instances or phases of the movement (apex, contact, interception, or throw) (see FIG. 5).

Reaching or intercepting motions rely heavily on visual information. The output side of behavior, i.e., the control of motion, therefore only describes part of the problem. The input side of the behavior, which encompasses the sensory and perceptual mechanisms, contributes to a complete understanding. These movements are in part driven by motor program and functional aspects such as the adaptation of the program to external task elements or dynamics represents a fundamental aspect of the skill acquisition. Goal-directed movements, such as in swing sports, are organized around what can be considered a goal state. In tennis, for example, the racket stroke motion is organized around the ball interception or impact. However, because the movement has to satisfy the constraints of the ball impact and the body and limb biomechanics, it is achieved through a complex coordinated pattern of motions. While the forward swing and impact phases are the most critical, these ancillary phases are required to create the best impact conditions needed to control the ball and also to adapt to the dynamic conditions of the task.

In other activities such as skiing, individual movements don't have such an explicit goal. Skiers use gravitational forces and body biomechanics to generate a turning motion to steer and control their path. These coordinated movements represent the primary unit of motion. While they may not be a distinct goal state such as in tennis or other swing sports, they often have a movement phase such as the apex of a turn, which together with the local environment interaction determine the primary outcome of the movement pattern. Skilled human movements, such as the tennis stroke, involve the sequencing of complex coordinated motions that are executed based on internal states and external cues. Their successful performance involves managing a range of contributions, including the effects of the tool or equipment (e.g., the tennis racket), the movement biomechanics, the interactions associated with the activity (e.g., tennis ball impact), and the interactions with the environment (e.g., aerodynamics or other medium) (see FIGS. 2, 7, and 9).

For a detailed and comprehensive assessment of the acquisition of movement skill, sufficient data from a description of the movement interactions with the larger task and environment elements may be required. Tracking and analyzing movement skills has long relied on visual techniques. Using these techniques means tediously observing video footage. Limitations in systematic data-driven skill evaluation and modeling are due to various complexities relating to the fundamental nature of complex movements and other task environment characteristics that were already discussed.

General Challenges and Requirements

Given the depth of hierarchical levels of the movement system, the scope of motion analysis can encompass multiple levels. For example, it could focus on low-level neuro-motor aspects, the movement technique and structure, the optimization of outcomes, all the way up to tactical and strategic levels (see FIG. 31). The range of motion sensors, available either embedded or deployed in the environment, can provide measurements of broad aspects of the movement dynamics surrounding the users, actors and their equipment. However, data alone is not sufficient to produce useful and actionable insights.

Detailed and comprehensive analysis of movement skill, in particular for open motor skills, has not been accomplished, especially for a larger population, because of both the practical issues of getting measurements, and the perceived complexities in analysis and assessments. In tennis, for example, comprehensive analysis has to consider the stroke motion as part of the larger system of coordination and interactions that include the ball trajectory, the footwork, going all the way to court motion, the game tactics, etc. (see FIG. 7)

Therefore, one of the general challenges in data-driven movement skill analysis has been the definition of a basic unit of analysis that provides a meaningful level of skill characterization and can be scaled to enable a more comprehensive scope of analysis for a single individual, and also generalizes across a population of performers. Basic analysis of the stroke motion usually focuses on the racket trajectory (i.e. end effector or equipment). Since that trajectory is the result of a kinematic chain that involves the upper body and the driving motion that starts from the feet, legs, and hips, by capturing the overall stroke pattern and its movement phases it is possible to infer deeper relationships between the larger biomechanical system and the end effector motion. As more measurements are available to track the various task elements and body segments, a more accurate and complete description of movement performance can be achieved (see e.g., motion tracking cameras or distributed motion sensors on the performers in FIG. 2). Ultimately, the depth of analysis depends on the available measurements, however, capturing the movement phase structure of the end effector motion that is used to produce the primary outcome in a task (tennis racket, or ski) can already provide comprehensive analysis and training interventions.

Another challenge in movement assessment and diagnostics is that of variability in performance. Viewed through direct observation, there is typically significant variability in human performance on repeated trials, making it difficult to apply quantitative models that describe an individual's technique and skill both comprehensibly and in details. In addition, because of individual differences in anatomy, style, fitness, and skill level, movement produced by different people targeting the same general outcome may turn out to be quite different. Therefore, it can be helpful to be able to capture a user's unique elements and features, and be able to continuously adapt the training method to the user's evolving skill.

The differences between individuals manifest both in their overall patterns and in their movement phase structure. The phase structure, as already discussed however, depends on biomechanical constraints, which are determined by individual characteristics such as body type, physical strength, and motor coordination, and therefore provides a more detailed understanding of an individual's movement. For example, for a beginning tennis player, a forward stroke will be a rudimentary movement including a forward swinging motion implemented from the shoulder joint. Over the course of skill acquisition and development, the brain will learn to better take advantage of their physical potential, range of coordination of their body segments, and other movement system components (FIG. 5).

Skilled behavior relies on organized strategies and builds on the well-defined hierarchical organization of neurological processes. The instances of observed movements belong to specific classes of movement patterns that are used to support interactions needed for a task performance. Therefore, capturing movements and aggregating them within classes provides a solution to systematic analysis even in the face of variations. These classes of movement correspond to the movement units.

Therefore, to enable the systematic data-driven training process, going from the skill assessment, to the diagnosis of skill deficiencies, to defining training goals and protocols, and the synthesis of various forms of feedback to help address those deficiencies, it can be helpful to define a comprehensive modeling language that captures the structure and organization of movement and is grounded on the fundamental principles of human movement science.

Following the example from natural language processing, conceptually, the core technology focuses on decoding movement data to extract relevant movement elements that can be used for skill analysis. The relevant elements in natural speech processing are the units of organization of speech production, known as phonemes. The decoded phonemes can then be used to identify words and eventually the meaning of a sound bite. To help extract movement units that are useful for skill analysis and diagnosis of an individual's movement technique, these units, similar to speech, have to be related to the process used for movement production. The result of this type of analysis can then be more readily translated into instructions and used to synthesize augmentation systems.

In parallel, to the analytical questions, a data-driven skill augmentation environment requires a system infrastructure to operationalize the various processes. The basis of the infrastructure is a data structure derived from the movement units that support efficient handling, processing, tracking, and managing of motion skill data. In addition, the data structure allows codification of skill components and their functional characteristics to design feedback mechanisms that target precise aspects of the movement skill performance and learning.

The proposed modeling language and skill model and accompanying technology infrastructure can accommodate the nuances that naturally occur in human performance, and build on the structural features inherent to the human movement system and its various functional and learning mechanisms. Moreover, the methods capture both the global skill components that give users its versatile performance in an activity domain, and the specific skill components needed for performance and adaptation to the specific task elements and conditions. And finally, it can be generalized to different activities and scaled to larger populations.

FIG. 4 shows examples of movement architecture for the primary movement unit for other movement activities (tennis 441, golf 442, rehabilitation 443, skiing 444, running 445, and swimming 446). The drawings also highlight the movement phases and the primary outcome.

Motor Learning

Behaviors are produced through a process of selection of a response (movement behavior), which is typically based on the observable environment state. A successful outcome of a behavior therefore depends on both the correct selection of the behavior type and its correct execution. Learning is defined as a change in behavior that results from experience. Learning is typically improved through feedbacks that reinforce correct behavior (Law of Effect).

Classic motor learning theory proposes that subjects have a repertoire of responses, some are rewarded and hence strengthened, increasing their probability of reoccurring (see Thorndike in Adams 1987). As a result of this process, the subjects develop and refine their repertoire of behaviors. More recent theories have investigated how movement pattern learning can be explained through neuro-plasticity. For example, the Theory of Neuronal Group Selection (Edelman, 1987) posits that the brain selectively reinforces the formation of patterns based on how adaptive the movement is given the prevailing environment or task conditions and constraints. Patterns that best support the task at hand are reinforced while unsuccessful ones are discarded. A movement that has a positive adaptive value will be reused more frequently. Through reuse, the pattern will be refined according to its adaptive value.

The learning process therefore depends on availability of signals that inform the subject of the success of its movement behavior. Moreover, for complex behavior, information about the outcome alone, or so-called knowledge of result, may not be sufficient. For complex movements, it can be helpful to combine an understanding of the movement technique—i.e., cognitive level—with feedback on specific aspects of that technique during and/or after the execution.

Augmented Skill Ecosystem

The data-driven training system builds on the augmented skill ecosystem that was previously described in U.S. Patent Application Publication No. 2017/0061817, which is hereby incorporated by reference in its entirety.

FIG. 21 illustrates a data-driven closed-loop training system including its primary processes organized according to three primary feedback loops. The assessment loop 200 is configurable to have five components. An extractor 201 extracts motion elements from a target motion. The extracted motion elements can be directed from an augmentation loop 202 which collects information from user training or play. The augmentation loop 202 can have a feedback loop between a movement process 222 and a cueing system 224. Additionally, the augmentation loop 202 can receive information from an instruction module 203. The instruction module 203 may receive a set of target skills 204 from a user or a trainer. Session data 226 can be provided to the extractor 201. The extractor 201 output generates a motion model 205 which can then be used for skill assessment and diagnostics 206 based on reference skill data 207. A measurement process can be provided that maps aspects of behavior or movement into one or more measurement signals.

The system operationalizes the training process and creates a systematic schedule that builds skills in following logical development, consistent with human learning principles. The training starts from a user's existing motor skills and proceeds by shaping these skills towards the specified goal skills.

The Assessment Loop (AL) corresponds to the process of data acquisition and processing associated with modeling subjects' movement technique and skills, skill diagnosis, and the organization of knowledge, for example in training lists and training schedules/plans, as well as the synthesis of augmentation laws. The Training Loop (TL) corresponds to interactions associated with the management and organization of the training activity, including reviewing the skill status, learning about the movement technique, selecting training elements and goals for the session, scheduling a training or performance session, and finally tracking the progress of the training process. The inner most loop is the Feedback Augmentation Loop (FL), which corresponds to actual performance of the movement activity and includes the effect of feedback cues communicated to the subject during the performance.

The cueing system 124 can include two components: a cue processor and a cue generator. The cue processor translates movement data into cue signals. The cue processor implements a finite state estimator and a cueing law calculator. The finite-state estimator is an approximation of the user's movement model (which is itself represented as a finite-state machine). The cue generator translates cue signals into physical stimuli; the system operates in real-time to provide feedback as the user participates in an activity. The cueing law calculator takes the state estimate and the motion data and operates on them to calculate if a cue will be delivered and what the cue should communicate. The feedback synthesis model determines how the cueing law calculator operates, whereas the finite-state estimator is defined by the user's current movement model. The cue generator takes the cue signal and translates it into feedback stimuli generated by a transducer (audio, visual, haptic, symbolic, or other type). The form of transducer is determined by the platform implementation details, user characteristics, equipment parameters, environment status, and/or other concerns.

The system receives input from a user's physical movement that takes place during a use or play session. The measurements can capture a range of movement behavior that was performed to complete the activity (e.g., all the motion associated with a tennis stroke, all the motion associated with a golf swing, etc.), associated task conditions, as well as the elements relevant to the broader functional components such as perception of task elements.

DETAILED DESCRIPTION

FIG. 1 illustrates one embodiment of a human augmentation system 101 applied to movement skill training or rehabilitation. The system in this example, combines existing devices such as a smart phone 102, a smart watch 103, or other processor in wired or wireless communication with a motion tracking device 104 attached to or embedded in the tennis racket 105. The device 104 streams motion measurements to the smart watch 102 and/or phone 103 or other processor. Motion measurements are typically obtained from MEMS IMUs (e.g., available from ST Microelectronics and InvenSense), which usually include 6-axes acceleration and angular rates and 3-axes magnetometers, which are often used to estimate absolute orientation in space (Attitude and Heading Reference System or AHRS).

As described in U.S. Patent Application Publication No. 2017/0061817, the motion data is processed at different levels in this system to render useful information for the subject's training or rehabilitation. The processing is distributed across typical internet of things (IoT) components, such as the wearable/embeddable devices, smart devices and cloud infrastructure. The segregation of these processes depend on the temporal requirements, such as acceptable delays or latencies, the required computational capacity, the availability of data, such as subjects' history and even larger population data and meta-date. Other factors include the streaming bandwidth and power requirements. All of these factors combine to determine the best network topology, data structure and management, as well as hardware selection.

To render useful information from collected movement measurement data collected, structural characteristics are identified that can then be related to particular motor events or actions. For computational analysis of technique and skills and ultimately synthesis of effective feedback for training instructions, it can be helpful to break down movement into movement elements (see FIGS. 3A-5).

Movement characteristics can be represented as geometrical and topological properties, which can be related to specific aspects of movement organization and skill. For example, movement characteristics can be observed in movement phase portraits such as that of the racket angular rate. Ensembles of movement data can be analyzed for patterns (e.g., using principle component analysis, phase-space analysis, and nonlinear time series analysis techniques such as state-space embedding). In addition, machine learning techniques can be applied to analyze the distribution of features and characteristics of the movement, as well as to aggregate and classify the data to determine patterns which in turn can be used to determine a deeper organization of the overall system. Given the variety of movement types and the variability in human performance, typically, the system is configurable to distinguish between different movement types before proceeding to deeper analysis of any individual movement or component thereof.

As shown in FIG. 2, which will be described in greater details later, one or more motion sensors, either embedded or deployed in the user's environment, can be used with the system to provide measurements of movement dynamics encompassing one or more users, actors, and their associated equipment (if any). As will be appreciated by those skilled in the art, given the depth of hierarchical levels of the movement system, the scope of motion analysis can be conducted at multiple levels. For example, it could focus on neuro-motor aspects, movement technique and structure, the specific outcomes of these movements, all the way up to tactical and strategic levels that describe how these movements are deployed in a task (see FIG. 31). The illustration in FIG. 2 delineates between different categories of measured or captured quantities. The output side (measurements and observations) includes behavioral quantities (movement such as the end effector, body segments; visual attentions; muscle activation); task and environment elements and objects. On the input side are motion tracking cameras 70, Gaze Tracking/AR device 80, and other sensor input.

Analysis of the intrinsic movement structure of the movement technique and functional characteristics can be used for skill analysis. This analysis can be formalized by focusing on the interactions of the movement with the environment and task elements. Operators or agents such as a tennis player organize their behavior in relationship to environment and task elements.

The resulting organization of the behavior combines the effects associated with the natural organization of the human movement system and the structure of the task and environment. FIG. 7 shows the different outcome levels, using tennis as an example, and some of the outcome measurements and FIG. 8 shows how these interactions produce the repertoire of strokes and their associated shot distributions. A particular distinctive characteristic of human behavior, which contrasts with robots and other engineered system, is that human behavior can be considered relational, i.e., movement behavior is produced through the action-perception loop and therefore is often anchored in a particular environment features and elements. In tennis for example, the stroke, which can be considered the primary movement unit, is directed at specific target areas in the court environment.

The specific human court environment perception, and the associated movement interactions, that can be formalized in terms of the tennis stroke and their associated shots (see FIG. 7), result in a specific discretization of the task environment as shown in FIG. 8. The characteristics of this discretization depend on the movement skills and the underlying motor, perceptual, and cognitive processes. For example, beginning players, because of their lack of control of the ball, may be able to consider only a very large target area such as the entire opponent court half. As the player improve, their perception of the environment and associated movements become more precise and therefore lead to a larger repertoire that spans the task environment with higher resolution and thus allows higher task performance.

FIG. 2 illustrates an exemplar augmented activity for tennis. The primary interaction is the tennis stroke, driving the tennis racket 20 toward a ball impact 30. The activity environment elements include the tennis court environment 50, with a net 52, as well as marking on the court 51. One or more motion tracking cameras 70 and/or other acoustic or RF motion sensors 90, can be used to track the subject's motion on the court environment 50, including the details of the individual body segments 15, the ball 30 and the racket 20. Other measurements can include the subject's visual gaze 81, which direction changes depending on the focus of visual attention, when tracking different visual cues, including the ball's ground impact 32, or net crossing 31 as well as desired court placement. The apparatus 40 shown in the same figure can be programmed to enable different forms of interactions. In one tennis example, the apparatus 40 is a ball machine that can be programmed to support the development of specific stroke patterns and therefore can be programmed in conjunction with the cueing system.

Augmented Movement Performance

The systems and devices disclosed herein augment movement skills at several levels, for example: 1) providing users feedback for training, including providing signals during the performance; 2) enhancing the athletic experience during performance to help focus; 3) providing protection from injury by helping users engage in optimal techniques; and 4) developing training protocols which are directed to developing skills related to the training.

Patterning characteristics are expected in many movement activities. In tennis, for example, the same general stroke pattern can be used to generate different amounts of top spin or pace. However, to maximize these different outcomes, distinct patterns have to be formed to fully exploit the biological capabilities. For example, a stroke for a top spin or slice has characteristic features in the temporal and spatial arrangement of movement phases. Movement patterning is due to how changes in movement outcomes or task conditions affect movement technique within a particular operating region of the state-space. As the desired outcome or task conditions change beyond a certain threshold, the biomechanics and motor-control organize differently to best take advantage of the system's capabilities. From a trajectory optimization perspective, the changes in outcome and condition alter the system's “operating point” and result in activation of a different set of constraints. Due to the nonlinearity, this leads to the emergence of a different motion pattern with distinct dynamic characteristics. Patterning corresponds to a tendency for the trajectories in each movement class of behavior to stay close together in spatial and temporal terms. This closeness can be described formally using techniques from nonlinear time series analysis. Using these techniques, measurement data describing racket state trajectories during a tennis stroke can be aggregated and clustered to identify different stroke patterns, and subsequently analyzed to determine their functional properties and characteristics.

Such performance data for an activity taken in its totality, for example from measurements of an entire tennis match, results in a repertoire of distinct movement patterns. This repertoire of distinct movement patterns is the result of the optimization of movements technique, i.e., achieving the range of outcomes and conditions required to be proficient in the particular activity. For instance, in tennis an individual will develop a repertoire of different strokes to optimize the desired outcomes (e.g., type and amount of spin, strength, etc.) and accommodate the range of impact conditions (ball height, speed, etc.; see FIG. 9). This repertoire essentially plays the role of a vocabulary of motion pattern that an individual can call upon when engaged in a particular activity. For example, FIG. 8 illustrates distributions of shots associated with different strokes and the effect of skills on the accuracy and granularity of the discretization of the task environment, which in this case is the tennis court.

The movement patterning and organization in repertoire, therefore, have implications for the assessment of skills. The skills of a particular tennis player, for example, can be assessed by: 1) extracting characteristics about the entire repertoire of strokes, e.g., how well they collectively achieve the range of outcomes and conditions in the activity domain, 2) determining how well and how consistently each class of strokes in the repertoire achieves associated outcomes, and 3) determining how well the strokes adapt to the impact conditions. The first analysis provides a comprehensive assessment, and the last two emphasize the technical implementation of the motion skills. Understanding human movement from this analysis provides a deeper assessment and diagnostic of the movement technique, that can be used to specify training goals and various feedbacks to help correct and optimize movement technique.

Improving Movement Learning

The following disclosure addresses the general question of how to improve movement learning using information technology, machine learning, and wearable devices. The disclosure also addresses specific questions including how to formulate training goals; how to manage the larger training process, in particular how to break up larger training goals into a sequence of goals; and how to dynamically update these goals based on data from the training activity such as skill acquisition stage and trends. In addition, the system determines what type of feedback to use to augment the experience and accelerate the learning process, when to present the feedback, how to determine the best type of feedback given the learning stage, and how to distinguish between different skill elements.

Furthermore, the disclosure also addresses how to best represent information to augment a subject's training experience. The resulting system takes into account what is learned by the subject as they make progress in an activity domain, what aspects of behavior to emphasize depending on learning stage, and also accounts for the characteristics of human information processing to provide feedback and information that can be processed and assimilated efficiently.

The central requirement for deliberate training is the specification of training goals and management of the training process using these goals. These processes are usually handled by human coaches or physical therapists. The contributions of this disclosure are the algorithms and system that enable training to be operationalized following a computational, data-driven process. The disclosure addresses two central capabilities: the computation of training goals, and scheduling and management of the training process.

The general approach is to use movement data to assess skill and identify deficiencies, followed by specification of training goals to address these deficiencies. Regarding training process management, the general approach is: i) leveraging the natural structure and organization of the human skill learning process; ii) using information from both individual subjects as well as from a larger population to extract knowledge to guide that process while accounting for individual characteristics.

The structure of the skill acquisition processes refers to the type of changes taking place over time as a result of activity (training or experience), which manifest as sequence of learning patterns characterized by specific changes in movement skill attributes and task performance. By applying population thinking, i.e., considering the skill acquisition process across the diverse group of subjects with different skill levels and movement technique, as well as accounting for the wide range of factors that affect this process, it is possible to extract knowledge about the larger skill acquisition process, which in turn can be used to guide training or rehabilitation.

Both of these goals require that skill be treated as an explicit concept that can be expressed quantitatively, e.g., decomposed into skill elements, that can be computed from performance data. Furthermore, this language of skill modeling should be applicable to a diverse population of performers so that spatiotemporal relationship in skill quantities can be extracted across the same as well as different individuals. And finally, this language should be valid across different forms of movement activities.

To enable these goals, the skill development process is formalized in terms of the hierarchical movement model detailed in U.S. Patent Application Publication No. 2017/0061817. Humans become proficient in a task or activity by developing a repertoire of movement patterns needed to interact with the task and environment elements involved in the overall goal of the task or activity. FIG. 11 illustrates the development of movement patterns over time. It is expressed as the differentiation of existing patterns as well as the formation of new patterns.

The model encompasses the repertoire of movement patterns, and the movement structure associated with the movement patterns used in the interactions with the task of environment. The specific Movement Functional Structure (MFS) also makes it possible to extract the wide range of movement skill attributes across the levels of organization of the movement system and the task structure.

Movement patterns that correspond to the primary movement units are typically associated with primary interactions found in an activity, some of those interactions produce specific outcomes on the environment or task elements, and hence can be characterized by their range of outcome and operating conditions. Therefore, the motion patterns associated with these primary movement units can be considered as the basic unit of skill, or skill element.

FIG. 6 gives an overview of the movement processing starting from the extraction of movement units, their classification, the movement model for each classes, following with the skill model that is used to determine relevant skill attributes used in the skill assessment and diagnostics. The figure also shows how these skill elements are then aggregated to produce the repertoire, which provides the basis for a subject's skill profile that can then be used for the analysis of the skill development (learning curve) and the population analysis.

The identification of these patterns in association with a skill development, and their delineation over the longitudinal acquisition process, makes it possible to relate the relevant movement skill attributes across the larger population; which in turn enables the systematic organization and management of the training process.

The quantitative definition of a unit of skill also provides the foundations to proceduralize training under an iterative learning scheme, which specifies how skill assessment, diagnostics and training goals are computed and updated over time. The system also incorporates the movement performance augmentations defined in U.S. Patent Application Publication No. 2017/0061817 (FIGS. 22 and 23) that are used to help induce changes in movement technique.

The central concepts needed for the realization of such a training agent system are reviewed next.

Skill Elements and Skill Profile

The first capability includes the precise and comprehensive assessment of an individual's movement skills, and more generally, data-driven training includes tracking various attributes of these skill elements. Using motion patterns as unit of skill enables the formulation of quantifiable, incremental change in movement technique, and its associated effect on measurable outcomes, as a result of experience or training. The sum of all changes in skill elements also ultimately produce incremental changes in some overall skill level that captures the larger impact of skill on the activity or task performance.

The skill element in the skill model represents the basic unit of skill acquisition. It is defined as the primary outcome associated with a particular class of movement pattern, and the associated attributes, that describe the relevant movement characteristics. These skill elements are derived from the movement system hierarchy specified in U.S. Patent Application Publication No. 2017/0061817. They encompass: (a) the repertoire of movement pattern classes, where each class is described by a movement pattern which is decomposed into phases; (b) the movement phases, which are the manifestation of the movement functional structure determined by the biomechanical constraints and other constraints arising from the properties of the environment interactions.

The skill elements can be combined to form a subject's comprehensive skill profile, which captures skill attributes associated with the skill elements. An individual's skill profile can be precisely and comprehensively characterized by the skill element attributes that can be derived from the hierarchical movement model and the functional structures underlying all movement patterns in a repertoire used in a domain of activity.

The present disclosure extends this movement pattern functional analysis and assessment, covered in U.S. Patent Application Publication No. 2017/0061817 to encompass the task-level performance, which is based on the fact that movement pattern classes support the interactions needed to perform the particular task or activity. Task performance metrics can be computed from attributes of the repertoire of movement patterns. Simple metrics can, for example, be determined from the use frequency of the various movement patterns.

More detailed models for higher-level assessment can be determined from the temporal sequence of movement patterns. Spatiotemporal patterns at the level of the repertoire, i.e., what movement patterns are used where and when, also enable the description of the high-level decision-making processes associated with planning and strategy which represent cognitive functions. This extended task performance analysis provides tools to compare players or performers, i.e., support the analysis competitive level performance. They can also be extended using population analysis (see concept of player profile).

Together, these elements make it possible to assess and diagnose some of the subject's higher-level functions, including the perceptual mechanisms, attention, and decision making. These quantities enable a comprehensive and precise quantification of skills, and therefore provide the basis for the computational framework to drive training at different levels of the movement system and task structure organization. For example, target reference values for the various parameters of the skill model (see Target Skills in FIG. 1) can be used to drive skill or performance attributes at different levels from features of the movement technique used to optimize outcomes to higher-level attributes such as success rates of tennis shots in specific areas of the tennis court.

Training Goals

To drive the training or rehabilitation process and enable quantitative data-driven training, it can be helpful to specify training goals. Training goals are a quantitative specification of a subject's target changes in the movement that will produce the desired increment in skill level. The training goal targets actionable characteristics in movement technique and therefore represents the drivers to achieve the larger skill level targets.

The goals typically combine expected changes in movement outcomes with the associated movement characteristic (functional element). To produce effective drivers for training, the training goals can be augmented by a range of instructions and feedback cues as defined in U.S. Patent Application Publication No. 2017/0061817, which can encompass different components of the information processing levels to best target the various attributes of the movement functional model.

To be useful, training goals should be: actionable, sufficiently broad in scope, effective, and realistic. By fulfilling these requirements, training goals enable subjects to train deliberately and achieve predictable, quantifiable changes in technique that result in improvements in skill level, relative to the existing skill level, but also provide a path for the long-term development of skills needed to attain the desired level of proficiency.

To be actionable, the training goals have to represent explicit changes in movement technique (and associated visual, perceptual, etc. processes). This is achieved by building on the movement and skill model just described.

To have sufficient scope, the training goals have to encompass the various characteristics in movement behavior engaged while operating in a particular activity or task. This is achieved by comprehensive assessment enabled by the hierarchical model and the movement functional structure.

To be effective, the training goals have to provide actionable milestones that lead to an incremental improvement in skill towards the next tier, and are aligned with the larger developmental or skill acquisition path. This is achieved by accounting for the larger skill development process.

Finally, to be realistic, the goals have to build on the subject's current skill level and the individual conditions (e.g., constraints that are imposed by health, age, physical fitness, etc.). This is achieved by accounting for the subject's specific location within the global training or rehabilitation path and specifying the training goals as precise incremental changes in existing skill attributes.

Computation of Training Goals

The present disclosure describes how training goals are identified and subsequently specified. The training goals are specified as target values of skill element attributes. The target skill values used to formulate training goals are computed from the individual's performance data, and extended by population data. The general approach is based on a statistical model describing the individual's skill elements and skill profile.

The variability intrinsic in performance naturally results in a range of values for these attributes. This statistical model provides the basis for the individual's skill analysis (see FIG. 19). The movement diagnostic is performed through the inference of relationships between specific movement technique features and selected outcomes relevant to the task performance.

FIG. 19 shows the distribution for some example skill attributes. Skill levels are captured through some objective function which is shown in terms of its level lines (shown here as low, medium, high, very high). The information specifies the direction the attributes have to be changed to achieve a higher skill level. The tiers can be derived from the individual's data or the data obtained from a larger population.

Skill Acquisition Process and Training Process Management

The present disclosure further describes the computational framework needed to determine training goals and manage the training process. The framework is based on a skill development or acquisition process model and, as already discussed, builds on the movement and skill model elaborated in U.S. Patent Application Publication No. 2017/0061817.

This training process model accounts for the development of the skill as the acquisition of a repertoire of skill elements. This process extends over larger periods of time and is influenced by a broad range of factors. Characterizing the skill development as a sequence of formations of movement patterns, i.e., skill elements, it is possible to analyze the acquisition process, and actually apply the gained knowledge to optimize an individual's skill acquisition process.

The present disclosure extends the skill model to account for the skill acquisition process. This process is formalized as series of transformations in movement technique, which describe, the longitudinal development or acquisition stages for each skill element (characterizing the brain's and motor system's natural learning process for the formation and consolidation of the movement patterns), and how these manifest into the movement functional structure, and overall skill profile. Typical learning processes are described by learning curves. However, these don't capture the details of structural changes associated with learning complex movements.

Based on the proposed model, as an individual becomes more proficient they can i) achieve more optimal behavior within an existing MFS; or more radically, ii) develop a new functional structure that better exploits the biomechanical capabilities and other supporting processes needed for the interactions in the task. The movement functional structure therefore provides the characteristic that helps delineate between stages in skill development, and also provides the basis to relate different performers or subjects.

The overall goal is to evolve a subject's MFS along the larger skill development process following stages that are best suited for an individual, and their overall performance or skill goals. The latter depends on a broad range of factors, including desire/motivation, needs (e.g., for professionals), as well as the various individual factors that are determined by biological and health conditions.

The specific sequence of acquisition can, on the one hand, be determined by the task requirements, specified by interactions (outcomes and conditions) that can be relevant to the performance of the task, and on the other hand, the individual's factors that determine what is feasible given, for example, the current skill level, the neuro-motor and physical factors involved in the development of coordination. Learning process can be characterized in terms of the skill acquisition stage, which provides the information to determine the best type of intervention, drivers, and activity to pursue the training goal.

Population data is able to capture the larger set of factors and therefore provides useful information to help orient and schedule this process, and at the same time account for these individual factors, i.e., how different body types, injuries, or health conditions affect the skill acquisition process, skill profile, and overall performance.

Population Analysis

The details of the larger skill acquisition process are determined based on movement data collected from a population of performers. The population data provides understanding about the global characteristics of the movement skill acquisition that emerges when taking into account the broad range of factors expressed in the population that affect this process. In essence, it makes the extracted information actionable by contextualizing it.

The general idea, therefore, is that learned global population characteristics can help support individualization of training and rehabilitation. The individualization is supported by providing reference data that relates an individual's skill attributes and skill profile to the larger population. This data provides both local reference about the skill attributes, e.g., how much specific attributes have to be improved to gain in skill level. It also provides more global reference about the longitudinal skill development from that local skill status, e.g., what aspects of the movement skills have to be optimized and in what order to produce favorable long-term development (e.g., faster progress in skill level and lower incidence of injuries). Therefore, information extracted from a larger population can help direct both the local performance and the more global, long-term training process (what aspects to focus on first, etc.).

The present disclosure also details a computational model of the skill assessment and diagnostics specific for population analysis and the extended task performance level. The population analysis builds on the skill profile derived from the movement hierarchical model in U.S. Patent Application Publication No. 2017/0061817. The skill analysis from the population perspective is defined under the concept of a player profile which describes the skill attributes in the context of a larger population to capture the type of player based on the type or style of movement technique. The player profile can also encompass the higher-level characteristics such as game strategy that captures how the movement patterns are utilized or exploited in the settings of a task or activity.

FIG. 30 describes the assessment and diagnostic process incorporating the player profile which is applied across the movement system hierarchy (see FIG. 6). The main components are: (a) determination of the movement classes in the repertoire that are in formation, consolidation, or optimization stage (based on the individual's skill acquisition stage); (b) determination of how these patterns are used in the performance of the task (e.g., based on use frequency and game or performance strategy); (c) identification of which aspects of the skill element needs to improved, e.g., the quality of the primary outcomes, which can be achieved by interventions at different levels of the motor control hierarchy, from task level attention to deeper movement technique (based on attributes).

The skill analysis incorporating the player profile, enhanced by the reference values derived from the population analysis, makes it possible to account for a broad set of factors needed to support individualized training.

Training Agent System and Process Organization

Finally, the present disclosure also delves deeper into the system architecture that supports data-driven augmented training. In particular, the delineation between the different modalities of augmentation (instructions, cues, apparatus), their deployment across the human information processing system, and the data and information management infrastructure.

FIG. 22 depicts the main elements of the augmentation system architecture delineating the augmented activity (with feedback cueing and/or apparatus interactions), the human system augmentation loop (with communications and UI systems), and the training management and configuration loop driven by the training agent system (not shown, which performs the modeling, assessment, and diagnostics to identify training elements that can be activated as training goals). FIG. 22 also highlights the primary tiers of augmentation that leverage human's natural information processing levels.

These include “cognitive level” information, which is communicated symbolically, verbally or visually (here as instructions or notifications provided by a visual UI or natural language such as a smart phone or smart watch, eyeglasses, etc.). Instructions and other forms of information such as notifications are provided by a communication system that can include a visual display for text and graphical objects and natural language processing system. Instructions are typically designed to help subjects' understanding of their technique and performance.

The “feedback cue level” describes information communicated via some cueing system (here an audible signal) but can also include visual or haptic systems. And the “signal level,” which includes both cueing signals and activity interactions (e.g., ball machine) that are typically delivered concurrently with the movement performance.

FIG. 23 illustrates the augmented human performance associated with the feedback cueing system emphasizing the low-level signal and cues within a typical perception-action loop. The movement data is processed in real-time by the cueing system to compute cue stimuli designed to help performer improve a specific aspect of the movement e.g., by acting as reinforcement signal.

Note that cueing can also be used to help focus attention to relevant elements of the task environment including, for example, the location of a task object (tennis ball) or features of that object (ball trajectory) or features of the adversary's movement that can be used to anticipate the result of the adversary's movement. Anticipatory information can for example help the subject select the movement pattern. Finally feedback cues also include cue signals that can be used by the subject to time the execution of the movement.

The various forms of feedback augmentations are computed by algorithms that have been synthesized based on the subject movement and skill model for the current epoch and historical records, and can also include reference data from larger population.

The training process is formalized within a computational framework with similarities to iterative learning. The framework describes the management of data sets used to support skill assessment and diagnostics, which include motion and skill model (skill profile and the player profile), and the training goals and synthesis of augmentations (instructions and cueing laws).

The data management process encompasses: i) the creation of data sets, models, baselines; ii) tracking their validity; iii) and updating these quantities to support an effective augmented training process. FIG. 49 illustrates the top-level logic diagram of this process and FIG. 25 illustrates the process flow diagram, highlighting the activity across the stack of processes of the assessment and training loop (data acquisition and processing, motion model, skill model, training goals, augmentation laws) as a function of time. During each session, activity data is collected and processed.

As shown in FIG. 25, the motion model, skill model, training goals, and augmentation laws are typically updated based on their validity with respect to the new session data. Note that the processing stack for the feedback augmentation (cueing system) isn't shown here. For example at n−3 a full update is implemented following changes in motion architecture. At session n−2 the motion model is still valid and the remaining parameters don't need updating. At n−1 the skill model is validated and progress on training goals prompts an update in training goal and augmentation laws. At session n the skill model is updated (skill status) and new training goals are determined along with augmentation laws. At session n+1 the complete update including the motion model.

The training process is delineated in sets, sessions, and epochs. The former two are time periods needed to organize activity and training (see FIG. 58). The epochs correspond to time periods that correspond to the use and updating of the data sets supporting the computation and processing of quantities. A new epoch starts when the movement technique and performance has evolved beyond the validity of the current models. Each epoch typically encompasses a set of training goals for the range of movement classes in a repertoire that will drive the next increment in skill level. The more recent sets of movement data are used to create new motion and skill model baselines, all the parameters used in the movement processing algorithms (e.g., classification), and the other algorithms supporting the computation of skill attributes, training goals, and synthesis of various feedbacks.

Larger scale time periods beyond epochs are defined based on patterns in the population data and typically would correspond to the characteristics in player or performer subgroups. Transitions between developmental stages typically involve deeper changes in movement skills, such as the re-organization of the movement functional structure (MFS).

The temporal structure introduced by this system, and derived from the natural acquisition process structure—and of course all the associated quantities—provides the basis for the management of the training process. The structural patterns in the acquisition process can inform how to compute trends, generate reference data, as well as other critical capabilities of the data-driven training system. Finally, the motion data, model, skill profile, and all the training elements, when extracted over time can also be used for bootstrapping recovery or rehabilitation following injury or other causes of interruptions in training or practice.

Human Movement Skills

Open Motor Skills and their Characteristics

Movement skill can be categorized into two primary groups. The first, the so-called closed motor skills, involve a stable environment where the performer initiates the action or movement. These conditions allow selecting the best movement or action to achieve the task objective. Closed motor skills therefore can typically be learned and perfected in a systematic way by identifying conditions and training movements in these corresponding conditions.

The second, so-called open motor skills, involve a dynamic environment with changing conditions and require responding to the task and environmental conditions. These conditions also require a broader range of movements and actions to adapt and achieve the task goals. Open motor skills typically involve learning a large repertoire of sensory-motor behaviors and associated perceptual mechanisms, as well as planning mechanisms. The broad range in system state and task conditions makes it difficult to understand what movement patterns to train. The performance under dynamic environments and conditions also makes it harder to create meaningful training task conditions. Furthermore, it is difficult to predict the specific range of conditions that need to be trained because the behavior results from the dynamic interactions between the performer and his or her environment.

Many human skills involve open systems which are characterized by the exchanges of energy between the subject and the environment (Davids, 2008; see, also Kugler 1982 in Davids p. 57). Skills in these systems require exploiting information both related to the physical performance (energy flows) and the control performance (system structure and behavior).

The neural system supporting motor control is organized hierarchically to enable efficient encoding and programming of the movements. A central theory in motor control is that to mitigate the complexity associated with the large amount of degrees of freedoms (DOF) (resulting from the numerous muscles and joints), movement patterns take advantage of so-called muscle synergies (see Bernstein, 1979). The synergies encode the coordination between groups of muscles and joints and thereby reduce the DOF that need to be controlled. They represent the functional elements of a hierarchical and modular representation that can be efficiently employed by the central nervous system to program and execute complex movements.

In addition to the complexity associated with the DOF problem, encoding and learning individual movements for every desired outcome and possible condition would result in an intractable amount of information to be stored. Another central concept in motor control theory is that instead of learning individual movement programs, humans and animals learn generalized motor programs (GMP). These GMP specify the muscle activation patterns for entire classes of motions. This concept was introduced for movement control under the so-called schema theory (see Schmidt, 1975). The GMP achieves efficiency by encoding common movement and perceptual characteristics as a form of schema. This provides flexibility by allowing variation in specific movement characteristics needed to adapt to variations in conditions or outcomes. As pointed out by Newell, it addresses the problem of endless variability and novelty in performance (Newell 1991).

Furthermore, most movements in open motor skills are coupled with dynamic elements of the task and environment. The resulting combination of conditions and states dramatically increases the complexity of learning to perform. For example, the tennis stroke is an action directed at returning a ball that is itself moving relative to the player and court, and the execution of a tennis stroke also depends on the body's motion relative to the court and the ball trajectory. Accounting for all these dimensions results in a potentially intractably large amount of information that needs to be extracted and encoded.

Coordination is defined over the organism and environment interaction, not just the organism. The coupling between movement and the environment was emphasized by the Ecological Theory of behavior. Unfortunately, this “extended” coordination problem adds considerable complexity. However, there are structural features that emerge from the interactions in system components that human and animal perceptual and control processes have evolved to exploit, greatly simplifying the extraction of information and internal representations needed to plan and organize behavior. Gibson's work on visual perception demonstrated that some elements of information involved in coordinating behavior relative to the environment are, in fact, readily available from the visual observations of environment (so-called direct perception, see Gibson, 1979).

Therefore, one aspect of operating effectively in skilled movement tasks is the automation of processes associated with environment perception and organization of behavior to exploit the natural structure of flow of information and behavior dynamics, respectively. Such a strategy minimizes the amount of information that needs to be explicitly processed from sensory signals, information that needs to be programmed for the motor actions, and information that needs to be stored.

Movement Patterns and Affordances

When operating in a new environment, or performing a new task, the organism has to identify opportunities for new actions to be learned. How are behaviors that are contributing to the task or goals learned and/or identified? In ecological psychology, the concept of affordances describes what the environment provides or furnishes the animal, implying the complementarity interaction between the animal and environment (Gibson, 1979). Based on this affordance concept, the animal or human has to essentially learn to recognize or perceive affordances. Therefore, learning skills can be conceived as the process of learning to recognize affordances and adapting behaviors to effectively exploit these affordances.

Affordances can take a broad range of forms. They can be static, such as chairs affording a person to sit, or dynamic, such as stairs affording climbing. Researchers have developed and adapted the concept of affordances to specific domains. Norman, for example, adapted it to the domain of human computer interfaces (HCI) where good interfaces convey action possibilities in forms that are readily perceivable by users (Norman, 1999).

For skilled movement tasks, the affordances are specified based on the dynamics of the agent-environment system. Such systems are typically complex, high-dimensional nonlinear systems, with numerous components interacting through their processes and physical components, including body segments. Complex, nonlinear dynamic systems are characterized by emergent behaviors (see Davids 2008 for emergent behavior in human movement). The physical system and the muscle and sensory-motor supporting movement coordination, along with the various processes needed to interact with the task and environment elements, form a complex system. Therefore, the overall evolution of the movement patterns and their properties are emergent phenomenon.

This perspective was investigated for spatial guidance behavior in (Mettler, 2015; see, also, Kauffman's 1993 and Van Gelder and Port 1995 pp. 31 and 32) using the concept of interaction patterns. Interaction patterns are agent-environment dynamics that are exploited to achieve efficient learning and programing of motion behavior. They have been shown to represent behavioral invariants that satisfy properties of equivalence relations (Kong & Mettler, 2013). Therefore, they provide an efficient decomposition of the complex, high-dimensional agent-environment dynamics into small sets of behaviors. Similar to muscle synergies in body coordination, but here describing the coordination of agent behavior relative to its environment.

These emergent interaction patterns can be exploited by humans or animals, and provide functional capabilities needed to achieve adaptive and robust performance in complex environments. Besides helping with the organization of behavior, where they play the role of unit of organization, the interaction patterns are manifestations of the functional structure of sensory-motor functions. Therefore, the interaction patterns also represent a type of functional unit that helps with the organization of the system-wide integration between different processes (control, perception, and planning) (Mettler, 2017).

Therefore, when considering advanced open motion skills, affordances can be formalized as emergent properties of a complex dynamic system. The understanding of behaviors as interaction patterns emerging from the agent-environment dynamics provides additional insights about what is learned, and therefore helps determine how this implicit knowledge acquired in a domain of activity can be modeled. Sensory-motor skills condition the interactions between the agent dynamics and the task and environment elements, and therefore, viewed from larger perspective they determine the affordances available to the operator or agent.

Viewing behavior as properties of the agent-environment dynamics suggests two components of the learning process. First, learning involves recognizing the affordances that are enabled by a subject's control and perceptual capabilities. Second, the subject has to learn to exploit and further shape these available affordances to improve the larger task performance, or more generally, adapt to the conditions and contingencies arising in the task environment.

And furthermore, for subjects to improve their skills in open motor tasks, they should identify: 1) the potential for new affordances, and 2) the opportunity of improving the quality of the interactions. The first leads to the development of the repertoire of behaviors, and the second leads to the refinement of movement patterns.

When an agent approaches a movement learning task, their existing sensory-motor capabilities determine the range of possible affordances, i.e., unexploited, latent affordances. The agent then should learn to identify and exploit these affordances. Once identified, fully exploiting these latent affordances require refining and optimizing the sensory-motor processes. The newly acquired sensory-motor capabilities, in turn, create new affordances. This process, therefore, describes the incremental learning process and explains the extensive training and experience needed to reach proficiency in a domain of activity.

Human Motor System

The human motor system has evolved to manage a variety of movement tasks that involve interactions with environment elements, while efficiently handling uncertainties, disturbances, and contingencies arising during the performance. While the human movement system has tremendous potential, systematic and dedicated training is required for high levels of motor facility. This requirement for training is similar in any domain of activity, such as athletics, music, or vocational tasks. Movement task constraints can be divided into extrinsic and intrinsic factors. Extrinsic factors include the interactions with the environment such as the foot strike or impact of the ball on the racket. Intrinsic factors include the biomechanics, human motor control, and effects arising from the manipulated equipment's dynamics. Most skilled behaviors are so-called deliberate behaviors that are directed at achieving specific outcomes. Therefore, learning skilled behavior in sports or vocational activities involves learning to master these interactions so as to achieve the desired outcomes or goals. The coupling of the human movement system and the task environment has to be considered as a coupled system. If the extrinsic and intrinsic interactions were considered separately, the complexity would be intractable.

An efficient and effective solution may be found in the use of strategies that structure and organize movement behavior to satisfy both the extrinsic and intrinsic factors. The brain evolves specific organizational structures and functionalities to efficiently work with these complexities. The brain and sensory-motor mechanisms that optimally deal with the coupling of the two domains, and achieve the sufficient level of adaptation, are the result of natural selection. Movement skills represent a critical aspect of a species fitness. As a result, the specific structure and organization of the brain, including the nervous system and larger biomechanical systems, support natural solutions that enable efficient and adaptive behaviors. Therefore, a portion of the movement system is genetically determined and are innate. However, motor skills, especially in deliberate specialized movement skills, are learned and perfected based on repeated interactions within the task and environment. Finally, learning movement skills involves changes in the cortex as a result of neuroplasticity. These changes, however, follow a specific process that is dictated by the organization of the various cortical structures (cerebellum, parietal cortex, pre-motor and motor cortices, and the prefrontal cortex). As a result, movement skills are best acquired early in life when the brain is still developing.

Three forms of units of behavior have been described for complex movement behavior. At the top level, motion primitives are related to the concept of “motor equivalence” which has been identified as one of the fundamental characteristics of motor behavior. The idea is that the same movement behavior can be repeated in various contexts and without changing the overall form of the motion. Therefore, segmentation of human movement behavior into motion primitives has been most successful from invariant characteristics in the performance that arise from symmetries and equivalences in the problem space.

Furthermore, since complex movements are obtained from a sequence of movement phases, the next level of primitive represents the segments that can be combined sequentially to compose movements. This is due to the brain's efficient encoding which exploits principles of modularity. Finally, the last level of decomposition is related to the so-called muscle synergies, which represent the movement components that describe the parallel combination of different muscle activations and the associated body segments displacements. The top-level primitives are considered the primary motion units, which support the interactions with the task and environment elements, and the lower two levels, the movement phases that specify the movement architecture and the synergies, provide understanding of the functional properties in relationship to the environment interactions and the biomechanical constraints.

These elements of the movement system can be derived from structural features extracted from measurements using pattern analysis. There may be a great variety of movement patterns that satisfy these constraints depending on the configuration and conditions; however, they all typically share characteristic features that enable the movement patterns to be identified and segmented. Muscle synergies can be obtained from factorization methods (e.g., principle component analysis or non-negative matrix factorization). The general idea is that many movements can be described as variations of a general model and once the general category of movement is specified, some of the mechanisms needed to achieve robust movement performance are those that allow adapting to those movement pattern to changes in conditions and transferring them to different contexts in a similar task or activity.

Learning Principles

In contrast to periodic and reflexive movements, which can be generated from low-level motor functions, skilled movements usually involve the deliberate expression of specific goals or outcomes that rely on higher-level motor, perceptual and cognitive functions. These movements may be completely self-initiated, e.g., picking up the phone to call someone; they may represent a stage in the context of a larger task, e.g., opening the dish cabinet when preparing food, or returning a tennis serve. As seen in these examples, movements are rarely an isolated behavior but are part of a larger set of interactions with the world and therefore are typically part of a chain of behaviors.

Learning deliberate, skilled movements involves learning the motor programs that determine the correct forms and sequences of actions as well as the perceptual cues that provide information to fine tune the movement characteristics that will enable to successfully accomplish the intended goal or outcome (e.g., reaching to grab an object or imparting momentum to a ball). Learning involves iterating on existing solutions as the task, or a similar task, is repeated. Therefore, learning has to be able to build on existing elements, and to change them incrementally to improve the outcomes, efficiency, and overall task performance.

As will be appreciated by those skilled in the art, teaching relies on two primary modalities: demonstration and practice. Demonstration in theory should focus on instructions to help the student understand what they need to know about the behavior or movement. Practice refers to the process of performance repetition. Studies have shown that demonstrations too often focus on the task outcome rather than on an analysis of the coordination of movement. Movement skill acquisition could therefore be accelerated by providing specific signals delivered during performance. Two signals in particular would be beneficial. First, signals that highlight the structural elements used in the composition of movement. Second, signals that indicate which characteristics of these elements play a role in movement outcomes. However, these signals have to be adapted to the skill level of the individual and to his or her specific movement technique.

Users form an abstract understanding of movement capabilities in terms of goals and outcomes. Users, for the most part, learn in which contexts to use a particular movement pattern. Therefore, at the highest level, people can assess how well they do from knowledge of the range and quality of their movement pattern repertoire. The technical details of movement skills are largely unconscious. This is in part because movement execution is too fast for humans to directly control their technique. Therefore, most learning follows a trial and error process. Movements that achieve goals are generally reinforced.

It is difficult to directly assess movement technique. Users typically only determine technique indirectly through outcomes. Therefore, it is hard to explicitly instruct the technical aspects of the motion skill system. Trainers and coaches increasingly use strategies to help users form so-called sense memory associated with a correct movement technique. A feedback signal that validates correct movement features can, through association, be used to reinforce memory of what such a movement should feel like. This form of feedback should hence accelerate the development and consolidation of a particular skill.

Feedback types can be delineated in terms of their temporal activity and the specific levels of the control hierarchy at which they operate: Real-time feedback, taking place during performance; feedback immediately following an action, such as based on information from the movement outcome; and feedback at the end of a training set or session. Inherent feedback associated with the feel, look, sounds, etc., of movement performance, as well as the movement outcome and interactions with the task and environment, can provide large amounts of information that can be used to assess performance and help train. Individuals, however, have to learn to recognize and evaluate those sources of information. Natural feedback describes feedback signals at each of these forms that are inherently present in the task-environment and movement associated with an activity. FIG. 24 illustrates the natural and augmented feedback based on cues and interactions. The cueing system operates by augmenting the natural cues that are available to the performer, e.g., from the movement outcomes, task environment (task elements and objects, adversary's movement, etc.). The augmented cue environment is designed to help the human perform and train for the task. Task interactions are produced by an apparatus that is coupled with the activity and possibly with the subject. Augmented feedback is information that is supplementary to inherent information about the task or movement. The two major categories of augmented feedback are recognized in the literature: knowledge of result (KR) and knowledge of performance (KP). KR represents post-performance information about the outcome or goal achieved. It is sometimes called reinforcement. Note, however, that not all movements have an outcome that is separable from the movement performance. KP represents information about the movement technique and patterning. This information is useful for the acquisition of complex movement skills, such as those requiring high-dimensional spatial and temporal coordination. Previously, it was difficult to measure and track performance in many activities. The advent of MEMS movement sensors has created a wide range of possibilities for using information about movement kinematics and dynamics (kinetics) from measurements.

Natural Feedbacks Supporting Learning and Performance

Several levels of natural feedback are involved in the support of skilled movement learning and performance. One consideration is that there exist different forms of feedback that act at distinct levels across the hierarchical organization of the nervous system. The cortical and subcortical systems are involved in the formation and implementation of movement patterns, and in the different feedback structures used to correct and modulate movement. At the lowest level, the spinal and subcortical systems physically implement movement by receiving commands from the cortical and subcortical structures. Feedback encompasses the information sensed at the level of the muscles, tendons and joints, and provides modulation at the level of spinal circuits. Between the spinal and subcortical is the system that controls posture. Feedback at that level encompasses information from the vestibular and proprioceptive systems, which also combines spinal and cerebellar contributions.

At the center of the neuro-motor system is a specialized system that deals with the formation of complex movement patterns, especially the chunking and sequencing of movement phases. Feedback mechanisms use information from cues extracted by visual, auditory, and haptic sources. The task of this system is to fine tune and synchronize behavior with external tasks and environment elements, such as adapting timing of movement phases, or modulating phase profiles. The phases are typically part of a sequence generated by the cortical circuits. The highest structure is the cortical system used for perception, planning and execution. This system combines the various sources of sensory and perceptual information to build representation that can be used to generate plans and monitor the performance and outcomes of the behavior. This system can also handle abstract information such as that in verbal or written form.

The human information processing model helps provide an understanding of what type of feedback information is most useful, and for which components of movement behavior these feedbacks apply. TABLE A summarizes the type of signals, cues/signs, and symbols in tennis as an example. At the highest level, knowledge-based behavior corresponds to the type of stroke and body positioning, etc. to use given the information about the overall situational awareness, such as adversary behaviors gained from exteroceptive information. At the intermediate level, cues trigger behavior. At the lowest level, signals are used to modulate muscle responses.

TABLE A
Example of signals, cues/signs and symbols in tennis.
Symbol:  The type/class of stroke as well
         as the desired ball placement.
Cue/sign When to initiate a stroke phase and
         the modulation of the stroke pattern
         based on the predicted ball impact and
         bounce trajectory, etc.
Signal   Coordination of the muscles during
         the stroke to conform to learned pattern
         based on proprioceptive feedback.

At the highest level, the rule-based behavior involves determining which pattern to activate based on the signs or cues typically obtained from the exteroceptive information. At the intermediate level, cues are used for time movement execution. For example, the particular state of the ball extracted visually, such as the impact, may be used to signal the instant to initiate the backswing or the forward stroke, and modulate the strength of the initial acceleration. Finally, at the lowest level, the skill-based behavior corresponds to movement patterns. Signals are primarily the proprioceptive information.

The delays and time constants of the sensory-motor system are too large to provide continuous feedback corrections for fast-paced skilled movements. The neuro-muscular time constant (time from the signal to go from the motor cortex to the muscle response) is of the order of 20-30 msecs; on the other hand, the response time from visual or auditory stimuli to a physical response is of the order of about 200 msecs.

Therefore, skilled movements rely on open-loop execution. Feedback is structured, for example, for intermittent actions based on specific cues and controlling the timing of phases. The largely open-loop execution implies that segments have to be learned in order to be reproduced accurately. And mechanisms to predict the outcome of the movement help enable modulation of the movement at the instant of execution. This natural functional structure of movement can be used as a general model for the design of augmentations to assist or train human movement behavior. In principle, augmentations can be designed across all three levels. Motion skills, assuming training within specific known outcomes, primarily involve the skill-based and rule-based behaviors. The symbol level is relevant to form mental models, for example movement architecture and functional characteristics including the environment cues. However, it is primarily relevant at the level of task and competitive performance, such as planning and strategy.

Augmented Feedback

FIG. 24 describes the main components used for real-time feedback augmentation in the augmented human movement system. The feedback augmentation comprises two primary forms of augmentation: feedback cues and interactions. The cueing system achieves its effect by augmenting the natural cues that are available to the performer, e.g., from the analysis of the movement outcomes, the real-time analysis of the movement technique, or event generating cues that pertain to the task environment such as the behavior of task elements and objects, opponents' movement or actions. One consideration is that the natural feedback environment is usually very sparse. Not many relevant quantities are directly observable by the subject or operator. Therefore, augmentation can be conceived as the supplementation of the useful signals and cues that the brain can take advantage of to improve movement performance and learning.

The augmented cue environment is designed to help the human perform and train for the task. Task interactions are produced by an apparatus that is coupled with the activity and possibly with the subject to enhance the scope of conditions. The apparatus can also include an assistive device that mechanically augments human movement. Note also that this configuration also applies to settings where humans operate in teleoperation such as a surgical robot, where the subject interacts with the system through a visual display and haptic interface, or even in the context of the operation of prosthetics.

The forms of KP feedback that are most useful are those that contribute to the understanding of the task or movement. This explains why providing a type of normative reference trajectory, e.g., to model after, is not necessarily useful. In that sense, KR has the advantage that it provides objective information about the implicit correctness of a movement.

Since human attention capacity is limited, it can be helpful to select augmentations that also account for these limitations and possibly organize these in ways that allows the brain to take advantage of the mechanisms used to operate efficiently with information (e.g., chunking).

Creating KP feedback contributes to understanding the task or movement. This can be achieved by using movement kinematic and dynamic measurements that produce KP that is connected to the movement outcomes, as well as organized in terms of timing and form, etc. in ways that are consistent with the movement's functional dimensions, including biomechanics, motor control, and sensing or perception mechanisms.

Therefore, to make it possible to generate feedback that helps to provide an understanding of how the movement technique contributes to specific outcomes and other attributes, a central requirement for motion analysis and cueing platform technology is the decomposition of movement into elemental movement units that are grounded in biomechanics and motor control principles including muscle synergies.

At the same time, using feedback that is structured based on the natural functional organization of movement, helps better overcome attention limitation since the movement units are part of a coherent movement language. Finally, a technology that reinforces and teaches this natural movement language will help acquire mental models that enhance the subject's movement intelligence. This type of cognitive enhancement would be difficult to develop using an ad hoc notation system.

By working within natural functional elements and features, it is also possible to factor out effects due to individual differences. Focusing on the structural characteristics of movements derived from performance data, and subsequently identifying features within the functional elements that contribute to the outcome, makes it possible to design feedback augmentation that targets individual movement characteristics, but generalizes over the range of skill and styles as well as differences that can arise due to injuries and other factors.

Formal Movement Analysis

The following builds on the understanding of movement structure and organization to briefly review select concepts involved in the formal analysis of human movement.

Most tasks are composed of a series of stages, and each stage involves complex movement patterns that are themselves delineated in distinct phases. This understanding of how movement behavior is composed provides the general approach to decomposing movement performance data needed to support analysis of movement skills. It is also beneficial to understand what aspects of skills can be assessed from these different elements and levels of analysis.

Movement analysis includes at least three components. The first component involves decomposing the movement into primary movement elements or units. Units are typically associated with subtasks or subgoals that depend on the elements of the task and environment giving rise to the task stages. These units manifest as movement patterns that emerge from the functional characteristics of the movement interactions with some elements of the task or environment within a task stage, and therefore these units are also named movement patterns in this disclosure. Second, is segmenting these movement units into the sequence of movement phases. And third, is decompose into components that can be combined in parallel to achieve the coordination of the body segments and muscles, i.e., muscle synergies.

Therefore, there are three primary levels of movement organization, including i) the movement profiles and their associated outcomes. This level corresponds, for example, to task level description and represents the overall movement element or unit such as a tennis stroke in tennis. Also, ii) the movement profiles are usually composed of series of multiple phases. This level corresponds to the biomechanical implementation, i.e., the coordination of the limb segments and joints to achieve a complex movement. And iii) The movement phase profiles can then be decomposed into muscle synergies. This level corresponds to the neuromuscular implementation, i.e., how the profiles are achieved by superposition of muscle units. The muscle synergies represent muscle activation patterns.

The first organizational level corresponds to the building blocks developed by the brain through interactions with the environment and task elements to partition the workspace efficiently, and achieve a range of outcomes relevant to a task. It is related to what could be considered the semantic characteristics, i.e., the meaning of the movement elements in relationship to the task goals, elements, contingencies, and the range of conditions.

The second level, the phase segmentation, corresponds to the functional structure of the movement, and is related to the strategy used by the nervous system to achieve the particular outcome given the available neuro-muscular system.

The third level, the muscle synergy, describes how the various muscles are activated to achieve the movement profile at the phase level. The synergies typically provide spatial and temporal components that can be combined to achieve a variety of movement. Therefore, it is expected that same set of synergies can be reused by other movements. Yet, for example, in tennis the arm segments configuration can be very different at different stroke phases, therefore it is likely that different sets of synergies are used in each phase.

As already discussed, complex human movements are high-dimensional, i.e., their description requires large numbers of state variables (position, velocities, angles). The representational complexity is in part due to 3-dimensional (3D) space which involves six degrees of freedom for the linear and angular motions. This number is multiplied when multiple body segments are involved, and becomes exponentially complex once ligaments and muscles are accounted for. In addition, the effects of dynamics dictate how these state variables evolve over time and interact through the action of forces (both internal effects such as inertial coupling and the external actions such as the muscles or aerodynamics, etc.). For this reason, tracking even a single segment or object in 3D space such as a tennis racket or forearm, requires a dozen state variables.

Their temporal evolution is described through coupled differential equations. These differential constraints and other constraints on joint configuration, etc. result in geometrical properties which can be exploited for analysis. From a control standpoint, the formulation of movement programming in engineering or robotics follows from the equations of motion, and the specification of an initial state (starting configuration) and a goal state (see FIG. 2). Such problems can be solved as a dynamic program, or a two-point boundary value problem. The trajectory is obtained by solving for a trajectory that minimizes a pre-specified cost function (e.g., trajectory duration or energy). This formulation leads to equations which provide conditions for an optimal trajectory. Thus, for a given initial state and goal state (e.g., that specifies the outcome), there typically exists a unique optimal movement trajectory. The control and trajectory optimization framework provide useful tools for the conceptualization and analysis of movement. For example, it is possible to define cost functions that characterize human trajectories, such as energy or more general physical performance. Furthermore, the calculus of variation used in trajectory optimization make it possible to investigate relationships between variations in trajectory and outcomes of the trajectory.

Movement measurements, such as from wearable motion sensors or optical motion capture systems, are typically given in the form of time series. Since these time series typically originate from nonlinear dynamic processes, their analysis relies on an understanding of the structural characteristics of the underlying dynamics. These structural characteristics are associated with the architecture of the movement, such as the movement phases in a tennis stroke or golf swing. Insights can be gained using computational visualization tools such as phase space; however, the states may have too many dimensions to be practical. Therefore, the data should be reduced. The behavioral data captured from the available measurements results in a high-dimensional state space. The dynamics driving the behavior, on the other hand, may be lower-dimensional.

Dimensionality reduction is a class of unsupervised learning techniques that can be used to discover the state dimension of the underlying behavior. The goal is to transform the original movement data time series which are described in terms of the high-dimensional time series x_t into a lower dimensional description that preserves the geometric characteristics of the underlying nonlinear movement dynamics. This can be done, for example, using Taken's embedding theory. Examples of recent applications of dimensionality reduction for movement analysis include gait analysis.

Movement Patterns Analysis

Although movements are usually high-dimensional behaviors, trained movements typically have specific patterns. Patterns have the useful property that even though the behavior relies on many degrees of freedom (DOF), they can be described by a few, dominant DOFs. The patterns form a lower-dimensional system as a result of the coordination provided by the neuro-motor processes, and other perceptual and control mechanisms. The lower dimensions however can hide a complex geometry and topology.

The movement architecture can be analyzed by focusing on the low dimensional manifold that are associated with the particular movements' dynamics pattern. Using a nonlinear dynamic systems formulation gives access to analysis and modeling tools that, under certain conditions, can reconstruct the pattern dynamics from measurements of the behavior. The reconstructed dynamics can then be analyzed to determine the underlying structure and geometry, which can then be used to determine useful abstractions or models.

Using mathematical tools used for the analysis of nonlinear dynamic systems, the movement patterns can be described by a nonlinear mapping F associated with discrete-time nonlinear dynamics:

x_t+1=F_t(x_t,t,ϵ_t)  [1]

where F_t is a map, x_t∈ⁿ is the state vector at discrete time t∈, and ϵ_t is a time-dependent noise. A continuous time representation could also be used. In the forthcoming discussion the dynamics are assumed to be autonomous and use a constant map F_t=F.

The nonlinear model of a movement pattern therefore can be described by a map F that captures the combined effects of the biomechanics, sensory, and motor-control processes. This model assumes that the learned movements result in deterministic dynamics. In this case, the dynamics are given as an ordinary differential equation (ODE) {dot over (x)}=f(x(t), ϵ(t)), which describes a vector field and is typically called the flow. The set of initial conditions which result in the same asymptotic behavior are referred to as the basin of attraction. Such nonlinear dynamic models can describe a broad range of phenomena. The model could be decomposed into subcomponents, giving access to the various contributing systems and processes. For example, it may be possible to explicitly model how the users adjust their movement pattern to changes in conditions, such as adjustment of a forehand topspin stroke for changes in ball height at impact. However, at this point in time the behavior is regarded as a closed-loop behavior which abstracts the various internal mechanisms.

The language of nonlinear dynamic systems makes it possible to describe the collection of movement patterns that composes a user's repertoire in a particular activity (tennis, skiing, surgery, etc.) by a collection of distinct dynamics or maps {F_α, F_β, . . . , F_γ}. In many nonlinear time series, the movement system state variable x is generally not directly observable. Instead, measurements y are acquired for example through motion sensors. The observations, or measurements can be defined as: y_t=h(x_t, η_t), where h is the output map and η_t is measurement noise.

A property of movement at the highest level is referred to as “motor equivalence.” The fact that the brain generates movements that are equivalent in terms of their accomplished outcomes underscores the idea that at the highest level the brain encodes outcomes and their relationship with task goals. The planning and monitoring functions associated with goals are part of the brain's executive system. For example, in tennis, the player selects a stroke type based on the desired outcome and the conditions (ball state including expected impact height, velocity, and spin of ball). Even within the continuum of conditions and outcomes, it is possible to recognize distinct classes of strokes. The invariant characteristics in movement features enables the delineation between movement classes, e.g., movements within one particular class can be related through some smooth transformation such as rigid-body translation and rotation, i.e., they are invariant under this class of transformation. The overall movement class can be subdivided into subclasses. For example, a hierarchical decomposition would group movements based on relative similarity.

In tennis, the overall stroke class can be subdivided into dozens of subclasses based on movement where the levels represent different types of features. For this example, a top hierarchical level is called the category level. It differentiates between groundstroke, volleys, serves, etc. The distinction between stroke categories is made primarily based on the height of the impact point. Further, subcategories can be created based on the side of the impact, i.e., forehand or backhand. Even further subclasses can be delineated based on the outcome (topspin, flat, slice), and strength. Beyond these common classes, finer distinctions can then be added based on additional aspects of stroke technique, such as open or closed stance. Most of the stroke characteristics can be determined entirely from the racket trajectory and therefore do not require additional measurements such as the position of the player on the court.

Each movement pattern class in a repertoire has different geometrical characteristics and their domain may occupy a different subspace of state-space (see FIG. 14). The shape and dimension are a result of the dynamics, which is given by the transition map F. The repertoire is the collection of these shapes or patterns. The precise geometrical characteristics of the movement patterns can be described via embedding theory. The idea is to determine the subspace of DOF that fully describes the movement. The dimensionality of the system and the geometry of the manifold that contains the trajectory describe the movement class structure.

As is often the case in nonlinear system dynamics, the state transition map F (the dynamics), the output map h, and the dimensionality of the state vector n are not known. Techniques of nonlinear time series analysis can (assuming deterministic dynamics F and smooth output map h) estimate the dynamics associated with a movement pattern from time series obtained from measurements of the behavior. Repertoire characteristics can provide a variety of information about skill. Movements are typically analyzed in specific classes without considerations about the overall repertoire structure. The movement repertoire for a particular activity domain describes how a user organizes the outcomes and technique in that task domain. The simplest way to classify movements into repertoire is to extract features from the time series and apply clustering techniques to determine classes.

Movement classification has been used in other applications unrelated to skill modeling, such as activity detection or gesture recognition. Gesture recognition is a growing aspect of natural human-machine interfaces. The general goal in the latter application is to determine motion primitives that provide a low-dimensional description of the various movements that can occur in that domain. The primitives can then be used to classify the movements. The library of primitives can then be used by other agents to identify the intent of a human or robotic agent and, for example, allow collaboration between agents. The emphasis of gesture classification is the identification of semantic characteristics. In the present application, the goal is classification based on characteristics that are related to movement technique and outcomes. Typically, the higher categories of the stroke classification can be considered in a semantic sense (e.g., groundstroke vs. volley or backhand vs. forehand), and the lower level classes are related to different techniques and conditions (see FIG. 9).

A particular ensemble or repertoire of patterns in a domain of activity arises through the effects of biomechanical, neuro-muscular constraints, as well as task-related constraints. In the most general sense, the patterns describe how an individual's movement techniques are used to achieve an outcome. One aspect of the movement characteristics is how they are broken down into phases. Overall movement pattern characteristics, therefore, are the result of the phase structure, and those can be used to classify motion patterns.

The serial order in behavior, i.e., the task stages, and the movement phase structure are usually distinct. As already discussed, the serial order is associated with the activity level, for example, characteristics related to the activity constraints such as process stages, rules, etc. The movement phases, on the other hand, are associated with the movement technique and are related to characteristics and constraints of the movement system and its interactions with the environment and task elements.

For example, in tennis, the movement stages associated with the serial-order of behavior, include serving, then moving to the anticipated return position, making adjustments in the positioning as the ball returns, setting up for the stroke and engaging the ball using the stroke type required for the desired outcome (see FIG. 7-9). Each stage can be parsed to extract the primary movement unit, and these movement patterns can then be analyzed to determine the movement functional characteristics, i.e., how the movement produces its specific outcomes while at the same time adapting to conditions. The functional analysis is facilitated by further decomposing the patterns into movement phases. The phase structure of the primary movement patterns defines the topological characteristics of the manifold, while the dynamics that drive the phases define its geometrical characteristics.

Phase Segmentation

As already discussed, many complex movements are achieved by combining several movement phases, leading to further temporal structuring of the movement. Examples include the phases in locomotion gait 445 or the phases in a tennis stroke 441 (see FIG. 4). Phase structuring of patterns typically arise from the intrinsic movement constraints (biomechanics), some aspects of task constraints, as well as functional factors related to motor-control and decision mechanisms as discussed elsewhere. For example, in gait, distinct phases are associated with the basic leg biomechanics and mechanics of ground interactions.

In tennis, the general goal of the user is to return an oncoming ball and further control the trajectory of that ball (see FIG. 7). This is accomplished by imparting precise linear and angular momentum to the ball with the racket. The user controls the ball primarily by modulating the amount of momentum imparted to the ball and selecting the precise interception point and time as the ball enters the half-court (FIG. 9). For accomplished players, the overall tennis stroke motion encompasses the kinetic chain formed by the legs, hips, shoulder and elbow, and wrist. These segments are coordinated to form a continuous movement starting from the backswing all the way to the follow through and recovery. At closer inspection, distinct phases can be recognized.

The exact phase characteristics depend heavily on skill level. Beginner players primarily swing the racket from the shoulder without very precise coordination with the rest of the body segments. Advanced players exploit the entire body kinematics to maximize the outcome. Ultimately, the phase characteristics reflect the combination of the body segments' biomechanics and neuro-motor strategies, including the muscle synergies that achieve the highest outcome reliability with best use of the physical capabilities. Different phases are associated with different biomechanical functions. For example, in walking, synergies that are activated at specific phases of the gait cycle (e.g., forward propulsion, swing initiation, deceleration, etc.) have been identified.

The role of constraints in creating distinct movement phases can be explained using concepts from constraint optimal control. In optimal control, trajectory segments are related to the concept of singular arcs, which correspond to segments where different sets of constraints are activated by the trajectory. In general, these systems are best controlled using switched control laws. The control law is determined based on a partitioning of the system's state. As the system is driven by the control action, and travels through the different partitions of the state-space, the control strategy switches to best account for the local characteristics of the dynamics.

Following the nonlinear dynamic systems description, trajectory phasing can be described mathematically as a sequence of dynamic models F₁, F₂, . . . , F_N. The overall trajectory is obtained by a series of initial values and asymptotic behaviors, where the next set of initial values corresponds to the terminal values of the previous phases (FIG. 3A). The dynamics associated with each phase result from different joint and limb segment configuration and force fields. Each dynamic model F_i, can therefore be assigned a state-space region specified by an initial state set and a goal or subgoal set. For example, once the dynamics are initiated from the initial set, the initial dynamics F₁ will take the state to its subgoal set χ₁, and from there, assuming the state satisfies the next initial state conditions for the next dynamics F₂, and the dynamics are triggered, the system dynamics will switch to the next phase, where it will evolve to the next subgoal, etc. This process can be cyclic, where the state transitions form a loop, such as for periodic movements (see e.g., running 445 or swimming 446 depicted in FIG. 4). In other activities such as tennis or skiing, the behavior can be quasi-cyclic, where for example the same general sequence of movement phase continues after a pause (see FIG. 3B). The dynamics can also switch between patterns that have different phase segments, such as a different stroke type or gait type, or altogether different movement patterns, such as in skiing when switching from a periodic turning sequence to a stopping maneuver, or switching between different stroke patterns in tennis.

The switching between dynamics at the phase transitions are typically determined by conditions on the terminal/initial states, e.g., χ_1i=χ_0j. As already discussed, the dynamics associated which each phase result from different joint and limb segment configurations and force patterns. These force patterns are determined by the spatio-temporal muscle activation patterns, i.e., muscle synergies. Within the phases, in particular for fast motions, the force patterns are specified in open loop, therefore the dynamics are specified by the force fields associated with the muscle synergies. The brain ostensibly learns to compensate for variations in initial conditions by adapting these force fields. This makes it possible to produce fast corrections in movement without relying on feedback. Feedback can be used intermittently, e.g., during phase transitions or during specific movement phases which can accommodate such effects, e.g., because of the slower dynamics and availability of sensory information.

The muscle synergies describe the coordination between the different muscle groups and limbs segments that are used to implement movements. The synergies are a type of motor primitive which is typically reserved for the neuro-muscular coordination. In examples discussed earlier, various movement profiles observed in an activity can be obtained through the combination of such primitives. Decomposition into synergies therefore can help determine the set of biomechanical and neurological components that participate in movement skill. In turn, this information can be used to gain understanding about the biological components, and could be useful for physical performance and injury prevention.

Popular techniques are based on non-negative matrix factorization, which decompose matrices that in this context represent the data for a movement phase into a product of matrices. Synergies have been characterized with a variety of measurements, including movement profiles of the end points, joints and/or body segments, as well as muscle and neurological activity, such as provided by surface electromyography (EMG). The type of measurements determines the accuracy of the results. For example, simple end effector or body segment measurements may not provide synergies that correlate strongly with the neuro-muscular activity. Synergy analysis has not yet been integrated in clinical settings where it could be used for assessment and rehabilitation. Since synergies have been identified at different levels of the neuro-motor hierarchy (motor cortex for grasping, brain stem for posture and spinal cord for locomotion), the muscle synergy analysis can provide a more precise picture of neuro-motor deficits.

Movement Functional Structure and Primary Outcome

Some movements have an explicit outcome or goal. This goal may be the movement's end state, i.e., χ_goal=χ_N, or it could be the state at an intermediate phase such as a subgoal. The latter is the case for the tennis stroke. While the ball impact is the primary goal or outcome of the stroke, this phase is not the actual end of the movement. The movement phase following the impact, the follow through, is one part of the overall movement pattern. Most complex movements involve many body segments or degrees of freedom. Therefore, the state trajectory is a multidimensional state vector and it can be helpful to add distinctions between the different state trajectories that participate in the action. Focal and corollary movements are distinguishable; the focal movement is, for example, in a piano performance, the finger movement that hits the key; the corollary movement is, for example, the motion of all other fingers that are part of the overall kinematic pattern involved in the task of hitting the key.

Not every movement behavior has an explicit goal or outcome. For example, most of the movements used in skiing 444 have as purpose to control the skier's speed and direction. From a dynamic system standpoint, this goal involves generating a centripetal acceleration though the interaction of the skies with the terrain. Depending on the skier's state and terrain conditions, different motion pattern of the legs and hips, etc. are used to achieve the best outcome (will be discussed elsewhere, see FIG. 4).

It is possible to define an optimal trajectory that takes the system through the phase sequence achieving the goal condition (outcome) while minimizing a performance objective such as jerk or energy. Given the biomechanical constraints, muscle synergies, etc., the optimal trajectory is associated with a specific phase sequence. The conditions at the phase transitions, i.e., the set of initial states, and subgoal states, χ_1i=χ_0j as well as the dynamics F_i describing the transitions, represent characteristic features of the optimal trajectory (see FIG. 3A).

The absolute optimal trajectory is the global optimal solution for a given outcome, while the local optimal trajectory corresponds to a given phase structure. The latter, for example, represents situations where due to a lack of flexibility or skills, or the presence of an injury, only a limited set of configurations and/or force fields is achievable. Therefore, movement phase characteristics provide valuable information for injury prevention and generally also for rehabilitation.

In optimal control theory, perturbation of the initial value leads to neighboring optimal trajectories. This is guaranteed if the initial value is within the so-called basin of attraction of the system. A similar idea can be used for perturbations in the dynamics F. Such perturbed dynamics lead to slightly different asymptotic behaviors; however, for small enough perturbations the trajectories stay close enough to the nominal trajectory that these perturbed trajectories belong to the same movement pattern. The range of perturbations in the initial values and dynamics for which the trajectories remain in the basin of attraction defines the admissible envelope. Perturbations in the dynamics and disturbances are captured by the time dependent noise term ϵ_t

FIG. 3A illustrates the trajectory envelope 113 for a hypothetical movement pattern delineating the movement phases that typically arise from biomechanical and neuromotor constraints. The figure also highlights a primary outcome and its associated phase (shown as a goal phase). It also shows an optimal trajectory across the movement phases, and different envelopes (optimal, admissible, feasible) resulting from the various movement constraints.

The trajectory envelope delineates a region of the state-space over time and highlights the feasible envelope and the envelope of admissible trajectories as well as the region for the optimal trajectory's initial conditions (x*_0i), and the optimal trajectory (x*(t)). The structure of the movement both in terms of patterning and the phase segmentation are given by its spatio-temporal characteristics. Movement characteristics are defined by the geometry and dimension of the manifold containing the trajectory.

Several phases are shown in FIG. 3A including: movement initiation, phase 1, phase 2, an intermediate goal phase, a follow-on phase, and recovery phase. For tennis, these phases correspond to the stroke initiation, backswing, back loop, forward swing, impact, follow through and recovery. The goal phase in tennis represents the impact phase, which is the phase during which the primary outcome is produced.

These movement pattern characteristics are usually determined from the topology of the movement pattern manifold obtained from analyzing the nonlinear time series. A user may choose “admissible movements” that belong to the same movement pattern and still reach the goal conditions or outcome. This could happen due to changes in movement goal conditions (impact height and velocity), or imperfect initiation of the movement. The suboptimal trajectories can still reach the desired end state or outcome; however, they will typically require more physical effort, may cause stress in some of the muscles or joints, or other undesirable effects. The physical performance can be described through models of the musculoskeletal system and cost functions such as for energy consumption.

Movements belonging to the same pattern can therefore be related through perturbations relative to a nominal trajectory. Moreover, the trajectory perturbations also result in perturbations in the primary outcome and any other secondary outcome characteristic such as the different phase outcomes. Using this data, it is therefore possible, for example through regression analysis or sensitivity analysis, to determine relationships between the trajectory perturbations (which correspond to the movement technique) and perturbations in outcomes. This information provides a quantitative basis to generate skill characteristics, such as what aspects of the technique contributes favorably to the outcomes and vice-versa what aspects are detrimental to good outcomes. This knowledge in turn can be used for training and eventually help synthesize feedback laws for real time cueing.

FIG. 3B is an illustration of the finite-state model representation 114 for the system shown in FIG. 3A. By modeling movement patterns as a sequence of phase segments with distinct dynamics F_i, the pattern dynamics can be abstracted as a finite-state model (see FIG. 3B and FIG. 5). In the present case, the finite states are the individual phase dynamics F_i which take the system from initial value x_i0 to the next subgoal state x_i1. More generally, the initial and subgoal states are represented by sets to account for the variations and disturbances that are typically expected in human behavior. With this model, the overall motion behavior is then given by some finite-state automata which gets triggered from the initial state and initial movement phase. The motion behavior combines both continuous dynamics and discrete variables that capture phase transitions and mode switching which may be associated with discrete decision variables. Hybrid models can be used in many modern engineering applications including robotics such as for autonomous systems, as well as human-machine systems. Once the structure of the motion is characterized, it can be described by finite-state models.

Statistical models, in contrast to deterministic models where the current state uniquely determines the evolution of the system (i.e., within the disturbance or model uncertainties), describe the evolution of the probability density of future states. Statistical models such as Dynamic Bayesian Networks have become increasingly popular in data-driven approaches. Popular applications in the movement domain are identification of human activities. These approaches typically require learning the phase of activities based on statistical pattern analysis; subsequently using this knowledge to discretize the state space into discrete states; and finally determining the state-transition probabilities. A common model is the Hidden Markov Model (HMM). Most of the notational systems focus on the discrete game structure and can be used to analyze game plans but currently do not reach down to the actual movement skill level.

Real-time movement phase estimation can be implemented by someone trained in the art. For example, a multi-layer HMM application to movement could be based on similar models to those used for real time speech recognition. Decoding sound recording for speech recognition typically proceeds on multiple levels. Most of those are associated with the levels of organization of the speech production system. The units of decomposition of speech is based on phones which combine to form the phonemes. The phonemes are the basic building blocks used to form words. The phones are related to features of the vocal movements. This model for movement corresponds to having, at the top level, a movement phase model which describes the probability distribution over possible sequences of movement phases. At the midlevel, a phase model that describes the composition of the movement phases in terms of movement components (c.f. synergies). And finally, at the bottom level, the movement model that describes the movement components based on features in the available measurements (IMU unit or other sensors).

Movement Skill Acquisition

Learning is about organization of information, which is a process that proceeds in stages. The following reviews some of the concepts related to the skill acquisition and its implications for training and concludes with an outline of the roles of technology to support learning complex movement skills.

Organization of Information

Organization of the learning process and codification and organization of information associated with movement is dictated by principles that can help mitigate complexity. These mechanisms are primarily directed at exploiting structure in task and interactions between the agent and the environment. Two major concepts have been proposed for dealing with complexity associated with representation of information: chunking and hierarchical representations. Chunking describes a general memory structure that applies to different domains.

Miller proposed the following “Chunking Hypothesis: A human acquires and organizes knowledge of the environment by forming and storing expressions, called chunks, which are structured collections of the chunks existing at the time of learning.” (Cited in Newell, 1981). This hypothesis is based on research on perceptual behavior and memory retrieval (see Miller, 1956) and earlier work by DeGroot in chess. The general idea of chunking is to achieve a more efficient encoding by combining individual bits of information into wholes. Gobet, for example, describes it as “a collection of elements having strong associations with one another, but weak associations with elements within other chunks.” For a review see (Gobet, 2001). A central assumption behind chunking of information is that the joint encoding reduces the latency of information retrieval, and more generally provides more economical information encoding and processing.

Chunks have been extensively studied in domains that involve static and discrete quantities, such as the perception or memorization of chessboard configuration. Early chunking theory has been studied as part of human perception and more generally information processing in (Miller, 1956). Many activities are described by a complex spatial and temporal structure. Later, the chunking theory has also been applied to improve our understanding of motor learning and more generally skill acquisition. There exist fewer investigations in the sensory-motor domain. In that domain, chunking is primarily associated with the concept of “serial order in behavior” introduced by Lashley (Lashley, 1951), and the general hierarchical learning theory.

The hierarchical models conceive complex skills as a “hierarchy of habits.” This model was introduced by Bryan and Harter (1897) studying Morse code learning. In that example, the telegrapher learns letters first, followed by sequences of letters to form syllables and words, and then phrases. This model applies to many motor skill domains. In most movement skills such as tennis, the elementary actions are movement phases (muscle synergies) that can be combined to form gross movements. Learning such skills, therefore, involves learning elementary movement units, and combining those into larger movement elements that are themselves nested into actions.

Lashley's serial order in behavior was a response to the linear sequencing that was suggested based on association learning theory (Terrace, 2001). Instead of a serial sequence, Lashley argues that skilled behaviors are planned, and plans have a hierarchical organization which combine multiple units of behavior into larger units. Some units are related to a movement's biomechanical and functional constraints, and others are related to task constraints (e.g., subgoals).

Following the hierarchical representation, it is possible to decompose the activity and the associated movements into a sequence of elements, which are themselves decomposed into smaller elements. Chunks are usually not made of arbitrary segments but have a functional purpose. Chunks, therefore, combine specific sensory and motor patterns that relate to the task environment interactions, as well as the constraints of the organism.

For example, in tennis, major behavioral chunks can include the “ready state,” “reposition,” “preparation,” and “stroke execution.” Each chunk can be described by a set of movement patterns with their associated perceptual process. During the ready state, the player orients himself or herself, extracting cues from the environment needed for court positioning, observes the motion of the ball and the opponent, etc. This information allows prediction of the location of anticipated ball interception selecting the desired outcome and planning the sequence of actions to achieve the desired outcome of the stroke. During repositioning, the player acquires the new court position and may start to bring the racket back (backswing). During the stroke preparation, the player adjusts his or her posture and extracts updated information about the ball and opponent needed to fine-tune posture and prime the stroke execution. Just before the stroke execution, the player obtains final ball trajectory information for the interception. The execution of the forward swing is synchronized with the arriving ball. Finally, after the execution of the stroke, the player returns to a ready state.

The behavioral chunks forming the larger program are typically subdivided into smaller sensory-motor units, starting with the elements such as muscle synergies that are combined to form larger movement patterns. For example, the stroke is composed of a sequence of body and arm movements (described elsewhere). Similarly, extracting information involves a type of perceptual chunking which describe how the various sensory stimuli are integrated to form the cues that can be used to predict the intentions of the opponent, anticipate the ball trajectory, and select and initiate the appropriate stroke type.

As an individual acquires experience in a task, they assimilate the movement units into procedural memory. Hence, less attention is required at the level of individual components that form the chunks, allowing increasingly automated processing. Proficient individuals are able to focus attention on task-relevant information, which enable better planned, more systematically organized behavior with fewer extraneous movements and smoother movement execution that takes advantage of the subject's physical performance.

The acquisition of open motor skills with their environment interactions, therefore, can be conceived as the acquisition of a library, or repertoire, of sensory-motor patterns, their associated perceptual cues, and the larger motor programs used to deploy these patterns and attain outcomes needed for successful task performance.

Following the general learning theory, learning proceeds as an evolutionary process. Behavioral patterns are associated with actions that produce outcomes for the task. Valuable outcomes are rewarded, and thereby produce reinforcement for learning patterns that are successful, i.e., have a positive outcome on the task. This process, however, depends on extensive practice and experience in the specific task domain.

The chunking theory of learning also provides additional understanding of the learning process. For example, it has been used to explain the so-called Power Law of Learning (see Newell, 1981). This law describes the improvement in skill (measured as response time) as a function of training and has been validated in many domains besides perceptual motor tasks, hence it is widely accepted as a universal law. However, the law has received criticisms, in particular that it does not explain qualitative changes in movement dynamics with practice (see Newell, 1991). As described in that reference, these may be due to the limited tasks used in studies (few degrees of freedom and limited perceptual environment).

The learning time to reach proficiency depends on the task environment and complexity of interactions associated with the movement production. To help appreciate this, it is useful to consider Newell's notion of environmental exhaustion, which can help describe the number of chunks required to cover the range of task conditions. Many unique chunks are required in task domains where there are many unique configurations (see for example, chess). Natural environments tend to have a statistical distribution of conditions, with large amounts of similar or related configurations and fewer unique configurations. This fractal, or self-similar, structure in natural environments means that these can be expressed as hierarchical structures that take advantage of modularity of the representation. However, even though efficient representations may exist, the individuals still have to experience the range of conditions to form an understanding of the patterns and develop a sufficiently rich repertoire. This explains why surgeons or athletes require thousands of hours of practice before they are proficient, and also, why they keep improving with additional experience (assuming the experience is sufficiently varied and rich).

Learning Process and Stages

Finally, it can be beneficial to understand the brain processes involved in learning skills, and, in particular, what changes take place in the brain, and how the brain processes and stores information as a function of the different stages of acquisition. Fitts proposed three major stages of acquisition (Fitts, 1964). The cognitive stage (also called verbal stage) is characterized by a conscious effort required to understand and control the movements. As a result, in this stage, movements are slow, they lack dynamic coordination, and have low success rate. Problem solving, by way of cognitive processes, is a critical aspect for the development of mental models, or representations that could be used to support this stage (Ericsson 2009). During the associative stage, the movements are partly automated. Conscious efforts are fewer but are still required to monitor and improve performance. Finally, in the autonomous stage, movements are stored in procedural memory which allows automatic execution. Movement in this stage may still require visual inputs to ensure accurate and consistent execution. However, these inputs are also automated and focus on very specific elements, i.e., cues.

The type of knowledge gained by subjects as they learn to be proficient in a task is directly related to the structure of the task, and the structure of the interactions between the movement and the task and environment elements. For spatial behaviors, the critical aspect is the structure of the interaction between the subject and the task environment and elements (see e.g., the interaction patterns in Mettler 2015).

These interactions combine the perceptual mechanisms used to extract information from the environment and the dynamics governing the agent's motion. Ecological principles (see Gibson 1979) suggest that humans and animals exploit information that can be obtained directly from the perceptual environment without relying on complex internal models. However, the brain can also learn more subtle patterns relevant to a task (see e.g., the squash study of Abernethy et al. 2001). Cues are determined by those features in an organism's perceptual environment that are directly relevant to the movement guidance and coordination with respect to the task environment. Cues can be viewed as sparse sensory stimuli such as, for example, in Tau theory (see Lee 1998).

As individuals familiarize themselves with a task, they develop a repertoire of automatic behaviors and mechanisms to deploy these behaviors (Ashby 2010). The repertoire represents a library of sensory-motor patterns that is stored in the brain's long-term memory. The structure associated with the task and interaction between the movement and task elements suggests that sensory-motor patterns are grouped hierarchically. The top sensory-motor chunks define larger categories of behavior, such as ground strokes and volleys; the intermediate level, which include the various stroke classes in a category; and at the lower level of the hierarchy are components of behavior which include muscle synergies and are shared by different classes.

The hierarchical and modular encoding has been known from early studies of the neural visual processing and encoding, and has been verified in the domain of movement encoding and control (Poggio 2004). For example, movement patterns within related movement classes (e.g., tennis forehand slice and top spin) share similar sub-movements. The movement phases result from the activation of muscle synergies that are encoded in part in the spinal circuits. Multiple studies have demonstrated the modular encoding of movement (Mussa-Ivaldi 1999).

Each class of movement learned has some operating range that defines the range of validity of the learned patterns. However, there are limits to generalization. These are due both to the neural encoding mechanisms (Kawato 1999), but also due to the structural and functional characteristics of the state-space. Therefore, to cover the range of outcomes and conditions typical of open motor skill, multiple movement classes may be employed. Hierarchical representations are used to efficiently encode these movement classes into motor programs.

Movement Skill Acquisition

In summary, movement skill acquisition results from the need to adapt to the task and environment, and thus learning proceeds incrementally with exposure and experience performing a task. Therefore, it is possible to conceive skill acquisition as an evolutionary process (see FIG. 11). The specific skill elements are classes of movement patterns that are evolving with their usage in the task or activity. Learning and perfecting skills are the result of an iterative process that takes place as these elements are repeated under different conditions, and modified based on the observed outcomes and effectiveness to the overall task goals and performance.

The acquisition process can thus be described as the evolution that involves two primary dimensions: 1) the diversification of the movement patterns to respond to the range of requirements and conditions called for by open motor tasks; 2) the refinement and optimization of individual movement patterns, which corresponds to the changes in those movements over the stages of acquisition.

The process therefore can be analyzed by tracking the movement repertoire over time. At any given time, an individual's skills are described by a repertoire with one or more classes of movement patterns (FIG. 11). The repertoire reflects both aspects of how the individual deals with the task and environmental structure, as well as the individual's perceptual and motor control abilities. Each pattern class can be at a different stage of acquisition.

Sensory-motor patterns serve as units of behavior used for organizing and planning the behavior toward the larger task goals (see Mettler, 2015). Identifying the repertoire of patterns therefore also provides the elements needed to analyze the skills at the planning level.

Challenges in Movement Acquisition and Training

The comprehensive understanding of movement skill acquisition highlights several challenges to efficient training. Forming a repertoire of movement patterns, and the associated perceptual and planning processes that enable task proficiency and versatility, depends on the training process and in particular the information available to support and guide this process. Without a coach, human skill development depends primarily on a trial and error approach. Typical information to guide the process includes the movement outcome. This so-called “knowledge of result” has been shown to help learning. However, this alone typically does not contain sufficient information to efficiently teach users how to improve their movement. It can also make individuals dependent on this (see Newell, Schmidt) type of feedback. Many common movements can be learned efficiently through trial and error; however, for complex movements such as found in surgery, music and many sports, trial and error are limited because many of the movements involved in these activities are unnatural. Additional information is needed for the discovery of the correct or optimal technique. Furthermore, some activities such as surgery also don't afford much opportunities for trial and error.

This situation is also similar for rehabilitation because the physical impairments acquired from an injury or disease can add constraints that make movements challenging to develop under natural conditions. For those situations, trial and error learning can be extremely time consuming and cannot guarantee that correct movement patterns will be discovered.

Therefore, movement skill training has depended on the expertise of a coach. For rehabilitation, the patient depends on the availability of a physical therapist. The traditional role of a coach is to help focus training efforts on correct technique, and attend to relevant aspects of the task and performance. However, even expert coaches are subject to limitations in perceptual and information processing. Most skilled movements involve coordinating many degrees of freedom that take place over short time scales (hundredth to even tens of milliseconds). These movements, such as a tennis stroke or golf swing, are highly dynamic behaviors that combine temporal and spatial dimensions into complex patterns.

Furthermore, motion patterns depend on complex biomechanical constraints and muscle synergies. These depend on musculoskeletal constraints, as well as the physical fitness and general health of an individual. Therefore, the training approach should be able to account for individual characteristics both in the movement technique and in the longitudinal skill development process. It takes great experience for a coach to be able to analyze movement and identify relevant characteristics of these patterns while taking into account the individual's constraints.

In contrast to closed motor skills, in which the conditions can be controlled, open motor skills require a broad movement repertoire in order to accommodate the varying conditions associated with the task and environment and produce the range of outcomes that help to control and pursue the task goals. In addition, not all movements associated with a task have the same importance to the task performance. Some movements are part of a basic repertoire that cover the general performance and conditions, and other movements are more specialized and allow actions in more specific conditions.

Skill acquisition is a parallel process where at any given time, a subject's repertoire will contain multiple movement patterns, each at different stages of development. The two primary directions in the skill acquisition process are: 1) the development of a sufficiently broad repertoire to cover the task requirements and conditions, and 2) the refinement of the movement technique within each class of the repertoire to achieve better outcomes and/or movement performance as well as adjust to conditions. These two directions are referred in this document as the longitudinal and vertical dimensions of skill acquisition. The longitudinal dimension represents the stage of development or acquisition, which is determined by skill characteristics in specific classes of movement. The vertical dimension represents the aspects of movement skills that have to be developed to cover the task conditions. At any given time, the training can be directed at refining a movement or diversifying the repertoire. The two dimensions are typically interrelated. The differentiation of the repertoire in the vertical direction often develops from the longitudinal process of refinement of an existing movement.

The expanding repertoire, with its collection of movement classes, and their associated outcomes, each at different stages of development, results in a complex picture for anyone to operate and train with, let alone comprehend. Therefore, efficient movement skill acquisition can depend on the availability of appropriate feedback targeting each movement type as well as on a systematic method to prioritize and plan training. Human subjects also should generate this information themselves, which requires mental workload. Extracting useful information for training depends on understanding how these movements satisfy the task constraints and help meet its goals.

Finally, human skills rely on multiple levels of human information processing, including signals, cues, and knowledge. The knowledge level supports reasoning about technique, such as particular details of the movement's spatial configuration. It also supports game strategy, taking into account the environment and task elements, etc. The cue level supports the efficient processing of information; for example, the visual perceptual system learns to focus on aspects of the scene and action that provide the most valuable information for the performance. The signal level typically encompasses the information used by brain processes to control movement such as proprioception or actual visual stimuli.

Human training does not effectively use the entire scope of information levels. The range of information involved in the movement skill process cannot easily be processed. Most human training takes place through, and is codified and communicated, using hands-on demonstrations and natural language. These modalities work reasonably well for the cognitive aspects of skills, such as introducing a new movement pattern. Many critical aspects, however, relate to movement characteristics that are too fast to observe, difficult to express verbally, or need to be generated concurrently with the unfolding movement to be effective. Even professional coaches cannot reliably generate cues and signals during performance to support the training process. This is in part due to limitations in human information processing, but also because it requires excessive attention and mental workload for a coach to simultaneously analyze and cue movement performance.

Roles of Technology

Technology can play a role in several areas of skill acquisition. Technology provides a means of collecting comprehensive information about human behavior that exceeds humans' sensory processes' spatial and temporal resolution. For example, the combination of distributed sensors in the form of wearable, implantable, and remote sensors can capture comprehensive dimensions of movement performance. This includes the movement of an end effector such as a piece of equipment, an individual body segment, muscle activity, as well as subjects' visual attention and task relevant quantities (see FIGS. 2 and 24).

Information technology enables the deployment of analytical and computational resources beyond humans' information processing capabilities. Algorithms can be designed to estimate various unmeasurable quantities, which can be used to provide feedback on outcomes (“knowledge of results”), as well as more complex aspects of performance such as those involved with the fast and high-dimensional dynamics, and coordination with the environment and task elements. This functional understanding can be used to design feedback augmentations that target movement technique (“knowledge of performance”). Information technology enables scalable deployment of analytical and computational resources across larger populations, where it can be deployed to identify patterns in movement technique and skill acquisition processes that can take into account broad range of individual factors. However, to be effective, these different augmentations and feedbacks should be provided within a system that is compatible with the natural movement mechanisms and learning process.

Aspects of technology for operation of the system-wide data-driven training include:

• • 1. Integration and analysis of comprehensive performance data to assess an individual's movement technique.
  • 1.1 Diagnosing movement technique and identify possible causes of outcome deficiencies, and other relevant characteristics such as enabling efficient use of musculoskeletal capabilities, as well as effects of fatigue, or onset of injury.
  • 1.2 Identifying most actionable characteristics in the movement performance that can be used to drive training.
  • 2. Design of feedback augmentations that precisely target the specific features of movement technique and help induce the changes needed to achieve the training goals.
  • 2.1 Selecting the feedback augmentation and communication modality that are adapted to the learning stage.
  • 2.2 And, communicating feedback signals such as real-time cues that leverage human information processing capabilities.
  • 2.3 Producing synergies between various forms of communication, including visuals, natural language, and cues across the human information processing hierarchy. Exploitation of feedback signals and cues, as well as skill and performance measurements, to stimulate attention and motivation.
  • 3. Operationalize training process driven by data to enable its systematic and quantitative management.
  • 3.1. Planning of the training process through specification of training goals that are based on the subject's skill and individual characteristics, including fitness, physical strength, and health.
  • 3.2 Tracking of the longitudinal and vertical dimensions of the skill development process.
  • 3.3 Tracking the effectiveness of the different augmentation modalities and of training effectiveness for the purpose of optimizing augmentation modalities and identifying issues that interfere with progress, such as physical injuries or psychological issues.
  • 4. Combining data from a population of subjects to discover global patterns in skill acquisition, movement skills, and related factors such as injury, aging, etc. that can be used to optimize performance training over larger training cycles.

Core Technology Capabilities

Open motor skills require the development of a variety of movement patterns to produce desired outcomes under changing task and environment conditions. These movements and their associated sensory-perceptual mechanisms are acquired from experience in a task domain. Depending on the task or activity complexity, learning motor skills can take several years.

Most of the motor skill acquisition in someone's life follows a trial-and-error process. For advanced motor skills, which rely on more complex movements, usually some forms of training methods are used. Efficient training of open motor skills depends on the availability of a range of feedbacks, including information about the movement outcome (knowledge of results or KR), the movement technique (knowledge of performance or KP), and overall training progress and process.

As described previously, training in open motor skills proceeds in two primary directions: the development of the range of movement patterns that help to accommodate the range of conditions and actions required by the task; and the development of optimal movement techniques for each movement class to allow reliable and efficient achievement of desired outcomes for a task. As a result, training or rehabilitation requires emphasis on both variability in conditions and outcome (see reference in Schmidt, 1975), and mastery of specific conditions and outcomes.

The central idea of this technology is that the movement performance at its various levels can be assessed computationally, i.e. it can be computed, and then further diagnosed to identify deficiencies at the various levels of the movement hierarchy, which are needed to determine training goals. The training goals can then be pursued through targeted training activities that can be augmented by various feedback modalities. The following provides a technical description of the capabilities needed to support comprehensive data-driven skill assessment, and diagnostic and training intervention for open motor skills. It introduces definitions of the relevant quantities and processes that will be formalized subsequently.

The section starts with the definition of relevant concepts used to describe and quantify the skill acquisition process; its assessments; the diagnosis and specification of training goals; planning training; and finally, augmentations that can be used to enhance training interventions. All of these concepts and capabilities are described in general terms. They will be further developed in the system's description and the process flow description.

Movement Pattern Classes and Outcomes

As already discussed, the fundamental element of movement behavior are the set of movement patterns that support the relevant interactions with the environment and task elements. These are also called primary movement units or skill elements. Most movement patterns are directed at producing an outcome or action toward the activity or task goal(s). The various movement patterns used by a subject in a task can be identified and classified.

As can be appreciated from this description, the quality of the skill assessment depends on the ability to extract relevant movement patterns that characterize relevant interactions in the task, and to classify these patterns according to their intrinsic characteristics, i.e., the movement technique and movement phases, and their relevance to the task, i.e., the movement outcomes and the task conditions. This is in particular critical for open motor skills, since the subject acquires a repertoire of movement patterns to produce a broad range of outcomes under a range of conditions. To ultimately provide feedback to help improve a subject's skills, performance can be contextualized, which may include identifying what movement technique is used under which conditions and to produce what outcome.

FIG. 11 illustrates the acquisition and evolution of movement patterns over time, highlighting the formation of movement patterns either from scratch or through a process of differentiation. At a different time in an individual's practice, training or performance history (shown as stages S0, S1, . . . ) the movement skills can be described as a repertoire of movement patterns (e.g., at S2 patterns P1-A, P1-B, P2-A, P2-B).

The width of the branches in FIG. 11 indicate the variability in movement characteristics in a given pattern. Beginner subjects tend to employ similar techniques to achieve a range of outcomes and conditions. With experience, subjects learn to perfect their control over the task conditions and can develop movement techniques that are more specialized and yield higher performance (more efficient, higher outcomes, more extreme conditions). Therefore the general trend is for a subject to start with a repertoire of a few movement patterns with fewer capabilities, and with experience and training develop a larger repertoire of more differentiated movement patterns.

A new pattern can form through differentiation of an existing pattern (i.e., core pattern), shown here as a dashed line that indicates the beginning of the differentiation process (e.g., differentiation of P1 into P1-A and P1-B at S1). Alternatively, the pattern can form “de novo” such as illustrated for P3 at S3 in FIG. 11. Newly differentiated patterns next go through a consolidation stage (shows as the bifurcation point at the end of the dashed line, e.g., P1-A and P1-B at S2) where they each become distinct patterns. After consolidation, patterns undergo a process of optimization, as shown by a tapering of each branch into a tighter distribution of patterns.

FIG. 12 shows several classes of movement patterns as clusters for some parameterization such as features from the measurement time histories. The clusters capture the pattern differentiation that takes place as the individual improves their skills. The example is based on the patterns at stage S3 in FIG. 11. The patterns that form following differentiation typically appear as a mixture of two patterns, such as shown for P1-A1 and P1-B2 in the original pattern P1-B. Patterns in an early stage of consolidation show distinct features such as P2-A and P2-B.

FIG. 13 shows the family tree highlighting the evolutionary relationship between movement patterns. Since some patterns form through differentiation, it is possible to track the based on features or attributes that are inherited. In FIG. 13, core pattern to refer to the pattern that inherits the main attributes in the development of the new patterns. The non-core patterns differentiate to create new attributes that are distinct from the core pattern.

Movement pattern classification is typically based on movement profile features (e.g., racket angular rate or acceleration). Movement outcomes are a consequence of the movement performance and conditions, and therefore a function of the movement characteristics (see FIGS. 3A and 3B). Consequently, some movement profile features can be used to predict or estimate the movement outcome. Viewed abstractly, therefore, the classification task corresponds to identifying the structure of the extended state-space X in FIG. 14. The state-space associated with the entire human or system performance combines the typical states of the systems, such as needed to describe the subject's or agent's movement, as well as states that are associated with the task and environment elements that participate in defining the conditions in which a particular movement performance or pattern takes place. Classification therefore can be conceived as the mapping from the extended state space into its co-domain V.

FIG. 14 shows the mapping between the movement performance state (state-space X) and the movement outcomes and other attributes f_i in V. The state space highlights the partitions associated with various movement pattern classes. Movement patterns are typically associated with features that originate from domain characteristics (such as geometrical characteristics of the manifold associated with task dynamics, interactions and various constraints). The classification maps the state-space features to the movement attribute space. Movement attributes include results or outcomes (e.g., spin, pace, etc.), as well as other attributes that can be used to assess the movement technique (consistency, timing, smoothness, etc.), or performance (energy, etc.). These attributes can be computed via analytical functions, estimated statistically, generated using neural networks or even directly measured (e.g., ball spin using computer vision). Each pattern has a range of values for the particular outcome metric shown as a partition.

Since the outcome is usually what the performer is trying to achieve or control, and is often what they are most conscious and deliberate about, it is helpful to depict the movement pattern classification relative to the outcomes. For example, for tennis strokes, a “stroke map” can be used to depict the different stroke classes (forehand, backhand) as a function of the outcome: the spin level (slice, flat, top spin) and pace (low, med, high) imparted on the ball.

This example is illustrated in FIG. 15, where the dimension O1 could represent stroke intensity and dimension O2 spin imparted on the ball. Repertoire of movement patterns depicted relative to the primary outcome dimensions (O1 and O2). The figure illustrates the relationship between movement patterns and outcomes described by the mapping f: X->V of FIG. 14.

The classes of strokes can be divided into subclasses. To be intuitive, these subclasses have to represent different regimes or conditions. FIG. 16 illustrates the relationship between the movement patterns and their outcomes quantized based on ranges defined by O₁₁, O₁₂, O₁₃ for dimension O₁, and O₂₁, O₂₂, O₂₃ for dimension O₂. Such relationships can be determined by embedding V into a subspace W that produces meaningful outcome categories (semantic interpretation), as illustrated in FIG. 14.

Furthermore, since the movement patterns and outcomes also depend on task conditions, movement pattern classes can be represented as a combination of outcomes and conditions. The performer has to compensate for effects of conditions or even exploit these conditions to their advantage in order to produce the desired outcome. For example, in tennis, the ball comes into the court with varying amounts of pace and spin. FIG. 9 shows three interception types that are characterized by the impact conditions.

Therefore, in addition to positioning the body to successfully intercept the ball, the player has to adjust the stroke execution to achieve the impact conditions that produce the desired outcome. Typical adjustments of the stroke impact conditions involve choosing the interception point relative to the ball's impact on the ground, such as interception while the ball is on the rise, when it is near or at the apex of the trajectory, or when it is descending toward the ground. The conditions can have dramatic effects on the ability to achieve certain outcomes. For example, intercepting the ball on descent makes it easier to produce top spin (because of the relative angle between the ball velocity vector and the racket face orientation).

More advanced players are generally more conscious about the conditions, since they will try to exploit the conditions to help improve the outcomes, for example in FIG. 9 where backing off from an oncoming ball affords the choice to intercept it on the descent, which is advantageous for producing top spin. The subject can also decide to intercept the ball on the rise or at the apex depending on the desired outcomes at the different levels of the task (e.g., producing a shallow power shot deep in the court and down the line, or clearing the player at the net). Therefore, extended repertoire representation can include conditions as well as outcomes to provide a more complete understanding of the subject's skills, which in turn can be used to determine more complete and precise training interventions.

Pattern Development and Learning Stages

To understand how to create meaningful interventions in the skill development process, it is beneficial to understand the brain's learning process. The acquisition of movement technique proceeds according to relatively distinct stages, which can be defined as follows:

• • Pattern formation represents the first stage of skill acquisition, the so-called cognitive stage. At this stage, the subject forms a model of the movement such as the outline of the movement spatial configuration. The movement at this stage cannot be performed reliably because it relies on conscious guidance and visual feedback, required to ensure that the movement conforms to the model.
  • Pattern consolidation refers to the process of consolidation of the movement pattern from spatial configuration (e.g., based on visual demonstration or verbal description) into sensory-motor patterns that can be executed dynamically without conscious effort. The movement patterns are encoded as motor programs that can be performed in an open loop (e.g., without visual feedback). This corresponds to the acquisition of procedural memory.
  • Pattern optimization refers to the stage where a given movement pattern undergoes further differentiation or refinement (e.g., by fine-tuning technique and perceptual mechanisms), as well as developing physical performance.

The acquisition stages manifest in movement characteristics that are captured by the skill model. Therefore, the acquisition stage can be assessed from statistics associated with the skill attributes.

Movement Patterns Optimization

Note that the acquisition stages assume that skill development takes place around a particular class of movement pattern, and proceeds in successive stages from formation to consolidation to optimization. It is helpful to realize that one particular pattern may not be optimal in an absolute sense, but the optimality of the movement is relative to a subject's particular biological constraints (biomechanical system, physical strength, health status). In this sense they can be considered as local optima. Achieving globally optimal movement patterns in the absolute sense requires building up various components involved in the human movement system, including physical strength, the neurological motor circuits that support response speed and movement coordination, and other functions such as perceptual mechanisms.

The overall capabilities of a subject, for example, manifest in the range of possible movement architectures and their corresponding functional capabilities. Therefore, movement skill acquisition can be assessed and modeled by tracking the evolution of the movement architecture, i.e., the sequence of movement phases that makes up each movement pattern.

Movement patterns may go through several generations of acquisition, each generation characterized by a specific movement architecture and its associated functional characteristics (see evolutionary process in FIG. 11). Within each generation, movement patterns may progress through the stages of formation, consolidation, and optimization. Newly acquired physical strength, or other changes in constraints, can also prompt a new iteration in movement architecture that will typically have to go through the formation, consolidation, and optimizations stages.

When a particular movement pattern reaches the optimization stage, limitations resulting from inefficiencies and other factors will typically become apparent. Once the potential improvement within the same pattern has been fully exploited, often the only way to further improve the performance and outcome is to form a new pattern. Therefore, it can be helpful to distinguish between the training skills within the same generation of a pattern class, and the training of a new pattern, or evolution of a pattern class into a new generation (see movement architecture in FIG. 5). In some cases, a new pattern generation emerges naturally from the optimization. When a new pattern is formed, it will typically lead to a momentary decrease in performance and consistency until it consolidates and is eventually optimized.

This staggered acquisition process allows individuals to perform optimally at the task level with their “suboptimal” architecture. The skill development is interrelated with the body's physical development. For example, a new movement architecture may require physical strength and coordination that is not sustainable. Therefore, some changes in movement technique can require development of physical strength.

One factor that drives the evolution of the movement pattern architecture is the opportunity to make movement more efficient. Efficiency is determined by how well a performer is able to use his or her biomechanics while protecting the body from wear and injury. Typically, evolution of the movement pattern architecture follows a development that proceeds from proximal to distal body segments. Therefore, the architecture usually evolves to involve the superposition of an increasing number of body segment motions.

In tennis, for example, the early generation of stroke pattern is characterized by a simple backswing and forward swing 441 (see FIG. 4). The pattern is then refined as a performer learns to exploit the multiple degrees of freedom afforded by their body (legs, hip, torso, shoulders, elbow, wrist). The overall pattern composed of multiple movement phases can be represented by a finite-state machine (FIG. 5). For example, in tennis a typical evolution in the stroke is starting from relatively simple, lower-dimensional motions that exploit the basic biomechanical capabilities, for example a basic backswing and forward swing phases (see e.g., 4-state system in FIG. 5), to learning to exploit and coordinate the larger degrees of freedom, for example using a more elaborate backswing, with a back loop that transitions optimally into the forward swing phase (e.g., 8-state system in FIG. 5).

This process eventually extends into the entire available body kinematic system. With training, subjects learn to exploit the whole body kinetic chains, which involves movement that originates at the feet, hips, torso, etc. Such movements are complex in the sense of the spatio-temporal characteristics of multiple joints and muscle groups. They also require more anticipation and therefore rely on advanced perceptual skills and planning. Given these levels of complexity, it is understandable why the complex movement skills develop in stages.

The process of movement pattern refinement simultaneously exposes the body to new and larger displacements with the potential to create undesirable stresses on the joints, ligaments, tendons, and muscles. Therefore, it is possible to conceive acquisition of more advanced movement patterns as a process that's geared at maximizing outcomes while minimizing strain and more generally injury risks. Increased loads are also drivers for the development of physical strength as well as musculoskeletal structure.

The process of movement pattern refinement simultaneously exposes the body to new and larger displacements with the potential to create undesirable stresses on the joints, ligaments, tendons, and muscles. Therefore, it is possible to conceive acquisition of more advanced movement patterns as a process that's geared at maximizing outcomes while minimizing strain and more generally injury risks. Increased loads are also drivers for the development of physical strength as well as musculoskeletal structure.

Repertoire Development and Pattern Differentiation

FIG. 13. illustrates the evolutionary relationship between movement patterns. Each movement pattern is identified in terms of its ancestor(s) or parent(s) (shown in bold). The patterns shown correspond to the ones in FIG. 11, ordered by which stage it was formed along the evolutionary process (S1-S5).

As shown in FIG. 13, each movement also can be assigned a degree of significance to the task that specifies how relevant the pattern is to the task performance and goals, and is indicated as primary, secondary, tertiary, etc. As described earlier, movement patterns can either be formed de novo, or through differentiation from an existing pattern. In the former case, the new pattern typically fills a new need for the task performance, such as a volley. In the latter case, new patterns typically form to expand the range of outcomes or conditions. For example, in tennis, a generic forehand stroke can evolve into several subclasses to achieve specific ball spin and pace such as to better control the outcome of the stroke (see FIGS. 11 and 12).

At the onset of skill learning in a new activity domain, a subject typically starts with some rudimentary movement capabilities. These early movements are typically adapted from the repertoire they have acquired in other activity domains, or by combining general movement primitives that are available from their neuro-motor repertoire. At the beginning (S0 in FIG. 11), consider the two movement patterns P1 and P2. For example, these could represent a forehand and backhand stroke. At this very early stage, the movements are not yet specialized. Beginners typically employ a few movement patterns they try to adapt over a broad range of outcomes and conditions. For example, in tennis, beginners may have one forehand and one backhand stroke pattern to accommodate a broad range of conditions such as return balls from an opponent under a variety of conditions (e.g., pace, spin, impact point, interception height, etc.).

Since beginner movement patterns have to accommodate broad conditions, they cannot exploit the subject's movement capabilities in an optimal fashion, i.e., using the same general movement pattern for a range of conditions compromises its performance. Therefore, to achieve optimal performance over a range of conditions, multiple specialized movement patterns have to be formed. These are optimized both for the perceptual conditions, and the biomechanical movement conditions needed to support the range of outcome.

With more experience, the subject learns to exploit his or her biomechanics and to identify conditions in which movement patterns can be specialized to produce more reliable outcomes. For example, in tennis, a player may learn to generate top-spin on a return, enabling a more aggressive return stroke with increased pace that requires tighter control over timing and conditions, or to return with a slice, which tolerates a broader striking area.

With more extensive experience, the player also learns to link the stroke patterns with the larger task hierarchy, in particular they focus on improving the task performance, i.e., producing outcomes at the task level. In tennis this includes the precise placement of shots on the court; at the same time, also broadening the regions that can be targeted, while also learning to target these regions from a range of impact location and conditions. The development of the repertoire at the task performance level can be assessed from the discretization of the court environment shown in FIG. 8.

Movement specialization or differentiation at the pattern level is illustrated in FIG. 11 at time S1, where the P1 patterns start its differentiation into two distinct patterns P1-A and P1-B. At the early stage of this differentiation process, the movements still have overlap in their characteristics, as shown as mixture in FIG. 12 for P2-B. Therefore, there will be variability in the technique and unreliability in the performance.

Eventually, as shown at S2 in FIG. 11, the two patterns begin to be sufficiently differentiated to represent distinct movements in terms of their technique. As described elsewhere, the movement technique is formed by sequencing movement phases that build on muscle synergies. The development of movement technique, therefore, also relies on the development of physical strength along with motor coordination.

As the different functions supporting movement are formed, a subject can begin optimizing their movement. At S3, following the same process as for P1-A, P1-B differentiates into more specialized patterns. Patterns can be further differentiated as a result of ongoing refinement or optimization of the technique. For example, S4 shows the optimization of pattern P1-A. Optimization requires narrowing down on operating conditions and technique; therefore, the patterns begin to have more restricted domains of operation which leads to two new sub-patterns P1-A1 and P1-A2.

As a result of the learning process with different learning stages (formation, consolidation, and eventually optimization), subjects operating in an open skill domain expand their repertoire, and, at any given time, subjects will have movements that are at different stages of development. Even a relatively proficient player in a sport may be required to form a new movement pattern, or change an existing pattern to such a degree that it loses much of its relationships with the original pattern.

To help describe the various phenomena in skill acquisition process and computational processes in its analysis, it is useful to define different time periods. The following terminology are used:

• • Epoch refers to time periods associated with a data set that is associated with a particular model (see assessment loop described later).
  • Learning/acquisition stage refers to time periods associated with transitions in a subject's neurological learning process for a particular movement pattern (formation, consolidation, and optimization).
  • Developmental stage refers to time periods associated with evolutionary milestones in the development of a player's larger movement pattern repertoire.
  • Generations refer to time periods associated with differentiation in a player's overall skill profile (e.g., as it relates to other player subgroups), based on the aggregate contribution of skill, technique, etc. This information can be captured by the player subgroups through population analysis described later.

Modeling Movement Pattern Development

As a movement pattern evolves, it can differentiate, and/or new patterns can be formed from scratch. Therefore, several patterns can coexist in the same class, i.e., supporting the same outcome and task interactions (see FIG. 11). Sometimes, the classes are differentiated by the conditions, i.e., they represent the same outcomes but under different conditions. Typically, these patterns evolve sufficiently to give rise to distinct classes for example specialized in a specific range of outcome and conditions.

Therefore, when processing and analyzing movement skills as a process of movement pattern evolution and development, inheritance relationship between patterns can be considered. In the following we define a core pattern (CP) as the primary pattern descending from an ancestor, in contrast to new patterns that emerge through differentiation. In FIG. 13 the core patterns are shown by a solid edge to underscore that they inherit the main attributes in the development of the new patterns. The non-core patterns, linked by dashed edges in FIG. 13 differentiate to create new attributes that are distinct from the core pattern.

The core pattern often corresponds to the predominant technique in that class, for example that is further consolidated in procedural memory. Under challenging conditions, the subject may tend to fall back on that pattern. The core pattern may also be more difficult to change because of its long-standing history.

This conceptualization of movement learning as an evolutionary process combining the development of new patterns through differentiation, as well as formation of patterns de novo, is useful to assess the longitudinal skill acquisition process. This involves relating the patterns through features that are inherited as they differentiate and tracking the stage of learning of the patterns through the differentiation process. The hierarchical classification of patterns can determine the hierarchical relationships between the classes. These structural characteristics can be exploited to design training interventions, and plan and manage the training process. For example, interventions that help new patterns form through differentiation and consolidation.

Movement Planning and Perceptual Mechanisms

For open motor skills, successful performance depends on extracting various forms of information from the task environment and elements. Many actions and movements need to be synchronized with the task environment and elements. Learning movements also involves learning the perceptual mechanisms used to extract relevant information and using this information to plan or adapt behavior.

Proficient movement technique and overall task proficiency rely on the formation and optimization of perceptual mechanisms, for example the ability to recognize the state of an incoming ball and adjust the stroke to these conditions. For example: a fast, high bouncing ball is returned with a slice, enabling a more reliable but less offensive return. In addition, if a player can extract early cues to estimate the return location (e.g., from an opponent's body and racket swing) they can control the point by selecting the return and positioning the body to precisely intercept the ball in the strike zone to achieve the desired return trajectory.

A broad movement repertoire allows a subject to select the best actions needed to control the state of the activity based on the task state and conditions. For example, a tennis player may take advantage of a slower, shorter return to intercept the ball earlier and produce large top-spin and pace as a way to surprise the adversary with a deep return in the open side of the court. Alternatively, under an offensive return from the adversary, the player has less time to prepare a stroke and uses a slice to gain time before the adversary's return. These changes reflect the subject's ability to assess the situation and use this information to control the task and achieve its goal, while at the same time adapting to the environment and conditions.

Assessment of Open-Motor Skills

Most open motor tasks involve dynamic interactions with the environment, combining different outcomes, at different levels of the motor system hierarchy, levels of the information processing hierarchy, and task structure hierarchy. Proficient performers are able to combine these processes and components into an organized whole. Therefore, skill assessment of open motor skill has to encompass these different levels and components, which presents some unique challenges both from an analytical and data acquisition, or practical standpoint.

Tennis is a good example where every stroke is executed at conditions that depend both on the player's control over their position relative to the moving ball and the stroke technique. Therefore, the assessment of skills encompasses different aspects of the performance, and is enabled by defining outcomes that can be defined based on how actions influence the task and environment state over multiple levels of the motor system and task structure hierarchy. A useful to way to formalize this analysis is to investigate the various interactions between elements of the human system, the participants, equipment, and environment and task elements.

As an example, based on the tennis use case, FIG. 7 shows relevant interactions in the larger system and illustrates the following outcome levels, which are also shown in FIG. 2:

• • 1. Stroke/racket ball impact: impact conditions.
  • 2. Impact and shot primary outcome: ball velocity and spin.
  • 3. Shot trajectory and type relative to the environment elements, e.g., net clearance, curvature, velocity, spin.
  • 4. Shot placement overall relative to the opponent and court landmarks.

These levels are defined based on the various interactions between the agent and relevant task and environment elements and form a nested closed-loop system. They underscore the general idea that human behavior is relational, i.e., the behaviors are anchored in specific object relations, which is both a result of how humans perceive and conceptualize the environment (in contrast to machines which are often based on discretization of certain dimensions).

Note that the outcomes at levels 2)-4) are all function of the performer's ability to control the ball and manage the impact conditions) (see FIG. 9). Therefore, controlling the ball and the stroke execution conditions depends on the ability to perceive and anticipate the task environment state, move on the court, prepare the stroke, and establish appropriate posture.

Note also that behavior and performance in open-motor tasks depend on the full range of human information processing: the abstracted task level rules and organization, the discrete elements and events associated with individual movement unit selection and execution, and the continuous process of the physical movement performance. It is possible to delineate the primary information processing components corresponding to each outcome level.

These primary information processing components are responsible for acquiring knowledge of the results and associated knowledge of performance that in turn can be used to help improve skills. Therefore, from a skill assessment standpoint, it is critical to understand which outcome levels are processed at each assessment level and, at the same time, provide knowledge of performance that can be converted into an actionable training intervention.

For example, stroke-impact and primary stroke outcome are most directly related to motor control processing (body coordination, and ball interception). The performer can assess these outcomes through proprioception, including how the racket “feels” at impact, and the resulting shot. But the latter does not provide as much information about the movement technique or knowledge of performance.

The shot trajectory and placement are most directly related to the planned shot and game strategy but also depend on the performer's performance and control of the ball and conditions. The performer assesses these by perceiving the ball trajectory relative to the court and the opponent and the impact on the game. Information from this level helps improve the positioning and shot selection and game strategy. However, training at this level relies on sufficient facility to control the ball and achieve sufficiently precise outcomes at levels 1)-3).

In activities that don't have an adversary, such as skiing or surfing, the strategy level is concerned with the negotiating the environment and conditions. This requires reading the terrain and conditions, and planning the deployment of a sequence of movement patterns.

The dynamic coupling and the multiple levels of information processing make it very challenging to assess and produce effective training interventions. The stroke and ball impact conditions manifest directly in the stroke-impact quality 1), and primary outcome 2), making this process the most directly observable; however, it also depends on the ability to predict the ball trajectory and anticipate and select the interception condition. The other outcomes 2)-4), on the other hand, accumulate other factors, making the diagnostic task difficult. With technology, it is possible to tear these confounded contributions apart, thereby delivering analysis and creating training interventions at the appropriate outcome levels and the appropriate information processing level.

The following describes the framework that was conceived to make improved data-driven assessment and training possible.

Assessment and Diagnostics

Training relies on the ability to 1) assess motor skills, which corresponds to the description of the movement outcomes and characteristics in relationship to task requirements, and to 2) diagnose skill, which corresponds to the identification of specific aspects of the movement technique that are deficient and reduce performance in the task through their effect on critical outcomes (diagnostics). The gained knowledge can subsequently be used to determine adequate interventions that address the specific skill deficiencies and lead to a higher skill level and hence task proficiency.

Skill assessment is responsible for characterizing the movement performance. Producing an assessment is essentially the challenge of defining metrics and features from collected movement data that provide a concise and useful description of a subject's performance outcome and technique (knowledge of result and performance). For example: “The ball spin produced by the impact is too low for the forehand top-spin high-strength (FHTSH) class.”

Skill diagnostic is responsible for identifying the causes of movement and task performance characteristics. It typically focuses on the deficiencies that need to be addressed or corrected to improve the skills towards a task performance. For the previous example: “Racket height at forward stroke initiation is too high and racket roll rate profile is too shallow.”

Assessment Levels

Building on the components of the movement and skill models in U.S. Patent Application Publication No. 2017/0061817 (illustrated in FIG. 6), skill assessment proceeds hierarchically, accounting for movement performance at the different organizational levels of the human movement system which also couple with the task structure hierarchy. FIG. 10 illustrates the relationship between the levels of the movement hierarchy and the task hierarchy. It defines the following levels of assessment:

• • Physical performance level: The assessment at this level focuses on the physical details of how the movement is produced. This level is best analyzed at the level of movement phase segments, including considerations such as the movement phases and relationship with muscle synergies, the musculoskeletal constraints, and sensory and perceptual processes used to execute and deploy the movement in respective task conditions.
  • Pattern performance level: The assessment at this level focuses on how well the movement pattern associated with the primary movement unit support the task and environment interactions, and more specifically produce outcomes that contribute to the task goals and adapt or take advantage of conditions. This level is best analyzed through the movement pattern and outcomes, for example in tennis the stroke and shot relative to the court, as well as the oncoming shot and conditions (see FIG. 9).
  • Task performance level: The assessment at this level focuses on the relationship between the acquired skill elements and the task requirements. This level of assessment is best analyzed through the repertoire. It includes considerations such as what types of patterns have been acquired to support critical task interactions such as producing the range of outcomes and adapting to conditions, and how these outcomes and interactions collectively contribute to the task or activity performance. In analogy to robotics or trajectory planning, this level corresponds to the assessment of the discretization of the task space, i.e., how the overall range of outcomes and conditions are quantized into distinct patterns that collectively provide the skill elements to perform the task proficiently.
  • Competitive performance level: The assessment at this level focuses on how the subject uses their acquired skill elements in a task, while considering the subject's strategy and more generally how they compare with other performers. It is best analyzed at the level of repertoire but taking into account how the movement patterns and capabilities are utilized to support and enable competitive performance. The assessment encompasses strategic characteristics that may, for example, be used to outperform an adversary, both in a static way as well as dynamic, which corresponds to modeling the temporal relationships between movement patterns and events in the task and environment, as well as participants.

Accounting for the hierarchical relationships between the levels in the assessments makes it possible to leverage these relationships in the design of training interventions.

Assessment Components

Assessment components refers to the different perspectives that can be taken on the movement performance and skills and follow from the assessment level analysis that was just discussed and is summarized in FIG. 10. The following components can be considered:

• • Outcome characteristics: The outcome assessment corresponds to the traditional knowledge of result and performance. The outcomes capture specific qualities of the movement pattern, their effects on the task environment, and the associated conditions in which they are executed. Outcomes are defined and analyzed at different levels of the movement system such as the different outcome levels defined in FIGS. 7 and 8.
  • Functional characteristics: The assessment focuses on the underlying mechanisms of the movement pattern classes and their effect on the task. The functional analysis is usually connected to the various outcome quantities and the range of conditions required for a task. For example, functional analysis at the pattern level considers how the movement phases combine to produce the movement pattern that support the interaction with the task and environment level, and produce the primary outcome for the task. Functional analysis also encompasses the perceptual mechanisms, for example those used to support synchronization with the environment and task elements. At the physical performance level, the functional characteristics can encompass the details of biomechanics and muscle activation (muscle synergy).
  • Perceptual characteristics: This assessment highlights the quantities that can drive the subject's behavior across the different assessment levels. For example, at the physical performance level, perceptual quantities correspond to the proprioceptive features of the movement phases that are critical to the execution of a particular movement pattern. Perceptual mechanisms are part of the functional characteristics, they are separated as a component to emphasize their potential role as part of cueing for example.
  • Memory and learning characteristics: Movement characteristics and skill level depend on the movement's acquisition stage, which refers to specific milestones associated with the brain's learning process. This assessment focuses on the identification of the learning stage of a movement pattern, which can help better select diagnostic tools and training interventions, such as cueing to reinforce sensory-motor patterns, or visualizations that can help form mental models.

FIG. 10 illustrates the different levels of assessment highlighting the representative elements 280 of the model at each level for the tennis example. The figure summarizes the assessment and diagnostic components 290 that are applied across the different levels. The illustration also conveys how the different levels are nested into one another, going from the movement segments at the bottom, which are used to form the stroke patterns; then, how these patterns enable the shot interaction with the court environment; next, how the different strokes and shots collectively discretize the task space; and finally, the decision making and strategy driving the task competitive performance.

FIG. 31 provides a different perspective with a description of the: a) the levels of assessment, b) the central elements that describe that level, c) the criteria and quantities that can be used to determine the skill characteristics at that level, d) the analysis or diagnostics to identify the critical characteristics, and finally, e) the drivers and mechanisms used to produce training interventions.

Assessment of Outcomes

The outcomes represent the primary result of the movement, as viewed from the perspective of the task. As already discussed and described in FIG. 10, outcomes can be defined at different levels of the movement system hierarchy and task structure hierarchy. Outcomes are quantities that provide relevant information for the task performance and skill assessment. They are typically designated based on the task requirements and available measurements.

One type of outcome is success rates. Success and success rate can be determined at different outcome levels (see FIGS. 2 and 7). For example, in tennis the success at the racket-ball interaction level (Outcome 1) is determined by the racket impact location and outcome level for the particular class such as spin and pace. At the court interaction level (Outcome 3) it is determined by the court impact location and state (see FIG. 8).

Every stroke class is characterized by the range of values that characterize the functional model, which includes states at phase transitions such as the racket states at the beginning of the forward swing, or the racket orientation, the racket angular rate, etc. at impact, etc. These characteristics can be used to determine outcomes at different levels, including the ball spin and pace, but also the shot trajectory. With the additional information about the player position and orientation, it is also possible to predict and estimate the ball impact location on the court.

An example of this in tennis includes a comprehensive motion capture system that measures the task object (tennis ball) relative to the task space in addition to the subject's body segments, body pose, motion of the equipment, etc. It is possible to more directly assess the outcome of the subject's motion. In addition to the quality of the outcome, another attribute is the success rate of movements for each specific movement class.

FIG. 8 shows respective shot placements based on ground impact distributions for a player and an opponent. The skills at the shot level manifest as different resolutions and precision in interactions with the task environment. The task level performance, which will be described next, while depending on the stroke, puts more emphasis on the shot outcome level such as how the stroke used by the player can control the ball relative to the court and opponent (see FIG. 7).

The collection of movement classes in the repertoire and the information extracted from the movements in the repertoire—including the outcome, success rate, and other metrics—form a subject's skill profile. The skill profile represents a holistic description of the subject's skill, which can be used to compare players as well as track how skill evolves over time.

The movement class technique assessment looks at the overall characteristics of the movement pattern. As described earlier, each movement pattern can be described by a so-called core pattern (CP). The idea is that the movement follows a motor program which has a template with specific variability due to disturbances and adjustments made to adapt to conditions. The CP therefore describes the nominal movement performance.

Deviation from a CP can therefore be used to assess technique and other attributes such as adaptability. Even under perturbation, movement pattern should be distributed around the nominal range of CP, i.e., normal range of variations. Movements that exceed the normal range can represent poor execution or may also be of a secondary pattern that may be due to differentiation of the core pattern as part of the normal skill learning.

It is expected that as an individual's skills improve, the range of variations of the CP decrease. This is in part due to the specialization and optimization of the pattern as well as the tighter compensation over effects of conditions.

Movement differentiation can be detected from the presence of a secondary pattern grouping, distinct from the CP, within a hierarchical movement pattern class. This type of differentiation is especially likely in the early stages of skill acquisition, when new patterns are derived from existing patterns.

Functional Assessment

Functional assessment was described in detail in U.S. Patent Application Publication No. 2017/0061817. In the following, it is extended to the different assessment levels and task hierarchy. FIG. 2 illustrates an interaction between a stroke motion and the task and environment elements, including the ball trajectory relative to the court, the impact of the ball, and its bouncing before the interception with the racket trajectory. The figure also illustrates the gaze of the player along different points of the ball trajectory and environment elements, and shows a ball machine as an apparatus that can be programmed to enable different forms of interactions.

FIG. 2 also illustrates details associated with the functional characteristics of the stroke pattern and the interaction with the environment to produce a desired outcome (e.g., Outcomes 1-3). The interactions include for example adapting to conditions, such as the timing of the movement phases relative to the ball state following ground impact 32 (see also FIG. 9).

FIG. 2 also shows examples of visual cues that are used to control the movement execution, such as the ball trajectory curvature, the magnitude and angle of the bounce or impact 32. The figure depicts the visual attention based on the gaze vector 81 to some of these cues, as well as the elements that are relevant for the Outcomes 1-3 indicated by labels 33-35.

Delving deeper, FIGS. 3A and 3B illustrate the movement as a sequence of phases and highlights phase transition characteristics, and phase profile characteristics. The phase profile characteristics refer to the dynamics during the phase segment. These characteristics are associated with the coordination of the movement segment and muscle synergies. The figure also shows the feasible envelope that results from musculoskeletal and other constraints, an admissible envelope that represents movements that produce acceptable outcomes but are suboptimal, and an optimal envelope that represents the range of motions that produce the best outcomes with the best use of the biological system.

The figure also introduces the concept of a goal phase, which represents the phase associated with the primary interaction relevant to the production of an outcome and environment and task element interactions. The goal phase in tennis is the movement segment that corresponds to the ball impact and extends throughout the ball interaction or contact. This phase is critical in the production of the outcome. Some details illustrating the functional analysis for the forward swing phase is discussed in a later section and illustrated in FIG. 37. The other phases (initiation phase, phase 1, phase 2, follow-on phase, and recovery phase) represent a sample of phases that can be used in a movement pattern such as the tennis stroke.

With the designation of a primary outcome phase, it is possible to conceive of the rest of the movement as the system organizes around that goal phase to support the outcome. The different segments play different roles in supporting the production of an outcome, as well as supporting the adaptation to conditions and interactions with the environment that may contribute to a robust and versatile performance.

From this more general perspective, every phase could have its own outcomes and interactions. In tennis for example, the forward swing phase (phase 2) is the phase that is the next critical one after the impact because the conditions achieved in the goal phase (impact) are determined by that preceding phase. Furthermore, in the case of the tennis stroke, the forward swing phase lasts about 100 ms and therefore is too fast for the player to make any corrections. Therefore, the forward swing phase is largely determined by its initial conditions x(t=t₀₂), which in turn are determined by the back-loop phase (phase 1). Similar general characteristics can be found in other movement activities.

Transition characteristics are determined by the movement configuration, including the state of the body segments and end effector such as the racket. These conditions also include timing characteristics, such as the synchronization with the environment elements. For example, in tennis, a relevant timing is the synchronization between the tennis stroke initiation phase and the tennis ball state, which itself can be delineated into different phases, such as the net crossing, ground impact, and various phases before the ball impact (see conditions in FIG. 9). Next, the timing of the forward swing phase initiation (phase 2) is determined similarly by ball state and the anticipated impact conditions but closer to the impact time. This synchronization and modulation of the movement phases are instrumental in achieving an accurate interception of the ball and producing the desired impact conditions (target phase) that will lead to a successful outcome. Note that similar considerations can be made about the rest of the body segments and configuration.

The skill element therefore can be defined formally in terms of these primary interactions and the skill characteristics and determined from the various attributes of these interactions, including: the movement functional characteristics (described by the movement phase characteristics and perceptual and motor interactions), the musculoskeletal characteristics, physical performance, and different levels of the task and motor system hierarchy.

Assessment of Learning Stages

The assessment of movement technique (knowledge of performance) to determine what and how to train, ideally requires taking into account some of the neurological properties of the motor skill acquisition process. Learning stages are defined based on motor learning theory, including memory representations and cognitive strategies (see Rosenbaum 2010).

The following movement acquisition stages can be defined from the three learning states described earlier: movement formation, movement consolidation, and movement refinement/optimization. The movement acquisition stages manifest in movement characteristics and can be described as follows:

Patterns to form (e.g., FIG. 52B, step 322): Patterns are missing from the repertoire or exist in unreliable form. The missing patterns typically are due to the lack of differentiation among existing motion patterns. For example, in tennis, the absence of subclasses in backhand topspin represent gaps in possible operating regimes and possible outcomes such as pace or spin. These gaps in movement repertoire preclude flexible production of outcome and adaptation to conditions and therefore manifest in task performance.

Patterns to consolidate (e.g. FIG. 52B, step 323): Movement phases are not yet sufficiently defined and integrated in the movement pattern to allow reliable execution under dynamic conditions. For example, the muscle synergies associated with the phases are not yet fully automated and their transition are not smooth. These deficiencies manifest as unreliable outcomes, variability in movement pattern, lack of smoothness, inefficient movement performance, and do not have sufficient flexibility to deal with changing conditions. After early formation and differentiation, patterns undergo automatization and refinement in their structure. These changes reflect the brain's learning mechanisms (e.g., procedural memory). The automatization allows repeatability and reliability. The refinement of the pattern structure is guided by the functional requirements, including achieving better outcomes and physical efficiency, as well as effectiveness with the task and environment constraints and conditions.

Patterns to optimize (e.g., FIG. 53B, step 324): Movement patterns do not achieve the outcomes efficiently and do not adapt sufficiently to environment or task conditions. For example, movement phases do not make optimal use of the subject's biomechanics. These deficiencies for example, may result in excessive use of force when seeking an increase in outcome.

Skill acquisition stages also manifest in physical changes, including gaining sufficient strength and endurance to sustain good technique over time.

The acquisition stage is captured by the concept of skill status. For each existing class of movement patterns in the repertoire, it is possible to assign a skill acquisition. The acquisition stage can be determined based on quantitative criteria or metrics. For example:

Missing patterns can be determined by the repertoire completeness, i.e., how well the movements in the repertoire cover the performance requirements associated with the task objectives and environment conditions. Typical pattern analysis tools, such as clustering combined with similarity measure (e.g., dendrogram) can be used to identify new patterns within an existing movement class. The degree of differentiation of a pattern relative to other existing patterns can provide a measure of its development.

• • Patterns to consolidate can be identified by success rates, variability in technique, and outcomes within a given class. At this stage, movements also tend to display particular physical performance characteristics, such as high jerk, lack of smoothness, and timing variability. These patterns can also be identified by inconsistency in movement phase structure, smoothness of phase transitions, as well as unreliable timing of some movement phases (e.g., forward swing acceleration profile). Finally, patterns at this acquisition stage can also be identified from the lack of flexibility in adapting outcomes to the range of conditions and outcomes.
  • Patterns to improve or optimize are already formed, but the movement structure does not utilize the subject's biomechanical potential efficiently, and does not achieve the theoretical range of outcome and level of flexibility helpful to deal with the range of conditions. Patterns to optimize are primarily analyzed from the functional characteristics (feature analysis described elsewhere), which provide a detailed understanding of the relationship between the movement technique and its relationship to outcomes. Frequently also relevant is movement efficiency, i.e., the work required to produce an outcome. One goal of movement optimization is refining movement technique to use the least energy and produce the least strain on the musculoskeletal system.

FIG. 41 provides an example of the acquisition stage assignment for the skill elements in the groundstroke repertoire.

TABLE 1
Qualitative characteristics used to determine the
acquisition stage of a movement class.
	Patterns	Patterns to	Patterns to
	to Form	Consolidate	Optimize
Outcome	Poor results	Variability in	Optimality
		outcome	Flexibility
Technique	Weak structure	Variability in class	Adaptability to
	Poor	Unreliable structure	conditions
	differentiation	(smoothness)	Functional
Performance	N.A.	High Jerk	Efficiency

TABLE 2
Quantitative criteria that can be used to identify the
acquisition stage of a movement class.
		Patterns	Patterns to	Patterns to
		to Form	Consolidate	Optimize
	Primary	Smoothness	Consistency	Outcome level
	Criterion		Timing	Efficiency
				Range of
				conditions
	Secondary	Outcome	Smoothness	Timing
	Criterion		Outcome	Smoothness
				Consistency
	Minimum	SR > 50%	SR > 70%	SR > 85%
	Requirements
	(success rate
	SR)

Population Analysis

Population analysis is valuable to understand the contribution of the broad range of factors intervening in the skill acquisition process. Population analysis can be used to determine player types based on skill levels and a variety of other factors such as body type, health, etc. The player type or profiling makes it possible to generate appropriate reference outcome values by accounting for groups of players with similar technique types and skill levels. The player profiling at the same time enables identification of the player characteristics or attributes, i.e., what skill attributes and other factors such as development stage, are characteristic traits of a particular player group. The player profile information can be used, for example, to determine weights in the composite scores that determine the larger player characteristics.

FIG. 29 illustrates the process of generating population groups based on the performance and skill data from the hierarchical movement model. The information extracted from the population analysis makes it possible to determine performer profiles.

FIG. 30 illustrates assessment across the skill-model hierarchy, incorporating player profile information to generate reference attribute values used to assess the skills at the different levels of the movement system and performance hierarchy. The reference values can be used to provide contextual information to determine what training interventions to pursue.

Physical Performance Assessment

The movement physical implementation describes how each movement is composed from distinct phase segments, where each segment is typically associated with the coordination of a specific set of body segments driven by so-called muscle synergies. The performance criteria at this level includes how the biomechanical system supports the phase segments, for example, which muscles and joints are involved in the motion as a function of segment profile dynamics and phase transitions.

The analysis at the movement phase level is based on identifying the components of motion such as muscle synergies and other musculoskeletal quantities. There are overlaps between the segment level analysis and the functional movement analysis, in particular when it comes to the critical movement phases, such as the forward swing in the tennis stroke.

Pattern Performance Assessment

The individual movement segments combine to form the entire movement pattern. This pattern represents the basic skill element supporting the various task interactions. At that level, the primary movement performance criteria are the movement outcomes relevant to the task performance, as well as how movement adapts to the task conditions. The analysis focuses on identifying features or attributes that explain the relevant qualities of the outcomes and conditions (e.g., using sensitivity analysis). These features provide the quantities that can be manipulated through training interventions to optimize movement technique. One question is to determine the most actionable features or attributes and synthesize feedback cueing or other augmentations such as instructions that can be used to produce effective training intervention. Note that other criteria relevant to movement technique and performance can be considered, such as movement efficiency or injury risks.

Movements involve the spatial and temporal coordination of multiple motion degrees of freedom. Detailed skill models focus on the functional aspects of movement characteristics that support the performance of the movement outcomes, the interaction with the relevant task and environment elements, and the adaptation to conditions.

More detailed assessments at the level of movement technique can be performed by decomposing the patterns into segments. The analysis of the movement's functional characteristics, for example, determines movement efficiency in producing specific outcomes, synchronization with task and environment event, and the ability to compensate for conditions.

An example of a functional skill model for a stroke is the coordination between racket roll and swing rate during the forward swing phase, which describes the technique of the subject that may be important for top spin. The model can be used to identify a subject's “spin envelope,” in a particular movement class (see details the following sections, see also FIG. 33).

Similar models can be derived for other characteristics of the forward swing and other phases of the stroke. For example, the racket motion results from the superposition of several components of body motions including the trunk, the shoulder, the forearm, and the wrist. The ability of the subject to achieve the desired impact conditions and result, as well as compensate for conditions, depends on the proper timing and coordination of the body segments. With sufficient measurements, it is possible to estimate the contribution of the movement components of the different body segments and determine outcome variables that characterize the biomechanical performance based spatial or temporal profiles.

FIG. 9 illustrates the interception and impact conditions and primary stroke and shot outcomes. These conditions affect the outcomes but also represent characteristics of the movement pattern classes, since these patterns are fundamental to the interactions in the task. Therefore, movement classes are typically characterized by the movement technique (stroke type), the stroke outcomes, and the conditions under which the action takes place (interception and impact conditions). Notice that the interception conditions are determined by the players' movement on the court and their ability to anticipate and plan their actions.

Temporal characteristics are also critical to the movement performance (see analysis of ping pong stroke timing in Bootsma 1990). For the tennis example, two timing characteristics are included for the assessment of the forward stroke: the instant of peak racket angular rate relative to the impact, and the time of the forward stroke initiation relative to the impact (see details in subsequent sections, see also FIG. 42).

Task Performance Assessment

Individual movement patterns combine to form a complete repertoire of skill elements that provides an individual subject the range of interactions needed to effectively perform a task. At the level of the task performance, the analysis determines how the movement patterns are deployed in a task and how they collectively contribute to task success.

At the repertoire or task level, the outcomes are related to how movement patterns change the state of the task and adapt to conditions and various contingencies that can arise in a task; for example, producing shot placements that drive the game and adapt to the opponent shots. Therefore, the repertoire of movement patterns describes the movements or actions available to a subject in an activity domain. Every subject acquires their own specific repertoire, encompassing a particular range and quality of movement patterns.

The elementary assessment at the task performance level is based on assessing how complete the repertoire is relative to the task requirements. Task requirements define the outcomes of actions helpful for the task. At this assessment level, skill analysis primarily focuses on identifying gaps in the repertoire. The absence of outcome and associated motion patterns in a relevant task area or condition can be used to identify “unformed patterns.” For example, in tennis, this may manifest as the absence of a high-strength backhand top-spin class. The completeness of a repertoire is determined by the extent to which it achieves a sufficient discretization of the task space. Typically, as a subject's skill level increases, the movement patterns become more precise and therefore enable a more granular discretization of the task environment (see FIG. 8). As the discretization level increases, more optimal levels of task performance can be achieved.

Note also that in some domains, the range of outcomes and actions can depend on the style of play or even the personality of the individual performer.

More advanced task performance analysis and assessment takes a more comprehensive perspective and is achieved by tracking attributes of entire sequence of actions. For example, in tennis, relevant attributes include sequence of strokes, the length of the rallies, what type of strokes are used, how they relate to the other player's actions, including movement on the court, and the overall performance of the activity or task. Statistical analysis of the sequence movement patterns can also be used to provide relevant information about the individual's skills and strategies, such as the frequency distribution of which movement patterns the subject uses in a task over a session, provides a signature of the activity and the subject's strategy.

FIG. 8 also shows the type of quantities, such as player's court motion and positioning, which can be used to model and assess the subject's game strategy, i.e., how the player can position the ball according to the opponent's pattern of play and position. These high-level skills also depend on perception of the court, and anticipation of the opponent's behavior. These dynamic characteristics of how movement patterns are used can be modelled by using techniques for learning temporal relationships and dependencies in the performance data. Popular techniques include Hidden Markov Models (HMM) or recurrent neural networks (RNN).

Competitive Performance Assessment

At the top level, the primary goal is to assess competitive performance, which is typically performed at the level of a population. The criteria therefore represent what can determine a form of population fitness such as actual performer rankings obtained from competitions. These may not always be available; therefore it is also possible to compute rankings based on player skill profiles, which can also take into account population groupings. When available, competitive rankings can be used to calibrate the ranking based on skill profile. The movement skill attributes characteristics include those included in the skill profile, how performers relate in terms of their individual characteristics, and how these contribute to their competitive performance. Analysis of the competitive performance provides information about what aspect of the skill profile (skill element and attributes) can be improved to make someone more competitive in a task.

The skill profile is designed to capture the comprehensive, composite characteristics of the individual's movement skills: account for the performance (repertoire), and how well it serves the task or activity; and the effectiveness of the individual movement patterns in the repertoire (movement technique and physical performance).

This can be accomplished though some composite cost or objective function (see equation for Q(a_i), below). The skill profile can then be used to compare performers. FIG. 17 shows the skill profile as a line graph with the contribution of the different skill components, i.e., movement patterns to the composite score. FIG. 40, which will be discussed later provides an illustration of the skill profile for groundstroke repertoire. Different objective functions can be used to emphasize different aspects of the performance. For example, task performance, efficiency, long-term injury risks. The illustration in FIG. 17 also highlights gaps in the repertoire, and the difference between two subjects (C and A) as a skill profile gap.

The skill profile and composite cost can be analyzed to determine which aspect of an individual's movement behavior, or skill attribute, has the largest impact on the overall performance. Since type of sensitivity information can be used as a guide to determine what training elements to focus on. Here again, the different objective functions that can be used for a skill profile provide a way to look at training from different perspectives (performance, efficiency, injury). They can also be combined in a multi-objective analysis to find the tradeoffs, such as performance vs. injury. In contrast to the analysis at the movement technique level, this analysis provides a more holistic perspective on these questions.

However, the skill profile is a static assessment in the sense that it does not account for the dynamics, i.e., how these skill elements are deployed as a task or activity unfolds, or in response to an adversary. Usage frequency of motion patterns provide simple model to assess strategy. The next level corresponds to the statistics that describe the sequence of patterns such as conditional probabilities of a pattern given the previous pattern or the opponent's pattern. A more complete competitive analysis accounts for the dynamics of the activity performance, i.e., the transition between actions and their associated events, e.g., pattern X is used when return from opponent is of a particular stroke type and ground impact conditions. Such a model takes into account the chain of events in the outcomes, which corresponds to determining a causal model. The dynamics capture the complete task performance or game strategy building on the skill elements and underlying details.

Population Analysis and Reference Values for Assessment

As already discussed, to capture the overall impact of the wide range of factors that play out in someone's skills and performance, and at the same time determine reference values for the various attributes and characteristics, it can be beneficial to take into account the data from a broad population of subjects.

The player or performer attributes provide information to characterize player type. Player groups can be determined by clustering the player attributes as illustrated in FIG. 18. Within groups of performers that share similar characteristics, it is then possible to analyze movement performance and skills across a broader range of conditions and identify subtle variations in technique that influence an individual's level within that group.

FIG. 18 also indicates the relationship between the groups and some skill level such as determined by the skill profile. This information can then be used to determine the player profile (see FIG. 29). FIG. 19 shows the distribution in attributes associated with a score or cost function for an entire population group described by the group distribution highlighting a member (subject A, described by distribution (e₁, e₂)), and the tiers ({low, medium, high, very high}) associated with the composite scoring function for the entire population subgroup (e_1,G, e_2,G).

The statistics from these groups depend on how well performer or subject subgroups sharing similar general movement technique and other common factors can be identified. Population level analysis can account for any possible relevant factors such as body proportions, sizes, health conditions, age, etc. The analysis could even be extended to genotype and thereby provide insights into possible innate differences.

The population analysis enables performing absolute assessments. The values obtained for the various skill attributes relative to a larger group of performers help contextualize a subject's performance. This allows more objective comparison between the skill profiles of groups of players (FIG. 17) and can be used to determine reference values.

The reference values from population analysis can be incorporated in the assessment and diagnostic of the skill elements and extended to the various levels of assessment. For example, the composite score used to capture skill elements can be normalized by reference ranges associated with the attributes for the subject's subgroup.

Information from the population analysis can also be used to rank players or performers, such as through leaderboards, which in turn can provide additional source of incentives for training. The leaderboard also enables the determination of which attributes in the composite and profile cost function differentiate players or performers. This corresponds to the competitive assessment (see FIG. 31). Therefore, this information, for example, describes which skill element and attributes have the largest impact on the ranking, and can be also used to prioritize training.

Finally, the combination of the population analysis makes it possible to find larger patterns in movement technique, performance, and even skill acquisition. One aspect of the assessment based on population data is the profiling of the player or performer. Player profile can be determined to characterize the player's performance or skills relative to the larger population. This profiling can include ranking of a player, e.g., based on different skill profile composites, as well as relating subgroups of performers with different but related movement technique.

System-Level Assessment Considerations

These assessments are combined to provide a holistic assessment using a composite analysis. The following summarizes how the different elements integrate to produce a comprehensive assessment that in turn can be exploited to achieve more effective training interventions.

This section emphasizes the role of system-level thinking and of critical quantities used for the assessment and diagnostics and what characteristics provide the basis for their integration. The system represented by FIG. 31 gives an overview of the holistic understanding required for systematic skill training. FIG. 10 illustrate some of these quantities in the tennis use case.

The vertical arrows going up indicate the bottom-up aggregation of the information and characteristics that participate in the formation of the characteristics at the next level, where additional elements also come into play. For example, at the functional performance level, the movement phases combine into a movement pattern that interacts with a task element to produce an outcome. These characteristics are critical in understanding the learning process, and therefore can be used to determine what movement characteristics have to be developed first (e.g., difficulty rating: basic, intermediate, and high level).

The downward arrows indicate the top-down influence of the higher-level assessment on informing the focus of the lower level assessments. The higher levels can provide top-down information to determine which specific assessments and characteristics drive training. For example, the skill profile characteristics provide understanding of which skill element and attribute have the most effects on the current profile level. Therefore, acting on this element and attribute will produce the most effect on the performance at the profile level. These characteristics are critical for understanding the task performance process, and therefore can be used to determine what movement characteristics are relevant for the task (e.g., core motion patterns, etc.).

Trend in Skill Acquisition

Skill status provides the basis for selection of skill elements that should be exercised during training, in what order these elements are trained, which goals are achievable, and what forms of feedback augmentation are most appropriate for training (see FIG. 22). Therefore, the skill status comprehension describes an individual's skills and can be viewed as the state of the skill acquisition process.

The determination of the acquisition stage also makes it possible to more precisely analyze the progress someone is making in an activity domain, which specific aspects are improving, and which ones are more resistant to change.

The criteria applied for the acquisition stage provides specific information about the skill element that can be used to measure progress toward their improvement.

Given that skill evolves over time based on practice and training activity—and also changes in fitness, health, etc. —the skill status should be continuously evaluated. Continuous skill evaluation makes it possible to adopt training activities that are adapted to the subject's specific skill deficiencies and fitness and health conditions.

Skill acquisition is a process that unfolds over time as a function of exposure to the task. Thus, to determine future training activity, it is also beneficial to be able to analyze the trends of the different skill elements. The trends provide information about the stability or susceptibility of these elements to improve under a given training activity.

Motion patterns or skill elements have varying degrees of stability. Some patterns are deeply solidified in a subject's procedural memory, and therefore will show less variations from session to session; other patterns are more malleable. Furthermore, due to variability in human performance, movement patterns will occasionally achieve superior outcomes and techniques. Therefore, the skill analysis method should be able to capture such changes that are inherent to movement behavior, be able to understand which features are associated with improvements, and finally, have feedback techniques to reinforce these features.

At any given time, it is possible to assess one's current skill status and the trend of the repertoire of skill elements relative to current and past times. Time windowing techniques can be used to highlight skill status and trends at different times or epochs in an individual's training history. Skill trends can be analyzed for different time scales, e.g., within sets, from session to session, etc.

Different time scales capture different aspects of the movement skill process. For example, the longer-term trends (months to years) can measure the physical characteristics associated with movement skills such as strength, effects of wear, injury (both development and recovery). The medium-term trends (weeks to months) can measure the assimilation of the training goals and the consolidation in procedural memory of the refinement and optimization of movement patterns. Short-term trends (days to week) measure the successful assimilation of the formation and consolidation, or optimization, of movement patterns. Micro trends (within sets or sessions) can measure the effectiveness of new instructions and the effectiveness of feedback cues.

FIG. 47 shows a plot displaying the progress along several training goals over a specified time range. The progress in the figure is described as a normalized gap w.r.t. training goal (e.g., improving the top spin or consistency, success rate, etc.). When the current training goal for a training element is attained (shown as a star), the system generates a new goal (shown as a square). The trend plots can be superposed for all the active training goals or a specific subset (e.g., what a subject is currently focusing on). The characteristics can be used to help identify which skill elements to prioritize. For example, more focused effort can be put on aspects that are difficult to improve, or on a training goal that is close to completion to get it done and move to a new training goal.

The trend can also show comprehensive skill elements as described by their associated metrics (outcome, technique, performance). The information from the skill status can be converted to a numerical score or grade to provide a summative assessment of skill and its evolution over time. Furthermore, it is possible to decompose the total score into their respective components, including outcome, technique, and performance.

Training Goals and Planning

One capability for data-driven training is the generation of target values for the different movement skill attributes that can then drive the training process and lead to improvements in the associated aspects of movement (see 204 in FIG. 21). FIG. 30 gives an overview of how target skills are generated across the levels of the hierarchy. Target skills are used to determine training goals that provide actionable drivers for training or rehabilitation.

FIG. 31 provides an overview of the integration of assessment and diagnostics across the levels of the movement system organization. It gives a description of the following: a) levels of assessment, b) the central elements that describe that level, c) criteria and quantities that can be used to determine the skill characteristics at that level, d) analysis or diagnostics applied to identify the critical characteristics to specify the training goals, e) the drivers and mechanisms used to produce training interventions, and f) the feedback modalities that can be used to augment the training intervention.

Training Goals

The skill assessment attributes and metrics and the skill status and trends provide the main elements to support the quantitative, data-driven approach to training. The relevant step to render an assessment actionable is to determine a training goal, and preferably some specifications for the pursuit of that goal. As already discussed, diagnostics are typically performed based on some causal models. In the context of this invention, the causal models are derived from the functional component of the assessment. As described previously, the functional components explain how the outcomes are produced at the different levels of the movement and task structure organization. The specification of training goals is also directly connected to the synthesis and selection of appropriate feedbacks (instructions and real-time cueing).

In the proposed system, a skill element becomes a training element once it gets assigned one or more training goals. Training goals can target any attribute across the movement model hierarchy (see e.g., FIG. 30). Training goals provide a way to direct and drive training activity, as well as basic element needed for the planning as well as the continued assessment and managements of the training process.

FIG. 48 shows the learning curve associated with the data driven training process. The learning curve shows the incremental improvement in some relevant attribute a_i of a skill element e_i over the training activity (sets and sessions). Typically, multiple skill elements can be improved concurrently in one training epoch. The training goals are expressed as a target change in attribute a_i of the skill element e_i. When the training goal is completed (or the underlying parameters such as motion model, skill model, are not valid anymore) new baseline data is generated and the training goal is updated. The figure also illustrates the acceleration of the learning curve provided by an update to models and augmentations, etc. As the model parameters are tracked and incrementally updated, the training goals and associated augmentations drive the learning process to achieve best efficiency.

The training goals are identified based on the assessment and diagnostics, which can include both the various skill and performance attributes as well as the skill status. The various sources of information from the assessment and diagnostics determine the forms of augmentation that are most effective for training the skill element (see FIG. 31).

The skill status (acquisition stage) provides relevant information for specifying general training goals. For example:

The training goal for unformed patterns is directed at helping subjects develop new movement patterns that help produce desired outcomes, taking into account each subject's physical and health status.

The training goal for pattern formation is directed at helping subjects differentiate the existing movement into separate patterns that can each better respond to task requirements (outcomes and conditions). It can take into account the existing pattern landscape in a class, e.g., the core pattern and the newly differentiating patterns to help guide and reinforce the desired attributes. The selection of which patterns to form may also depend on a subject's physical and health status, e.g., patterns that are causing stress or contribute to an injury.

The training goal for pattern consolidation is directed at helping subjects refine movement patterns and create procedural memory to enable automatic and repeatable execution.

The training goal for pattern optimization is directed at helping subjects maximize outcomes, improve efficiency, and improve ability to adapt to conditions.

The development stage also provides information to help select appropriate augmentation forms and determine which movement characteristic to emphasize. The augmentations, in particular real-time feedback or apparatus, allow more effective learning and therefore influence the training goals specification.

As can be appreciated by this description, training goals can be determined based on a functional analysis of a subject's own existing performance. The variability in performance ensures that there is a range of performance level and associated attributes contained in the data. A general approach for the training system is to identify the best performance within the individual's range of data, and then help the subject consolidate or optimize their technique so that they operate at this new level. Incrementally, with new data available from subsequent sessions, this process can be pursued and the subject's performance therefore can be incrementally improved.

This data-driven, analytical approach to formulating training goals ensures that these goals are realistic for a particular individual; however, working off an individual's own data can be limiting. A broader sample of performance and attribute can help form new movement patterns or techniques that are not necessarily available in a subject's own repertoire. This is in particular critical to extend the technique beyond what is currently used by the individual. Population data extends the performance, conditions, and the range of factors that are known to contribute to skills.

With sufficient data from a population of performers and data encompassing various other relevant factors (such as body type, physical fitness, health, or age), this framework also makes it possible to predict the time that may be needed for achieving the goals, taking into account the particular feedback augmentations.

Specification of Training Goals

The specification of more targeted training goals can be based on skill analysis of the attributes across the hierarchical model as shown in FIGS. 30 and 31. FIG. 30, for example, illustrates example assessment, diagnostics, and training goals across the skill-model hierarchy, incorporating player profile information to generate reference values for the attributes used to assess the skills at each level of the movement system and performance hierarchy.

Training goals take different forms depending on the level in the hierarchy (see assessment levels in FIG. 10). For example at the physical level, the training goal synthesized for the improvement of an outcome can be encoded as a change in features of movement technique that has been shown to produce improvement in a specific outcome.

At the pattern performance level, the training goal could consist of improving movement technique in the deployment of the stroke such as producing more precise court shot placement. The training goal is specified in terms of skill attributes that have been shown to produce improvement of shot level outcomes, such as timing (FIG. 42).

The target values for the quantitative specification of training goals can be determined from the statistical analysis. For example, for the optimization of movement technique, the training goals to improve outcomes can be determined from functional feature analysis at the movement physical level. For example, see FIG. 37, which illustrates key features for the forward swing phase along with some example stroke phase profiles. FIG. 20 shows a model of the statistical distribution for two technique features, which can be used to analyze the forward swing phase. For example, the features could be the angle of attack or phase length shown in FIG. 37, and the outcome could be the topspin imparted on the ball. The level lines in FIG. 20 can be computed based on percentile ranking from the individual's data.

A similar analysis can be conducted at higher levels, such as taking into account any skill attribute that is relevant for the task or activity performance. The training goal can be set to achieve the next performance tier, or a fraction of the existing variation in performance (see the ellipsoid e₁, e₂) in FIG. 19, which illustrates the relationship between attributes distribution and some measure of performance that is shown as level lines in the context of a larger population or some selected subgroup based on player profile information. The level lines in this case can be computed based on percentile ranking from population data.

The specification of training goals at higher levels follows the characteristics from the movement system's and task structure's hierarchical organization. As already discussed, these types of training goals are derived from diagnostics, which are typically performed using some causal models. The features and attributes, and therefore the form and encoding used to specify training goals, depend on the level of the movement and task hierarchy.

FIG. 43 gives an overview of integrated perspective on the system's main components based on the tennis use case, organized in terms of the levels of assessment (physical 510, pattern 520, task 530, and competitive 540), how the criteria can be expressed with cost functions (512, 522, 532, and 542), and how these elements relate across the different levels. Together with FIG. 10, they highlight some of the key elements and quantities that can be used to drive the diagnostic and ultimately the training process. In particular see the assessment criteria at each level and the diagnostic components shown in FIG. 10.

In this example, at the task level 530, the diagnostics are concerned about how movement patterns are deployed across a larger task environment (see FIGS. 8 and 10). As shown in FIG. 10, the functional model at the task performance level can be formulated to describe conditions that can be exploited to produce the desired outcomes, including proper positioning on the court to control the impact conditions (shown in FIG. 9). The training goals at that level therefore can be specified based on deficiencies in these functional characteristics.

The diagnostics and training goal specifications can also include perceptual aspects, such as extracting the cues from the environment and elements (court landmarks and ball trajectory) needed to anticipate the oncoming ball, as well as generating targets for the shots across desired court areas. Similarly, they can include aspects of memory/learning, also shown in FIG. 10, such as mental representations of these environment elements (see FIG. 8) and corresponding movement patterns.

The specification of training goals at the competitive level follow a similar logic but focus on the dynamic characteristics, i.e., the temporal sequence of shots driving the game. As already discussed, the functional models at that level can for example be formulated using Dynamic Bayesian Networks or Hidden Markov Models. These models can then be used to assess the individual's strategy from the temporal patterns in shots, and identify deficiencies that are responsible for, for example the loss of points in a game. This understanding can then be used to generate training goal specifications that address these types of strategic or tactical deficiencies.

Planning Training Activity

At any given time in a skill assessment cycle, the skill status typically includes a repertoire of movement patterns, each in one of three learning stages. The potentially large number of movement types and the variety of challenges specific to learning stage can make assessment and training challenging. A combination of training goals is usually beneficial to effectively drive skill training, including training goals for forming new patterns, consolidating patterns, and optimizing patterns. Furthermore, there are the questions of deciding which training goal to emphasize at any given time, and keeping track of the changes in movement as learning process unfolds.

Training should follow a systematic process that accounts for the relative importance of the various skill elements to the movement activity and, at the same time, accounts for the natural skill acquisition process, i.e., how the brain naturally forms, consolidates, and refines movements. The training process should be able to distinguish between what aspects of skill to preserve and build on, what aspects of skill to eliminate, and when to form and consolidate new movement patterns.

Planning corresponds to the selection and scheduling of training goals. Training activity can be planned using the following criteria:

• • 1. The significance of a movement pattern and associated outcome to the particular domain of activity (e.g., where the relevance of the tennis strokes is denoted in terms of three categories: primary, secondary, and tertiary). Based on this consideration, training should account for the importance of a movement to task requirements and conditions.
  • 2. The relationship between movement patterns and, in particular, how some patterns can be understood as derivatives of others (see differentiation in FIG. 11 and evolutionary relationship in FIG. 13). Based on this consideration, training should emphasize patterns that are fundamental to repertoire development.
  • 3. Available augmentation modalities.
  • 4. Predicted difficulty of each goal, and time required to achieve training goals.

The training elements can, for example, be arranged in a list sorted in the order of priority that takes into account the above criteria. The training list (see FIG. 45A) is a list of training elements ordered by priority. The training list serves as a type of “working memory” for the skill elements that a user wants to focus on and track at a given period in a training activity.

For example, within each skill acquisition stage category, it is possible to rank movement patterns with the highest deficiency, as well as account for the hierarchical ordering of the movement units and outcomes for each movement activity.

The elements of the training list can also be arranged in a training schedule (see FIG. 45B). A typical schedule is defined by time units such as a session subdivided into sets, and each set is assigned with one or more training goals. The training schedule makes it possible to organize the training activity for a session. The sequence of training elements can be determined based on the acquisition process, i.e., how the skill elements build on one another and their respective acquisition stage.

Typically, the first set focuses on warming up, during which movement patterns that are technically less challenging and emphasize the range of motion and timing. Once warmed up, the subsequent sets can focus on specific technical aspects. At the end of a session, players can play freely or play points, which acts as a test for how well the focused training activity is translated into the task or activity performance. For each training goal in a set, relevant aspects of the performance can be monitored and augmented.

Planning can be done manually, with the assistance of an expert, or by an algorithm. In one scenario, the user can select training goal(s) to pursue based on skill status, trend, and overall goals. In another scenario, a coach can use their domain expertise in combination with the skill status and other quantities to help select training goals. In yet another scenario, an algorithm (training agent) can suggest and manage the training goals and schedule.

FIG. 46 shows the state machine showing the active training element and the criteria for the issuance of notifications to the performer. Also shown are stopping conditions for the training element, including, the number of strokes performed, the time elapsed, the incremental (e.g., percentage) progress toward the associated training goals. Typically, the subject is notified of the incremental progress milestones and notified when the stopping criteria of the training goal has been attained. At that point, the next training element can be initiated.

FIG. 44A shows the skill status with elements ranked by order of priority within each training stage category. The lists in each acquisition stage category can be ordered based on the contribution to the overall skill profile (based on the skill element composite score).

FIG. 44B illustrates an example of skill status that shows how training activity over several training sessions (e.g., Set 1-3) lead to a change in the skill status of skill elements. For example, BHTSH increases its ranking within the pattern to form (from 6^th to 4^th). Or the top skill element in the “patterns to form” BHSLH improves and gets re-staged to “patterns to consolidate.” Similarly, another skill element BHFLM is upgraded from “patterns to consolidate” to “patterns to optimize.” (Note that the training effect is exaggerated for purpose of illustration.)

Other Approaches to Skill Assessment and Diagnostics

The system-level understanding of skill, with its different levels of characteristics and assessments, essentially provides a rich data set that can be processed using a variety of other analytical techniques, in particular statistical modeling and learning, including neural networks. The systems approach taken here was motivated by the need to identify the different components of a data-driven system and the various forms of assessment and information. It is conceivable to generate these quantities using statistical learning techniques, which can even help discover additional skill attributes from patterns in the performance data.

A well-known class of diagnostic processes is based on so-called diagnostic expert systems. FIG. 26 shows an example of a diagnostic system building on the assessment system. The assessment system used to extract the various skill attributes can be used to drive such a system. Such diagnostic networks can be configured to generate the types of assessments presented herein (skill status, skill profiles), as well as training goals and even feedbacks and instructions and the configuration of the augmentation (cueing and apparatus interaction laws).

Typical diagnostic expert systems reason backward, through Bayesian inference, from observations to determine probable causes of specific phenomena. Traditional expert systems are built around a production system which provides the mechanisms to support user interactions. The core component of these mechanisms are rules (e.g., expressed using propositional logic), which are typically deterministic.

FIG. 27 shows details of the diagnostic system. It combines a knowledge representation, observations, and an inference mechanism to produce a diagnostic of the movement performance. The domain knowledge from an expert (e.g., tennis stroke motion and game) is encoded in a representation (e.g., Bayesian Network). An inference algorithm uses the Bayesian Network and the observations to determine the most likely explanation for the observations, i.e., diagnostic.

Complex behaviors such as human movement in open motor tasks, depend on a broad range of factors (sensory, physical, environmental, etc.); these relationships are complex and uncertain. Statistical inference systems such as Bayesian Belief Networks, which are graphical knowledge representation of a decision problem, make it possible to capture non-deterministic knowledge and uncertainties, as well as account for the larger patterns in the combination of factors or attributes.

FIG. 28 shows an example of an influence diagram for tennis. The diagram captures various factors across the different levels of the movement system hierarchy, including perceptual processes, court motion and positioning, stroke technique, and ball impact. The observations correspond to the example metrics detailed in the specifications. Other observations can be considered depending on their availability as measurements. For example the ball trajectory or the subject's gaze. The diagram can be structured as a Bayesian Belief Network and used as part of the diagnostic system. Note that the observations can also include general features.

The diagnostic system can combine expert knowledge, such as shown in the influence diagram in FIG. 28, with detailed movement functional analysis and direct diagnostics based on assessments. Note also that while some features—for example, the skill attributes illustrated for the tennis use case—are deterministic, movement in the real world usually involves more complex interactions such as adaptation to conditions. Therefore, statistical models can provide deeper insights into movement mechanisms. These models can be further extended using large amounts of data available from a diverse population of subjects spanning a broad range of skill levels, styles, and physical attributes.

An instruction generator, for example, converts the diagnostic results to verbal or visual communications (FIG. 27). Information from the diagnostic system, when applied to the larger control hierarchy, can also be used to analyze games or task performance and even be used in real-time to recommend actions; for example, which strokes to choose and which locations to target on the court, given the current states of the system conditions.

Versatile and effective movement skills depend on the seamless integration of all the functions or skill components required to perform the task, including perceptual skills, anticipation, planning (positioning), etc. Therefore, additional measurements to capture body posture, as well as perceptual functions such as gaze, may be required to assess a subject's skills comprehensively (see described elsewhere). And, conversely, feedback provided at all those levels is beneficial if it is integrated systematically. TABLE 3 summarizes the primary elements of feedback and instructions at different levels of the skill hierarchy.

Other data-driven techniques such as deep learning, which use multi-layered deep neural networks (DNN), can in theory produce the data-processing capabilities described in this disclosure. The main components of such as DNN may include: At the lowest level, delineating between movement phases to produce movement functional structural that would allow detailed characterization of the skills and task performance. Next, learning the movement and broader performance features (conditions and contingencies associated with contextual details) that are associated with the pattern classes and explain the repertoire structure and characteristics that describe a player's performance. Furthermore, higher level layers can identify the technique features that best delineate between movement classes and outcomes at the task level to predict player task performance. Finally, learning structured relationships between features and other factors or conditions that explain a subject's skill and performance at the task and competitive levels, which includes the temporal relationships characterizing task dynamics and for example game strategy.

Augmentations

The final category of capabilities for comprehensive data-driven training are the augmentation methods described in FIGS. 22-24. The general purpose of augmentation is to produce various forms of feedbacks (instructions, cues, and signals) and interactions that enhance the subject's performance and maximize training effectiveness for a given set of training goals.

The augmentations achieve these effects by: 1) providing information to the subject that help them assimilate the knowledge and/or learning process associated with a training goal (e.g., forming new mental models); 2) providing reinforcements that help induce specific changes in movement characteristics; and 3) creating or extending interactions with the task or activity performance that drive the operational envelope associated with the range of conditions under which a subject can successfully produce an outcome. The former is typically achieved through instructions, the second through feedback cueing, and the third through the use of an apparatus or cues in the task environment.

The human augmentation ideally follows an architecture that builds on our knowledge of human information processing (see e.g., Rasmussen 1983). Feedback augmentation can operate at any of the three primary information processing levels (see FIG. 22): the knowledge, rule, and the signal level.

The knowledge level includes instructions that explain training elements and training goals, bringing attention to specific movement characteristics and explaining what and how to correct these characteristics. This level of information is typically communicated verbally, in writing, or through visual representations. It helps form representations needed to monitor and correct performance.

The rule level includes feedback cue stimuli that encode information to help select the correct movement, or the timing of a specific movement phase, and/or focus attention on relevant aspects of the performance or environment. This level of feedback is typically communicated through visual, audio, or haptic signals.

The signal level includes continuous feedback, such as sonifications of the movement based on specific parameters that can be used to communicate relevant aspects or features of the movement profile. This type of feedback can also include extraneous physical effects such as a force field produced by an exoskeleton or often robotic device. They may also include functional muscle stimulation. Signal-level feedbacks are typically generated concurrently with movement execution.

Through their combined actions, feedback creates interactions that can stimulate subjects' learning process and/or assist in the movement performance. It is useful to distinguish between feedback that is produced about the subject's movement performance, and feedback that is produced about the task environment and its elements. The latter includes the interactions enabled by an apparatus, e.g., robot manipulator in rehabilitation or a ball-machine in tennis.

The following describes examples of specific forms of feedback augmentations that are used to enable augmented training, including Instructions and Notifications, Real-time Augmentation, and Apparatus Augmentation (see FIGS. 22 and 24).

TABLE 3 details possible instructions and feedback across the levels of the control hierarchy illustrated for tennis including: game plan; task environment, orientation, positioning and action selection; stroke-environment coordination; and stroke execution (see influence diagram FIG. 28).

TABLE 3
Feedback and instructions at different levels
of skill hierarchy with examples for tennis.
Level	Instructions	Cueing
Game plan	Game rules, strategy, point	Game plan, point
		construction
Task environment,	Task elements and	Perceptual cueing
orientation, positioning,	orientation, etc.	(e.g., ball trajectory
and action selection		anticipation)
		Positioning cueing
		Action selection (e.g.,
		stroke selection)
Stroke-environment	Elements of stroke-ball	Perceptual cueing
coordination	trajectory coordination	Impact timing
		anticipation
		Position adjustments
Stroke execution	Stroke architecture	Racket state at specific
	Relevant movement	movement phases
	phases and racket	(phase transition
	configuration	features)
		Outcome validation

Instructions and Notifications

Instructions operate at the cognitive information processing level, and are associated with the symbolic encoding of information. Instructions can help contribute to the formation of mental models or representations that support the skill acquisition process. Instructions are typically communicated verbally or visually.

Graphical instructions include plots, schematics describing the spatial outline of a movement, maps, etc. For example, a repertoire map (see FIG. 15) shows the distribution of different movement classes depicted relative to their primary outcomes (e.g., pace and spin imparted on the ball). The graphical description can be distilled based on a given set of movement pattern classes (see FIG. 16).

For example in tennis, the repertoire of ground strokes can be shown as a stroke map that highlights attributes such as the use frequency of the movement pattern during a session; number of movement executions; and statistics about outcome, success rate, etc. This information can be extracted and displayed for different time periods such as the current set or session. Additional information can be communicated in the stroke map, such as the relevance of the movement class to the task or the difficulty of the movement pattern, which can be determined from the evolutionary relationship shown in FIG. 13, as well as from the complexity of the movement architecture (see e.g., the number of states of the finite-state model in FIG. 5).

Another example of graphical instructions includes phase profile plots for a particular movement class to highlight relevant movement characteristics such as phase transition features. Or, an illustration or simulation of the movement showing the spatial configuration of the equipment over certain phases of the movement execution. FIG. 37 shows an example of the forward swing phase highlighting features associated with the spin outcome including trajectory curvature at the beginning of the phase and angle of attack at impact.

An example of verbal instructions would include validation of an outcome or instructions describing which phase transition feature to focus on. Or, it could walk through the movement phases describing features that are critical to performance. Textual instructions also include information layered on graphical instructions or displayed on the screen of a smart watch to display outcome information and progress toward training goals. Instructions and notifications are communicated on a display such as a smartwatch, smartphone, or tablet. It is also possible to use verbal communication via a natural language processor. The training agent determines when and what type of information is presented to the subject.

Instructions and notifications provide the interactions needed to run the training process. In an automated training mode, training activity operates as an autonomous (or semi-autonomous) program. As a training program, the system determines training goals and schedules, then tracks and updates the training goals and schedule based on the progress and trends. Notifications and instructions are used to communicate information to run such a program, provide instructions about the active training goals, how they are pursued through the activity (e.g., drills), and when to switch training goals, etc. Under an autonomous training program, the training goals and schedules are updated dynamically.

Real-Time Augmentation

The disclosure builds on the real-time augmentation technology for movement training described in U.S. Patent Application Publication No. 2017/0061817. The three primary categories of augmentation forms that can be used to help induce changes in movement technique specified by training goals include:

• • Outcome validation: Signals provide instantaneous assessment of the overall movement performance and outcomes. Validation cues are generated immediately following the action to indicate a successful outcome. The outcome validation is not limited to movement outcomes, but can be used to reinforce other relevant aspects of movement performance including smoothness, timing, etc., and those captured by performance criteria.
  • Alerts: Alerts augment the natural proprioceptive signals to enhance the subject's sense of movement with respect to specific training goals. They can also be used to implement injury prevention using the relationship between movement characteristics and biomechanics.
  • Outcome improvement and optimization: Real-time audio feedback during the movement execution helps reinforce and refine features of the movement technique that contribute to the outcome.

A central aspect of learning good movement technique is learning the sensory consequence of correct performance. Real-time cueing, therefore, can provide validation signals that augment the natural signals to reinforce learning the sensory consequence (see FIG. 24). Feedback that validates movement features provide an associative reinforcement of some sensory dimensions.

In addition, real-time augmentations can be designed to help with:

• • Training movement architecture: Real-time feedback assists in the formation of new movement structure through the use of visuals (e.g., simulation), as well as real-time cues e.g., that signal the conformance of the pattern to a template.
  • Forming anticipatory perception: Provides signals to learn to identify critical environment and task cues that are used to anticipate critical states and conditions, such as timing of the movement phases that enable synchronization of movement behavior with the task elements or objects.

Real-time feedback augmentations are communicated by the cueing system (see FIGS. 22 and 23) and include audible, visual, and haptic signals.

Apparatus Augmentation

The natural variations in a training environment combined with the variability in a subject's performance may not be sufficient to expose the subject to all relevant conditions that help drive skill acquisition. Particularly for deeply solidified patterns, highlighting erroneous features in a movement or providing feedback cues may not sufficiently change the movement pattern. In these situations, it may be more effective to actively produce new training conditions and thereby force the subject to acquire new movement patterns.

Since movement skills are developed for the purpose of adapting to task and environmental conditions, it is possible to force the development of new patterns by manipulating task and environmental elements and conditions. Varying the operating conditions beyond the natural range can be used to force the subject to develop new patterns and/or extend the range of operation of a given pattern. For example, a tennis ball machine can be used to produce ball trajectories that force the player to form a new stroke technique or adapt an existing one beyond its operating range.

An apparatus can also be used to help form new movement patterns by physically guiding the movement. This technique is already used in robotic movement rehabilitation.

Generalization to Other Activities

Since the training system is derived from the understanding of human movement learning and movement organization and performance, the training system can be implemented for a broad range of movement domains, including sports such as tennis (described in detail), rehabilitation, as well as professional activities such as surgery. Most of the concepts and quantities such as movement repertoire, their outcomes, etc. are derived from the theory of open motor skills acquisition. The training system can also be used for various forms of human-machine systems, including tele-robotics, humans equipped with prosthetics, or other forms of physical augmentations such as exoskeletons.

Human-Machine Systems

Since humans are increasingly integrated within human-machine systems, the augmented training system can be conceived as an integral part of such HM systems.

The robotic surgery system such as the da Vinci is an example of such a HM system. Many of the relevant quantities (operator inputs, manipulator or tool motion, visual gaze, etc.) are measured and recorded; therefore, the training system can be incorporated into the surgical robot's operating system. A data-driven skill assessment and training system integrated into such a robotic system can fulfill many functions, including: 1) train surgeons for new procedures, where they would benefit from accurate tracking of their skill learning process and feedback to help that process; 2) opportunities to formalize the certifications of surgeon training for different procedures, etc.

Report System Description

The following illustrates data visualizations for some of the concepts and quantities described as part of the data-driven analysis and training system in the context of a tennis application. These plots show some of the elements of the assessment and diagnostic processes illustrated in FIG. 30.

Overview of Data Visualizations

FIGS. 32-39 give a sample of the processed performance data. Starting with FIG. 35, which illustrates the activity data for a time period, highlighting sessions and sets over a calendar period. FIG. 39 then provides a close-up into a specific session and shows the event diagram that displays select stroke types ST used over the session timeframe (12:13 to 12:50). It also displays the pace SP and spin S outcomes as time histories TH to visualize trends in those outcomes over the play duration.

FIG. 36 then gives a more detailed look at the period of activity on a stroke-by-stroke basis 381. Additional outcome quantities are illustrated first, including the impact variability 382 and success cumulative progress 383. Below, it includes the separate time histories for the pace 384 and spin 385. The time histories are filtered to smooth out the stroke-by-stroke variations that can make these plots more difficult to read. Note, however, that since there is no inherent continuity in outcomes such as spin from one stroke to the next, the filtering can create artifacts. The plots in FIG. 36 also highlight the reference tiers 360-364 for the outcome quantities to help their interpretation (corresponding to low, medium, high, and very high values achieved by a player population).

FIG. 37 illustrates details of the functional analysis at the level of the stroke pattern. It displays the forward swing movement segment phase for the forehand topspin medium (FHTSM) stroke class, highlighting the path 710 of the racket relative to the origin or impact point 720. The stroke analysis based on this phase segment allows for the identification of features such as the angle of attack 730, the curvature of the path at the beginning of the forward swing 740 (transition from back loop phase), and the length of the swing phase 750. The figure also illustrates sets of segments corresponding to the core pattern 760 of this stroke class, and a sub-pattern set 770 that represents the subclass of strokes with the highest spin outcomes. This representation of the stroke technique can, for example, be used to investigate the efficiency with which the subject generates the spin outcome. The results of this analysis provides the basis for the specification of training goals and synthesis of real-time feedback and instructions to help the subject form, consolidate, or optimize the technique for that particular outcome.

Continuing with the functional analysis, FIG. 42 shows impact timing for the different groundstroke classes GC, which is defined as the timing relationship between the impact time and the time of the peak acceleration (or angular rate) of the forward swing movement phase T. Impact timing depends on the movement technique, motor coordination, as well as proper anticipation of the impact point and the player's preparation for the stroke. Therefore, it provides critical information to diagnose stroke technique.

FIG. 33 shows an aggregate view of the relationship between the swing rate R (horizontal axis) and spin S (vertical axis) produced for an ensemble of strokes in topspin, flat, and slice classes C for a particular subject. The quantities define the so-called spin envelope SE, which describes the range of spin S that can be produced by the subject as a function of the racket swing rate R. The spin represents the outcome and the swing rate represents a movement technique attribute, which in this case can be considered as the effort applied by the subject to produce the outcome. The spin envelope is parameterized based on the slope of the two linear boundaries (k_max MX and k_min MN), each are depicted along with reference lines corresponding to low, medium, high, and very high ranges, which again can be computed from a population.

The data representation then shifts to FIG. 38, which depicts the composite score for a specific stroke class (skill element) as a radar chart, which is an illustration of the skill-element composite score. It shows individual cost components based on extracted performance and skill attributes (impact precision IP, consistency CC, impact SR, efficiency EF, smoothness SS). The composite score, which can be visualized as the area covered by the polygon PG, represents the overall assessment of the skill element (stroke class). This polygon compared to the less opaque polygon CP illustrate what could be a comparison between two players, or between different skill elements, or the same skill element at different times in a subject training history.

FIG. 40 then takes a more comprehensive view and depicts the overall skill profile as a bar graph of composite scores CS for the groundstroke repertoire GR. This chart makes it possible to assess the overall repertoire strength and weaknesses (see FIG. 17). Similar to the skill element composite score, this skill profile can be used for comparisons between different players or between different times in a subject's training history. As already discussed, different composite costs can be used to emphasize different characteristics relevant to a task performance. FIG. 41 displays the acquisition stage of the strokes in the groundstroke repertoire based on the criteria described in TABLE 1 and TABLE 2.

Finally, FIG. 34 shows the leaderboard, which synthesizes the entire assessment at the population level. Note that these data visualizations are a sample of quantities described in this disclosure, and are used here to illustrate the types of quantities that can be used for the assessment and diagnostics for different levels and components, and how they can be used in conjunction with reference ranges to support the identification of training goals and eventually the feedback synthesis. These visualizations can then also be used for tracking progress and for updating the training elements and cueing laws, etc. as someone's skills evolve relative to their own history as well as to that of a larger population.

As already discussed, FIG. 43 gives an integrated perspective on the system's main components, organized in terms of the levels of assessment (physical 510, pattern 520, task 530 and competitive 540). The figure highlights some elements and quantities that drive the training process, in particular it highlights examples of assessment criteria at each level, and how the criteria relate across levels.

Starting from the physical performance level 510, the stroke forward swing phase profile with features (shown with more detail in FIG. 37) depicts an example of a skill model that can be used to analyze a subject's movement technique, taking into account the different assessment components (outcome, biomechanical, functional, perceptual, memory, and learning). Each component can be used to generate attributes for assessment and characterization of the skill element (i.e., the stroke class). The efficiency attribute EA captures the relationship between the spin outcome and forward-swing energy. In some situations, the attributes can be formally captured by a cost function 512, 522, 532, 542. FIG. 37 emphasizes the model describing the spin outcome and relevant functional characteristics. Similar models for the biomechanical characteristics, for example to identify features that can predict joint loads or muscle strain, can be developed and then converted to an attribute, e.g., injury index, that can be included for the skill element composite score (see FIG. 38).

At the pattern performance level 520, FIG. 43 shows how different attributes associated with the assessment components contribute to create the overall skill element score (see FIG. 38).

At the task performance level 530, FIG. 43 shows how the skill elements contribute to create the subject skill profile, highlighting the forehand top-spin medium stroke FTSM depicted in levels 510 and 520. It also shows how the skill profile is obtained through a composite cost function 532 combining the skill elements in the stroke repertoire.

At the competitive performance level 540, FIG. 43 shows how an individual's skill compares at the level of a population. In this example, the comparison is based on percentile rank computed from the skill profile composite score. The figure highlights how the individual's skill profile ranks SPR are relative to the population PP.

The material illustrated in FIGS. 32-43 can be embedded within a web-based or mobile app reporting system to allow a subject to navigate their skill elements and characteristics. The content below is organized into three sections:

• • I. Activity Session Report provides a description of the movement activity for a given session in terms of the skill elements, how these are used throughout the activity period, and various performance and skill attributes. The session report can also include training elements in the training list. The knowledge also provides the data to generate training goals and to plan and schedule training activities.
  • II. Detailed Pattern Class Report is a class-by-class detailed description of the various assessments, including pattern level assessment, as well as functional analysis and diagnostics at the level of the skill elements. The assessment can also include historical trends of how different outcomes and attributes of the individual skill elements evolved over the subject's recorded activity history. The class-by-class description can also provide information about active training elements as well as suggested training goals.
  • III. Comprehensive Player Report provides a summary of the player's activity and how the skill elements combine to create the subject's overall facility in the domain of activity. This is illustrated here using the repertoire, an overview of the different skill elements and their outcomes and attributes, the skill profile, and skill status. The player report can be augmented by population data to describe the relationship to other players in the subgroup as well as related subgroups that can represent longer-term skill targets for training.

I. Activity Session Report

The activity session report focuses on the overall description of the movement performance in a given session, focusing on the activity performance characteristics. The purpose of the session report is to convey understanding of high-level patterns in the activity performance, such as the evolution of various attributes over the period of the session; the use of particular movement patterns; and the trend in their outcomes, such as energy and success rate. The session report can enable the identification of the onset of fatigue or loss of concentration. This information can, for example, be used to help improve the training session, or even fitness or physical strength.

Play Activity Summary

The activity summary for a session can be presented as a table that describes statistics and trends for attributes of the most frequently used movement patterns in the recorded session. The statistics for the tennis use case can include: a) Pattern usage frequency (%); b) Impact success rate; c) Pace (m/s); and Spin (rpm). A trend symbol (up, down, or equal) and a trend value can be appended next to each metric to highlight the change in the respective metric for the session or relative to a selected time period.

A similar table can be used to summarize the activity for the training elements currently in a training list. The table may include the activity level for each training element during the session, when the element was created, the progress toward the goal during the last session or relative to a selected time period, etc. This information can be used to verify the effectiveness of previous training goals, training lists, and training schedules, and to help update the subsequent training plans. These summaries can be linked to visualizations of the session activity that enable more detailed insights into trends of select attributes of skill elements or training elements.

Trends in Movement Patterns Usage

FIG. 39 depicts a time history TH of player movement pattern usage. The usage trend plot depicts movement patterns on a stroke-by-stroke basis, where each vertical line L is a stroke occurrence. The movement class membership of a stroke, representing a skill element, is indicated by the vertical position of each line L. This example uses a subset of six stroke classes 30 to describe the main movement pattern trends in this activity.

This data used for the usage trend can also be analyzed to identify rally segment statistics, such as the average stroke counts or rate of return for each class used during the rally. The rate of return describes the probability of the opponent making a return. This probability can be computed for a specific pattern class. Furthermore, by analyzing rally ending strokes that lead to either points or losses, it is possible to identify the strong or deficient pattern classes, which can be used to identify deficiencies in the repertoire.

Trends in Movement Outcomes

The movement outcome trends in this section shown in FIG. 39 focus on the evolution in primary stroke outcomes across the different movement patterns during an activity session (pace SP and spin S). The session report gives the breakdown of how the subject used their time during a session, and therefore provides a composite view of the activity in the session. This information can reveal patterns in the technique and outcomes associated with the activity performance at different stages, such as during the warm up, while training on a specific training element.

The information in this chart can enable automatic identification of the type of the sets in a session. Sets can be identified by the intermittent rest, and for example, the deliberate training sets will have specific features such as concentration of strokes belonging to the same movement pattern, or the stroke pattern transitions forming a repeating pattern.

Moreover, the chart, and the information underlying it can convey information about the intensity or even the competitivity or competitiveness of the play in a set. The information can also can reveal patterns within a set, or across sets, that are related to physiological or psychological processes, such as the onset of fatigues, or deterioration in concentration.

These insights and knowledge can then be incorporated into the system, and used to plan and schedule training sessions. For example, this knowledge can help determine limits on the duration of certain activities in sets, or the total number of repetitions of particular skill elements in a set, or it can be used to set dependencies in the sequence of training elements in an activity period. All of these patterns can be identified using statistical modeling techniques.

As already described, activity at the task and competitive level can be further analyzed and assessed using statistical algorithms, such as a Hidden Markov Model. For example, such techniques can be used to build a state-machine that represents the most likely transitions between movement patterns based on various factors including a player's own prior activity. It can also include information from opponent activity performance, and be set up to capture the extended temporal patterns encompassing the task and environment elements.

II. Detailed Pattern Class Reports

The pattern class report is organized at the level of the individual skill element or movement pattern. It tracks the multivariate attributes and characteristics for each movement pattern, and therefore can provide insight into the skill acquisition process of each movement pattern, and help identify the specific deficiencies, which in turn can be used to help determine training goals.

The play activity of the pattern class is presented as the stroke counts by set, by session, and across the entire recorded history (see FIG. 35). The bottom histogram 351 in FIG. 35 shows the stroke counts by date over the entire recorded activity history of a player. The shaded bar 354 on the histogram can be moved by the user to select a set of consecutive dates to be presented in top chart 355. In the top chart, sets are shown as stacked shaded bars grouped by date 352. (This chart can also be used in the Play Activity Summation section in the player report, with stacked bars representing sets or movement pattern classes.)

The stroke counts 353 of the specific movement pattern indicates how frequently the pattern has been used. If use frequency is correlated with a decline in outcomes, for example, it prompts the diagnosis to identify causes which in turn could be used to formulate a training goal.

Movement Outcomes Trends

Movement outcome trends for a specific class are shown in FIG. 36. It focuses on the longitudinal dimension of the movement pattern development process by presenting the trends of a selection of select movement outcomes and attributes (for example: pace 384, spin 385, cumulative success progress 383, and impact variability 382) across the entire recorded activity history (see FIG. 35). The plot background shades 70 in the x-axis delineates the different sets. The plot background shades 360 in the y-axis encode the information about the reference ranges or tiers (e.g., very high 361, high 362, medium 363, and low 364).

The success rate trend plot depicts the cumulative summation of the impact success variable. The trend plot takes the form of a stair function 370 (up one step for a successful impact, down one for a missed impact). The dashed line 371 provides a reference for 100% success rate trend; a horizontal trend line would correspond to a 50% success rate. A subject can easily determine success rate trend by looking at the slope and contour of the trend line.

Impact variability is one of the class ensemble statistics. It is calculated for every set and presented as a staircase function across sets 377. The other trend plots (e.g., pace 384, spin 383) depict the evolution of the movement outcomes over time on a stroke-by-stroke basis. However, the time history can be smoothed to remove large variations that can make the interpretation more difficult.

At the scale of set and sessions, the diagram of the trends enables the investigation of the range of variation of the movement pattern. When multiple sets or sessions are combined as in FIG. 36, it is possible to determine variations in movement pattern performance as a function of the various types of sets, such as training, free play, or competitive play. The long-term longitudinal perspective also provides insights into the larger skill development process.

Moreover, this visualization can be used to verify the effectiveness of previous training goals, training lists, and training schedules, and help update the subsequent training plans.

Movement Functional Analysis

The movement functional analysis focuses on the details of movement technique used by the player to achieve their outcomes across the various movement patterns or skill elements. It also encompasses other relevant mechanisms that are used to modulate the outcomes or adapt to conditions. Functional analysis at the level of movement phases provides detailed insights into the movement technique that is valuable for the determination of training goals. This is illustrated in FIG. 37 for the forehand topspin medium (FHTSM) stroke class.

For example, the forward-swing phase, occurring immediately before the target phase of impact, contributes to the realization of desired movement outcomes of the motion pattern. Therefore, it provides both information about the outcome, and the more general organization of the movement. This phase lasts about 100 ms, which means that most of this movement segment is executed in open-loop, i.e., without opportunities for corrections. Therefore, its success depends on the motor program stored in so-called procedural memory. This program encodes the coordination and perceptual cues, the muscle synergies that support the physical execution, and the correct movement phase initiation and configuration (see FIG. 3A).

FIG. 37 presents the stroke trajectory profile of the forward-swing phase of the forehand topspin class. The figure compares core-pattern strokes with a subset of pattern strokes identified as having the highest movement spin outcomes. As already described, several features can be extracted for this movement phase (angle of attack, curvature of the path at the beginning of the forward swing, and the length and the elevation of the swing phase).

The forward swing profile also provides a visual description of the movement technique that can be used to generate visual instructions, such as a target movement profile shape. Real-time feedback cues can be generated to reinforce the desired features. The efficacy of these cues can be enhanced by combining them with visual descriptions of the target profile shape, which serves as a template for a mental model. The integration between the sensory-motor and cognitive levels can accelerate the consolidation.

Other functional metrics can be defined that focus on the overall range of outcomes. For example, FIG. 33 compares the overall spin envelope SE, which is defined by the racket swing rate R and imparted spin S. The spin envelope describes the efficacy of the stroke technique as ratio or spin/swing rate. A larger angle for the line delineating the envelope indicates that a player can achieve a higher spin outcome with an equal racket swing rate. The dashed lines DD correspond to the reference ranges from the population analysis. The spin envelope helps identify the deficiencies in stroke technique; as shown here the cause of the spin deficiency is due to an insufficient racket roll rate at impact. Generating larger roll rate at impact requires optimizing movement coordination, i.e., the movement architecture between the backswing and impact phase.

Another functional metric is the timing of the impact during the forward swing. The timing metric is defined as the relation between the instant of the peak racket swing rate and the impact. Correct timing of the forward swing depends on the player's anticipation of the interception, as well as other factors such as anticipation of the ball trajectory, footwork, and preparation.

Composite Analysis

This section integrates the attributes statistics of the movement pattern to determine a composite skill score using a cost function, e.g., the weighted sum of the attributes:

Q(a_i)=Σ_e^Naw_ea_i,e/Σ_e^Naw_e,  [2]

where w_e are the weights indicating the relative importance of the attributes.

The attributes can be normalized based on some characteristic values. These values can also be obtained from the individual's data reference ranges computed through population analysis, with the advantage that the composite score then provides more meaningful information.

A radar chart, as shown in FIG. 38, enables an intuitive interpretation of the multivariate contributions of attributes to each skill element. The figure shows a subset of select attributes depicted as a dimension 10-50. Under certain conditions, the total area of the polygon 60 formed by the outcome or attribute values can be viewed as a description of the composite score of the movement pattern.

The composite skill score can be used to rank the movement patterns and can be combined across patterns to form the player skill profile (see FIG. 40 and FIG. 17), which provides an overview of the repertoire, enabling the identification of player strengths and weaknesses.

This representation also enables the comparison of skill elements over different time periods, or between different skill elements. The two polygons 60, 70 shown in the chart can represent the statistics of the current epoch versus that of the entire recorded history, or the statistics of the player versus that of a subgroup that the player belongs to.

III. Comprehensive Player Report

The example player report combines the different assessments to create an overview of a player's overall skill status and skill development progress. The player report is organized at the level of the repertoire. It includes the following four sections:

Total Play Activity History

The play summation presents a player's activity statistics, which is a summary of performance activity over the subject's entire recorded history in terms of the following: 1) total number of sets, 2) total number of sessions, 3) total time duration, 4) last time of play, and 5) overall success rate.

Task/Repertoire Level Skill Assessment

The repertoire level skill assessment focuses on how complete the repertoire is relative to the task requirements. The repertoire completeness can be determined from use frequency (stroke counts) and the overall movement outcomes of a performer's repertoire relative to a nominal repertoire of motion patterns for the task. In this example, the nominal groundstroke repertoire is defined by a fixed number of groundstrokes expressed in terms of the spin and pace. Each of these outcomes are discretized in three levels (“slice,” “flat,” and “topspin” for the spin imparted on the ball and “low,” “medium,” and “high” for the pace) (see FIG. 16 and FIG. 32). In addition to these primary outcomes, the impact success rate (defined based on sweet spot area) and impact location variability are evaluated to measure the impact quality.

A more comprehensive assessment would cover different outcome levels (see FIG. 7), extending across different shot types as described by their trajectories and relationship to the court (see FIG. 8) as well as the broader repertoire of strokes and interception conditions (see FIG. 9). As previously discussed, these levels can be assessed using additional data, such as provided by vision-based tracking system.

FIG. 32 illustrates the overall movement outcomes using pace and spin as example. Movement patterns are divided into backhand and forehand, and sorted by the average outcome values. The data is visualized as a histogram chart. The lighter color bars correspond to the movement patterns without sufficient stroke counts and low statistical significance.

Background shades in FIG. 32 indicate different tiers/reference ranges (e.g., low, med, high, and very high). These reference ranges can either be determined based on the player's own statistics or derived from the population analysis that extracts the statistics from a subgroup of players sharing similar movement techniques and skill level. In this example, common reference ranges for all movement pattern classes are depicted since the emphasis is the overall repertoire. A more precise assessment can be achieved by extracting reference ranges that are specific to different pattern classes, including other relevant factors such as impact conditions. The more detailed contextual information is available, the more precise and actionable assessments can be achieved.

Skill Status

Skill status captures the skill acquisition stages of the movement patterns in the repertoire. FIG. 41 illustrates an example for the groundstroke repertoire. Each movement pattern is determined to be at one of the three stages: pattern formation, pattern consolidation, and pattern optimization. The qualitative characteristics and quantitative criteria that can be used to identify the acquisition stages are listed in TABLE 1 and TABLE 2 respectively. Skill status can be presented as a table with acquisition stages as columns, and movement pattern classes, or skill elements, are arranged in a sorted order (see FIGS. 44A and 44B).

Player Skill Profile

The information of use frequency, movement outcomes, and skill status of all the movement patterns can then be used to determine the player's skill profile as a histogram of sorted scores of motion patterns (see FIG. 17). The skill profile provides the information to build a leaderboard (see FIG. 33) and the larger population analysis.

Moreover, this section also presents the rally statistics, such as the average number of strokes in a rally and the cadence (number of strokes per minutes). This provides information to identify the player style in the gameplay.

General System Description

The disclosure includes a system to help individuals train or rehabilitate movement through and using targeted augmentations designed to stimulate learning through feedbacks and interactions. These augmentations are further adapted to the specific skill deficiencies that occur at different stages of the movement learning process, and account for the human information processing hierarchy. The system builds on movement sensing, skill modeling and diagnosis, and feedback synthesis, which are described previously described in U.S. Patent Application Publication No. 2017/0061817.

The general goal of training augmentation is to help guide the development of skills by providing feedback during training or performance. Since skill learning is an ongoing, dynamic process, a valuable feature of systematic data-driven skill training is the capability to model and diagnose skills in a way that captures the longitudinal and vertical dimensions of skill development. Recall, the longitudinal skill dimensions refer to the process of skill acquisition over time, through transformations of existing skill elements, and the vertical skill dimensions refer to formation of new skill elements.

Augmented Skill Ecosystem

The augmented skill platform is configurable to create an integrated environment for training, maintaining, and rehabilitating motion skills by combining motion capture technology, skill modeling and analysis tools, and a set of feedback modalities that can target precise aspects of movement performance. The system trains movement techniques to optimize a set of outcomes that are relevant to the activity over its domain of operation. FIG. 2 illustrates the elements of the augmented tennis activity environment that serves as a use case for this disclosure.

Any task can be described by environment elements EE, and task elements TE. For example, a person manipulates a device (e.g., tennis racket), end effector or piece of equipment, to interact with the task elements TE (e.g., tennis ball). In addition, there may be miscellaneous accessories Z such as shoes or clothing that may be relevant for the description of the activity. The workspace W is contained in the environment and is specified by various constraints and rules that characterize the task's success and performance (e.g., the tennis court and tennis game).

In tennis, the person is the player (or players); the task environment is the tennis court; the task element is the tennis ball; and the equipment is the tennis racket, and the accessories Z are the shoes and other pieces of attire such as an arm or head band. In addition, a variety of output devices can be included, including graphical displays (e.g., LCD, OLED, etc.), haptic devices (e.g., embedded in the racket grip), speakers. Finally, consider a variety of input devices, including touch sensitive display (user interface), keyboard, etc. The input and output devices may be integrated in the form of a smart watch, tablet, or a wearable device that can be worn by the person.

The overall elements, agents and other components used, including the measurement, input and output devices, are referred to as the augmented human system or simply the system S. Other examples of systems that have this general setup include a robotic system, a cybernetic system (e.g., a human fitted with a prosthetic), and a human-machine system (human operating a robot through tele-operation). For example, a robotic surgical system such as the DaVinci® Surgical System (available from Intuitive Surgical, Inc.) is a robot that is an example of an integrated augmented movement skill system.

Measurements y that contribute to the recorded performance data can be acquired from different components of the human actors, equipment, or system. Typically, instrumentation is designed to obtain measurements that encompass relevant variables for the particular level of analysis. For example, as illustrated in FIG. 2 in the analysis of human tennis stroke path 25 and performance, the states, or a subset of the racket motion may be sufficient. However, to enable a complete analysis of the movement on the court, the footwork, or the body motion such as the kinematic chain or other movement units, additional measurements about the environment and body segments 15 (e.g., arm, legs, feet, etc.) can be added.

These measurements can be obtained using a variety of technologies, including inertial measurement unit (IMU), visual or optical tracking systems, etc. Examples include the use of video cameras 70 that capture the broader agent behavior and the task environment 50. Vision processing can also be used to extract information about the motion of individual body segments 15.

Another category of performance data measurements is one that captures physiological quantities. For example, a gaze tracking system 80 to measure the visual attention. Thus, as shown in FIG. 2, a user 10 (or player or other subject) holding a tennis racket 20 which impacts a ball 30 during the swing or stroke of the racket 25. One or more motion tracking or video cameras can be attached to the performer, such as integrated with the gaze tracking system. These so-called first-person cameras capture data related to the interaction of the subject 10, the tennis racket 20, the ball 30, as well as the motion of other participants such as the opponent 53, and other relevant environment elements such as the court 51 and net 52. Combined with measurements of the gaze direction or vector 81, video cameras on the subjects and/or environment make it possible to determine which elements or events the performers are attending to at any given time, or at specific instants of the performance such as during specific movement phases or phase transitions 26, 27 on the path 25, opponent behaviors, or task elements such as related to the ball trajectory 36.

Inertial sensors 21 or similar measurement units can be embedded or affixed to the equipment; worn by the user, subject, or other agent 10 to measure the movement of body or body segment 62; or even placed on the user's, subject's, or agent's skin or implanted in the body to measure muscle activity or neural signals involved in the control of muscles 15.

Note that additional behavioral measurements such as gaze can be used to analyze the perceptual functions. For example, the gaze follows the ball trajectory 36, which has several notable events during the motion, such as the ground impact 32, the racket impact 30, and the interception by the opponent. The gaze (described by gaze vector 81) also typically can fixate on target areas on the court (outcome 3, ref 35), in between the court (outcome 2, ref 34), as well as anticipated racket impact or post-impact location (outcome 1, ref 33).

In addition to the measurements, data fusion and state estimation techniques may be implemented to determine states x that are not directly measured. For example, in most applications using IMUs, the orientation of a body segment 15 or piece of equipment 20 requires an attitude estimator which combines angular rate data from the gyroscopes, the accelerations from the accelerometer and the magnetic field strength from the magnetometer. An example of data fusion and estimation is the use of a vision-based tracking algorithm, applied to video data from video cameras, combined with IMU data from a device on either the body segment or equipment, to extract body segment or equipment motion information. Such a data fusion system can be used to provide an accurate estimation of absolute pose of body segment or equipment. The combination of motion processing such as based on IMU and computer vision enables the extraction of video frames associated with certain events in the agent-environment interactions. For example, the identification of a phase transition 27, such as the forward swing initiation 26, can be used to extract larger contextual information from the environment such as the location of the ball or opponent at that instant. Or vice-versa, a specific event in the task or environment such as a ground impact of the ball 32, can be relevant in assessing the agent strategy, taking into account visual attention (gaze 81), body location of the subject 10, footwork (shoes or feet) 60, and movement preparation, or particular phase segment initiations 26, 27. These interactions provide the basis for the task performance modeling, for example using Hidden Markov Models (HMM).

In terms of outputs, various wearable devices can be configured to generate a range of communication modalities such as audio, haptic, or visual. These devices can operate along different levels of the information processing hierarchy discussed earlier. Such cueing devices can be worn on the body, skin; integrated in the equipment such as in the racket grip 21, shoes 60; or even implanted in the skin or body such as muscles 15. They can be configured to provide different modalities of feedbacks such as audio, haptic stimuli, or visual cues. Another class of output devices include an augmented reality (AR) system 80 that can be configured to provide visual cues superimposed on the natural environment. Speakers, or visual signaling devices such as cones, markers, etc. can also be deployed in the environment itself 50 or on the object such as the tennis ball 30. Finally, implantable devices can also be used as part of the augmented system and for example provide functional muscle stimulation 15. Outputs can also be communicated via the typical wearable devices, mobile and portable devices and computers that are part of the augmented skill ecosystem, such as smart watches, phones, or tablets.

Typical human cyber-physical systems are described formally using hybrid system notation. This notation system combines continuous and discrete quantities. For example, the movement of a user, subject, or other agent may be governed by physical laws that result in nonlinear continuous time differential equations. Discrete variables may be used to evaluate conditions associated with specific events, such as counting strokes in a tennis game or scoring the game based on ball trajectory relative to the task environment and rules. Categories of state variables include: controlled variables, specific behavioral variables such as the visual gaze vector, and features used as cues by the agent to make decisions.

Actions are typically taken by the user and represent the addition of force or energy to the system such as the racket ball impact 30. Actions are typically applied to specific locations such as the end effector or equipment. As already discussed, actions are often motivated by a deliberate desire to achieve particular outcomes 33-35. In tennis, for example, the player wants to impart a specific effect on the ball (velocity and spin) 33, with the ultimate goal of driving it to a specific location on the opponent's court side 35. Events can be defined by particular state conditions. For example, in tennis, a major event is the impact of the ball on the racket 30. Events can be expressed formally by constraints on the system states, e.g., racket acceleration exceeding a threshold due to the impact, or alternatively, the impact can be detected when the ball and racket velocity are equal. Other relevant events in tennis include contact of the ball with the ground and when the ball crosses the net (see FIGS. 7 and 9).

As already discussed previously, outcomes are defined as quantities that capture the relevant characteristics of the agent's behavior in the performance of task. To provide a concise description, outcomes can be categorized hierarchically, e.g., primary outcomes, secondary outcomes, etc. (see FIGS. 7 and 10). The definition of outcomes are a function of the scope and level of the analysis. Expressed formally, outcomes are a subsect of the system states (e.g., at specific times, defined by events) or a function of the states. For example, in tennis, primary outcomes are the characteristics associated with the racket-ball impact 30, such as the spin of the ball when it leaves the racket or the ball's velocity 33. Primary outcomes could also include the location of the ball on the racket's string bed 30. Depending on the level of analysis (and available measurements), more comprehensive outcomes include the location of the ball's net crossing 34 or impact on the court 35.

The skill of an agent A is the effectiveness with which the agent is using its body and/or tool, equipment, etc., to achieve desired task outcomes TO and more generally interact with, and/or adapt to the environment elements EE and task elements TE.

Miscellaneous additional quantities that can be added to the description of the task or activity performance include task or game rules (e.g., rules of the tennis game), which provide the basis to determine task success or completion and various task performance characteristics, as well as various decision rules and control laws for other computer-controlled or autonomous agents, apparatus, or equipment or accessory. For example, control law, rules, and algorithms that specify the behavior and actions of the apparatus in the environment. These systems can include a prosthetic limb, an apparatus that reacts to the environment or task interactions, or even the various components of a robotics system such as a surgical tele-robotic system.

Note that once formalized as a dynamic and augmented agent-environment system, many movement activities include similar elements such as a human agent, primary equipment, the environment and its elements and, potentially, other human or robotic agents, and apparatus. These elements participate in the activity and combine to produce a scope of dynamic interactions. Such activities also follow the same general organization and therefore can be described using equivalent quantities and general modeling language as described here for tennis.

Augmented Skill System Overview

The following provides a systems level description and abstraction of such augmented human systems on which the data-driven skill analysis and training system is built. FIG. 21 illustrates an overview of the system and is followed by a description of the “augmented human system,” and finally, the general motion model, skill model, and the different augmentation modalities illustrated in FIGS. 22, 23, and 24.

The iterative training process illustrated in FIG. 21 illustrates three primary feedback loops: 1) A skill assessment loop (AL) 200 that tracks the overall progress in movement performance in the task domain, updates information about the user's skills, including motion models and skill models, as well as diagnostic tools used to identify specific deficiencies in movement technique that provide the basis for the synthesis of training goals; 2) A training loop (TL) 208 that tracks the progress in specific areas of the skill captured by training goals and configures the augmentation system; 3) A feedback augmentation loop (FL) 202 that provides relevant information during the movement performance.

The identified motion and skill models, combined with the diagnostic assessment, provide the basis for generating a set of instructions, which can be used to organize the training process, and synthesize cueing laws used to drive the augmentation. A user receives two primary forms of feedback: instructions and real-time cues. The instructions are typically generated during a session at particular intervals, e.g., completion of a training set, or after a training session. Instructions are typically presented in visual form and emphasize more comprehensive aspects of performance and skill.

The augmentation loop can be used to exercise movement on movement characteristics that have been identified through the diagnostic tools. The cueing process targets specific characteristics to directly impact movement outcome and performance. The cueing system computes feedback signals using algorithms that are synthesized based on the motion and skill models derived during the assessment. These cues are communicated in real-time to the user. The assessment and augmentation feedback are delivered following the hierarchical organization that takes into account the hierarchical structure of skill development and the temporal characteristics of the movement and skill attributes.

The training assessment loop is managed by a training agent. The augmentation loop is managed by a cueing agent. These agents operationalize the two processes and are able to track progress at these two levels and provide user with the interactions to run this system (see FIG. 21).

Data-Driven Training System Capabilities

The motion model captures the comprehensive movement performance through the movement repertoire which organizes the range of movements as classes of movement patterns and their associated outcomes. The repertoire model provides the ability to identify gaps or weaknesses in patterns. Gaps in the repertoire, i.e., missing motion patterns, manifest as the inability to produce actions and outcomes in areas that are relevant to the task performance. Gaps can also manifest as the inability to deal successfully with the range of prevailing operating and task conditions that are required to enable high level of task performance or from contingencies or environmental disturbances. Movement patterns are represented to describe relevant functional characteristics, such as phases and their associated biomechanical constraints.

The primary functions needed to support data-driven augmented training include:

• • 1. Assess and guide formation, consolidation, and optimization of patterns at the level of the repertoire. This function focuses on the actions and outcomes that support task performance.
  • 2. Assess and track the quality of movement outcomes. Deficient patterns don't achieve the required outcomes consistently or efficiently, or don't achieve them under a sufficient range of conditions.
  • 3. Diagnose movement technique for deficient patterns, which corresponds to determining aspects of the movement technique that are favorable or detrimental to the outcomes.
  • 4. Diagnose movement skills based on their development stage to determine the appropriate types of training (formation, consolidation, or optimization).
  • 5. Formulate training goals to address the specific deficiencies in skill elements.
  • 6. Determine appropriate forms of feedbacks across the scope of human information processing levels, including: instructions, real-time cues, and apparatus interactions.
  • 7. Monitor the learning process, track and update the skill models, the derived training goals, augmentation forms, etc. based on changes in a subject's movement skills and other factors including health and fitness.

The system provides a range of feedback types that act as drivers to modify subjects' behavior toward improving their skills. The feedbacks are based on information and knowledge extracted from the motion and skill models, as well as from the extended analysis based on performer population, which make it possible to account for broader factors.

The feedback, as already discussed, operate at various levels of the human information processing systems. These encompass a broad range of neuro-cognitive mechanisms. For example, the highest-level feedbacks are based on drivers that are rooted in social aspects of performance. These include leaderboards with ranking, side-by-side comparisons between players (e.g., via the skill profile, see FIG. 17), or role models that can be selected from the population analysis.

TABLE 4 summarizes the drivers, derived from the data-driven modeling and assessment, according to their levels of operation in the hierarchy.

TABLE 4
Drivers for training derived from movement
and skill model across the hierarchical levels.
Top-level	Mid-level	Low-level
drivers (cognitive)	drivers (cues)	drivers (signals)
Patterns to	Quality of	Feedback
develop/form	outcomes (e.g.,	augmentation
(accommodate	pace, spin,	(functional
conditions and	etc.)	movement
outcome	Perceptual cues	characteristics).
types)	(movement	Apparatus
Outcomes types	coordination	(interactions
and relevance	with environment	that drive
for task (e.g.,	and task	sensory
effect of stroke	elements)	and motor
outcome on	Mid-level cues/	processes,
shot outcome,	feedback	e.g., expand
etc.)	(outcome	conditions,
Movement	validation)	etc.)
architecture
Cue environment
(e.g., perceptual
mechanisms)
Population
reference ranges
“Peer-pressure”
(e.g., ranking
or leaderboard).

With these functions, it is possible to operationalize the training process as a training program with variable degrees of user interaction. From manual—where the user uses the features to guide his or her decisions, to completely automatic—where the system guides the user through the training process generating and updating the plan according to the evolving skill status.

Finally, the entire modeling, assessment and feedback process can be extended by population-level analysis. The specific features include:

• • 1. Perform population skill analysis by clustering the individuals based on their skill level, movement technique, skill attributes, and other potential factors (health, age, etc.).
  • 2. Identify the subject's population subgroup membership and the related groups with respect to skill development.
  • 3. Compare the subject's skill attributes to the subgroup's. Statistics provide appropriate reference values to help rate each performer within the group, perform diagnostics, and specify training goals to drive and track the training process.
  • 4. Check the related subgroups to determine possible benefit of forming a new movement technique, which would help the subject transition into a “better” subgroup.
  • 5. The population group capturing the skill development provides the direction for the orientation of the training, such as movement architecture

System Architecture

As described elsewhere, the system relies on a movement capture and measurement system (shown in FIG. 2 and FIG. 21). This system collects data from relevant movement quantities, including movement of equipment and body segments; physiological quantities, including electrical muscle activity (e.g., via surface or implantable electrodes); and other relevant quantities from the recorded performance data. Data also includes task relevant quantities, such as outcome of the action or movements, as well as its effect on the larger task outcomes. The system can track multiple users and their interactions.

The three primary feedback loops 200, 202, 208 shown in FIG. 21 are closed around the augmented human system detailed in FIGS. 22 and 23. The human movement activity is augmented at three primary feedback levels which are communicated to the user through different modalities. The feedback forms are organized according to the primary levels of human information processing and include: instructions or notifications, feedback cues, and feedback cue signals.

As already described, communication modalities include audio, visual or haptic stimuli (potentially also direct functional muscle stimulation or even stimulations of the subject's peripheral and central nervous system). In addition, feedback augmentation also includes activity interactions provided by an apparatus.

The purpose of these feedback augmentations are as follows:

• • Instructions provide information about the training elements and the associated training goal. This information contributes to the formation of mental representations. They are typically communicated verbally, symbolically, or graphically.
  • Notifications provide information about progress with respect to a training goal. These are considered at the knowledge level of human information processing and can be communicated verbally, symbolically, or graphically.
  • Cues provide information to highlight specific features about the performance or outcome. They are typically communicated through discrete audible, tactile (haptic), or visual signals and contribute to the formation of rules that allow efficient processing of information both for motor and perceptual functions.
  • Signals provide real-time information to guide movement and enhance relevant movement features. They are typically communicated through continuous, or piece-wise continuous audible, tactile (haptic), or visual signals (potentially also direct functional muscle stimulation or even stimulations of the subject's peripheral and central nervous system).
  • Apparatuses enable interactions at the activity level to emphasize particular task states or conditions, such as a ball machine that can throw balls with different trajectories (pace, spin, height, depth, etc.) to the player. An apparatus can also be used to physically guide movements, such as with an assistive robotic device.

The system is configured to receive various inputs from the user, coach, or physical therapist. The user interactions are enabled by a graphical user interface (GUI) and/or natural language interface (NLI). The GUI or NLI enable the user to browse or interrogate the skill assessment and configure the training process. For example, users can select which training elements to track and which feedback forms (notifications, cues, signals) are preferred. The user can also provide inputs related to the outcomes or technique of the movement during performance. For example, they can tag a particular action or movement they believe is relevant for further analysis. Users can also rate individual sets in a session, for example, based on their perceived training effectiveness. These feedbacks about performance can be used to highlight particular qualities during the assessment and diagnosis process. For example, they can serve as additional assessment signals.

The training loop is managed by a training agent which provides various degrees of autonomy and provides functions to assess skills along specific skill elements that have been designated as training elements. The TL helps structure practice by organizing training goals in schedules. It also manages the configuration of the different components of the augmentation system.

The feedback augmentation loop is managed by a cueing agent, which tracks effectiveness of the selected cueing profiles (cueing law, apparatus interactions).

General Operational Model

The general operation follows the block diagram shown in FIG. 21 that lays out the three primary feedback loops introduced earlier.

The Assessment Loop (AL) describes feedback that takes place over longer periods, spanning one session to multiple sessions, associated with the skill acquisition process. The unit of time for the AL is an epoch, which as already discussed is defined by the data set requirements for modeling and assessment, denoted by superscript k. The primary functions of the AL are computing and updating the movement models (M^k) and skill model (skill status S^k). The skill status S^k is a collection of skill element extracted from the repertoire that are assessed with respect to the skill learning stage. Information about movement and skills are used to plan training activities and synthesize the various forms of augmentations. The training activities are codified by training elements and goals. These are represented as projected changes in skill elements. Overall changes in skill are measured as incremental changes in skill status ΔS^k.

The Training Loop (TL) encompasses feedback around training elements, including selection of active training goals g=Δa, the instructions relative to the training goals, and the tracking and progress reported on the selected training goals. The unit of time for the TL is the set, denoted by the subscript n. Changes in training elements during a session are measured as changes toward training goals Δs_n.

The Feedback Augmentation Loop (FL) encompasses the feedback during movement performance, including the various forms of cueing, and those mediated by the apparatus to support the active training goals. The FL focuses on the interactions that take place during performance and directly impacts movement behavior such as provided by real-time cueing. Users dispose of a range of instructions and feedback modalities to augment their training or playing experience including instructions, feedback cues, and/or apparatus interactions.

Given the multiple hierarchical levels that contribute to successful motion skill performance—limb segment coordination, movement architecture, body posture, positioning, movement outcome; all the way to movement planning, task, or game strategy—augmentation can potentially encompass a wide scope of skill components and interactions. As described elsewhere, the feasible level of analysis and interactions depends on the information that can be extracted from available measurements.

Typical training or play sessions can be described as periods of performance interrupted by pauses (see FIG. 58). Pauses subdivide the session into sets. Usually, users start a session by planning their activity and setting active training goals. Not all sessions are explicitly structured or planned. Even if this is the case, users can improvise and at any time enable various augmentations and access skill analysis and training management features.

The main user interaction application supports browsing functions to: review past and current data, view existing movement skill status, select active training elements, view details of training goals, and enable augmentation profiles.

Closed-Loop Data-Driven Training Process

The system components of the closed-loop training framework of FIG. 21 operate according to three primary units of time. A training epoch k is a time scale ranging from one to several sessions. The set n associated with the training loop is a time scale ranging from a few to many occurrences of a motion pattern set, i.e., provide a unit of time to organize sessions. A set n can have one or more active training elements. The time t corresponds to the actual time and is typically associated with the augmentation loop (real-time feedback from the cueing system or the apparatus based on measurements y_t).

The motion measurement data y is processed to determine the movement state data x. The data can also include other behavioral data (e.g., visual gaze) and task specific data (e.g., movement and location of task elements and objects, and various types of outcomes). Raw measurements are often extended through an estimation process to determine relevant state information based on available measurement data y.

Different data are emphasized depending on their role in the system shown in FIG. 21, e.g., monitor the feedback augmentation (y_t), training loop (session Y_n), or assessment loop (Ψ^κ).

The motion data is processed to extract the primary movement units associated with the actions performed in a task or activity (described elsewhere). This process can be formalized based on human movement system theory or principles. The movement repertoire R^k can be obtained through classification of the ensemble of movement units into a collection of movement patterns {P_i}, which can be divided into classes (FIG. 12). The movement patterns result from sensory-motor schemas or programs (described elsewhere). Through motion modeling, these movement patterns can be described by a sequence of movement phases that are related to the functional characteristics, including muscle synergies, biomechanical constraints, perceptual mechanisms, and task constraints.

The result of the motion modeling at a given epoch k is a set of motion models M^k={δ_i, i=1 . . . N_c} that combines the movement repertoire, the phase decompositions, and functional aspects that can, for example, be described by finite-state machines or statistical models such as a hidden Markov model (HMM) or some other model form learned from data (e.g., through deep learning). The elements of the motion model M^k provide the basis for skill assessment and diagnosis to extract skill attributes encompassing competitive performance, task performance, pattern performance, and physical performance levels (see FIG. 14).

The movement pattern P_i, the motion model δ_i, and skill attributes a_i in combination enable the definition of skill element:

e_i=(P_i,δ_i,a_i).  [3]

The skill status S^k contains a collection of sorted skill elements. The skill elements are sorted based on their acquisition stage. The collections correspond to the three acquisition stages: formation, consolidation, and optimization:

S^k=S^k_form∪S^k_con∪S^k_opt,  [4]

where e.g. S^k_form is the subset of skill elements that contain motion patterns satisfying the criteria discussed earlier for the formation stage.

Skill profile p_skill(S^k) describes how different skill elements combine to create the subject's overall performance. This information can for example be determined by adding up the composite scores for each skill element across the repertoire:

p_skill(S^k)={p_skill,d(e_i), d=1 . . . N_p, e_i∈S^k}  [5]

where N_p is the dimension of the skill profile and p_dkill,d(e_i) is a composite score of the skill element e_i.

Each skill element e_i can be selected and combined with a training goal g_i to form a training element γ_i=(e_i, g_i). Simultaneously, analysis of the training element can determine feedback augmentations that are appropriate for achieving particular training goals. The augmentations include instructions, real-time cueing, as well as interaction modes mediated by an apparatus.

Training goals take into account specific skill characteristics. For example, statistical analysis of skill metrics associated with a skill element can be used to predict the expected progress along the skill metrics. When population data is available, additional statistics from the subject's sub-group can be used to provide reference values and goals for the various skill attributes a_i. Training goals are expressed as the desired changes in skill attributes: g^k+1_i=a^k+1_i−a^k_i=Δa^k_i.

The training goals for an epoch k are arranged in an active training list Γ^k=γ₁→γ₂→ . . . →γ_Nb where N_b is the length of the training list. This can be used to plan or schedule the training session. Since human information processing is limited, it can be helpful to focus training on a limited set of skill elements. The primary purpose of the training list is to designate which skill elements to focus on, and also to configure the augmentation system. The active training list describes an order of importance, with the top-listed training goal representing the most significant training goal. These elements have priority on using system resources such as notifications or real-time feedback.

As described earlier in skill status, skill elements are organized hierarchically to describe their acquisition stages, which reflects their relative importance to the activity performance. The active training list can be generated automatically from the skill status taking into account the relevance of skill elements, or selected by the user taking into account information such as their preference, available time, and conditions.

Training goals can be explicitly pursued, e.g., during a dedicated training set. Alternatively, performance related to the training goals can be tracked during the “free” performance of the activity. Relevant information about these goals can be used to notify the subject. Such notifications can, for example, highlight when significant progress toward a goal has been achieved.

The longitudinal analysis coupled with the population data provides both the microscopic and macroscopic information to support training planning. Specifically, the population sub-group and its association with the subjects' individual characteristics (physical, training history, skill status, etc.) provide the information needed to manage the skill development: at the microscopic level, by providing references of realistic and preferred skill and performance characteristics relative to the group at a given level; and at the macroscopic level, by providing directions on movement architecture, and other attributes such as movement functional characteristics, to adopt for efficient and safe performance.

Another population analysis is performer profiling. Specific characteristics of a subject groups can be captured by their skill profiles, which can be described by composite metrics that emphasize different attributes. These profiles make it possible for the assessment and diagnostics that drive a subject's behavior in the direction of that subgroups' style.

In conclusion, the motion model, skill status, skill elements, and training elements provide the quantities needed to implement training as a data-driven, iterative process. For each performance set, the training goals in the active training list are tracked to provide progress reviews or notifications. As the performance of several training elements has improved substantially (e.g., when one or more training goals have been met), the motion model and skill status can be re-assessed, leading to an update in skill status. At that point, the user may continue with the remaining elements in the training list or re-assess which aspects of skill to emphasize.

Training Modes

One disclosed capability is the management of comprehensive information relevant to a user's movement performance and its application to drive and manage training. The disclosure also addresses the problem of how this information is communicated to the performer. The system can support several modes of interactions. These modes distinguish themselves by the levels of augmentation (types of feedback) and how the training elements are used to direct training.

The following training modes are considered for illustrative purposes:

• • Fully guided training: training agent selects the training elements and provides a training plan that specifies which training elements are exercised and when to switch training elements. This mode also includes drills.
  • Partially guided training: training agent selects the training elements and user determines the order of training elements to exercise and when to switch training elements.
  • Interactive, augmented play: training elements are selected by user, user determines the order of training elements to exercise and when to switch training elements. In augmented play, training elements can be integrated within regular playing sessions. Tracking takes place in the background and the training agent provides notifications on various milestones for the selected elements.
  • Free augmented play: users can take advantage of feedback augmentations during regular play.

The technology can also be used by a coach as a tool or training assistant. In this scenario, the coach will essentially become an element in the feedback training loop. The “augmented” coach within this system can play several functions, including interpreting the results of the skill assessment, planning the session, and providing verbal and other instructions such as demonstrating the movement.

Quantities and Variables for Implementation of the Training System

The following describes the primary system components and their system-wide integration in terms of logic diagrams shown in FIGS. 49-58. FIG. 49 shows the top-level logic diagram for the overall system and its primary processes, depicted in FIG. 21. The main blocks in the diagram are as follows.

Data Acquisition 110 represents the process of capturing performance data, which includes movement measurements from the activity, and other relevant activity data. The movement measurement covers the motion of the agent and his or her segments. The activity data covers the quantities useful to assess outcomes related to the performance and goals of the task or activity.

Modeling 120 represents the processes of modeling the subject or agent's movement and the relevant interactions with the task and environment elements. It includes extracting and processing primary movement units (PMUs), which are subsequently decomposed into movement components (described earlier), such as movement phases and muscle synergies that are relevant to the functional understanding of the movement patterns.

Skill Assessment and Diagnosis 130 represents the processes used to determine the parameters that characterize a subject's skill elements, which subsequently determine a subject's skill profile and skill status. These processes can also take population data as inputs, denoted here as reference data. This additional data enables the determination of the player's profile.

Training Goal and Feedback Synthesis 140 represents the processes involved in designing the various feedback augmentations (instruction and notification, cueing laws for real-time feedback, interaction laws for the apparatus).

Planning 150 represents the process of selecting the training elements, as well as planning and possibly scheduling of the training goal sequence, that will be used to manage the training or activity session.

Finally, Activity Management 160 represents the actual performance of the activity, including processes through which the feedback acts on the users during the various interactions. It also includes the process of tracking the training, managing the session, and configuring the overall system. The configuration determines the feedback profiles and how the training goals are tracked during the performance.

Additional details associated with these processes are provided in the following sections.

I. Data Acquisition

Data Acquisition 110 is the collection of all relevant performance data from a given performance. It is achieved through a variety of motion capture technologies, including IMUs that are worn by the subject or embedded into equipment or clothing, and optical or vision-based motion tracking systems. Data Acquisition also encompasses measurement of other activity or task relevant quantities such as outcomes and performers' behavioral data (e.g., visual gaze data). In addition, estimation techniques can be applied to estimate unmeasured quantities from the available measurements.

Motion Activity Measurements

Consider y_t=y(t*ΔT), 0<=t<=N_t, where ΔT is the measurement time interval and N_t is the number of measurement samples. The measurements y encompass all relevant data for the desired level of analysis. They can include other behavioral measurements, such as gaze or muscle activity, as well as contextual data (information about set, session, player/subject, task, and environment conditions, etc.). Some quantities can be estimated. In the following, the notation, y encompasses any type of data, measured or estimated.

The performance data set for an activity set n: Y_n={y_t, t=1 . . . N_t,n} where N_t,n is the number of measurement samples in set n. The data set for the entire performance epoch k is: Ψ^k={Y_n, n=1 . . . N^k_n}, where N^k_n is the number of sets in epoch k.

II. Modeling

Movement modeling 120 (see FIG. 50) uses the captured performance data and possibly previous movement models 260 to form the subject's up-to-date movement model. Movement modeling is an ongoing process that evolves in parallel with the skill acquisition. Therefore, it typically accounts for previous model information, in an iterative process.

The modeling process described in FIG. 50 includes the following steps:

• • Extraction 210
  • Classification 220
  • Phase segmentation 230
  • Synergy decomposition 240
  • Motion model formation 250

Movement Patterns

The first step in movement modeling is the identification and extraction 210 of the movement patterns associated with the primary movement units (PMU). The PMUs are described in terms of movement unit profiles, which can be represented as time histories or trajectories in the state space. These profiles are then classified 220 to determine the membership information needed to determine the movement repertoire.

The measured movement data is segmented to extract the set of primary Motion Units {s} for the activity and their associated actions, events, and outcomes.

The set of extracted motion units for an activity set is Ξ_n={s_j, j=1 . . . N_s,n}, where N_s,n is the number of motion units in set n. Similarly, the set of extracted motion units for an epoch k is Ξ^k={s_j, j=1 . . . N^k_s}=∪Ξ_n, n=1 . . . N^k_n, where N^k_s is the number of Motion Units in an epoch k (e.g., one or more sessions).

The repertoire is obtained through classification of the set of primary movement units Ξ over the period of activity, for example the set of movement units Ξ^k for an epoch k.

The movement repertoire for an epoch k: R^k={P_i, i=1 . . . N_c}, where N_c is the number of primary classes or clusters. The classification can be hierarchical, where one primary pattern class P_i can be decomposed into pattern subclasses: P_i={P_i,A, A=1, . . . N_c,i}, where N_c,i is the number of subclasses under P_i, and P_i,A={P_i,A,B, B=1 . . . N_c,(i,A)}, where N_c,(i,A) is the number of sub sub-classes.

Movement Model

The classified movement units can then be further analyzed to determine additional information relevant to the selected level of analysis. For example, PMUs can be further segmented into movement phases 230 associated with the muscle synergies, or other forms of segments relevant to the execution and functional analysis of movement. The logic to select the level of analysis is shown in the inset in FIG. 50. The phase segmentation 230 generates finite-state motion models, used in the functional analyses of movement, as well as a finite-state estimator, which is used in the cueing system. Finally, the synergy decomposition 240 generates muscle synergies that can be used for physical and musculoskeletal analysis.

The result of the motion modeling is a set of motion models M^k={δ_i, i=1 . . . N_c} for an epoch k. This set describes the entire repertoire R^k, with each motion model δ_i describing a primary movement unit P_i. For example, a motion model can be a finite-state statistical model (HMM, etc.) such as:

δ_i:X×U−>X,  [5]

where δ_i is the state-transition function for the pattern P_i, U is the input alphabet, and X is the set of states. Each model typically accounts for relevant functional details, such as the movement phases and associated actions or events for a specific movement pattern class (muscle activation, environment or task state, sensory or perceptual state, etc.).

The synergy decomposition 240 uses the movement phase profiles to determine components of muscle activation patterns that are combined to produce the resulting movement throughout a phase. Typically, adequate determination of movement synergies requires measurement or estimation of the individual body segments movement, and possibly other relevant quantities, including physiological quantities such as electro-myographic measurements of electrical muscle activation. Modeling the relationship between movement components and the musculoskeletal system provides information that can be used to estimate the biomechanical load and in turn help prevent excessive wear and injury.

The various modeled quantities are combined to form the motion model 250.

III. Skill Assessment and Diagnostics

Skill assessment and diagnostics 130 (see FIG. 51) and the underlying skill modeling, as described elsewhere, builds on the elements of the motion model (repertoire, movement phases, etc.) and the skill and performance attributes that can be generated through various metrics.

Movement pattern classes P_i, and associated motion models 8, provide the structure to perform skill modeling, various forms of assessments, and diagnostics. The assessment is primarily a descriptive process of various skill characteristics relevant to the movement activity. As shown in FIG. 52A, the overall movement skill assessment metrics encompasses several levels: physical performance 312, pattern performance 313, task performance 314, and competitive performance 315. Each level, if selected 311, will be assessed across several components described earlier: outcomes, functional, perceptual and memory and learning (see FIG. 10).

Skill modeling uses the attributes generated in the assessment process and integrates them 316 to enable movement diagnosis. The assessment step includes determining relevant quantities from the movement data, elements of the movement model, and movement activity performance. Reference values 317 from population analysis or individuals can also be incorporated in the assessment of the skill elements.

The diagnostic step includes interpreting these quantities to identify which aspects of the movement technique or other physical attributes need to be changed to improve the outcomes and other behavioral characteristics critical for movement activity performance. This process is achieved by determining the relationship between outcomes and the various skill attributes. The movement functional analysis plays a critical role in movement technique diagnostics since it describes the mechanics of how movement accomplishes its outcomes.

This information is then used in a subsequent step to formulate training goals and to synthesize the augmentation that can be used to drive training (FIG. 53).

Physical Performance Assessment

At the physical performance level 312, the assessment evaluates skill in terms of the physical effort required to achieve the outcomes and in terms of characteristics associated with the biomechanical constraints, such as the strain on the muscles or torques and forces on the skeleton, ligaments and joints. The movement physical performance assessment is based on metrics such as energy or jerk. These quantities can then be related to the outcomes, or used to determine movement efficiency.

This assessment level also evaluates the relationship between movement patterns, specific movement phases, and wear and strain on the associated musculoskeletal structures. The features extracted from this assessment can then be used to generate feedback to help modify aspects of the associated movement execution and thereby help mitigate injury.

The output of the physical performance assessment 312 are metrics p_i=h(P_i, δ_i), such as peak power, energy use, and joint torques associated with either select movement segments or the overall movement pattern.

Pattern Performance Assessment

At the pattern performance level 313, the assessment evaluates movement technique, as well as all other supporting functions, such as perception, that a subject uses to achieve outcomes under changing and uncertain conditions. The pattern performance assessment provides critical information for the movement diagnostics.

Basic movement skill assessment includes the analysis of movement technique by extracting attributes of the movement trajectories within a given movement class. Typical movement skill attributes include:

• • Smoothness: Many skilled movements are obtained as a sequence of movement phases. Phases typically represent individual synergies (described elsewhere). Skill acquisition involves the consolidation of the phases into units of movement behavior. Proficient subjects, therefore, are able to execute the sequence seamlessly, while execution for beginners are more disconnected and discrete.
  • Consistency: The movement profiles in a class represent general motor programs (described elsewhere). Therefore, proficient subjects are expected to display consistent trajectory characteristics within a class.
  • Timing: The successful execution of movements and their associated outcomes depends on accurate spatial and temporal coordination. Critical timing characteristics can be evaluated and used as skill metrics.

More advanced movement skill assessment builds on the movement structure (e.g., phase decomposition) and is based on derivatives computed using sensitivity analysis. The primary functional metrics are derivatives that capture how different features describing the movement technique, participate in outcomes and adaptation to task conditions.

The output of the pattern performance assessment 313 are features f_i=g(P_i, δ_i) that capture relevant characteristics of the movement technique. These can be determined for relevant skill and performance characteristics, and can be expressed as features of the finite states X, such as timing characteristics, movement and body configuration at state transitions, or movement phase profiles during phases.

Comprehensive functional movement skill assessment can also include the perceptual functions or events that are relevant for the coordination of the movement phase with task and environment elements. As stated elsewhere, the scope and depth of the skill assessment, and therefore also the scope of diagnostics and feedback augmentations, depends on the available measurements.

Task Performance Assessment

At the task performance level 314, assessments evaluate a subject's skills in terms of the range of movement patterns in the repertoire developed to tackle the movement activity requirements and handle the range of conditions prevailing during performance.

As described elsewhere, in open skills, a range of different movement behaviors have to be acquired to successfully deal with task and environmental conditions. Diverse movement patterns are needed to achieve different outcomes, and to attain those outcomes under a variety of conditions.

Therefore, the primary skill metrics at the task performance level focus on the range and quality of outcomes associated with the actions or movement patterns in the repertoire. The output of the task performance assessment 314 are task and outcome metrics m_i=f(P_i, δ_i) which typically represent descriptive quantities determined from the movement model and outcome measurements and/or estimations. They can include: success rate, movement outcome/result, variability, precision, as well as statistical characteristics, for a specific session and/or relative to historical data.

Competitive Performance Assessment

At the competitive performance level 315, assessments evaluate a subject's skills in terms of overall strategies developed to tackle the task and handle the range of conditions prevailing during performance.

The motion patterns can be used as the state of the agent to describe the agent-environment interaction dynamics at the task and competitive level. For example, a player's sequence of motion patterns in a game or set can be described by an HMM model, P_k+1=Ψ(P_k, c_k), where Ψ is the conditional probability of transitioning from motion unit P_k to P_k+1 given subject observation of cues c_k at time k, which includes perception of environment conditions such as opponent shot and incoming ball trajectory, and a subject's perception of their own position on the court. The function Ψ can capture a player's strategy, as well as their ability to perceive the game status and opponent actions and intention. Therefore, the function Ψ contains information that can be used to assess player's competitive performance. Such an HMM model can be extended to include any relevant state information such as the position of the subjects or the position of the ball.

Integration of Skill Attributes and Definition of Skill Element

The Skill Attributes for a particular movement pattern provide information for the overall assessment of the movement skill, performance, as well as other relevant considerations such as injury risk and the learning process itself. The skill and performance attributes extracted for a movement pattern P_i are combined a_i=c_i□m_i□f_i□p_i to define the set of so-called Skill Elements 318.

A Skill Element provides the formal definition of the concept of a unit of skills. The skill element e_i combines the pattern class P_i, its movement functional structure MFS (e.g., specified by the motion model δ_i), and various relevant attributes a_i:

e_i=(P_i,δ_i,a_i)  [6]

The collection of attribute a_i, in particular the outcomes, the attributes relevant to technique and performance, and the range of operating conditions, provide a comprehensive description of each skill element. This information can be used to score the skill elements, which helps determine which ones a subject can perform more proficiently. An example of such a score is the use of a composite cost function.

For example, the cost function Q can be defined as the weighted sum of attributes, with the weights indicating the relative importance of each attribute:

Q(a_i)=Σ_e^Naw_e·a_i,e/Σ_e^Naw_e.  [7]

Determination of Skill Status

A critical aspect of skill assessment is the acquisition stage (e.g., formation, consolidation, optimization). This information is described by the concept of skill status (FIG. 52B), which provides information about the acquisition stage of each skill element. This information is useful for the determination of training or rehabilitation intervention.

The skill status can be determined by applying criteria derived from a variety of skill attributes a_i and their associated statistics 321. TABLES 1 and 2 describes examples of criteria that can be used to determine the acquisition stage of movement patterns from the repertoire.

The Skill Status S^k for an epoch k can be represented as a partition on the set of skill elements that covers the movement repertoire:

S^k=S^k_form∪S^k_con∪S^k_opt,  [8]

where e.g., S^k_form is the subset of skill elements that contain motion patterns satisfying the criteria discussed earlier for the formation stage.

Determination of Skill Profile

As described earlier, the repertoire combines the collection of movement patterns that have been acquired by an individual to deal with the task requirements and environment conditions. The motion model encompasses the movement repertoire, its movement classes, and movement phases and synergies. The extracted attributes from the various skill metrics provides additional information to determine other quantities relevant to learning and training.

The comprehensive description of an individual's skill can be determined based on the set of skill elements associated with all the classes (and potentially sub-classes) of movement patterns in the repertoire:

S^k={e_i,i=1 . . . N^k_c}.  [9]

This set is shown here for a particular epoch k.

The skill elements e_i combined with the skill status provide comprehensive information about the subject's movement technique and performance. This information can be used to generate a so-called skill profile 330, which describes the overall skill and performance of a subject.

Skill Profile p_skill(S^k) describes how the full set of skill elements combine to create the subject's overall performance. This information can for example be determined by adding up the composite scores for each skill element across the repertoire:

p_skill(S^k)={p_skill,d(e_i),d=1 . . . N_p,e_i∈S^k},  [10]

where N_p is the dimension of the skill profile and p_skill,d(e_i) is a composite score of the skill element e_i. In some conditions p_skill,d(e_i) can be simplified to be Q_d(a_i).

The skill profile can be illustrated graphically for example by displaying the skill composite of each skill elements (see FIG. 17). One potential output of this assessment process is to generate a list of the movement patterns sorted by skill level based on composite score and development stage 326. This list provides the basis to define the training elements.

As already described, higher-level assessment such as task performance and competitive performance can be determined by how elements are deployed in a task or game.

Determination of Other Forms of Statuses and Profiles

Movement classes can be arranged relative to physical and biomechanical criteria. Typically, skill and physical attributes evolve in parallel during learning; however, subjects can adopt techniques that may be effective in achieving outcomes but detrimental to their musculoskeletal health. Possible physical development stages include, “physical build up,” i.e., patterns where the technique is affected primarily by a lack of adequate strength, “endurance,” i.e., patterns that exhibit premature wear, and “excessive load,” i.e., patterns that are executed with a level of force that produces excessive wear and strain on the body. This information can for example be used to determine an injury index for each skill element. This index can then be added across the repertoire to determine the injury profile.

Similar ideas of profiles, which are based on some composite assessments, can be generated for other characteristics besides movement skills. For example, the motion and skill model and attributes can be combined to compute quantities such as Physical/Fitness Profile p_fitness(e_i) or Injury Profiles p_health(e_i).

Determination of Player Profile

Note that it is possible to add more importance to some outcomes or actions or skill elements in the task by setting different weights in the skill profile, e.g., giving more importance to a forehand top-spin high compared to a forehand slice low. Therefore, the skill profile can be tuned to particular task requirements or styles of performance. For example, some of the strokes and outcomes are more fundamental to player performance. These can be characterized as core strokes. Different tiers of skill elements or strokes can be defined, and the skill profile therefore can be decomposed into different profile components to capture different characteristics.

The relative weights assigned to the skill elements in the skill profile enable characterizations of performer or player types, which can be used to define a player profile 340. For example, in tennis, strokes that are used in defensive play as opposed to offensive play provide the information to characterize the player type. This information is further developed under population analysis.

Reference Ranges and Population Analysis

One aspect of assessment is the definition of reference ranges that make it possible to more objectively assess a subject performance or skills (see 317 in FIG. 52A). Reference values can be used to provide absolute references, for example to measure how the various extracted attributes compare to a representative group of players. For example, in tennis, this allows a subject to understand if their topspin amount produced for a particular stroke class is high or low.

Reference values can be determined by extracting statistical distributions across attributes for a group of subjects with similar movement technique. The statistical data can then be used to generate percentile ranks for any relevant attribute, and using those for example to discretize the reference ranges into tiers (such as low, medium, high, very high).

These various forms of performance, skill, health, or injury profiles provide information that can be used for high-level feedback on various aspects of performance including strategy, physical fitness, as well as injury prevention.

Population information can be used to determine leaderboards that can be helpful for a coach or a physical therapist. It can also help motivate subjects to understand how they compare with other individuals, e.g., in an absolute ranking, as well as to understand the specific aspect of their skills or performance that is responsible for their ranking and which aspects of their movement technique or performance is the most actionable to help them progress within their group.

IV. Training Goals and Feedback Synthesis

Training Goals and Feedback Synthesis 150 represents the determination and specification of training goals, discussed earlier, and associated augmentations that can be used to drive training. These are selected across the different feedback modalities.

The specification of a training goal 410 can be viewed as a dual problem to the determination of an augmentation that will help drive the training process toward the goal. Ideally, goal and feedback synthesis is performed simultaneously.

Feedbacks target the skill elements, as well as other aspects of skill and performance such as insights from the skill profile, identified in the assessment. It can also use information from diagnostic data such as from the skill status (S^k) 130.

The synthesized feedback (instructions, real-time cueing laws, etc.) determine the “augmentation space” available to skill goals (FIG. 21). These augmentations define the scope of user interactions within which the user can then choose to operate. FIG. 23 describes how the augmentation environment is enabled and operated during performance.

The acquisition stage in the skill status computation provides information that allows to determine the appropriate feedback forms. For example, the formation of new patterns requires different augmentations than the refinement or optimization of an existing movement pattern. In general, several feedback modalities can be combined (e.g., instructions, feedback cues, and apparatus interactions). The feedback configuration 426 describes how feedback modalities are combined to produce the user augmentation.

Training Goals and Elements

Training goals help make training actionable, and enable subjects to focus their attention during performance. The quantitative specification of the training goal also means that it can be measured or estimated, which allows to objectively track the training progress for the particular skill element.

The computation of training goals 410 (FIG. 54A), as previously discussed, is based on the skill elements, the larger system in FIGS. 10, 30, 31, and can also account for the skill status to help specify a meaningful training goal.

Training goals can be derived from the statistical analyses of a subject's skill at the various assessment levels such as task performance level, for example, based on the attributes within a skill composite score, taking into account population reference data (see FIG. 19). Or, for example at the physical performance level based on the functional analysis (see FIG. 20 and FIG. 37). The training goal can for example be derived based on an increment (or a fraction increment) in a percentile tier for a skill level or outcome level, respectively.

A training goal at the performance level can be determined from the global score ranking shown in FIG. 34. One can proceed, for example, by identifying the skill element in the repertoire that has the largest impact on ranking (critical skill element). And from there determine the skill attribute in the skill profile composite (FIG. 38) that has the highest impact on skill profile. It is possible to determine the target skill attribute a* associated with a target increment in composite score for the critical skill element e_i using the statistical distribution shown in FIG. 19 as follows:

g^ki=a*_i−a^k_i=Aa_i,k  [11]

where a* is the goal value for the feature that would result in the target skill profile level, and where k stands for the epoch.

For a training goal at the physical performance level, one can proceed following an outcome optimization based on functional analysis, as given in the example of the forward swing analysis shown in FIG. 37. The target skill feature f* associated with a target increment in outcome level (spin) for a skill element e_i can be determined from:

g^k_i=f*_i−f^k_i=Δf_i,k  [12]

where f* is the goal value for the feature that would result in the target outcome level, and where k stands for the epoch.

Since skill deficiencies often manifest across multiple attributes, one or more component, or even some combination of components of attribute a_i, can be selected as the critical attribute to drive a particular goal g_i. Furthermore, the attributes can require targeted movement technique optimization. Therefore, the attribute goal can be combined with the functional analysis.

Note that the relationship between attributes and target increment in skill level was described at the diagnostic level 130 (see distribution and tiers in FIG. 19). Dimensionality reduction or embedding techniques can be used to determine the functional relationship. This level of analysis is typically conducted during functional movement modeling.

One question for the specification of training goals is the determination of how actionable an attribute or feature is. Functional analysis usually provides enough information to determine causal relationship and identify the critical driving attribute for training.

Which attribute to select to drive training 411 can also depend on the acquisition stage (Skill Status). Training goals can have different formats, depending on the level in the hierarchy (outcome, or functional characteristic), and also the acquisition stage of the targeted training element.

Training Element γ_i,b=(e_i,g_1,b) describes a Skill Element e_i combined with a Training Goal g_1,b.

Levels and Types of Training Goals

For the formation patterns 412, the specific goals include the spatial definition of the movement configuration. This corresponds to the cognitive stage of skill acquisition where the subject forms an understanding and representation of the movement primarily focusing on its spatial configuration. The knowledge for example includes the movement phases, including the configuration of the body segments, and end effector and equipment (system state), at phase transitions. Also relevant at the formation stage is the understanding of the relationship between the movement phases and their synchronization with the environment and task elements and objects.

For the consolidation patterns 413, training focuses on consolidating the sequence of movement phases into a smooth trajectory. This stage corresponds to the consolidation of the procedural memory where the movement knowledge is translated into an automatic pattern that can be performed dynamically under various conditions. This stage is mostly unconscious and relies on feedback to validate the correct technique.

For the optimization patterns 414, the specific goals include the refinement of the movement patterns and associated functions to achieve best outcomes within the subject's bio-mechanical constraints. The quantities that are optimized include the functional characteristics (features associated with movement outcome) and physical performance characteristics (musculoskeletal loads). At this stage, the subject can form mental representations that enable them to focus on features in the technique that influence the outcome, or gain an understanding of which elements of the task convey information that helps with the movement modulation or timing.

These training goals can be codified based on the parameters associated with the acquisition stages that are relevant to the movement activity. These parameters include statistical characteristics of the relevant parameters such as consistency, smoothness, and timing (described earlier). Augmentations are selected to target the aspects that are critical for the particular acquisition stage.

Feedback Synthesis

The feedback laws are synthesized 420 using the training elements (combining skill elements and training goals), including information from skill profile and status (FIG. 54B). The terminology of feedback is used in the larger sense, with the following two primary feedback types: instructions 421, and feedback cues 422. In addition, an apparatus 423 (see e.g., ball machine in FIG. 2) can be used to provide additional interactions for movement performance and training (see later discussion, see FIG. 23).

Instructions are synthesized from the skill model parameters and assessment 424, in particular the skill profile and player profiles.

For instructions (see FIG. 55A), the communication modalities include visual 431, verbal 432, and text 433. Instructions 434 represent feedback that operate at the “knowledge” level. They include aspects such as descriptions of the training elements for the next training set, or details about the movement features that will be augmented through feedback cues. Instructions can also include visual descriptions or simulations of the spatial configuration of formation patterns.

Cueing mechanisms 439 are synthesized from the motion model and in particular the functional movement model 425. For cues (see FIG. 55B), the cueing mechanisms include validation cues 435, outcome optimization cues 436, alerts 437, and pattern formation cues 438. These cues are used as feedback augmentations. The cueing laws for the real-time feedback cues are determined from the functional movement modeling and analysis.

If an apparatus is available, such as a ball machine, 427, apparatus interaction modes are synthesized 423.

The instructions, cues, and apparatus can be combined to create different augmentation profiles that lead to different interaction forms. The synthesized instructions and cueing mechanisms are first integrated to determine best combinations. The goal is to combine these feedback modes to achieve synergy. The settings and parameters define the available feedback configurations 426 (FIG. 54B). These combinations are then used to determine configuration parameters for the communication, cueing, and apparatus systems.

General Augmentation Levels

Augmentations can operate at the symbolic/cognitive level, cue level, and signal level. The augmentation laws and programs for an epoch k are denoted as a collection of feedback laws K^k={κ_i, i=1 . . . N_c} and programs.

At the cognitive level, feedback is in the form of instructions prior to performance, reports following the performance, and notifications during the performance. Instructions can be used to help subjects form mental representations of the movement pattern focusing on aspects that are relevant to current training elements.

Also relevant at the high-level are feedback related to population analysis such as leaderboard. This type of feedback plays at the psychological level.

Notifications can be used to provide feedback on training progress, e.g., on a specific training goal. Reports provide a summative overview of the subject's skills and training activity. The generation of textual and other symbolic or graphical information is performed via a communication system with an instruction generator such as an expert system. Notification can be implemented in the form of a state machine, or even using a conversational agent which can output either text or natural language.

At the cue and signal level, feedback is provided by the cueing system (described elsewhere). Feedback cues target the movement characteristics associated with the training goal (through outcome validation, feature validation, etc.), as well as associated sensory and perceptual processes. Cueing and signal level feedback can operate as reinforcement or deterrents.

The cueing system can also provide visual cues to help form visual attention needed to support a particular interaction for the task or activity. The cueing system combines a cueing law specific to the training element γ_i that computes cueing signals and a cue generator that translates and encodes these signals into understandable signal forms (audible, visual, haptic, etc.). The cueing laws are implemented for example by a state machine which uses the movement measurement data y_t, states x_t, and/or movement features f_i to compute the cue signal.

The cue and signal level also encompass the interaction laws for a possible apparatus. The primary role of the apparatus interactions is to expand the operating range, for example to help form new patterns. The apparatus can also provide physical interactions such as those provided by an assistive robotic device or exoskeleton. The apparatus action is driven, similarly to the feedback cueing, by a feedback law and/or program.

V. Planning

Training planning addresses the question of which aspects of movement performance are to be emphasized during training. The plan or schedule describes the organization of a session in terms of the training elements and associated training goals. The plan also provides the structure to schedule and manage the session during the performance of the activity 160 (see FIG. 56). Planning typically takes into account the overall training goals, available time, and other resources.

The prioritization can be determined from the stage of skill acquisition of the skill elements and for the significance of the skill elements to the task objective. The training elements can be prioritized based on the skill status (S^k) of each skill element 415.

To facilitate the planning and management of a session, several training elements can be selected. These selected training elements produce a so-called training list. By selecting the active training element, it indicates to the augmentation and tracking system which aspects of movement performance have to be monitored and actively cued.

The training process is formalized as an iterative learning process. This model makes it possible to determine how the data is managed. For example, epochs can be defined to coincide with major developmental changes during a training cycle over which a new skill level is reached with significant changes in attributes to result in a new profile. This epoch has an associated data set with motion model, skill model and various skill attributes, and statistical descriptions. For each epoch, there are associated training elements and goals that when completed will lead to a new skill level. The delineation between epochs can be arbitrary. More objective criteria can be used to determine training epochs, for example, the validity of the motion model used for motion pattern classification. As an individual's movement technique changes sufficiently to compromise the motion processing, the training system can prompt the user and the assessment cycle can be re-initialized, which provides a new baseline for training. The motion model enables the analysis of the skill acquisition process for an individual and also across the larger population. Therefore, patterns in skill acquisition can also be used to manage the training process and determine the larger-scale training goals.

Training List

Training list for a current epoch k, Γ^k can be represented as an indexed set (list) of training elements Γ^k=γ₁→γ₂→ . . . →γ_Nb, where N_b is number of training elements in epoch k.

The training list provides a way to emphasize a group of training elements. The goals at the top of the list have the highest priority. Training priority can be determined from the skill status parameters and criteria (see TABLE 2), the development stage, and information about the relevance of particular movement patterns (skill elements) and associated outcome for the task (see FIG. 13). The designation of the priority of a training element in the training list can be performed either manually by the user, or automatically based on assignment of the skill elements (see primary, secondary, tertiary in FIG. 13).

Training Schedules

A training schedule for an epoch k, Σ^k can be represented as a sequence of subsets of training elements Σ^k=Σ^k₁→Σ^k₂→ . . . Σ^k_n→ . . . →Σ^k_Nn where N_n is the number of activity sets in epoch k and Σ^k_n is a subset of the training list Γ^k, Σ^k_n=γ_n,1→γ_n,2→ . . . →γ_n,Nbn.

Each training element may include a stopping criterion to signal when to transition to the next training goal. Stopping criteria could be the number of movements to repeat in that particular class, performance over a time duration, given progress toward the goal (given fraction), or the accomplishment of the goal, which can be determined statistically such as in clinical significance tests.

The ensemble of training elements and goals can be used to systematically plan and manage training or playing sessions. For example, a training schedule composed of sets can be generated before the session (see FIG. 45B). Each set can emphasize one or more training goals.

VI. Activity Management

As discussed earlier, a training or play session is typically divided into time periods. These periods are designated as sets. Each set can have one or more training goals. These elements are arranged across several sets to form a training schedule. This structure makes it possible to decompose longer-term training goals into intermediate goals.

The implementation of the training process takes place through the augmented human system (see FIGS. 22 and 23).

Different feedback modalities call for different frequencies of user interactions. Instructions for example are presented intermittently typically following the selection of a training element. Real-time feedback cues, on the other hand, are applied concurrently with the movement performance. Real-time feedback can also be communicated continuously or at discrete time periods during the execution, or just following the movement outcome.

In some activities, an apparatus is used as part of the performance. A typical apparatus in tennis is the ball machine. The apparatus can be programmed to work conjointly with the feedback cues and instructions.

System Configuration

The main parameters for the augmentation system configuration 620 are designating the targets (e.g., subject, coach, etc.), and specifying the type of instructions (e.g., verbal, audio, etc.) and the type of feedback cues. The primary systems that mediate interactions are shown in FIG. 22. They include the communication systems (e.g., tablet or smart watch), the cueing systems (e.g., wearable device), and the apparatus system (e.g., ball machine).

For the instructions, different targets can be selected based on the training format. For example, in one scenario, a coach interprets and communicates the instructions to the subject. In this case, the coach would receive information about the subject's performance during a particular set and use this information to coach the subject before the next set. In another scenario, the subject uses instructions to assess the progress on a given training element.

The feedback forms under instructions include visual, verbal, or text. These forms provide different modes of interactions. For example, they can invite the user to browse the movement repertoire. Or, they can invite the user to learn about technique for a particular movement pattern.

A typical scenario includes the refinement or optimization of a movement pattern. In this scenario, the cue profile combines phase transition cues with outcome validation cues. Yet, in another scenario, the subject could use cueing during the performance to assist the formation of a new movement pattern or to optimize an existing pattern.

Once the system is configured, the subject can start with the activity performance 630. During performance, movement and system behaviors are monitored 640 and data acquisition continues. However, the emphasis of the assessment is characterizing the movement skills with the augmentation and the performance relative to the training goals. The activity can be paused at any instant 690.

Planning may take place ahead of the session or proceed incrementally. Initial training elements and schedule are defined based on current skill status. The training goals and elements for the subsequent set are defined as a function of the subject's completion of the training goals and overall performance, as well as other factors such as wear, fatigue, or motivation. To support possible changes in goals and configurations, the training system enables interactive management during the performance of the activity.

Session Management

Managing the training activity is an interactive process. The management of the session 610 includes specifying which training goals are pursued at a given time period in the activity and updating the configuration of the augmentation system (instructions, feedback cues, apparatus mode of action). The training goals are typically provided as part of a training schedule specifying training elements and associated goals. The goals are pursued through the interaction of the subjects with the augmentations. As discussed in the section on feedback synthesis, training goals provide a quantitative description of the change in a training element and can take into account the augmentation profile available for the element. The augmentation systems are configured based on the goals of the next period of activity. The system configuration 620 (FIG. 57A) determines how the different feedback modalities 621, 622, 623 are combined to create performance interactions that are most effective for the pursuit of the training goals.

FIG. 58 describes a session temporal structure delineating the different periods of activity, shown as sets #1 to #4. A set can be followed by a pause in movement activity. During activity periods, the performer receives cues, and or notifications. During pauses, the performer can review the performance data, and if needed modify goals and system configurations.

Activity Monitoring

Once the training activity is initiated, the progress towards the training goals can be tracked during the training activity 640. The change in training elements provides the basis to provide feedback on progress. The monitoring system 640 (FIG. 57B) provides notifications 644 to the performer (or coach).

Notification criteria 643 can be used such as the number of repetitions of a training element 641, the achievement of a certain fraction of the training goal, or the time elapsed. Notifications 644 indicate if a training goal has been achieved 642, which can be determined using a form of clinical significance test. The significance test determines when the subject's technique has progressed sufficiently for the skill attribute to have stabilized near the target level.

Depending on the specific system implementation, a subject's movement skill profile and skill status can be assessed at various time intervals to accommodate for the different rates at which various aspects of movement skills evolve. Therefore, the assessment loop is closed (updated) at different rates for different system configurations and different assessment levels.

While movement technique can be modified through instructions, demonstration, or feedback cues, the changes that result from these inputs first need to be assimilated. For example, the movement repertoire does not change rapidly since it requires consolidation of movement into procedural memory. Therefore, the assessment at the task performance (repertoire) level is typically made at the interval of the epoch, spanning sessions to days or months. Epochs may be linked to changes in a subject's movement repertoire, but as described earlier the associated time periods are defined based on the creation and maintenance of sets of movement data and models (see FIG. 25).

The notification 644 can be issued using a range of communication devices and signals. For example, the subject can be prompted 645 through an audible signal and a message can be displayed on a smart watch. Alternatively, notifications can be translated by a natural language processor and via voice communication. The message can indicate the progress toward a specific training goal, or attainment of a particular outcome threshold. The system can also prompt the user for an input 645. For example, this allows the performer to make a note or comment 646, or to simply tag a particular movement pattern, for example to indicate an issue or an outstanding result. At any time, the user can also prompt the system, e.g., via a smartwatch to tag an event.

Activity Interruptions

Depending on the notification and the status of the training or activity, performance may be paused 690. Interruption in the activity can be prompted by the subject, the systems, or the coach. Typical scenarios include the following:

• • The subject briefly interrupts the session to gather more detailed information about a particular movement pattern that was just performed.
  • In another situation, the subject wants to review his or her performance over the last set.
  • In another scenario, the cueing system detects a decrease in effectiveness in one of the active cueing mechanisms.
  • In another scenario, the cueing agent notices that the movement performance has achieved the target level of the training goal. The user receives instruction to pause to select the next training goal(s).
  • In another situation, the coach monitors the performance via the communication system and decides to interrupt the performance to change the configuration of the augmentation system.
  • In yet another scenario, the system detects changes in the outcomes or attributes that may be due to the onset of fatigue or wear, or even injury.

The user can receive instructions to pause, for example through a smartwatch, and subsequently pauses the session. Once the activity is paused, depending on the reason for the interruption, the activity can be resumed immediately 690, or suspended for a longer period to allow for data review and changes in plans and configuration. At this point, the performance data can be reviewed in greater detail 650, and then depending on the required attention, the performance is resumed or the session can be suspended.

Before the session is resumed, the augmentation profile 670 and training goals 680 can be updated. The change in performance associated with an active training element may require updating the training priorities within the existing skills status and thus can prompt a review of the training goals 680 in the training list. Large changes in skill status may require an update of the motion and skill models (iteration of the assessment loop leading to a new skill status S^k+1).

Activity Suspension

Once the activity is suspended, a more detailed review can be initiated 650. The review is typically mediated by the communication system, i.e., tablet or smart phone. The purpose of the review is for the user or coach to go over the progress for the current training or to address issues that have been brought up by the training agent.

After the review, the user has two options: to end the activity or to resume it 660. If the user decides to end the activity, it closes the session. If the user resumes the activity, it can be done under the same training list and augmentation profile 670, or a new augmentation profile can be selected which leads to the system re-configuration 620. Alternatively, a new activity or training plan can be selected before resuming performance 610.

In the event of a system induced interruption, the activity review provides details on the cause of the interruption. The user will then typically be prompted to return to the system configuration 620 or activity planning 610.

Examples

In the data-driven movement skill training systems disclosed herein, the systems may use movement skill assessment and diagnostics at distinct levels of the human movement system hierarchy to specify training goals. The systems may provide different forms of augmentations synthesized to help pursue the training goals. The system may also include a system to track and/or manage the learning process.

Efficient movement training may require a systematic way of organizing the training process. Training may be most effective if it targets specific areas of weakness of an individual's movement skill, accounts for individual health and fitness, and/or unfolds according to plans that are compatible with the structure and principles of natural skill development. Precisely assessing skills before planning a training activity, and/or providing adequate forms of feedback before, during, and after movement performance may benefit a training process. Movement skills depend on a broad array of functions and capabilities, which may make skill assessment and/or modeling difficult.

The systems disclosed herein may employ a movement skill model that may help identify skill deficiencies quantitatively. The model may also analyze the relationship of the skill deficiencies to the skill development process. This information may be used to synthesize feedback augmentations and/or determine training goals, which may be designed to induce changes in movement technique and guide training during performance. The components of this system may form a framework that allows planning and organizing training activity in a data-driven manner. This system may include a systematic and individualized approach to movement training suited to subjects' physical characteristics and health.

In one embodiment, a system for processing a variety of movement and performance data from an activity is provided. The system may extract movement elements that support task interactions. The system may classify movement elements according to type and outcomes. The system may decompose movement elements into segments associated with biomechanical and functional characteristics of movement.

In one embodiment, a system for assessing and diagnosing movement techniques of a subject is provided. The system may assess movement technique and outcome for movement classes. The system may identify development or learning stage of classes based on skill and outcome attributes.

In one embodiment, a system for synthesizing feedback appropriate for a subject's skill development stage is provided. The system may determine training goals based on performance criteria and learning stage of the subject. The system may synthesize feedback augmentation specific to a development stage to assist training towards training goals.

In one embodiment, a system for operationalizing or augmenting training of a subject is provided. The system may schedule training elements or training goals based on development stage, which may include intervals for memory consolidation. The system may track a subject's performance relative to training goals. The system may provide feedback on one or more of training elements, skill development, injury, physical wear, and fatigue. The system may track overall skill development. The system may update a training goal and/or a training schedule for the subject. The system may determine augmentation from a combination of feedback modalities that improve training effectiveness. The feedback modalities may include one or more of instructions, cues, and signals.

Additional examples and embodiments include an apparatus for movement skill training, the apparatus comprising: a sensor system comprising one or more sensors configured to obtain movement data for a subject performing an activity; a processor system in communication with the one or more sensors, the processor system having a microprocessor and memory configured to: receive the movement data from the one or more sensors, wherein the subject performs a primary movement unit associated with the activity; identify one or more movement patterns from the movement data, wherein the movement patterns are associated with the subject performing the primary movement unit; analyze the movement patterns to identify one or more skill attributes descriptive of the subject performing the primary movement unit; and assess the one or more skill attributes to specify one or more training goals for the subject, wherein the training goals are selected to address deficiencies in the skill attributes.

The apparatus, wherein the one or more sensors comprise one or more inertial sensors, accelerometers, gyroscopes, or inertial measurement units, and wherein the movement data comprise one or more of velocity, rotational velocity, acceleration or rotational acceleration descriptive of movement patterns. The apparatus, wherein the one or more sensors comprise a magnetometer configured to acquire direction or orientation data descriptive of the movement patterns, a transducer configured to acquire one or more of position, velocity, pressure, strain, or torque data descriptive of the movement patterns, an acoustic sensor configured to acquire acoustic wave data descripting of the movement patterns, a visual sensor or camera configured to acquire image data descriptive of the movement patterns, and a video sensor configured to acquire video data descriptive of the movement patterns.

The apparatus, where the one or more sensors are configured to obtain the movement data from one or both of the subject and an associated object used by the subject to perform the primary movement unit, the movement data selected from one or more of angle, angular velocity, direction, distance, force, linear acceleration, position, pressure, rotation, rotational speed, speed, strain, and torque. The apparatus, where the one or more sensors are further configured to obtain activity data descriptive of the subject performing the activity over a number of sessions distributed over a calendar period, the processor system being further configured to assess outcomes related to performance of the training goals over the calendar period, based on the activity data and the skill attributes. The apparatus, where the activity performed by the subject is selected from badminton, baseball, cricket, golf, rehabilitative exercises, running, skiing, snowboarding, surfing, surgery or other medical procedure, swimming, table tennis, tennis, and volleyball.

The apparatus, where the processing system is configures to extract the one or more skill attributes from the one or more movement patterns to define one or more skill elements, the skill elements characterizing movement patterns for the subject to form, movement patterns for the subject to consolidate, and movement patterns for the subject to optimize. The apparatus, where the processing system is configured to determine a skill status by applying criteria derived from the skill attributes, the skill status defining the movement patterns for the subject to form, movement patterns for the subject to consolidate, and movement patterns for the subject to optimize. The apparatus, where the processing system is configured to combine the skill elements with the skill status to generate a skill profile describing an overall skill and performance of the subject. The apparatus, where the processing system is configured to analyze the skill attributes taking into account the skill status to produce the training goals. The apparatus, where the processing system is configured for a user to select one or more of the training goals on the skill status and further configured to track and update the one or more training goals based on changes in the one or more skill elements. The apparatus, where the processing system is configured to derive one or more training elements from the skill elements, wherein a skill attribute associated with one or more of the one skill elements is assigned to one of the training goals. The apparatus, where the processing system is configured to generate a training schedule for the subject that comprises the training element and associated training goal.

The apparatus, where the processing system is configured to configure one or more of a communication system, a cueing system, and an apparatus system. The apparatus, where the communication system is configured to provide symbolic, verbal, or visual information. The apparatus, where the cueing system is configured to provide audible, visual, or haptic feedback.

The apparatus, wherein one of the training goals comprises a pattern to form, wherein the pattern is absent from the movement patterns in the collected data. The apparatus, where the training goal comprises developing the pattern to form from scratch or through modification of an existing movement of the subject.

The apparatus, where a one of the training goals comprises a pattern to consolidate, wherein the pattern in the collected data is not sufficiently defined in the collected data to allow reliable execution under dynamic conditions. The apparatus, where the training goal comprises refining the pattern or creating procedural memory to enable automatic or repeatable execution of the refined pattern by the subject.

The apparatus, where one of the training goals comprises a pattern to improve, wherein the pattern in the collected data does not achieve a desired outcome. The apparatus, where the training goal comprises refining a movement technique to use the least energy or to produce the least strain on a musculoskeletal system of the subject. The apparatus, where the collected data comprises population data.

Additional embodiments and examples include a method of training comprising: assessing movement skills of a subject; identifying deficiencies in the movement skills; specifying training goals for the subject; providing augmentation to the subject; and tracking a training process of the subject; wherein identifying deficiencies comprises relating the movement skills of the subject to population data; and wherein specifying training goals comprises using the population data to determine which movement skills can be improved by the subject to improve skill level and to produce long-term development.

The method, where producing long-term development comprises identifying which aspects of the movement skills can be improved and in what order. The method, where the training goals are associated with training elements, and a training list comprises a plurality of training elements. The method, where selecting the training element indicates to a tracking system which aspects of movement performance are to be monitored. The method, where selecting the training element indicates to an augmentation system which aspects of movement performance are to be actively cued.

The method, further comprising developing a training plan, wherein the plan describes the organization of a training session in terms of training elements and the training goals. The method, where the training elements are compiled in a training list arranged as a training schedule. The method, where the training schedule comprises at least one session, each session is divided into a plurality of sets, and each set is assigned at least one training goal.

Further embodiments and examples include a closed-loop system for data-driven training, the system comprising: an assessment loop configured to collect data from a movement performance by a user; a training loop configured to track progress in at least one skill of the movement performance; and an augmentation loop configured by the training loop to provide information to the user during the movement performance.

The system, where the collected data comprises one or more of movement data from a body segment of the user, movement data from equipment used by the user, physiological data of the user, outcome of the movement performance, and effect of the movement performance. The system, where the physiological data comprise electrical muscle activity collected from a surface or an implantable electrode. The system, where the system is configured to track at least one movement performance from a plurality of users. The system, where the system is configured to track interactions between the movement performances of the users.

The system, where the assessment loop comprises an extractor configured to extract motion elements from a target motion of the movement performance. The system, where the augmentation loop collects movement information from the user and provides motion elements to the extractor. The system, where a motion model is produced from an output from the extractor. The system, where skill assessment and diagnostics are performed on the motion model to produce a skill model. The system, where the skill model further comprises reference skill data.

The system, where session data is provided to the extractor and the motion model further comprises the session data. The system, where the augmentation loop comprises a movement process, a cueing system, and a feedback loop between the movement process and cueing system. The system, where an instruction module is configured to receive a set of target skills from the user. The system, where the instruction module processes the target skills and provides the processed target skills to the training loop.

The system, where the cueing system comprises a cue processor configured to translate movement data into a cue signal. The system, where the cue processor implements a finite state estimator comprising an approximation of a movement model of the user. The system, where the cue processor implements a cueing law calculator and the calculator operates on the finite state estimate and the collected data to calculate if a cue will be delivered. The system, where the cueing law calculator determines what the cue should communicate. The system, where a feedback synthesis model determines operation of the cueing law calculator.

The system, where the cueing system comprises a cue generator configured to translate cue signals into physical stimuli. The system, where the cue generator translates the cue signal into a feedback stimulus generated by a transducer. The system, where the feedback stimulus is selected from audio, visual, haptic, and symbolic. The system, where the cueing system operates in real time to provide feedback to the user during the movement performance. The system, where the augmentation loop provides feedback to a user that mimics human information processing hierarchy. The system, where the feedback comprises one or more of an instruction, a notification, a feedback cue, and a feedback cue signal.

The system, where the instruction is generated from at least one of a motion model, a skill model, and a diagnostic assessment. The system, where the assessment loop comprises an extractor configured to extract motion elements from a target motion of the movement performance and the motion model is produced from an output from the extractor. The system, where the skill model is produced from assessing the motion model. The system, where the diagnostic assessment comprises identifying deficiencies in the movement performance of the user. The system, where the instruction provides information about a training element and an associated training goal. The system, where the instruction organizes a training process. The system, where the instruction synthesizes one or more cueing laws that govern the augmentation loop.

The system, where the instruction is generated during a training session at an interval or after the session. The system, where the interval is upon completion of a training set. The system, where the instruction is presented verbally, symbolically, or graphically. The system, where the cue is provided in real time to the user. The system, where the cue targets specific movement characteristics to directly impact movement outcome or performance. The system, where the cue comprises a discrete audible, tactile, or visual signal.

The system, where the feedback cue signal is provided in real time to the user. The system, where the feedback cue signal guides a user's movement and enhances a movement feature. The system, where the feedback cue signal comprises a continuous or semi-continuous audible, tactile, or visual signal or stimulation of the user's muscles or nerves. The system, where the notification provides information about a user's progress towards a training goal. The system, where the notification is presented verbally, symbolically, or graphically.

The system, where the feedback further comprises an activity interaction provided by an apparatus. The system, where the apparatus comprises a ball machine or an assistive robotic device. The system, where the skill assessment loop is further configured to update information about the user's skills. The system, where information about the user's skills includes a motion model and a skill model. The system, where information about the user's skills includes a diagnostic tool for identifying deficiencies in a movement technique. The system, where the identified deficiencies are synthesized into training goals. The system, where the training loop is managed by a training agent, and the training agent is configured to identify training elements that can be activated as training goals. The system, wherein the training agent suggests training goals for a user and manages a user's training schedule.

This application is described with respect to certain embodiments. Equivalents can be substituted and changes can be made to adapt these systems and methods to other problems and applications, without departing from the scope of the invention as defined by the claims.

Claims

1. An apparatus for movement skill training, the apparatus comprising:

a sensor system comprising one or more sensors configured to obtain movement data for a subject performing an activity;

a processor system in communication with the one or more sensors, the processor system having a microprocessor and memory configured to: receive the movement data from the one or more sensors, wherein the subject performs a primary movement unit associated with the activity; identify one or more movement patterns from the movement data, wherein the movement patterns are associated with the subject performing the primary movement unit; analyze the movement patterns to identify one or more skill attributes descriptive of the subject performing the primary movement unit; and assess the one or more skill attributes to specify one or more training goals for the subject, wherein the training goals are selected to address deficiencies in the skill attributes.

2. The apparatus of claim 1, wherein the one or more sensors comprise one or more inertial sensors, accelerometers, gyroscopes, or inertial measurement units, and wherein the movement data comprise one or more of velocity, rotational velocity, acceleration or rotational acceleration descriptive of movement patterns.

3. The apparatus of claim 1, wherein the one or more sensors comprise one or more of:

a magnetometer configured to acquire direction or orientation data descriptive of the movement patterns,

a transducer configured to acquire one or more of position, velocity, pressure, strain, or torque data descriptive of the movement patterns,

an acoustic sensor configured to acquire acoustic wave data descripting descriptive of the movement patterns,

a visual sensor or camera configured to acquire image data descriptive of the movement patterns, and

a video sensor configured to acquire video data descriptive of the movement patterns.

4. The apparatus of claim 1, wherein the one or more sensors are configured to obtain the movement data from one or both of the subject and an associated object used by the subject to perform the primary movement unit, the movement data selected from one or more of angle, angular velocity, direction, distance, force, linear acceleration, position, pressure, rotation, rotational speed, and speed of the object, or from one or more of strain, pressure and torque on the object.

5. The apparatus of claim 1, wherein the one or more sensors are further configured to obtain activity data descriptive of the subject performing the activity over a number of sessions distributed over a calendar period, the processor system being further configured to assess outcomes related to performance of the training goals over the calendar period, based on the activity data and the skill attributes.

6. The apparatus of claim 1, wherein the activity performed by the subject is selected from badminton, baseball, cricket, golf, rehabilitative exercises, running, skiing, snowboarding, surfing, surgery or other medical procedure, swimming, table tennis, tennis, and volleyball.

7. (canceled)

8. The apparatus of claim 1, wherein the processing system is configured to extract the one or more skill attributes from the one or more movement patterns to define one or more skill elements, the skill elements characterizing one or more of movement patterns for the subject to form, movement patterns for the subject to consolidate, and movement patterns for the subject to optimize.

9. The apparatus of claim 8, wherein the processing system is configured to determine a skill status by applying criteria derived from the skill attributes, the skill status defining one or more of the movement patterns for the subject to form, the movement patterns for the subject to consolidate, and the movement patterns for the subject to optimize.

10. The apparatus of claim 9, wherein the processing system is configured to:

combine the skill elements with the skill status to generate a skill profile describing an overall skill and performance of the subject relative to population data for a population of such subjects; and;

analyze the skill attributes taking into account the skill status to produce the training goals using the population data to determine which of the movement skills to be improved.

11. (canceled)

12. The apparatus of claim 10, wherein the processing system is further configured for a user to select one or more of the training goals based on the skill status and further configured to track and update the one or more training goals based on changes in the one or more skill elements relative to the population data.

13. The apparatus of claim 8, wherein the processing system is configured to:

derive one or more training elements from the skill elements, wherein a skill attribute associated with one or more of the skill elements is assigned to one of the training goals; and

generate a training schedule for the subject, wherein the training schedule comprises the training elements and training goals to which the associated skill elements are assigned.

14. (canceled)

15. The apparatus of claim 1, further comprising:

a cueing system configured to provide audible, visual, or haptic feedback cues to the subject and

an apparatus system programmed to provide activity interactions concurrently with the subject performing the activity, to support development of specific movement patterns in conjunction with the cueing system.

16. The apparatus of claim 15, further comprising a communication system configured to provide symbolic, verbal, or visual instructions to the subject, wherein the apparatus system is programmed to work conjointly with the feedback cues and instructions to support the development of the specific movement patterns.

17. (canceled)

18. The apparatus of claim 8, wherein the training goals identify one of the movement patterns to form through modification of an existing movement pattern of the subject, wherein the movement pattern is absent from the movement patterns in the collected data due to lack of differentiation among such existing movement patterns.

19. (canceled)

20. The apparatus of claim 1, wherein the training goals identify one of the movement patterns to consolidate by creating procedural memory to enable repeatable execution of the movement pattern by the subject, wherein the movement pattern is not sufficiently defined in the collected data to allow reliable execution under dynamic conditions and the deficiencies manifest as variability in the movement pattern, lack of smoothness in the movement pattern, inefficient performance of the movement pattern, or insufficient flexibility in the movement pattern with changing conditions.

21. (canceled)

22. The apparatus of claim 1, wherein the training goals identify one of the movement patterns to optimize to minimize strain on a musculoskeletal system of the subject, wherein the pattern in the collected data does not achieve a desired outcome and the deficiencies result in excessive use of force when seeking the desired outcome.

23. (canceled)

24. The apparatus of claim 1, wherein the collected data comprises population data providing reference values for the skill attributes and the training goals for the subject.

25. A method of training comprising:

assessing movement skills of a subject performing a task;

identifying deficiencies in the movement skills of the subject relative to population data for a population of such subjects;

specifying training goals for the subject to address the deficiencies in the movement skills;

providing feedback augmentation selected to induce the subject to achieve the training goals, using the population data to determine which of the movements skills to be improved; and

tracking a training process of the subject wherein identifying the deficiencies comprises relating the movement skills of the subject to the population data and specifying the training goals comprises using the population data to determine which of the movement skills to improve for a skill level of the subject to attain a desired level of proficiency.

26. The method of claim 25, further comprising identifying aspects of the movement skills to be improved and in what order, wherein information extracted from the population data directs the training process to the aspects to focus on first to produce progress in the skill level.

27. (canceled)

28. The method of claim 30, further comprising selecting a training element from the training plan, wherein selecting the training element indicates to a monitoring system aspects of movement performance characterizing movement skills of the subject to be monitored and the monitoring system provides notification for achievement of the at least one training goal, or a fraction thereof.

29. The method of claim 30, wherein selecting the training element indicates to an augmentation system aspects of movement performance characterizing movement skills of the subject to be actively cued and the augmentation system provides real-time feedback cues concurrently with the movement performance, at discrete time periods during execution of the movement performance, or following an outcome of the movement performance.

30. The method of claim 25, further comprising developing a training plan, wherein the training plan describes organization of a training session in terms of training elements associated with the training goals;

wherein a plurality of the training elements are compiled in a training list arranged as a training schedule; and

wherein the training schedule comprises at least one such training session, each training session divided into a plurality of sets, and each set assigned at least one of the training goals.

31-32. (canceled)

33. A closed-loop system for data-driven training, the system comprising:

a sensor configured to collect one or more of movement data describing motion of a body of a user or equipment used by the user during performance of a movement, physiological data collected from muscle activity of the user during the performance, and data collected from an outcome or effect of the performance by the user;

a movement processor configured to execute: an assessment loop configured to collect the data from the sensor during the performance by the user; a training loop configured to track progress in at least one movement skill of the user during the performance; and an augmentation loop configured by the training loop to provide information to the user during the performance;

a cueing system comprising a cue processor configured to translate the movement data into a cue signal and a cue generator configured to translate the cue signal into a physical feedback stimulus generated by a transducer, wherein the feedback stimulus is selected from audio, visual, haptic, and symbolic; and

a feedback loop executed between the movement processor and cueing system, wherein the cueing system operates in real time to provide the feedback stimulus to the user during the performance.

34-35. (canceled)

36. The system of claim 33, wherein the system is configured to track at least one such performance from a plurality of such users, and wherein the system is further configured to track interactions between the performances.

37. (canceled)

38. The system of claim 33, further comprising an extractor configured to extract motion elements from a target motion of the performance;

wherein the augmentation loop collects the movement data from the user and provides motion elements to the extractor;

wherein session data are provided to the extractor and a motion model of the user is produced from output from the extractor, the motion model comprising the session data;

wherein skill assessment and diagnostics are performed on the motion model to produce a skill model, the skill model comprising reference skill data for the at least one movement skill.

39-44. (canceled)

45. The system of claim 33, further comprising an instruction module configured to receive a set of target skills from the user, wherein the instruction module processes the target skills and provides the processed target skills to the training loop.

46-47. (canceled)

48. The system of claim 38, wherein the cue processor implements a finite state estimate comprising an approximation of the motion model of the user;

wherein the cue processor implements a cueing law calculator that operates on the finite state estimate and the data collected from the sensor to determine a cue to be delivered;

wherein a feedback synthesis model determines operation of the cueing law calculator and the cueing law calculator determines what the cue should communicate to the user.

49-55. (canceled)

56. The system of claim 33, wherein the augmentation loop provides feedback to mimic a human information processing hierarchy of the user, wherein the feedback comprises one or more of an instruction, a notification, and a cue, or the physical feedback stimulus provided in real time to the user.

57. (canceled)

58. The system of claim 56, wherein the feedback comprises the instruction generated from at least one of:

a motion model, wherein the assessment loop comprises an extractor configured to extract motion elements from a target motion of the performance by the user and the motion model is produced from output of the extractor;

a skill model, wherein the skill model is produced from assessing the motion model; and

a diagnostic assessment, wherein the diagnostic assessment identifies deficiencies in the performance of the user.

59-61. (canceled)

62. The system of claim 58, wherein the instruction synthesizes one or more cueing laws that govern the augmentation loop and provides information about a training element and an associated training goal to organize a training process for the user;

wherein the instruction is generated at an interval during the training session, upon completing within a training set of the training session, or after the training session; and

wherein the instruction is presented to the user verbally, symbolically, or graphically.

63-67. (canceled)

68. The system of claim 56, wherein the feedback comprises the cue provided in real time to the user, and wherein the cue comprises a discrete audible, tactile, or visual signal selected to target specific movement characteristics of the user to impact the performance or the outcome of the performance.

69-70. (canceled)

71. The system of claim 56, wherein the physical feedback stimulus comprises an audible, tactile, or visual stimulation of a muscle or nerve of the user provided in real time to guide the movement of the user and enhance a feature thereof.

72-73. (canceled)

74. The system of claim 56, wherein the notification is presented verbally, symbolically, or graphically and provides information about progress of the user towards a training goal.

75. (canceled)

76. The system of claim 56, wherein the feedback further comprises an activity interaction provided by an apparatus concurrently with the user performing the movement to support development of a specific movement pattern in conjunction with the cueing system.

77. The system of claim 76, wherein the apparatus comprises:

a ball machine configured to throw balls with different trajectories selected for the subject to form a new stroke technique or adapt an existing stroke technique; or

an assistive robotic device or robot manipulator used to physically guide movements of the subject.

78. The system of claim 33, wherein the assessment loop is configured to update a diagnostic tool for identifying deficiencies in the performance of the movement by the user;

wherein the deficiencies are synthesized into training goals for the user; and

further comprising a training agent configured to identify training elements associated with the training goals, wherein the training agent suggests the training goals to manage a training schedule for the user.

79-83. (canceled)