INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND COMPUTER PROGRAM
An information processing device according to the present technology includes an action recognition unit that recognizes an operation action of a user based on sensor information, and an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
Latest SONY CORPORATION Patents:
- IMAGE PROCESSING APPARATUS AND METHOD
- Terminal device, information processing device, object identifying method, program, and object identifying system
- Drive circuit, electronic apparatus, and method of controlling drive circuit
- Overcurrent determination circuit and light emission control apparatus
- Non-zero random backoff procedure
The present disclosure relates to an information processing device that processes a user's action records, an information processing method, and a computer program.
BACKGROUND ARTA technology to recognize a user's operation action from sensor information acquired by using various sensing technologies is proposed. The recognized user's operation action is automatically recorded as an action log and can be represented by various techniques, for example, reproducing the operation action by animation such as an avatar, showing a user's movement locus on a map, or using an index abstracting various operation actions for representation.
CITATION LIST Patent Literature
- Patent Literature 1: JP 2008-3655A
However, when an action log is reproduced by animation such as an avatar using an action recording device like, for example, a motion capture, a very large-scale device will be needed. On the other hand, an action log generated by using a small sensor containing recording device such as a smartphone limits the types of action that can be recorded/recognized and thus, it is difficult to present an action record that is valuable to the user. Therefore, an action log is generally shown as a user's movement locus on a map or displayed as an action record converted to the amount of activity like a health index.
Therefore, a proposal of the representation technique to present an action log recorded by a small sensor containing recording device to the user in a manner that is easy to understand has been sought.
Solution to ProblemAccording to the present disclosure, there is provided an information processing device including an action recognition unit that recognizes an operation action of a user based on sensor information, and an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
According to the present disclosure, there is provided an information processing device including an action recognition unit that recognizes an operation action of a user based on sensor information, an action representation generation unit that generates an action segment constituting an action log from operation action data showing the operation action of the user recognized by the action recognition unit based on operation action estimation information that decides the operation action, and a feedback adjustment unit that corrects the operation action estimation information based on correction feedback from the user to the action segment generated by the action representation generation unit.
According to the present disclosure, there is provided an information processing method including a step for recognizing an operation action of a user based on sensor information, and a step for analyzing operation action data showing the recognized operation action of the user to generate an action segment represented by a meaning and content of the operation action from the operation action data.
According to the present disclosure, there is provided an information processing method including a step for recognizing an operation action of a user based on sensor information, a step for generating an action segment constituting an action log from operation action data showing the recognized operation action of the user based on operation action estimation information that decides the operation action, and a step for correcting the operation action estimation information based on correction feedback from the user to the action segment.
According to the present disclosure, there is provided a computer program for causing a computer to function as an information processing device including an action recognition unit that recognizes an operation action of a user based on sensor information, and an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
According to the present disclosure, there is provided a computer program for causing a computer to function as an information processing device including an action recognition unit that recognizes an operation action of a user based on sensor information, an action representation generation unit that generates an action segment constituting an action log from operation action data showing the operation action of the user recognized by the action recognition unit based on operation action estimation information that decides the operation action, and a feedback adjustment unit that corrects the operation action estimation information based on correction feedback from the user to the action segment generated by the action representation generation unit.
According to the present disclosure, operation action data showing a user's operation action recognized by an action recognition unit based on sensor information is analyzed by an action representation generation unit to generate an action segment represented by the meaning and content of the operation action from the operation action data. By displaying an action log with the action segment represented by the meaning and content of the operation action, information can be presented to the user in a manner that is easy to understand.
Advantageous Effects of InventionAccording to the present disclosure, as described above, a recorded action log can be presented to the user in a manner that is easy to understand.
Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the appended drawings. Note that, in this specification and the drawings, elements that have substantially the same function and structure are denoted with the same reference signs, and repeated explanation is omitted.
The description will be provided in the order shown below:
<1. Overview of Action Log Display System>
<2. Functional Configuration of Action Log Display System>
[2-1. Action Recording Device]
[2-2. Action Log Server]
[2-3. Analysis Server]
<3. Action Segment Generation Processing>
[3-1. Relationship between Operation Action and Meaning/Content thereof]
[3-2. Action Segment Generation Processing]
(Example 1: Generation of an action segment by the contextual analysis)
(Example 2: Generation of an action segment by combining the contextual analysis and the time width)
(Example 3: Generation of an action segment by combining the contextual analysis, the time width, and position changes)
[3-3. Action Recognition Processing]
[3-4. Processing Content of Action Representation Generation Unit]
<4. Action Recording and Display Application>
[4-1. Representation of Action Log Based on Action Segment]
[4-2. Browsing Action]
[4-3. Correcting Action]
[4-4. Combining Actions]
[4-5. Dividing Action]
[4-6. Representation of Action Segment Based on Segmentation Grain Size]
[4-7. Deleting Action]
[4-8. Posting Action]
[4-9. Action Log Acquisition Stop Processing]
[4-10. Updating Display Content]
<5. Reflection Processing of Correction Feedback>
[5-1. Properties of Correction Feedback]
[5-2. Action Recognition Processing]
[5-3. Reflection Processing of Correction Feedback]
(5-3-1. Overview of reflection processing of correction feedback)
(5-3-2. Reflection processing of correction feedback of an action)
(5-3-3. Reflection processing of correction feedback of an action and position information)
<6. Others>
[6-1. Personal Modeling of Action Pattern]
[6-2 Position Display Technique by Moving Medium/Means Determination]
(6-2-1. Line estimation processing)
(6-2-2. Station name selection processing)
<7. Exemplary Hardware Configuration>
<1. Overview of Action Log Display System>
First, an overview of an action log display system according to an embodiment of the present disclosure will be provided with reference to
The action log display system according to the present embodiment realizes a representation technique that presents an action log recorded by a recording device 100 containing a small sensor (hereinafter, referred to as an “action recording device”) to the user in a manner that is easy to understand. As the action recording device 100, for example, a mobile terminal such as a mobile phone, PDA (Personal Digital Assistant), and smartphone can be used. The action recording device 100 is provided with at least one sensor to sense conditions or an action of a user holding the device. The action recording device 100 estimates an operation action of the user based on sensor information acquired by the sensor and transmits the operation action to an action log server 200 as an action log. In this manner, the action log of the user is accumulated in the action log server 200.
An action log analyzed by the action recording device 100 and stored in the action log server 200 records an operation like, for example, “meal”, “movement”, and “sleep” together with the action time, position information and the like. An action log display system according to the present embodiment further analyzes an action log representing the operation content by an analysis server 300 to recognize the meaning of action and generates information (action segment) to which the meaning of action is added. The action segment is unit information as an easy-to-understand representation for the user of an action log. Instead of simply presenting an action log to the user, the action segment can present an action log in a manner that conveys the meaning of action.
An action log analyzed by the analysis server 300 and presented to the user can be corrected by the user. In addition, data of the presented action log can be combined, divided, or deleted by generating an action segment. The presented action log can also be posted to a posting site. Thus, by using an action log display system according to the present embodiment, an action log acquired as an operation can be analyzed and presented to the user in an easy-to-understand manner. The configuration and function of an action log display system according to the present embodiment will be described in detail below.
<2. Functional Configuration of Action Log Display System>
[2-1. Action Recording Device]
The action recording device 100 includes sensors 110, an action recognition unit 120, a client interface unit 130, an action representation processing unit 140, a display unit 150, and an input unit 160.
The sensors 110 are devices that sense a user's action or conditions and are installed in the action recording device 100. As the sensors 110, for example, an acceleration sensor, gyro sensor, magnetic field sensor, atmospheric pressure sensor, illuminance sensor, temperature sensor, microphone and the like can be used. As the sensors 110, a latitude/longitude acquisition sensor that acquires the latitude/longitude can also be installed. As the latitude/longitude acquisition sensor, for example, not only GPS (Global Positioning System) or WiFi, but also base station information of other communication networks or information such as RFID and images may be used. The sensors 110 output detected information to the action recognition unit 120 as sensor information.
The action recognition unit 120 estimates a user's action based on sensor information. The action recognition unit 120 includes a sensor controller 122 and an operation action recognition unit 124. The sensor controller 122 controls the sensor 110, the CPU or the overall system to cause sensing by the sensor 110 to operate effectively. The sensor controller 122 controls the above devices based on recognition results by the sensor 110 or the operation action recognition unit 124.
The operation action recognition unit 124 recognizes a user's action or conditions by performing signal processing or statistical processing of sensor information. The action recording device 100 holds a correspondence between an action model as information about a user's action obtained as a result of processing sensor information and an operation action in advance. When action parameters are obtained by processing sensor information, the operation action recognition unit 124 identifies an operation action corresponding to the parameters. Then, the operation action recognition unit 124 associates the identified operation action and the action time period, action time, position information and the like and outputs the associated information to the client interface unit 130 as operation action data. The operation action data is uploaded from the client interface unit 130 to the action log server 200.
The client interface unit 130 transmits/receives information between the action recording device 100, and the action log server 200 and the analysis server 300. For example, the client interface unit 130 transmits operation action data input from the action recognition unit 120 to the action log server 200 or outputs an analysis result received from the analysis server 300 to the action representation processing unit 140. Also, the client interface unit 130 transmits feedback information from the user input through the input unit 160 to the analysis server 300.
The action representation processing unit 140 is a functional unit that displays an action log or processes feedback information from the user and includes a display processing unit 142 and an input information processing unit 144. The display processing unit 142 performs processing to display an analysis result by the analysis server 300 input from the client interface unit 130 in the display unit 150. The input information processing unit 144 performs processing to transmit feedback information from the user for an action log input from the input unit 160 to the analysis server 300 via the client interface unit 130.
The display unit 150 is an output device that displays information and can be configured by, for example, a liquid crystal display, organic EL display or the like. For example, an action log processed for display by the display processing unit 142 is displayed in the display unit 150.
The input unit 160 is an input device to input information and, for example, a touch panel, keyboard, hardware button or the like can be used. In the present embodiment, it is assumed that the display surface of the display unit 150 is provided with a touch panel as the input unit 160. In this case, the user can input information by, for example, bringing an operation body such as a finger or touch pen into contact with the display surface of the display unit 150 or moving the operation body brought into contact with the display surface. Information input from the input unit 160 is output to the input information processing unit 144.
[2-2. Action Log Server]
The action log server 200 includes a log server interface unit 210 and an action log DB 220.
The log server interface unit 210 transmits/receives information between the action log server 200, and the action recording device 100 and the analysis server 300. For example, the log server interface unit 210 records operation action data received from the action recording device 100 in the action log DB 220 or acquires operation action data in accordance with a transmission request from the analysis server 300 from the action log DB 220 and transmits the operation action data to the analysis server 300.
The action log DB 220 is a storage unit that stores operation action data of the user acquired by the action recording device 100. In operation action data stored in the action log DB 220, as described above, the operation action identified by the operation action recognition unit 124 and the action time period, action time, position information and the like are associated and stored in the action log DB 220 in, for example, chronological order.
[2-3. Analysis Server]
The analysis server 300 includes an analysis server interface unit 310, an action representation generation unit 320, and a data management unit 330.
The analysis server interface unit 310 transmits/receives information between the analysis server 300, and the action recording device 100 and the action log server 200. For example, the analysis server interface unit 310 receives an analysis instruction (analysis request) of an action log from the action recording device 100 or transmits a transmission request of necessary operation action data in accordance with an analysis request The analysis server interface unit 310 also receives feedback information from the user of an action log from the action recording device 100.
The action representation generation unit 320 analyzes operation action data to understand the meaning thereof and generates an action segment to which the meaning and content is added. The action representation generation unit 320 includes a living action recognition unit 321 and a hierarchical structure judgment unit 322. The living action recognition unit 321 generates an action segment from an action log including operation action data. The living action recognition unit 321 analyses the meaning and content of operation action data arranged in chronological order based on relationships between data and the time period, time and the like of data. Then, the living action recognition unit 321 selects data classified as the most detailed meaning and content of analyzed meaning and content as the action segment. The generated action segment is output to the data management unit 330 and held there.
The hierarchical structure judgment unit 322 judges a hierarchical structure about the meaning and content of an action segment generated by the living action recognition unit 321 and attaches hierarchical information representing a hierarchical relationship of the meaning and content to the action segment. Hierarchical information is hierarchical meaning information attached to an action segment by processing described later. Hierarchical information may be, for example, information using a normalized value as a key or information using ID identifying the level of meaning information as a direct key. An action segment to which hierarchical information is attached is also called a hierarchical information attached action segment. Hierarchical relationships of the meaning and content of action are stored in the data management unit 330. The hierarchical structure judgment unit 322 outputs a hierarchical information attached action segment to the data management unit 330 via the living action recognition unit 321. The function of the action representation generation unit 320 and details of processing content thereby will be described later.
The data management unit 330 manages an action segment generated by the action representation generation unit 320. The data management unit 330 includes a data acquisition unit 331, a feedback adjustment unit 332, an analysis parameter DB 333, a unit data storage DB 334, and a hierarchical information attached data storage DB 335.
The data acquisition unit 331 transmits/receives data to/from the action representation generation unit 320. The data acquisition unit 331 records an action segment transmitted from the action representation generation unit 320 in the unit data storage DB 334 or records a hierarchical information attached action segment in the hierarchical information attached data storage DB 335. The data acquisition unit 331 acquires the specified action segment in accordance with a request from the action representation generation unit 320 from the unit data storage DB 334 or the hierarchical information attached data storage DB 335 and outputs the action segment to the action representation generation unit 320.
The feedback adjustment unit 332 reflects feedback information received from the action recording device 100 in analysis parameters used for analyzing the meaning and content of operation action data. The feedback information represents content of processing such as corrections made by the user on an action log displayed in the display unit 150 of the action recording device 100. The feedback adjustment unit 332 corrects analysis parameters using feedback information so that the meaning and content of a user's action can be recognized more correctly.
The analysis parameter DB 333 is a storage unit that holds analysis parameters used for analyzing the meaning and content of operation action data. In the analysis parameter DB 333, for example, a correspondence between an operation action and the meaning and content is stored as analysis parameters. Information stored in the analysis parameter DB 333 can be referenced by both of the living action recognition unit 321 and the hierarchical structure judgment unit 322. Analysis parameters are updated when necessary based on feedback information from the user.
The unit data storage DB 334 stores an action segment generated by the action representation generation unit 320. The action segment stored in the unit data storage DB 334 is a segment (unit segment) of the minimum unit necessary for recognition.
The hierarchical information attached data storage DB 335 stores an action segment in which hierarchical information is attached to an action segment generated by the action representation generation unit 320. The action segment stored in the hierarchical information attached data storage DB 335 is a hierarchical information attached action segment to which hierarchical information representing a hierarchical structure of the meaning and content of action is attached by the hierarchical structure judgment unit 322. The recording timing of an action segment to which hierarchical information is attached may be, for example, when requested by an application or analysis results of a plurality of segmentation grain sizes may be recorded in advance by the action representation generation unit 320.
That is, the action representation generation unit 320 and the data management unit 330 function as information processing devices that analyze the meaning and content of operation action data generated by the action recording device 100 to present information that is easy for the user to understand.
<3. Action Segment Generation Processing>
In an action log display system according to the present embodiment, the meaning and content of an operation action generated by the action recording device 100 is analyzed by the analysis server 300 to generate an action segment based on the meaning and content of action. Hierarchical information about the meaning and content of action can also be attached to an action segment and the display form of an action log can also be changed easily based on the hierarchical information. First, generation processing of an action segment will be described based on
[3-1. Relationship Between Operation Action and Meaning/Content Thereof]
The action recording device 100 analyzes an operation action, for example, “meal”, “movement”, or “sleep”. The analysis server 300 analyzes content of each operation action more deeply using operation action data containing the operation action. The analysis of the meaning and content of the operation action is conducted by using, for example, as shown in
If, among the operation actions of “meal”, “movement”, and “sleep”, the operation action of “movement” is taken up, As shown in
Then, a further action of “walking” can be associated with a “walking” action or “changing trains” action and a further action of “stopping” can be associated with a “halting” action, a “waiting for means of transport” state, or a “train stop” state. A “train” as a means of transport can further be associated with a state of “moving by train”.
Thus, action meta information at an operation action level could change to, as shown in
In the present embodiment, the ontology/semantic technology is applied to the recognition of the meaning and content of an operation action to segment the action recognition that judges “context” in an “action” and operation action data. Ontology systematically represents the concept of relations between words and in the present embodiment, for example, as shown in
[3-2. Action Segment Generation Processing]
Example 1 Generation of an Action Segment by the Contextual AnalysisAs a concrete example of generation processing of an action segment using the relationship between the operation action and the meaning and content, generation processing of an action segment by the contextual analysis will be described based on
As shown in
For example, a state of short “stopped” of a predetermined time or shorter between operation action data of “got on a train” is estimated to be a “train stopped (at a station)” state. A state of short “walked” of a predetermined time or shorter between operation action data of “got on a train” is estimated to be a “changing trains” action. Further, “stopped” of operation action data immediately before operation action data of “got on a train” is estimated to be a state of “waited for a train”.
By using the action time of operation action data, the action can be estimated more appropriately. Regarding an action of “movement by train”, for example, the meaning of action of “going to office” or “going to school” can be estimated if the action time is a morning hour (for example, from 6 am to 10 am) and the meaning of action of “going home” can be estimated if the action time is an evening hour (for example, from 5 pm to 8 pm). Similarly, regarding an action of “meal”, the meaning of action of “breakfast” can be estimated if the action time is a morning hour, “lunch” can be estimated if the action time is an hour around noon, “supper” can be estimated if the action time is an evening hour.
Thus, an action log including action segments as shown in
Then, by acquiring hierarchical action meta information at an operation action level from the context level dictionary shown in
For example, as shown in
By changing the segmentation grain size based on hierarchical action meta information at the operation action level in this manner, the action log can be displayed at an operation action level that is easy for the user to view.
An action segment concerning movement is described with reference to
In the example shown in
Next, a concrete example when an action segment is generated by considering, in addition to the contextual analysis, the time width will be described based on
As shown in
When action segments are generated, like in the above case, the display of the action log including the action segments can easily be changed by changing the segmentation grain size. For example, in a state of a grain size 2-1 reached by increasing the segmentation grain size from the segmentation grain size of the unit segment, a series of operations of “desk work”, “short walking”, “advance arrangements”, “short walking”, and “desk work” are represented as one action segment of “worked”. In this case, “short walking” is combined into one action segment of “worked” and thus, the action segment of “movement between premises” may be displayed simply as “movement”.
In a state of a grain size 2-2 reached by further increasing the segmentation grain size, a series of operations of “worked”, “meal”, “worked”, “movement”, and “worked” are represented as one action segment of “was in company”. By changing the segmentation grain size based on hierarchical action meta information at the operation action level in this manner, the action log can be displayed at an operation action level that is easy for the user to view.
Example 3 Generation of an Action Segment by Combining the Contextual Analysis, the Time Width, and Position ChangesNext, a concrete example when an action segment is generated by considering, in addition to the contextual analysis and time width, position changes will be described based on
As shown in
If, for example, the operation actions before and after the operation action data of “walked” are “did shopping” and a movement time t is t1 (for example, 35 s) or more and position changes of the action recording device 100 carried by the user are measured, the user is estimated to “move between shops”. Also, if, for example, the operation actions before and after the operation action data of “walked” are “did shopping” and the movement time t is t2 (for example, 20 s) or more and shorter than t1, and no position change of the action recording device 100 is measured, the user is estimated to “move between floors” during shopping. Further, if, for example, the operation actions before and after the operation action data of “walked” are “did shopping” and the movement time t is t3 (for example, 5 s) or more and shorter than t2, and no position change of the action recording device 100 is measured, the user is estimated to “move in a shop” during shopping.
Thus, if the meaning and content of operation action data is recognized by combining the contextual analysis, time width, and position changes, as shown in
When action segments are generated, like in the above case, the display of the action log including the action segments can easily be changed by changing the segmentation grain size. For example, in a state of a grain size 3-1 reached by increasing the segmentation grain size from the segmentation grain size of the unit segment, among action segments concerning walking, the action segment SG1 with the shortest walking time is combined with the action segments of “did shopping” preceding and succeeding the action segment SG1. These action segments are represented by an action segment as a series of operations of “did shopping”. In this case, “movement in a shop” is combined into one action segment of “did shopping”, other action segments concerning walking may be displayed simply as “movement”.
In a state of a grain size 3-2 reached by further increasing the segmentation grain size, among action segments concerning walking, the action segment SG2 with the shortest walking time next to the action segment SG1 is combined with the action segments of “did shopping” preceding and succeeding the action segment SG2. Then, in a state of a grain size 3-3 reached by further increasing the segmentation grain size, among action segments concerning walking, the action segment SG3 with the longest walking time is also combined with the action segments of “did shopping” preceding and succeeding the action segment SG3. Accordingly, a series of operations of “did shopping” and “walked” are represented as one action segment. By changing the segmentation grain size based on hierarchical action meta information at the operation action level in this manner, the action log can be displayed at an operation action level that is easy for the user to view.
[3-3. Action Recognition Processing]
Processing to generate an action segment from operation action data will be described in detail based on
Action recognition processing includes, as shown in
The operation action recognition unit 124 of the action recognition unit 120 having acquired sensor information from the sensors 110 starts creation processing of operation action data (S100). The operation action data creation processing can be performed by using an existing technique. After creating operation action data, the operation action recognition unit 124 outputs the operation action data to the action log server 200 (S110). In this manner, operation action data constituting an action log of the user is accumulated in the action log server 200. Incidentally, the action recognition unit 120 may generate, as operation action data, not only action information at the operation level, but also information including, for example, time information, location information, an operation history of devices and the like.
When operation action data is created, the action representation generation unit 320 of the analysis server 300 analyzes the meaning and content of the operation action data through the living action recognition unit 321 (S120). The living action recognition unit 321 segments the operation action data into data of a preset unit length and attaches living action meta information to each piece of segmented data. The unit length of the operation action data is defined by a predetermined time T (for example, T=1 min). The segmentation order of the operation action data is set as i (i=1 to N).
After segmenting the operation action data into the unit time T in chronological order, the living action recognition unit 321 first determines whether an integrated value of the unit length (T) and the parameter i is smaller than the length (time) of the operation action data (S121). If it is determined in step S121 that the integrated value of the unit length (T) and the parameter i is smaller than the length (time) of the operation action data, the living action recognition unit 321 attaches living action meta information to the segmented data between time T*i and time T*(i+1) (step S122). The symbol “*” indicates integration processing. In step S122, the meaning and content (living action meta information) applicable to the segmented data at the time can be attached by using, for example, ruled-based branching processing. Alternatively, living action meta information can also be attached the segmented data using machine learning such as the Hidden Markov Model (HMM) or Neural Network. The number of pieces of living action meta information attached to the segmented data is not limited to one and a plurality of pieces of living action meta information may be attached.
When living action meta information is attached to the segmented data of the operation action data in step S122, the living action recognition unit 321 adds 1 to the parameter i (S123) to repeat the processing from step S121. If it is determined in step S121 that the integrated value of the unit length (T) and the parameter i is equal to or greater than the length (time) of the operation action data, the living action recognition unit 321 outputs each piece of segmented data to which living action meta information is attached by the processing in step S122 as living action data (S124). An output result of the living action recognition unit 321 may be recorded in a predetermined storage unit (not shown) or may be output directly to the functional unit (in this case, the hierarchical structure judgment unit 322) that performs the next processing.
To return to the description of
If none of these conditions is satisfied in step S131, that is, the highest hierarchical information is already attached, the hierarchical structure judgment unit 322 terminates the processing shown in
Next, if the chronological arrangement order of segments generated in step S132 is set as j (j=1 to M), the hierarchical structure judgment unit 322 determines whether j is smaller than the number of segments generated in step S131 and dictionary information about the action of the j-th segment can be acquired (S134). If both of these conditions are satisfied in step S134, the hierarchical structure judgment unit 322 selects and attaches dictionary information optimum to the relevant segment of the acquired dictionary information (S135). Then, the hierarchical structure judgment unit 322 determines whether still higher hierarchical information can be attached to the segment (j) based on the selected dictionary information and temporal context (S136). If it is determined in step S136 that higher hierarchical information can be attached, the hierarchical structure judgment unit 322 attaches higher hierarchical information to the segment (j) and adds 1 to the parameter j (S137). Then, the processing in step S134 and thereafter is repeated.
On the other hand, if it is determined in step S134 that j is equal to or greater than the number of segments generated in step S132 or dictionary information about the action of the j-th segment cannot be acquired, the hierarchical structure judgment unit 322 repeats the processing in step S131 and thereafter. Also when it is determined in step S136 that higher hierarchical information cannot be attached to the segment (j), the hierarchical structure judgment unit 322 repeats the processing in step S131 and thereafter.
As shown in
Then, the hierarchical structure judgment unit 322 sets the parameter k (k=1 to K) representing the chronological order of segments generated by merge processing to the initial value 1 (S206) and determines whether the action time of the segment (k) is shorter than a predetermined time T1 (for example, T1=3 min) (S208). If the action time of the segment (k) is shorter than the predetermined time T1 in step S208, the hierarchical structure judgment unit 322 accumulates the segment in a buffer (S210). On the other hand, if the action time of the segment (k) is determined to be equal to or longer than the predetermined time T1 in step S208, the hierarchical structure judgment unit 322 further determines whether the action time of the segment (k) is shorter a predetermined time T2 (T2>T1; for example, T2=10 min) (S212).
If the action time of the segment (k) is determined to be shorter than the predetermined time T2 in step S212, the hierarchical structure judgment unit 322 merges the segment (k) into the action immediately before (S214). On the other hand, if the action time of the segment (k) is equal to or longer than the predetermined time T2 in step S212, the hierarchical structure judgment unit 322 decides the operation action of the segment as “another action” (S216). Then, the hierarchical structure judgment unit 322 determines whether the processing of steps S208 to S216 has been performed for all segments (S218) and if there is any unprocessed segment, the hierarchical structure judgment unit 322 adds 1 to k (S220) and then repeats the processing in step S208 and thereafter,
On the other hand, if the processing of steps S208 to S216 has been performed for all segments, as shown in
Next, the hierarchical structure judgment unit 322 determines whether action content of the segment is “walking” (S226) and, if the action content is other than “walking”, accumulates the segment in the buffer (S228). On the other hand, if the action content of the segment is “walking”, the hierarchical structure judgment unit 322 determines whether any vehicle action is accumulated in the buffer (S230). If a vehicle action is accumulated in the buffer, the hierarchical structure judgment unit 322 sets the operation action of the segment as an action of vehicle with the maximum share from “walking” (S323). On the other hand, if no vehicle action is accumulated in the buffer, If a vehicle action is accumulated in the buffer, the hierarchical structure judgment unit 322 sets the operation action of the segment as “another action” (S324).
Here, hierarchical information added to an action segment will be described based on
An action segment holds hierarchical information by being attached to the action segment combining unit segments or by being attached to the unit segments. When hierarchical information is attached to an action segment combining unit segments, it is assumed that, for example, an action segment SG17 of the action B in
On the other hand, when hierarchical information is attached to unit segments as action segments of the minimum unit, it is assumed that, for example, an action segment SG01 in
Hierarchical information may be attached in any form and can also be attached in other forms. The case of attaching hierarchical information to an action segment combining unit segments is superior in terms of the amount of data and the case of attaching hierarchical information to unit segments is superior in terms of a database search.
Returning to the description of
[3-4. Processing Content of Action Representation Generation Unit]
The analysis server 300 can accumulate an action log by action segments in real time and at the same time, can generate an action representation based on the meaning and content of an operation action. The analysis server 300 can also generate an action representation from a past action history. A detailed configuration of the action representation generation unit 320 and the data management unit 330 of the analysis server 300 is shown in
As shown in
The hierarchical processing unit 323 performs subsequent processing of a judgment result of the hierarchical structure judgment unit 322. The hierarchical processing unit 323 functions based on a hierarchical structure when only a portion of data to be attached to action segments is recorded in the storage unit for slimming down or speedup of data or hierarchical information of the specified action segment is delivered to an application.
As described above, hierarchical information may be attached to an action segment combining unit segments or to the unit segments. When hierarchical information is attached to a combined action segment, the hierarchical processing unit 323 processes the action segment of the hierarchical information selected by the user via the input unit 160. On the other hand, when hierarchical information is attached to unit segments, the hierarchical processing unit 323 generates an action segment by combining unit segments based on the hierarchical information selected by the user via the input unit 160. The hierarchical processing unit 323 a processing result of information to the registration processing unit 324 and the comment creation unit 325.
The registration processing unit 324 records the action segment generated by the hierarchical processing unit 323 in the data management unit 330. The registration processing unit 324 outputs an action segment to the data acquisition unit 331 to record the action segment in the hierarchical information attached data storage DB 335.
The comment creation unit 325 creates and attaches a comment such as the meaning and content of an action to a generated action segment. A comment created by the comment creation unit 325 is output to the data acquisition unit 331. The data acquisition unit 331 associates the comment with the corresponding action segment and records the comment in, for example, the hierarchical information attached data storage DB 335.
The acquisition unit 326 acquires a predetermined action segment from the unit data storage DB 334 or the hierarchical information attached data storage DB 335. When, for example, processing that needs to use a past action log is performed by the action representation generation unit 320, the acquisition unit 326 past data recorded in the unit data storage DB 334 or the hierarchical information attached data storage DB 335. Data to be acquired is decided based on instructions from the user.
<4. Action Recording and Display Application>
As described above, by analyzing operation action data acquired by the action recording device 100 by the analysis server 300, an action segment to which the meaning and content of an operation action is attached is generated. An application function to represent an action log of the user using action segments will be described below.
[4-1. Representation of Action Log Based on Action Segment]
First, the representation of an action log using action segments will be described. An example of the action log is shown in
An action log includes action segments arranged in chronological order. For each action segment, for example, the start time and the end time of the operation action and operation content are displayed. When the operation content is a movement action like “movement by train”, a position representation like, for example, from the start location to the goal (for example, “from Gotanda to Ohsaki”) is added to the operation content. When the operation content is other than a movement action like “work” and “meal”, the location (for example, “in Ohsaki”) where the operation is performed is added to operation content.
Further, to notify the user of the operation of such an action segment in an easy-to-understand manner, an object of operation content may be displayed or an object showing the feeling of the user when performing the operation may also be displayed. Also, the number of steps (step value) for the user to perform the operation or a value (exercise value) indicating energy consumption by the operation may be displayed. The content displayed in each action segment constituting an action log is not limited to the example of
In the action log shown in
As the non-display filter, for example, a filter that prevents the display when the action time is short or a filter that prevents the display of an action segment judged to be unimportant to the use can be considered. Also, a filter that prevents the display when the precision of recognition is low or a filter that allows the display of an action or location specified by the user may be set.
[4-2. Browsing Action]
When an action log display application is activated in a browsing terminal (for example, the action recording device 100), for example, the user can browse the action log of the user in predetermined units, for example, in units of days.
The action log 410 includes action segments 412 arranged in chronological order, for example, from the upper end of the screen toward the lower end. In each of the action segments 412, as described in
By touching a Prev button 422 to display the action log of the previous day of the action log currently displayed or a Next button 424 to display the action log of the next day of the action log currently displayed, the display unit 150 can be caused to display an action log of another day. If the action log of the previous day is not present when the Prev button 422 is pressed, the display unit 150 may be caused to display an action log of the day when an action log is acquired next by further going back to the past. Similarly, if the action log of the next day is not present when the Next button 424 is pressed, the display unit 150 may be caused to display an action log of the day when an action log is acquired next by further moving to the present. Operation buttons 430 for browsing, editing and other operations of the action log 410 are displayed in the display unit 150 and the user can touch the button corresponding to desired processing to perform the processing.
If, for example, a calendar button 434 is touched, as shown in
The display of the calendar 440 is changed by a previous (<) button 442 or a next (>) button 444. If the previous (<) button 442 is operated, the calendar of the previous month is displayed and if the next (>) button 444 is operated, the calendar of the next month is displayed. If no action log of the previous month is present when the previous (<) button 442 is pressed, the calendar 440 of a month when any action log is acquired next may be displayed by further going back to the past. Similarly, if no action log of the next month is present when the next (>) button 444 is pressed, the calendar 440 of a month when any action log is acquired next may be displayed by further moving to the present.
If, for example, a map button 431 of the operation buttons 430 is touched, the action log display application activates a map 450 to display position information corresponding to the action log 410 in the map 450.
If, when the action log 410 is displayed, the map button 431 is touched while none of the action segments 412 constituting the action log 410 is selected, for example, a history of all position information of the action log 410 of the day is displayed on the map 450. If the user is on the move, a movement locus thereof is displayed on the screen.
On the other hand, if, when the action log 410 is displayed, the map button 431 is touched while one action segment 412a is selected from the action log 410, a history of position information of the action segment 412a is displayed on the map 450. If, for example, as shown on the left of
If the user is not on the move, an icon or the like may be displayed in a location where the operation is performed. In
[4-3. Correcting Action]
In the foregoing, the method of displaying the action log 410 in the display unit 150 using the action segments 412 analyzed and generated by the analysis server 300 has been described. However, display content of the generated action log 410 may be erroneous. In such a case, the user can correct content of the action log 410. The correction feedback is reflected in action recognition determination processing. First, the method of correcting the action log 410 will be described based on
To correct content of the action segment 412, the user selects the action segment 412a to be corrected from the action log 410 displayed in the display unit 150 and touches an edit button 435. Then, as shown on the right of
The operation content can be corrected in an operation content correction area 461 of the correction screen 460. If, for example, the operation content correction area 461 is selected, as shown in
After selecting operation content from the operation content candidate list 461a, the user continues to correct the start location and end location of the operation. At this point, if the selected operation content is a movement action like, for example, “movement by bus”, corrections of a start location correction area 462, a start location description correction area 463, an end location correction area 464, or an end location description correction area 465 can be made.
A location name list may be displayed for the start location correction area 462 and the end location correction area 464 so that the user can select and input the location name or the user may be enabled to directly input the location name. In the location name list, for example, location names to be a landmark such as a building name, station name, or shop name may be displayed. If there is no location to be a landmark, place names (addresses) may be displayed in the location name list.
In addition the display unit 150 may be caused to display a start location map 462a and an end location map 464a that display a map by being linked to input content of the start location correction area 462 and the end location correction area 464. The start location map 462a and the end location map 464a can be caused to display a map of any location by a scroll operation on the map. When a touch operation is performed on a map displayed on the start location map 462a or the end location map 464a, the location name corresponding to the position where the touch operation is performed may automatically be input into the start location correction area 462 or the end location correction area 464.
The start location description correction area 463 and the end location description correction area 465 are areas where what kind of location the location input into the start location correction area 462 and the end location correction area 464 is for the user is input respectively. When the start location description correction area 463 or the end location description correction area 465 is touched, for example, as shown in
As the description content of a location, for example, “location to go back to” like the home, “location to work” like a company, and “location to learn” like a school can be cited. By inputting the description of such a location, what king of meaning the location has for the user can be grasped and a contribution can be made to improve the precision of action recognition for the user. If no correct description is found in the description candidate list 463a or 465a, a description may directly be input into the start location description correction area 463 or the end location description correction area 465.
When the operation content selected from the operation content candidate list 461a in
An object indicating the feeling of the user when an operation is performed can be corrected by, for example, as shown in
When all corrections are completed, correction content can be reflected in the action segment 412a by pressing a save button 468a at the bottom or a save button 468b at the top of the correction screen 460. When the save button 468a or 468b is pressed, a transition to the screen before the transition to the correction screen 460 occurs. When a transition to the screen before the transition to the correction screen 460 should occur without reflecting input content in the correction screen 460, a cancel button 467a at the bottom or a cancel button 467b at the top of the correction screen 460 may be pressed.
[4-4. Combining Actions]
According to the present technology, the display of the action log 410 can easily be changed not only by correcting content of each of the action segments 412, but also by correcting relationships between the action segments 412. For example, a plurality of the action segments 412 may be combined to display the resultant segment as the one action segment 412. The combination of the action segments 412 is a function to combine the plurality of the action segments 412 that are temporally consecutive into the one action segment 412. The time range of the combined action segment 412 extends from the start time of the oldest action segment 412 to the end time of the newest action segment 412.
The action of an action segment 412b selected first among the action segments 412 to be combined can be set as operation content after the combination. In
In the example of
For example, the operation content of any action segment to be combined may be set as the operation content of the action segment after the combination. If, for example, the operation content of “walked in Ohsaki” is selected, the operation content of an action segment 412d1 after the combination becomes “walked in Ohsaki”.
The operation content of the action segment after the combination may be decided, for example, by majority of action segments to be combined. In the example of
Alternatively, the operation content of the action segment after the combination may be decided, for example, by reanalyzing action segments to be combined. In the example of
Therefore, the action segments 412 can easily be combined by selecting the action segments to be combined.
[4-5. Dividing Action]
Also according to the present technology, for example, a plurality of the action segments 412 can be divided and displayed as a plurality of the action segments 412 as a correction of the relationship between the action segments 412. The division of the action segment 412 is a function to segment the one action segment 412 into a plurality of the action segments 412. As the division method of the action segment 412, for example, a method of setting the time to divide the action segment 412 and a division method using hierarchical information are known.
For example,
For example, it is assumed that an action segment 412e is selected for division in
Also, for example,
If, for example, an action segment 414a is selected in
[4-6. Representation of Action Segment Based on Segmentation Grain Size]
In the present technology, the action segments 412 constituting the action log 410 hold a hierarchical relationship based on the meaning and content thereof as hierarchical information. The display roughness of the displayed action log 410 can be changed by changing the segmentation grain size using the hierarchical information. The display roughness can be changed by using, for example, a slider or a zoom button.
Thus, the display roughness of the action log 410 can easily be changed based on the segmentation grain size attached to the action segment 412 so that the user can view the action log 410 in the desired display roughness.
The display roughness of the action log 410 is changed in
When, for example, as shown in
When, for example, as shown in
Further, when, for example, as shown in
When the action segments 412 should be displayed in the same display grain size regardless of the action, for example, as shown in
Thus, because the display roughness of the action segments 412 can be changed independently in accordance with the type of action, only the action the user wants to check in detail can be displayed in detail.
Incidentally, the method of changing the display roughness shown in
[4-7. Deleting Action]
According to the present technology, the action segment 412 can be deleted from the action log 410. If, for example, as shown in
[4-8. Posting Action]
According to the present technology, content of the action segment 412 of the action log 410 can be posted. If, for example, as shown in
[4-9. Action Log Acquisition Stop Processing]
According to the present technology, when the acquisition of an action log should be stopped for some reason, for example, as shown in
[4-10. Updating Display Content]
The action log display application in the present technology automatically uploads operation action data acquired by the action recording device 100 to the action log server 200 in predetermined timing (for example, twice per day). Also, the analysis server 300 automatically generates an action segment in predetermined timing (for example, twice per day). While an action log is displayed based on generated action segments, an action log displayed in accordance with the system function or circumstances may not correspond to the latest results. Thus, by pressing an update button 493 that updates the action log displayed in the settings screen 490 of
<5. Reflection Processing of Correction Feedback>
In an action log display system in the present technology, the meaning and content of an action is analyzed by the analysis server 300 and an action log is displayed by action segments. However, as described above, content of the displayed action log may not all correct. Thus, according to the present technology, the user can make corrections to correct content by using the action log display application. In the present technology, correction feedback of the user is reflected in the next analysis processing by the analysis server 300 and used to improve the precision of the next and subsequent analysis results. The reflection processing of correction feedback will be described below based on
[5-1. Properties of Correction Feedback]
In the present technology, the precision of analysis results is improved by reflecting correction feedback of the user in analysis processing, but the user may not correct all errors of analysis results by the analysis server 300. That is, content of an action log that is not corrected may not necessarily be correct. Thus, in the present technology, it is necessary to assume a system capable of collecting substantially biased information only. In addition, analysis results before corrections by the user do not necessarily match the latest analysis results. Thus, by reflecting information showing which action segment is corrected in what way in analysis processing for each user, the action specific to each user can be learned, which is considered to effectively work to improve the precision of analysis results.
[5-2. Action Recognition Processing]
In consideration of the above points, according to the present embodiment, an action pattern is decided based on characteristic amount analysis results in recognition processing of an operation action and acquires a plurality of probability distributions corresponding to the action pattern, time, and position information (location). In this case, a weight of a histogram is assigned and an operation action is recognized based on results of assigning weights depending on the location. If position information cannot be acquired or there is no need to acquire position information, uniform weights may be assigned or specific weights like “no location can be acquired” or “there is no need for location” may be assigned.
After an action log in the unit time is acquired, the living action recognition unit 321 starts processing to recognize the action of the action log. First, as shown in
Next, in steps S302 to S306, the living action recognition unit 321 performs processing to decide operation content of the action log in the unit time. First, it is assumed that the number of pairs of the probability distribution and the weighting factor acquired in step S300 is n and the parameter representing the processing number is i (i=0 to n) (S302). Then, the living action recognition unit 321 multiplies the probability distribution by the weighting factor of each action for the first (i=0) pair of the probability distribution and the weighting factor (S304). If, for example, in
When the processing in step S304 is completed, the living action recognition unit 321 adds 1 to the parameter i (S306) and repeats the processing in step S302 and thereafter. In the example of
Then, the living action recognition unit 321 adds the integrated value in the second row to the integrated value in the first row for each action. This results in integrated values of “shopping”: 110, “work”: 310, “meal”: 70, “others”: 210. Similarly, integrated values are calculated for the pairs of the probability distribution and the weighting factor in the third and fourth rows and these integrated values are added to the above integrated values of each action to finally obtain added values of “shopping”: 260, “work”: 460, “meal”: 420, “others”: 460.
The living action recognition unit 321 decides the action of the maximum final added value as the operation content of the action log. In the example of
Therefore, in the example of
[5-3. Reflection Processing of Correction Feedback]
(5-3-1. Overview of Reflection Processing of Correction Feedback)
As described based on
An overview of reflection processing of correction feedback will be provided based on
The analysis server 300 having received the correction feedback from the action recording device 100 through the analysis server interface unit 310 reflects content of the correction feedback in the operation action estimation information through the feedback adjustment unit 332. At this point, the feedback adjustment unit 332 corrects the probability distribution of the operation action estimation information if the content of the correction feedback concerns an action and corrects the weighting factor depending on the location if the content of the correction feedback concerns position information (location).
It is assumed that, for example, as shown in
It is assumed, on the other hand, that an analysis result of “location to work” is acquired, but correction feedback to change the location description to “location to do shopping frequently” by the user is received. In this case, the feedback adjustment unit 332 corrects, among a plurality of probability distributions, the weighting factor of the probability distribution having the maximum probability of “shopping”. For example, the feedback adjustment unit 332 makes a correction of increasing the weighting factor in the first row with the maximum probability of “shopping” by a factor of a predetermined number (for example, 10).
By correcting the operation action estimation information in this manner, correction feedback is reflected in analysis results of action segments so that the precision of analysis results of the operation content can be expected. The reflection processing of correction feedback will be described in more detail below based on
(5-3-2. Reflection Processing of Correction Feedback of an Action)
First, the reflection processing of correction feedback of an action will be described based on
When correction feedback is received from the action recording device 100, the feedback adjustment unit 332 first recognizes correction content. It is assumed here that operation content of an action segment is corrected. The feedback adjustment unit 332 acquires the action segment to be corrected from the unit data storage DB 334 or the hierarchical information attached data storage DB 335 and starts processing shown in
The feedback adjustment unit 332 first acquires the probability distribution (partial probability distribution) used to recognize the operation content of the action segment to be corrected from operation action estimation information stored in the analysis parameter DB 333 (S310). Next, the feedback adjustment unit 332 calculates a value M(i) obtained by multiplying the maximum probability of each probability distribution by the weighting factor of the row for the partial probability distribution and sorts these probability distributions (S311).
The parameter indicating the order of sorted probability distributions is set as i (i=0 to n) and the number of probability distributions constituting the partial probability distribution is set as n. Then, the feedback adjustment unit 332 determines whether the parameter i is smaller than n and the multiplied value M(i) is larger than a predetermined threshold th (S312). If the conditions in step S312 are not satisfied, the processing shown in
The loss ratio calculation function is assumed to be a single comprehensive measure representing losses caused when some available decision is made. In the present embodiment, the loss ratio calculation function is used to set, for example, a correction ratio table representing a correction ratio C between the action of analysis results and the correct action as shown in the lower portion of
Then, the feedback adjustment unit 332 subtracts the correction ratio C(i) acquired in step S313 from the value of the probability distribution of the action of the maximum value of probability distribution, adds the correction ratio C(i) to the value of the probability distribution of the correct action, and reflects these corrections in the operation action estimation information (S314). If, for example, the processing in step S314 is performed for the probability distribution in the fourth row of the operation action estimation information on the left of
If, for example, the processing in step S313 is performed for the probability distribution in the fifth row of the operation action estimation information on the left of
Similarly, if the processing in step S313 is performed for the probability distribution in the sixth row of the operation action estimation information on the left of
In this manner, the operation content of correction feedback is reflected in the operation action estimation information. The reflection processing shown in
In the reflection processing of correction feedback shown in
The feedback adjustment unit 332 first acquires the probability distribution (partial probability distribution) used to recognize the operation content of the action segment to be corrected from operation action estimation information stored in the analysis parameter DB 333 (S320). Next, the feedback adjustment unit 332 calculates a value M(i) obtained by multiplying the maximum probability of each probability distribution by the weighting factor of the row for the partial probability distribution and sorts these probability distributions (S321). The processing in steps S320, S321 can be made the same as the processing in steps S310, S311 in
If the parameter showing the order of sorted probability distributions is set as i(i=0 to n), the feedback adjustment unit 332 determines whether the parameter i is smaller than n (S322). If the condition in step S322 is not satisfied, the processing shown in
Thus, by using, instead of the correction ratio table, learning processing such as the neural network technique, content of correction feedback can be reflected in each value of operation action estimation information without the need to set the correction ratio table in advance.
(5-3-3. Reflection Processing of Correction Feedback of an Action and Position Information)
Next, the reflection processing of correction feedback of an action and position information will be described based on
The feedback adjustment unit 332 acquires the action segment to be corrected from the unit data storage DB 334 or the hierarchical information attached data storage DB 335 and starts processing shown in
In step S331, whether any correction related to position information is made on action segments accompanied by movement is determined. If a correction related to position information is made on action segments accompanied by movement, representative coordinates of end points (two representative coordinates like position X to position Y) are calculated (S332). On the other hand, if no correction related to position information is made on action segments accompanied by movement, representative coordinates of the movement are calculated (S333). Incidentally, representative coordinates can be calculated by using the center, center of gravity, most frequent point and the like.
Next, the feedback adjustment unit 332 records representative coordinates calculated in step S332 or S333, the precision, and attached attributes in a feedback DB (not shown) (S334). The feedback DB is a storage unit provided in the analysis server 300. Then, the feedback adjustment unit 332 analyses operation content using new position information recorded in the feedback DV in step S334 and determines whether the analysis result matches the correct action input by the correction feedback (S335). If it is determined in step S335 that the operation content analyzed by using new position information matches the correct action, a judgment can be made that correction feedback about position information is correctly reflected and also there is no error in the action content. Therefore, the feedback adjustment unit 332 judges that the reflection processing of correction feedback is completed and terminates the processing in
On the other hand, if it is determined in step S335 that the operation content analyzed by using new position information does not match the correct action, a judgment can be made that with corrections of position information alone, correction feedback is not correctly determined. In this case, processing in steps S336 to S341 is performed to reflect operation content of the correction feedback in operation action estimation information. The processing in steps S336 to S341 can be made the same as the processing in
That is, the feedback adjustment unit 332 first acquires the probability distribution (partial probability distribution) used to recognize the operation content of the action segment to be corrected from operation action estimation information stored in the analysis parameter DB 333 (S336). Next, the feedback adjustment unit 332 calculates a value M(i) obtained by multiplying the maximum probability of each probability distribution by the weighting factor of the row for the partial probability distribution and sorts these probability distributions (S337).
The parameter indicating the order of sorted probability distributions is set as i (i=0 to n) and the number of probability distributions constituting the partial probability distribution is set as n. Then, the feedback adjustment unit 332 determines whether the parameter i is smaller than n and the multiplied value M(i) is larger than a predetermined threshold th (S338). If the conditions in step S338 are not satisfied, the processing shown in
Then, the feedback adjustment unit 332 subtracts the correction ratio C(i) acquired in step S339 from the value of the probability distribution of the action of the maximum value of probability distribution, adds the correction ratio C(i) to the value of the probability distribution of the correct action, and reflects these corrections in the operation action estimation information (S340). Then, the feedback adjustment unit 332 adds 1 to the parameter i (S341) and repeats the processing in step S338 and thereafter. By performing the above processing, the operation action estimation information after the correction feedback being reflected can be obtained.
Incidentally, instead of the processing in steps S336 to S341, the processing shown in
The correction feedback of position information may be reflected by, as shown on the lower left of
The added amount of weighting factor for position information may be decided based on, for example, original position information or changed for each attribute type of position information. Further, a probability distribution specific to position information may randomly be generated and added to operation action estimation information. Accordingly, over-learning can be prevented.
<6. Others>
[6-1. Personal Modeling of Action Pattern]
In an action log display system according to the present embodiment, an action log is displayed by using action segments to which the meaning and content is attached. By performing, for example, autocorrelation processing or filter processing using these action segments, temporal or action errors can be absorbed. Then, a user's typical action pattern can be extracted from a small amount of data.
As a functional unit to extract a user's typical action pattern, as shown in
In the example of
[6-2. Position Display Technique by Moving Medium/Means Determination]
If an action is recognized as a movement action when an action segment is generated, the living action recognition unit 321 identifies position information of the user based on which medium of transport the user uses to move or which means of transport the user uses to move (
The living action recognition unit 321 assigns weights to the lines and stations using the above information to identify the nearest station. Weights may be assigned to lines and stations by, for example, increasing weights of nearest station candidates with a decreasing distance or assigning weights preferentially to lines and stations that are continuously acquired in action logs. Alternatively, weights may be assigned in consideration of distance differences or time differences that can be acquired from information up to the last time. Accordingly, if the fact of being a predetermined distance apart or that a predetermined time has passed is recognized from the information up to the last time and information this time, the possibility of having changed trains to another line can be considered.
(6-2-1. Line Estimation Processing)
The estimation of line can be determined from, for example, the number of passed stations recognized from an action log. In addition, the movement locus of the user can be estimated by considering the possibility of changing trains at a station identified from position information or whether a direct service between a plurality of lines is available. If a plurality of lines runs between the same stations, which line is used can be identified by estimating a more likely line from the user's past movement locus or acquiring more detailed position information from a position information acquisition sensor.
As a result of performing the above line estimation processing, for example, as shown in
(6-2-2. Station Name Selection Processing)
The station name is selected by, as described above, identifying the nearest station. In this case, even if the user does not actually move, changes in latitude/longitude may erroneously be recognized due to an error of radio field intensity of a sensor. Thus, for example, as shown in
If the medium/means of transport is not movement by train, for example, priority may be given to the identified nearest station as a landmark to represent the location of operation by the station name (excluding “station”). For example, it is assumed that, as a result of analyzing an action log, movement by car is recognized and the “Higash-Koganei station” and the “Shin-Koganei station” are identified as landmarks. In this case, it is not natural to move between stations by car and thus, action content can naturally be expressed by representing the start location and the goal as the “Higash-Koganei station” and the “Shin-Koganei station”.
<7. Exemplary Hardware Configuration>
A process of the action recording device 100 in accordance with this embodiment can be executed either by hardware or software. In this case, the action recording device 100 can be configured as shown in
The action recording device 100 in accordance with this embodiment can be implemented by a processing device such as a computer as described above. As shown in
The CPU 101 functions as an arithmetic processing unit and a control unit, and controls the entire operation within the action recording device 100 in accordance with various programs. The CPU 101 may also be a microprocessor. The ROM 102 stores programs, operation parameters, and the like used by the CPU 101. The RAM 103 temporarily stores programs used in the execution of the CPU 101, parameters that change as appropriate during the execution, and the like. These units are mutually connected via the host bus 104a including a CPU bus or the like.
The host bus 104a is connected to the external bus 104b such as a PCI (Peripheral Component Interconnect/Interface) bus via the bridge 104. Note that the host bus 104a, the bridge 104, and the external bus 104b need not necessarily be arranged separately, and the functions of such components may be integrated into a single bus.
The input device 106 includes an input means for a user to input information, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, or a lever; an input control circuit that generates an input signal on the basis of a user input and outputs the signal to the CPU 101; and the like. The output device 107 includes a display device such as, for example, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp; and an audio output device such as a speaker.
The storage device 108 is a device for storing data, constructed as an example of a storage unit of the action recording device 100. The storage device 108 can include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 108 includes, for example, a HDD (Hard Disk Drive). The storage device 108 stores programs and various data for driving the hard disk and executed by the CPU 101.
The drive 109 is a reader/writer for a storage medium, and is incorporated in or externally attached to the action recording device 100. The drive 109 reads information recorded on a removable storage medium such as a magnetic disk, an optical disc, a magnetooptical disk, or semiconductor memory that is mounted, and outputs the information to the RAM 103.
The connection port 111 is an interface for connection to an external device, and is, for example, a connection port for connection to an external device that can transmit data via a USB (Universal Serial Bus). The communication device 113 is, for example, a communication interface including a communication device and the like for connection to the communication network 10. The communication device 113 may be any of a communication device supporting a wireless LAN (Local Area Network), a communication device supporting a wireless USB, or a wire communication device that performs wire communication.
In the foregoing, a preferred embodiment of the present disclosure has been described in detail with reference to the appended drawings, but the technical scope of the present disclosure is not limited to the above examples. A person skilled in the art may find various alterations and modifications within the scope of the appended claims, and it should be understood that they will naturally come under the technical scope of the present disclosure.
In the above embodiment, for example, the action representation generation unit 320 and the data management unit 330 are provided in the analysis server 300 and the action representation generation unit 140 is provided in the action recording device 100, but the present disclosure is not limited to such an example. For example, these functional units may all be provided in the analysis server 300 or in the action recording device 100.
Additionally, the present technology may also be configured as below.
(1)
An information processing device including:
an action recognition unit that recognizes an operation action of a user based on sensor information; and
an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
(2)
The information processing device according to (1),
wherein dictionary data defining a relation of the higher meaning and content for the operation action is held, and
wherein the action representation generation unit estimates the meaning and content of the operation action from relations before and after the operation action data arranged chronologically based on the dictionary data to generate the action segment.
(3)
The information processing device according to (1) or (2),
wherein the action representation generation unit estimates the meaning and content of the operation action in accordance with a time period and a time of the operation action data to be analyzed to generate the action segment.
(4)
The information processing device according to any one of (1) to (3),
wherein the action representation generation unit estimates the meaning and content of the operation action in accordance with positions of the operation action data before and after the operation action data to be analyzed to generate the action segment.
(5)
The information processing device according to any one of (1) to (4),
wherein hierarchical information showing a hierarchical relationship about the meaning and content is attached to the action segment.
(6)
The information processing device according to (5),
wherein the action representation generation unit displays the action segments based on a segmentation grain size deciding roughness of segmentation of the action segments and the hierarchical information.
(7)
The information processing device according to (6),
wherein the action representation generation unit combines or divides the action segments based on a size of the segmentation grain size and displays the combined or divided action segments.
(8)
The information processing device according to any one of (1) to (7), further including:
a typical action pattern generation unit that extracts one action pattern from a plurality of action segment groups including the action segments of a predetermined unit based on a correlation between the action segments.
(9)
The information processing device according to any one of (1) to (8),
wherein the action representation generation unit displays, in a display unit, the action segments represented at least by a start time, an end time, position information, and operation content of the operation action by chronologically arranging the action segments.
(10)
The information processing device according to any one of (1) to (9), further including:
a feedback adjustment unit that corrects operation action estimation information that decides the operation action based on correction feedback from the user to the action segment generated by the action representation generation unit,
wherein the action representation generation unit generates the action segment constituting an action log from the operation action data based on the operation action estimation data, and
wherein the feedback adjustment unit corrects the operation action estimation information based on the correction feedback.
(11)
An information processing device including:
an action recognition unit that recognizes an operation action of a user based on sensor information;
an action representation generation unit that generates an action segment constituting an action log from operation action data showing the operation action of the user recognized by the action recognition unit based on operation action estimation information that decides the operation action; and
a feedback adjustment unit that corrects the operation action estimation information based on correction feedback from the user to the action segment generated by the action representation generation unit.
(12)
The information processing device according to (11),
wherein the operation action estimation information includes a plurality of combinations of a probability distribution and a weighting factor depending on a location for a plurality of the operation actions, and
wherein the feedback adjustment unit corrects the probability distribution or the weighting factor for each of the operation actions based on the correction feedback.
(13)
The information processing device according to (12),
wherein, when the correction feedback concerns action content, the feedback adjustment unit corrects the probability distribution of the operation action estimation information in accordance with content of the correction feedback.
(14)
The information processing device according to (11) or (12),
wherein, when the correction feedback concerns the location, the feedback adjustment unit corrects the weighting factor of the operation action estimation information in accordance with content of the correction feedback.
REFERENCE SIGN LIST
- 100 action recording device
- 110 sensors
- 120 action recognition unit
- 122 sensor controller
- 124 operation action recognition unit
- 130 client interface unit
- 140 action representation processing unit
- 142 display processing unit
- 144 input information processing unit
- 150 display unit
- 160 input unit
- 200 action log server
- 210 log server interface unit
- 220 action log db
- 300 analysis server
- 310 analysis server interface unit
- 320 action representation generation unit
- 321 living action recognition unit
- 322 hierarchical structure judgment unit
- 330 data management unit
- 331 data acquisition unit
- 332 feedback adjustment unit
- 333 analysis parameter db
- 334 unit data storage db
- 335 hierarchical information attached data storage db
Claims
1. An information processing device comprising:
- an action recognition unit that recognizes an operation action of a user based on sensor information; and
- an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
2. The information processing device according to claim 1,
- wherein dictionary data defining a relation of the higher meaning and content for the operation action is held, and
- wherein the action representation generation unit estimates the meaning and content of the operation action from relations before and after the operation action data arranged chronologically based on the dictionary data to generate the action segment.
3. The information processing device according to claim 1,
- wherein the action representation generation unit estimates the meaning and content of the operation action in accordance with a time period and a time of the operation action data to be analyzed to generate the action segment.
4. The information processing device according to claim 1,
- wherein the action representation generation unit estimates the meaning and content of the operation action in accordance with positions of the operation action data before and after the operation action data to be analyzed to generate the action segment.
5. The information processing device according to claim 1,
- wherein hierarchical information showing a hierarchical relationship about the meaning and content is attached to the action segment.
6. The information processing device according to claim 5,
- wherein the action representation generation unit displays the action segments based on a segmentation grain size deciding roughness of segmentation of the action segments and the hierarchical information.
7. The information processing device according to claim 6,
- wherein the action representation generation unit combines or divides the action segments based on a size of the segmentation grain size and displays the combined or divided action segments.
8. The information processing device according to claim 1, further comprising:
- a typical action pattern generation unit that extracts one action pattern from a plurality of action segment groups including the action segments of a predetermined unit based on a correlation between the action segments.
9. The information processing device according to claim 1,
- wherein the action representation generation unit displays, in a display unit, the action segments represented at least by a start time, an end time, position information, and operation content of the operation action by chronologically arranging the action segments.
10. The information processing device according to claim 1, further comprising:
- a feedback adjustment unit that corrects operation action estimation information that decides the operation action based on correction feedback from the user to the action segment generated by the action representation generation unit,
- wherein the action representation generation unit generates the action segment constituting an action log from the operation action data based on the operation action estimation data, and
- wherein the feedback adjustment unit corrects the operation action estimation information based on the correction feedback.
11. An information processing device comprising:
- an action recognition unit that recognizes an operation action of a user based on sensor information;
- an action representation generation unit that generates an action segment constituting an action log from operation action data showing the operation action of the user recognized by the action recognition unit based on operation action estimation information that decides the operation action; and
- a feedback adjustment unit that corrects the operation action estimation information based on correction feedback from the user to the action segment generated by the action representation generation unit.
12. The information processing device according to claim 11,
- wherein the operation action estimation information includes a plurality of combinations of a probability distribution and a weighting factor depending on a location for a plurality of the operation actions, and
- wherein the feedback adjustment unit corrects the probability distribution or the weighting factor for each of the operation actions based on the correction feedback.
13. The information processing device according to claim 12,
- wherein, when the correction feedback concerns action content, the feedback adjustment unit corrects the probability distribution of the operation action estimation information in accordance with content of the correction feedback.
14. The information processing device according to claim 12,
- wherein, when the correction feedback concerns the location, the feedback adjustment unit corrects the weighting factor of the operation action estimation information in accordance with content of the correction feedback.
15. An information processing method comprising:
- a step for recognizing an operation action of a user based on sensor information; and
- a step for analyzing operation action data showing the recognized operation action of the user to generate an action segment represented by a meaning and content of the operation action from the operation action data.
16. An information processing method comprising:
- a step for recognizing an operation action of a user based on sensor information;
- a step for generating an action segment constituting an action log from operation action data showing the recognized operation action of the user based on operation action estimation information that decides the operation action; and
- a step for correcting the operation action estimation information based on correction feedback from the user to the action segment.
17. A computer program for causing a computer to function as an information processing device comprising:
- an action recognition unit that recognizes an operation action of a user based on sensor information; and
- an action representation generation unit that analyzes operation action data showing the operation action of the user recognized by the action recognition unit to generate an action segment represented by a meaning and content of the operation action from the operation action data.
18. A computer program for causing a computer to function as an information processing device comprising:
- an action recognition unit that recognizes an operation action of a user based on sensor information;
- an action representation generation unit that generates an action segment constituting an action log from operation action data showing the operation action of the user recognized by the action recognition unit based on operation action estimation information that decides the operation action; and
- a feedback adjustment unit that corrects the operation action estimation information based on correction feedback from the user to the action segment generated by the action representation generation unit.
Type: Application
Filed: Jun 6, 2012
Publication Date: Apr 10, 2014
Applicant: SONY CORPORATION (Tokyo)
Inventors: Masatomo Kurata (Tokyo), Masanori Katsu (Tokyo), Takashi Ogata (Tokyo), Yasutaka Fukumoto (Tokyo)
Application Number: 14/123,886
International Classification: G06F 17/30 (20060101);
