MESSAGE TRANSMISSION SYSTEM, MESSAGE TRANSMISSION METHOD, AND PROGRAM FOR WEARABLE TERMINAL

Abstract

When a person 500 is present within a field of view of a user, a user terminal 10 acquires the field of view of the user as image data, and transmits the field-of-view image to a field-of-view image analysis server 100. The field-of-view image analysis server 100 extracts a feature amount from the field-of-view image, and identifies the person near a center of the field-of-view image by referring to a feature amount database 150. Further, the field-of-view image analysis server 100 reads information of the person from a person information database 151, and sends the information to the user terminal 10. The user terminal 10 transmits the inputted electronic message or image data via the message transmission server 200 based on the received person information such that the user can communicate with the person within the field of view by a message.

Description

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to and the benefit of U.S. Provisional Application No. 62/135,771, filed on Mar. 20, 2015 in the U.S. Patent and Trademark Office, the entire contents of which are incorporated herein by reference.

BACKGROUND

(a) Field

The present invention generally relates to a message transmission system, a message transmission method, and a program for a user to communicate with a person existing in a field of view.

(b) Description of the Related Art

Today, an electronic message such as an e-mail or an instant messaging has become popular as means for communication between users. The electronic message composed of text data or image data is capable of performing accurate input and browsing in cluttered places compared with a conventional voice exchange and is particularly excellent in use in cities with moving

However, the electronic message is typically used for communication with the other party at remote, and it is common to perform conversations by normal utterance upon directly facing with each other. In this case, since a speaker states while watching the other party of the conversation, it can be inferred that the statement has been made for a person close to a center of a field of view of the speaker.

For example, Japanese Patent Application Publication No. 2000-350192 describes an archive apparatus for appropriately handling information retrieval from a database by determining a person who is in front of a speaker's eyes at a meeting, converting audio data into text data, and storing the text data together with person data.

However, in the archive apparatus described in Patent Document 1, the actual voice is merely stored together with the person who is in front of the line of sight of the speaker serving the statement, and a providing function is limited as the simple archive apparatus.

SUMMARY

An embodiment of the present invention is focused on a fact that information can be accurately and quickly transferred, by transmitting an electronic message or an image of a field of view of the user to a person that exists in the field of view of the user, with taking a format that is similar to a direct dialogue, without an intervention of a voice, and even in a cluttered place such as a city or a party venue.

An embodiment of the present invention provides a message transmission system, a message transmission method, and a program for performing a communication with the person that exists in the field of view of the user, by acquiring the field of view of the user as image data, identifying the person present in the field of view, and transmitting an electronic message or the image data.

According to an embodiment of the present invention, pa message transmission system for performing a communication with a person that exists in a field of view of a user is provided. The message transmission system includes a field-of-view image acquisition apparatus including a field-of-view image acquiring module configured to acquire a field of view of the user as an image; a feature amount extracting module configured to analyze a field-of-view image acquired by the field-of-view image acquiring module and to extract a feature amount associated with an identification of a person; an person identifying module configured to identify a person within the field-of-view image from the feature amount extracted by the feature amount extracting module; and a message transmitting module configured to transmit an electronic message to the person identified by the person identifying module.

According to the above embodiment, in a message transmission system for performing a communication with a person that exists in a field of view of a user, a field-of-view image acquisition apparatus can acquire a field of view of the user as an image; and the message transmission system can analyze the acquired field-of-view image, extract a feature amount associated with an identification of a person, identify a person within the field-of-view image from the extracted feature amount, and transmit an electronic message to the identified person.

According to another embodiment of the present invention, the message transmission system further includes a field-of-view image transmitting module configured to transmit a part or whole of the field-of-view image acquired by the field-of-view image acquiring module to the person identified by the person identifying module.

According to the above embodiment, the message transmission system can transmit a part or whole of the acquired field-of-view image to the identified person.

According to yet another embodiment, the message transmission system further includes a person image position calculating module configured to calculate a position of the person identified by the person identifying module within the field-of-view image and a person information displaying module configured to display information on the identified person within the field of view of the user based on the position calculated by the person image position calculating module.

According to the above embodiment, the message transmission system can calculate a position of the identified person within the field-of-view image and displays information on the identified person within the field of view of the user based on the calculated position.

The above embodiments of the present invention relate to a message transmission system but may be applicable to a message transmission method and a message transmission system program in the same manner.

According to still another embodiment of the present invention, a message transmission method executed by a message transmission system for performing a communication with a person that exists in a field of view of a user is provided. The message transmission method includes acquiring a field of view of the user as an image by a field-of-view image acquisition apparatus; analyzing the acquired field-of-view image and extracting a feature amount associated with an identification of a person; identifying a person within the field-of-view image from the extracted feature amount; and transmitting an electronic message to the identified person.

According to further embodiment of the present invention, a message transmission system program for executing a message transmission method on a message transmission system for performing a communication with a person that exists in a field of view of a user is provided The message transmission method includes acquiring a field of view of the user as an image; analyzing the acquired field-of-view image and extracting a feature amount associated with an identification of a person; identifying a person within the field-of-view image from the extracted feature amount; and transmitting an electronic message to the identified person.

According to an embodiment of the present invention, a message transmission system, a message transmission method, and a program for performing a communication with the person that exists in the field of view of the user, by acquiring the field of view of the user as image data, identifying the person present in the field of view and transmitting an electronic message or the image data can be provided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram for explaining an outline of a message transmission system 1 according to an embodiment of the present invention.

FIG. 2 shows a system configuration of a message transmission system 1 according to an embodiment of the present invention.

FIG. 3 shows a functional block diagram of a user terminal 10, a field-of-view image analysis server 100, and a message transmission server 200 according to an embodiment of the present invention.

FIG. 4 is a flowchart showing a procedure of an identifying process of a person in a field-of-view image executed by a user terminal 10, a field-of-view image analysis server 100, and a message transmission server 200 according to an embodiment of the present invention.

FIG. 5 is a flowchart showing a procedure of a message transmission process executed by a user terminal 10 and a field-of-view image analysis server 100 according to an embodiment of the present invention.

FIG. 6 shows a schematic diagram for explaining an overview of a user terminal 10 imitating glasses according to an embodiment of the present invention.

FIG. 7 shows an example of personal information displayed on a user terminal 10 by a personal information displaying module 19 according to an embodiment of the present invention.

FIG. 8 shows an example of personal information displayed in a case that a smart phone terminal is used as a user terminal 10 according to an embodiment of the present invention.

FIG. 9 is a diagram showing an example of a feature amount table in a feature amount database 150 according to an embodiment of the present invention.

FIG. 10 is a diagram showing an example of a personal information table in a person information database 151 according to an embodiment of the present invention.

DETAILED DESCRIPTION

In the following detailed description, only certain embodiments of the present invention have been shown and described, simply by way of illustration. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive. Like reference numerals designate like elements throughout the specification.

An overview of embodiments of the present invention is described. When a person 500 is present within a field of view of a user, a user terminal 10 acquires the field of view of the user as image data, and transmits the field-of-view image to a field-of-view image analysis server 100. The field-of-view image analysis server 100 extracts a feature amount from the field-of-view image, and identifies the person near a center of the field-of-view image by referring to a feature amount database 150. Further, the field-of-view image analysis server 100 reads information of the person from a person information database 151, and sends the information to the user terminal 10. The user terminal 10 transmits the inputted electronic message or image data via the message transmission server 200 based on the received person information such that the user can communicate with the person within the field of view by a message.

Overview of Message Transmission System

An overview of a message transmission system 1 according to an embodiment of the present invention is described with reference to FIG. 1. A message transmission system 1 according to an embodiment of the present invention includes a user terminal 10, a field-of-view image analysis server 100, and a message transmission server 200.

First, the user terminal 10 acquires a field-of-view image of the user. When the user sends a message to a person 500, the user terminal 10 acquires the field-of-view image including the person 500 as shown in FIG. 1 (Step S01).

Next, the user terminal 10 transmits the acquired field-of-view image to the field-of-view image analysis server 100 (Step S02). The field-of-view image analysis server 100 analyzes the received field-of-view image and extracts a feature amount for determining a face, a system, or the like.

Information about a feature amount for identifying a person is stored, in advance, in a feature amount database 150 in a storage unit that is provided with the field-of-view image analysis server 100. The field-of-view image analysis server 100 identifies the person in the field-of-view image based on the extracted feature amount by referring to the feature amount database 150 (Step S03). The field-of-view image analysis server 100 queries information about the identified person to a person information database 151 in the storage unit (Step S04) and reads the personal information (Step S05). Then, the field-of-view image analysis server 100 sends the read person information to the user terminal 10 (Step S06).

The user terminal 10 transmits to the message transmission server 200 the identified person information and a message content to be sent to the identified person (Step S07). Here, the user terminal 10 may include the field-of-view image acquired on the current and past in the message content. The message content to be sent may be inputted by catching the person 500 in the field of view by using an input device such as a keyboard, or may be completely inputted prior to the step S01. That is, the person 500 may be included within the field-of-view image at the time of step S01 and the input of the message may be completed at a stage where the step S05 is executed.

Also, when transmitting a message via the message transmission server 200, a transmitting scheme is not limited to a regular e-mail protocol and an instant message (IM) may be used. In addition, a scheme for calling a message function provided with a specific software or a social networking service (SNS) by an API or the like may be used. The message may be electronically transmitted and received, and may include at least character information. The user terminal 10 transmits to the message transmission server 200 the message content and the identified person information serving as a destination in a format that is based on a specific message transmission standard, and the message transmission server 200 transmits the content to the identified person as a message (Step S08).

The person 500 receives, at a message receiving device 501, the message transmitted by the message transmission server 200. The message receiving device 501 may receive the message transmitted by the message transmission server 200. The message receiving device 501 is not limited to an information terminal such as a smart phone, a personal computer, or the like and may be a terminal with the same type as the user terminal 10.

The above is the overview of the message transmission system 1 according to a preferred embodiment of the present invention.

System Configuration of the Message Transmission System 1

FIG. 2 shows a system configuration of a message transmission system 1 according to an embodiment of the present invention. A message transmission system 1 according to an embodiment of the present invention includes a public line network 3 such as an internet network, a user terminal 10, a field-of-view image analysis server 100, and a message transmission server 200. The user terminal 10 is communicably connected to the field-of-view image analysis server 100 and the message transmission server 200 via the public network 3.

Here, the field-of-view image analysis server 100 and the message transmission server 200 may be a physically same server. In a case that all of functions of the field-of-view image analysis server 100 and the message transmission server 200 are provided with the user terminal 10, the message transmission system 1 may be formed by only the user terminal 10.

Description of Each Function

A configuration of each device is described with reference to FIG. 3. The user terminal 10 according to an embodiment of the present invention includes, as a controller 11, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM) or the like, and includes, as a communication unit 12, for example a WiFi (wireless fidelity) enabled device conforming to the IEEE standard 802.11. Further, the user terminal 10 includes, as an input-output unit 13, a photographing device such as a camera or the like for acquiring a field of view of a user as an image and further includes a display unit such as a liquid crystal display or a projection screen for displaying an image or a text message and an input unit such as a keyboard or a touch panel for receiving an input from the user.

In the user terminal 10, the controller 11 reads a predetermined program, thereby realizing a field-of-view image transmitting module 14, a personal information receiving module 15, and a message content transmitting module 16, in cooperating with the communication unit 12. Further, in the user terminal 10, the controller 11 reads a predetermined program, thereby realizing a field-of-view image photographing module 17, a field-of-view image adjusting module 18, a personal information displaying module 19, a destination determining module 20, and a message content input receiving module 21, in cooperating with the input-output unit 13.

Like the user terminal 10, the field-of-view image analysis server 100 according to an embodiment of the present invention includes, as a controller 101, a CPU, a RAM, a ROM, and the like, and includes, as a communication unit 102, for example a WiFi enabled device conforming to the IEEE standard 802.11 or a device for performing a data communication with a radio access point and other electrical appliance such as a device capable of achieving a connection by a wired cable. The field image analysis server 100 includes, as a storage unit 103 for storing data and files, a data storage unit formed by a hard disk, a semiconductor memory, a recording medium, a memory card, or the like. The storage unit 103 further includes a feature amount database 150 and a person information database 151.

In the field-of-view image analysis server 100, the controller 101 reads a predetermined program, thereby a field-of-view image receiving module 104 and a personal information transmitting module 105, in cooperating with the communication unit 102. In the field-of-view image analysis server 100, the controller 101 reads a predetermined program, thereby realizing a feature amount extracting module 106, a person identification module 107, and a personal information reading module 108, in cooperating with the storage unit 103

Similarly to the field-of-view image analysis server 100, the message transmission server 200 according to an embodiment of the present invention includes, as a controller 201, a CPU, a RAM, a ROM, and the like, and includes, as a communication unit 202, for example a WiFi enabled device conforming to the IEEE standard 802.11 or a device for performing a data communication with a radio access point and other electrical appliance such as a device capable of achieving a connection by a wired cable.

In the message transmission server 200, the controller 201 reads a predetermined program, thereby realizing a transmission message receiving module 203 and a message transmitting module 204, in cooperating with the communication unit 202.

Process of Identifying Person in Field-of-View Image

FIG. 4 is a flowchart showing a procedure of an identifying process of a person in a field-of-view image executed by a user terminal 10 and a field-of-view image analysis server 100 in a message transmission system 1 according to an embodiment of the present invention. The identifying process of the person in the field-of-view image is described.

First, a user terminal 10 acquires a field-of-view image of a user. A field-of-view image photographing module 17 photographs a field-of-view image of the user (Step S11). FIG. 6 shows a schematic diagram for explaining an overview of a user terminal 10 imitating glasses according to an embodiment of the present invention. A camera 51 is present approximately in the center of the user terminal 10 and photographs a field-of-view image of the user by taking a photograph toward the front of the user.

In one embodiment, under a circumstance that is similar to an circumstance where the user is always directed to right in front, an image of a front direction taken by the camera 51 may be directly used as the field-of-view image of the user. However, in another embodiment, when the user is actually facing at two or more persons at the same time, may occur a circumstance for acquiring a center of the field-of-view image of the user more accurately in consideration of a case where the user moves his or her eyeball while fixing the face. A field-of-view image adjusting module 18 of the user terminal 10 may acquire a line of sight of the user and adjust a center or a range of the field-of-view image (Step S12). For example, a scheme described in Japanese Patent Application Publication No. 2010-204823 may be used by installing an Infrared light source 52 and an infrared camera 53 on a back of a frame, photographing an eyeball image of the user, and calculating the line of sight of the user.

In some embodiments, a lens 54 of the user terminal 10 may be formed by a reflection process that is special to a face of the user side such that an image projected by a small projector 55 can be displayed on the field of view by overlapping. The user terminal 10 may include a keyboard 56 and a touch panel as an input unit.

Next, a field-of-view image transmitting module 14 of the user terminal 10 transmits the adjusted field-of-view image to the field-of-view image analysis server 100 (Step S13). A field-of-view image receiving module 104 of the field-of-view image analysis server 100 receives the field-of-view image (Step S14).

A feature amount extracting module 106 of the field-of-view image analysis server 100 extracts a feature amount of the image in order to identify a person included near the center of the received field-of-view image (Step S15). In some embodiments, the feature amount may be a value serving as evaluation criterion when performing pattern recognition that recognizes as a pattern the person's face or body previously stored from the received field-of-view image. More specifically, the feature amount extracting module 106 may identify an area corresponding to the human face or body from the vicinity of the center of the received field-of-view image, equally divide the identified area into vertically a predetermined number and horizontally a predetermined number, and calculate an average value of colors of pixels in each portion. Then, the feature amount extracting module 106 may regard as the feature amount an average value of colors which greatly varies depending on a position or size of a facial contour, a nose, an eye, an eyebrow, or the like.

In some embodiments, a person identification module 107 of the field-of-view image analysis server 100 may compare the extracted feature amount with a feature amount of each of persons stored in advance in the feature amount database 150 to calculate its difference. Further, the person identification module 107 may determine a person whose probability is highest from among the persons by multiplying each difference by a weighting coefficient which is set for each feature amount and normalizing the multiplied results together, and identify the person with the highest probability as the person that is near the center of the field-of-view image (Step S16).

FIG. 9 is a diagram showing an example of a feature amount table in a feature amount database 150 according to an embodiment of the present invention. Here, it is assumed for simplicity that the number of person is two and the field-of-view image has 256 colors, and five feature amounts from feature amount 1 to feature amount 5 are represented by a number between 0 and 255. However, it is common to improve the accuracy by representing the more feature amounts by the wider range.

Thereafter, a person information read module 108 of the field-of-view image analysis server 100 reads information of the identified person from a person information database 151. A person information transmitting module 105 transmits the person information to the user terminal 10 (Step S17). In some embodiments, the person information transmitting module 105 may simultaneously transmit a position within the image of the portion that the feature amount extracting module 106 identifies as a portion corresponding to the face or body.

FIG. 10 is a diagram showing an example of a personal information table in a person information database 151 according to an embodiment of the present invention. The person information table stores personal information including a person name, a gender, and an age of each person, and destinations of messages.

A personal information receiving module 15 of the user terminal 10 receives the personal information (Step S18). A personal information displaying module 19 of the user terminal 10 may display the received person information in the vicinity of the portion corresponding to the face or body of the person by using the position information within the image received at the same time.

FIG. 7 shows an example of personal information displayed on a user terminal 10 by a personal information displaying module 19 according to an embodiment of the present invention. FIG. 7 shows a case that the user terminal 10 is viewed from a user side. A person 61 is not a displaying content but is a part of a field of view penetrated through a lens 65. Personal information 62 of the person 61, a pointing rod 63, and a description text 64 are displayed with being overlapped with the person 61. These displaying contents may be projected onto the lens 65 by a compact projector installed in a string of the glasses.

In some embodiments, a smart phone terminal or the like having a display device such as a liquid crystal display or an organic light emitting display and a camera may be used as the user terminal 10 instead of the glass-type terminal shown in FIG. 7. FIG. 8 shows an example of personal information displayed in a case that a smart phone terminal is used as a user terminal 10. If the user sets up a smart phone terminal 10b in front of eyes, an image photographed by a camera of the smartphone terminal 10b is viewed on the field of view of the user and is displayed on a touch panel 75 of a display device such as a liquid crystal panel. That is, the user can see an appearance of a person 71 through the touch panel 75 without directly looking at the person 71 with naked eyes. Similarly to FIG. 7, person information 72, a pointing rod 73, and a description text 74 are displayed. In addition, if a software keyboard 76 by the touch panel is used, it is possible to view the software keyboard 76 while fixing the camera serving a viewpoint. Accordingly, the input can be easily performed as compared to the glasses-type terminal. The user can transmit a message content that is created to the person 71 by tapping the description text 74. When attaching the field-of-view image to the message, the current or past field-of-view image can be attached by tapping a field-of-view image attachment button 78.

The above is the identifying process of the person in the field-of-view image that is executed by the user terminal 10 and the field-of-view image analysis server 100 in the message transmission system 1.

Message Transmission Process

FIG. 5 is a flowchart showing a procedure of a message transmission process executed by a user terminal 10 and a message transmission server 200 in a message transmission system 1 according to an embodiment of the present invention. The message transmission process is described.

First, a destination determining module 20 of the user terminal 10 receives an input from a user and determines a destination of a message (Step S21). The destination of the message may be selected from a destination included in person information acquired an identifying process of a person in a field-of-view image described above or destination to which a message is sent in the past.

Next, a message content transmitting module 16 of the user terminal 10 determines whether a message content to be transmitted is inputted. If the input of the message content is not completed (Step S22: NO), a message content input receiving module 21 receives an input of the message content from the user (Step S23). When the message content has been already inputted to message content input receiving module 21 before the destination of the message is selected (Step S22: YES), the process continues without performing the step S23.

Thereafter, the message content transmitting module 16 transmits the message content and the destination of the message to a message transmission server 200 (step S24). A transmission message receiving module 203 of the message transmission server 200 receives the message content and the destination of the message (step S25).

A message transmitting module 204 of the message transmission server performs a transmission process of the message based on the received message content and destination of the message (step S26). Further, the transmission process may be for example, a process for processing a mail transmission request if the message is a mail or a process for writing the message content into a message box area of a user corresponding to the destination if the message is a message function in an SNS.

The above is the procedure of the message transmission process that is executed by the user terminal 10 and the message transmission server 200 in the message transmission system 1.

The above-described means and functions may be realized by reading and executing a predetermined program by a computer (including a CPU, an information processing apparatus, and various terminals). The program may be recorded in a computer-readable recording medium, for example, a flexible disk, a CD (e.g., a CD-ROM or the like) and a DVD (e.g., a DVD-ROM, a DVD-RAM, or the like). In this case, the program may be read from the recording medium by the computer and be transmitted to an internal storage unit or an external storage unit to be stored and executed. Further, the program may be pre-stored in a storage unit (recording medium) such as a magnetic disk, an optical disk, or an optical magnetic disk and be transmitted from the recording medium to the computer through a communications line.

While embodiments of the present invention have been described, the present invention is not limited to these embodiments described above. Further, the advantageous effects described in embodiments of the present invention are merely a list of the most preferable effects resulting from the present invention and the advantageous effects of the present invention are not limited to those described in embodiments of the present invention.

DESCRIPTION OF REFERENCE NUMBERS

1: Message transmission system, 10: User terminal, 100: Field-of-view image analysis server, 150: Feature amount database, 151: Personal information database, 200: Message transmission server

Claims

1. A message transmission system for performing a communication with a person that exists in a field of view of a user, the message transmission system comprising:

a field-of-view image acquisition apparatus including at least a field-of-view image acquiring means configured to acquire a field of view of the user as an image;

a feature amount extracting means configured to analyze a field-of-view image acquired by the field-of-view image acquiring means and to extract a feature amount associated with an identification of a person;

an person identifying means configured to identify a person within the field-of-view image from the feature amount extracted by the feature amount extracting means; and

a message transmitting means configured to transmit an electronic message to the person identified by the person identifying means.

2. The message transmission system of claim 1, further comprising a field-of-view image transmitting means configured to transmit a part or whole of the field-of-view image acquired by the field-of-view image acquiring means to the person identified by the person identifying means.

3. The message transmission system of claim 1, further comprising:

a person image position calculating means configured to calculate a position of the person identified by the person identifying means within the field-of-view image; and

a person information displaying means configured to display information on the identified person within the field of view of the user based on the position calculated by the person image position calculating means.

4. A message transmission method executed by a message transmission system for performing a communication with a person that exists in a field of view of a user, the message transmission method comprising:

acquiring a field of view of the user as an image by a field-of-view image acquisition apparatus;

analyzing the acquired field-of-view image and extracting a feature amount associated with an identification of a person;

identifying a person within the field-of-view image from the extracted feature amount; and

transmitting an electronic message to the identified person.

5. The message transmission method of claim 4, further comprising transmitting a part or whole of the acquired field-of-view image to the identified person.

6. The message transmission method of claim 4, further comprising:

calculating a position of the identified person within the field-of-view image; and

displaying information on the identified person within the field of view of the user based on the calculated position.

7. A message transmission system program for executing a message transmission method on a message transmission system for performing a communication with a person that exists in a field of view of a user, the message transmission method comprising:

acquiring a field of view of the user as an image;

analyzing the acquired field-of-view image and extracting a feature amount associated with an identification of a person;

identifying a person within the field-of-view image from the extracted feature amount; and

transmitting an electronic message to the identified person.

8. The message transmission system program of claim 7, wherein the message transmission method further comprises transmitting a part or whole of the acquired field-of-view image to the identified person.

9. The message transmission system program of claim 7, wherein the message transmission method further comprises:

calculating a position of the identified person within the field-of-view image; and

displaying information on the identified person within the field of view of the user based on the calculated position.