Information processing device, information processing program, and information processing method

- PREFERRED NETWORKS, INC.

An information processing device includes a memory, and processing circuitry coupled to the memory. The processing circuitry is configured to acquire gradation processing target image data, and perform gradation processing on the gradation processing target image data based on a learned model learned in advance.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to Japanese Patent Application No. 2017-248963, filed on Dec. 26, 2017, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

Embodiments of the present disclosure relate to an information processing device, an information processing program, and an information processing method capable of automatically performing gradation processing on image data.

BACKGROUND

In recent years, machine learning using a neural network having a multilayer structure, called deep learning has been applied in various fields. The machine learning has also been prominently utilized and has achieved a remarkable result in a field of image processing such as image recognition and image generation.

For example, “Automatic coloring of black-and-white photograph by learning of global feature and local feature using deep network” (Satoshi Iizuka, Simosela Edgar, and Hiroshi Ishikawa)

(http://hi.cs.waseda.ac.jp/˜iizuka/projects/colorization/ja/) achieves processing of automatic coloring of a black-and-white photograph by a deep network, and achieves coloring processing of the black-and-white photograph by machine learning.

As in “Automatic coloring of black-and-white photograph by learning of global feature and local feature using deep network” (Satoshi Iizuka, Simosela Edgar, and Hiroshi Ishikawa) (http://hi.cs.waseda.ac.jp/˜iizuka/projects/colorization/ja/), in addition to a need for users to perform coloring on image data, there is a need for users to perform gradation processing on image data such as line drawing data that does not include gradation information or to adjust gradation of image data that already includes gradation information. Here, the gradation processing refers to shadow color processing for adding a shadow, oblique line processing for applying expression by an oblique line to a portion that becomes a shadow, tone processing for adding a tone used for comics and the like, highlight processing for changing a part of the shadow that has already applied, tone conversion processing for converting a tone place of an image to which a tone is added into gray scale expression, special print processing for generating data used for special printing such as embossing processing or gold foil processing and the like. Conventionally, a process of completing an image by performing the gradation processing on the image data has been performed by a skilled person, but there was a problem that a target number of sheets cannot be treated (gradation-processed) in a limited time in the case of work by a human hand. For this reason, a structure capable of automatically performing the gradation processing has been demanded.

SUMMARY

Embodiments of the present disclosure have been made in consideration of the problems described above, and an object of the present disclosure is to provide an information processing device, an information processing program, and an information processing method capable of automatically performing gradation processing. The terms “learned model” and “trained model” as used in the present disclosure to describe various embodiments may be used interchangeably with each other. Similarly, the terms “learned” and “trained” as used in the present disclosure to describe various embodiments may be used interchangeably with each other.

An information processing device according to some embodiments of the present disclosure may include a gradation processing target image data acquiring unit configured to acquire gradation processing target image data, and a gradation processing unit configured to perform gradation processing on the gradation processing target image data based on a learned model learned in advance.

An information processing device according to some embodiments of the present disclosure may include a gradation processing target image data acquiring unit configured to acquire gradation processing target image data, and a gradation processing unit configured to output at least one or more mask channels for performing gradation processing on the gradation processing target image data based on a learned model learned in advance.

In addition, the information processing device according to some embodiments of the present disclosure may further include a processing content designation receiving unit configured to receive one or more designated gradation processing contents which is selected by a user from a plurality of gradation processing contents. The gradation processing unit may perform the gradation processing on the gradation processing target image data based on a learned model corresponding to a designated gradation processing content received by the processing content designation receiving unit among a plurality of learned models learned in advance.

In addition, the information processing device according to some embodiments of the present disclosure may further include a processing content designation receiving unit configured to receive one or more designated gradation processing contents which is selected by a user from a plurality of gradation processing contents. The gradation processing unit may output at least one or more mask channels for performing the designated gradation processing on the gradation processing target image data based on a learned model corresponding to a designated gradation processing content received by the processing content designation receiving unit among a plurality of learned models learned in advance.

In addition, the information processing device according to some embodiments of the present disclosure may further include a hint information acquiring unit configured to acquire hint information of gradation processing on at least one place of the gradation processing target image data. The gradation processing unit may execute processing based on the gradation processing target image data and the hint information.

In addition, in the information processing device according to some embodiments of the present disclosure, the learned model may perform learning based on both of training data to which the hint information of the gradation processing is not given and training data to which the hint information of the gradation processing on at least one place of the gradation processing target image data is given.

In addition, the information processing device according to some embodiments of the present disclosure may further include a light source position information acquiring unit configured to acquire light source position information from a user for designating a position of a light source for the gradation processing target image data. The gradation processing unit may execute processing based on the gradation processing target image data and the light source position information.

In addition, in the information processing device according to some embodiments of the present disclosure, the learned model may perform learning based on both of training data to which the light source position information is not given and training data to which the light source position information on at least one place of the gradation processing target image data is given.

An information processing program according to some embodiments of the present disclosure may cause a computer or a processor to achieve the following functions: a gradation processing target image data acquiring function of acquiring gradation processing target image data, and a gradation processing function of performing gradation processing on the gradation processing target image data based on a learned model learned in advance.

An information processing method according to some embodiments of the present disclosure may include a gradation processing target image data acquiring step of acquiring gradation processing target image data, and a gradation processing step of performing gradation processing on the gradation processing target image data based on a learned model learned in advance.

An information processing device according to some embodiments of the present disclosure may include a gradation processing target image data input form displaying unit configured to display a form region for a user to input gradation processing target image data on a display screen, a gradation processing target image displaying unit configured to display an image indicated by the input gradation processing target image data in a gradation processing target image display region provided on the display screen, and a gradation-processed image displaying unit configured to display an image indicated by gradation-processed image data obtained by performing gradation processing on the gradation processing target image data based on a learned model learned in advance, in a gradation-processed image display region provided on the display screen.

In addition, the information processing device according to some embodiments of the present disclosure may further include a hint information input tool displaying unit configured to display a hint information input tool on the display screen and receive input of hint information, the hint information input tool being configured to designate a place at which the gradation processing is to be performed on the image indicated by the gradation processing target image data, displayed in the gradation processing target image display region. The gradation-processed image displaying unit may display the image indicated by the gradation-processed image data obtained by performing the gradation processing on the gradation processing target image data in a state in which the hint information is input, based on a learned model learned in advance, in the gradation-processed image display region provided on the display screen.

In addition, the information processing device according to some embodiments of the present disclosure may further include a light source position information input tool displaying unit configured to display a light source position information input tool on the display screen and receive input of light source position information, the light source position information input tool being configured to designate a light source position for the image indicated by the gradation processing target image data, displayed in the gradation processing target image display region. The gradation-processed image displaying unit may display the image indicated by the gradation-processed image data obtained by performing the gradation processing on the gradation processing target image data in a state in which the light source position information is input, based on a learned model learned in advance, in the gradation-processed image display region provided on the display screen.

According to some embodiments of the present disclosure, it is possible to automatically perform the gradation processing on the gradation processing target image data based on the learned model learned in advance using the training data with respect to the gradation processing for the gradation processing target image data. In addition, it is possible to properly perform the gradation processing after giving the hint information on the position at which the gradation processing is performed on the acquired gradation processing target image data or the light source position information for designating the light source position. It is possible to perform the gradation processing in which the hint information or the light source position information is designated for the gradation processing target image data by advancing the learning in a form in which the designation for the hint information or the light source position information is included in the learning process of the learned model used for the gradation processing.

In addition, according to the present disclosure, since the gradation processing target image display region and the gradation-processed image display region are displayed on the display screen viewed by the user by the information processing device, the user can compare and observe the original gradation processing target image data and the gradation-processed image data, and thus, it is possible to directly compare an atmosphere of an image before the gradation processing and an atmosphere of an image changed after the gradation processing with each other. In addition, the hint information for designating the place at which the gradation processing should be performed on the gradation processing target image indicated by the gradation processing target image data displayed in the gradation processing target image display region or the light source position information can be input, such that the gradation processing can be executed in a state in which the hint information or the light source position information is added. Therefore, it is possible to give the hint information on the place at which the user desires to freely perform the gradation processing on the gradation processing target image or the light source position information to execute the gradation processing.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration of an automatic gradation processing device according to a first embodiment of the present disclosure;

FIG. 2 is a block diagram showing a hardware configuration required for realizing the automatic gradation processing device;

FIG. 3 is a block diagram showing an example of a system configuration for the automatic gradation processing device;

FIG. 4 is an explanatory diagram showing a concept of a neural network;

FIGS. 5A and 5B are explanatory diagrams showing a method of giving light source position information for gradation processing target image data;

FIG. 6 is a flowchart showing a flow of gradation processing in the automatic gradation processing device according to the first embodiment;

FIGS. 7A and 7B are explanatory diagrams showing an example of a display screen displayed by a program for a graphical user interface according to a second embodiment; and

FIG. 8 is a flowchart showing a flow of processing in the case of providing an automatic gradation processing tool based on a GUI according to the second embodiment.

 DETAILED DESCRIPTION

The terms “learned model” and “trained model” as used in the present disclosure to describe various embodiments may be used interchangeably with each other. Similarly, the terms “learned” and “trained” as used in the present disclosure to describe various embodiments may be used interchangeably with each other.

First Embodiment

Hereinafter, an example of an automatic gradation processing device according to a first embodiment of an information processing device of the present disclosure will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an automatic gradation processing device 10 according to some embodiments of the present disclosure. As shown in FIG. 1, the automatic gradation processing device 10 includes at least a gradation processing target image data acquiring unit 11, a hint information acquiring unit 12, a light source position information acquiring unit 13, a gradation processing unit 14, a storing unit 15, and a processing content designation receiving unit (not shown). The storing unit 15 may be implemented with EPROM, EEPROM, SDRAM, and flash memory devices, CD ROM, DVD-ROM, or Blu-Ray® discs and the like. At least one or more of the gradation processing target image data acquiring unit 11, the hint information acquiring unit 12, the light source position information acquiring unit 13, the gradation processing unit 14, and the processing content designation receiving unit may be implemented with a special circuit (e.g., processing circuitry of a FPGA or the like), a subroutine in a program stored in memory (e.g., EPROM, EEPROM, SDRAM, and flash memory devices, CD ROM, DVD-ROM, or Blu-Ray® discs and the like) and executable by a processor (e.g., CPU, GPU and the like), or the like.

In some embodiments, the automatic gradation processing device 10 may be a device designed as a dedicated machine. In some embodiments, the automatic gradation processing device 10 may be a device that can be achieved by a general computer. FIG. 2 is a block diagram showing a hardware configuration required for realizing the automatic gradation processing device 10. In some embodiments, as shown in FIG. 2, the automatic gradation processing device 10 includes a central processing unit (CPU) 51, a graphics processing unit (GPU) 52, a memory 53, and a storage 54 such as a hard disk drive or a solid state drive (SSD), which are generally included in a general computer, and includes an input device 55 such as a mouse and a keyboard, an output device 56 such as a display (or a display screen) and a printer, and a communication device 57 for connection with a communication network, and these components are connected to each other through a bus 58.

FIG. 3 is a block diagram showing an example of a system configuration for the automatic gradation processing device 10. In FIG. 3, a server device 60 and a terminal device 70 (or a plurality of terminal devices 70-1 to 70-n) are configured to be connectable to each other through a communication network 80. For example, in FIG. 3, the server device 60 may be allowed to function as the automatic gradation processing device 10 and any one of the plurality of terminal devices 70-1 to 70-n may be connected to the server device 60 functioning as the automatic gradation processing device 10 through the communication network to use the automatic gradation processing device 10. In this case, a configuration in which a graphical user interface program for using the automatic gradation processing device 10 is installed in the terminal device 70 may be adopted, a configuration in which a graphical user interface program on a server is used through a browser may be adopted, or an information processing device having a function of displaying various displays as a graphical user interface may be adopted.

In addition, the same device need not include all of the components of the automatic gradation processing device 10 to be described below, and some components are included in another device. For example, some components may be included, respectively, in the server device 60 and any one of the plurality of terminal devices 70-1 to 70-n connectable to each other through the communication network, such that the automatic gradation processing device 10 may use components in another device while performing communication. In addition, the server device 60 is not limited to a single server device 60, and a configuration using a plurality of server devices may be adopted. In addition, learned models to be described below may be distributed and included in the server device 60, the plurality of terminal devices 70-1 to 70-n and the like, as another device. In some embodiments, the automatic gradation processing device 10 may be connected to a device including a used learned model through the communication network each time it uses the used learned model. In some embodiments, the learned models are stored in a device as the automatic gradation processing device 10. In some embodiments, when a learned model stored by any learned model storage means can be used, it is not inquired whether the learned model storage means is included in the automatic gradation processing device 10 itself or is included in another device.

The gradation processing target image data acquiring unit 11 may have a function of acquiring gradation processing target image data. Here, the gradation processing in the first embodiment refers to at least one of shadow color processing for drawing a shadow in an image in which the shadow is not drawn, oblique line drawing processing for drawing oblique lines for expressing a shadow, tone processing for adding a tone to image data, highlight processing for changing a gradation of image data, tone conversion processing for converting a tone place of an image to which a tone is added into gray scale expression, special print processing for generating data used for special printing such as embossing processing or gold foil processing, or various processing for changing gradation information. In some embodiments, the gradation processing target image data may include image data to which a user desires to add gradation information from now, such as line drawing data that does not include gradation information and is drawn by binary of black and white. In some embodiments, the gradation processing target image data may include image data that already includes gradation information and is a target to further modify the gradation information, for example, a colored image, a shadow-processed image, a tone-added image, a photograph and the like. For example, the user may acquire the gradation processing target image data by the gradation processing target image data acquiring unit 11 (see FIG. 1) by selecting image data on which he/she desires to perform the gradation processing.

The hint information acquiring unit 12 may have a function of receiving and acquiring hint information including designation for at least one or more of places at which the user desires to perform the gradation processing on the gradation processing target image data, from the user. As an example of a method of designating the hint information by the user, the designation may be performed by a sequence of receiving designation for types of gradation processing such as the shadow color processing, the oblique line drawing processing, the tone processing, or the highlight processing and a sequence of designating places at which the user desires to perform the designated gradation processing for the gradation processing target image data displayed on a display screen. The method of designating the hint information may be performed by a processing content designation receiving unit, which may be provided as one of components of the automatic gradation processing device 10. In some embodiments, designation is not only performed for one type of gradation processing, but may also be simultaneously performed for two or more types of gradation processing, and designation for a place at which the gradation processing is performed in this case may be performed for a plurality of places. In some embodiments, when the gradation processing is executed without acquiring the hint information, the acquisition of the hint information in the hint information acquiring unit 12 can be omitted. The acquired hint information may be used for gradation processing in a gradation processing unit 14 to be described below. In some embodiments, the hint information acquiring unit 12 is not an essential component, and the gradation processing can be performed by a learned model learned using training data that does not include the hint information even though the hint information is not given.

The light source position information acquiring unit 13 may have a function of acquiring light source position information in the case of giving the light source position information when the gradation processing is performed on the gradation processing target image data. The gradation processing can be performed by depicting an influence from a virtual light source on the gradation processing target image data, and a position at which the gradation processing is performed may be changed depending on where a light source is positioned. Therefore, the light source position information may be given for the gradation processing target image data. As a method of giving the light source position information for the gradation processing target image data, various methods can be considered. For example, the method of giving the light source position information to the gradation processing target image data may include a method of designating a light source position by the user. In some embodiments, the method of giving the light source position information to the gradation processing target image data may include a method of automatically discriminating the light source position in the gradation processing target image data based on a learned model for discriminating a light source position in which it is learned in advance to discriminate the light source position in the gradation processing target image data. The acquired light source position information may be used for gradation processing in a gradation processing unit 14 to be described below. In some embodiments, the light source position information acquiring unit 13 is not an essential component, and the gradation processing can be performed by a learned model which may be learned using training data that does not include the light source position information even though the light source position information is not given. In some embodiments, when light source position information disagreeing with the hint information acquired by the hint information acquiring unit 12 is designated, because there is a risk that gradation processing in which both of the hint information and the light source position information are reflected cannot be executed, any one of the hint information and the light source position information may be prioritized.

FIGS. 5A and 5B are explanatory diagrams showing a method of giving light source position information for gradation processing target image data. FIG. 5A is an explanatory diagram showing a case in which the light source position information is two-dimensionally given for the gradation processing target image data 500, and FIG. 5B is an explanatory diagram showing a case in which the light source position information is three-dimensionally given for the gradation processing target image data 500. In the case in which the light source position information is given two-dimensionally as shown in FIG. 5A, the light source position information from any direction on the same plane as that of the gradation processing target image data is given. In the case in which the light source position information is three-dimensionally given as shown in FIG. 5B, the light source position information is given by designating any position on a surface of a sphere 600 centering on the gradation processing target image data. In either case of FIG. 5A or FIG. 5B, designation places of the light source position information may be set to be designated in advance based on predetermined divisions. For example, in the case in which the light source position information is two-dimensionally given, the designation places of the light source position information may be set to eight directions 510 as shown in FIG. 5A, may be set to sixteen directions of double fineness, or may be set to be more finely designated. In the case in which the light source position information is three-dimensionally given, the designation places of the light source position information may be set to a total of 64 directions divided into eight directions (x-y plane, e.g., a plane 610 in FIG. 5B)×eight directions (x-z plane), may be set to a total of 256 directions divided into sixteen directions (x-y plane)×sixteen directions (x-z plane), or may be set to be more finely designated. As described above, the designation may be received from the directions of the predetermined divisions, such that it is possible to perform the gradation processing using an example learned based on training data. In some embodiments, the training data includes a tag of light source position information of directions of the same divisions in a learned model.

The gradation processing unit 14 may have a function of outputting at least one or more mask channels for performing the gradation processing on the gradation processing target image data, based on a learned model in which a mask channel overlapped with the gradation processing target image data is learned in advance using training data to generate a gradation processing result as the mask channel. In order to execute the gradation processing in the gradation processing unit 14, designation for at least one type of the gradation processing target image data and a gradation processing content may be given to the gradation processing unit 14. In some embodiments, the hint information, which is acquired by the hint information acquiring unit 12, on the position at which the gradation processing is performed, or the light source position information acquired by the light source position information acquiring unit 13 may be given to the gradation processing unit 14. The gradation processing unit 14 to which these pieces of information is given may execute the gradation processing using a learned model in which it is learned in advance to perform the gradation processing. For some gradation processing such as the shadow color processing, it is possible to perform image processing directly on the gradation processing target image data without outputting a mask channel. In processing for expressing a predetermined range with a specific texture, such as the tone processing, a mask channel for instructing a type and a place of tone to be applied may be outputted. Based on information of the mask channel output from a learned model, the automatic gradation processing device 10 may perform actual tone addition processing and the like, or may add a tone to a layer different from the gradation processing target image data using a layer to be described below. In some embodiments, the server device as the automatic gradation processing device 10 may execute these processing, or the terminal device as the automatic gradation processing device 10 may acquire a mask channel outputted from a learned model on the server device and perform the actual tone adding processing and the like.

Here, a learning method of the learned model will be described. The learned model may be a model for learning to output at least one or more mask channels for performing the gradation processing using the gradation processing target image data as an input, and may refer to, for example, a model composed of a neural network including a convolutional neural network (CNN). FIG. 4 is an explanatory diagram showing a concept of a neural network. A model for learning can be composed of, for example, an encoder that extracts an abstract feature quantity of the input gradation processing target image data and a decoder that outputs a mask channel for gradation processing based on the extracted feature quantity. This is only an example, and various configurations of the neural network can be applied. Learning is performed on the neural network having such a configuration.

At the time of the learning, training data may be used. As the training data, for example, gradation processing target image data and correct answer image data in which gradation processing is performed on the gradation processing target image data are used as one set. In some embodiments, the gradation processing target image data set in the training data includes various image data such as line drawing data, a colored image, shadow-processed image, a tone-added image, a photograph and the like. In some embodiments, the correct answer image data in which the gradation processing is performed on the gradation processing target image data is prepared together with gradation processing target image data. In some embodiments, the gradation processing target image data and the correct answer image data are set as a set of training data. In the learning processing, the gradation processing target image data may be input to a model that the user desires to learn, and a gradation processing result may be outputted as a mask channel. A loss function may be calculated using gradation-processed image data obtained by overlapping the mask channel of the gradation processing result with the gradation processing target image data and the correct answer image data, and a parameter of the neural network may be updated in a direction in which loss is reduced. A learned model may be obtained by executing the update of the parameter as described above based on a plurality of training data.

In some embodiments, at the time of the learning, hint information on a position at which the gradation processing is performed may be given together with the gradation processing target image data to perform the learning, thereby learning gradation processing with the hint information. For example, a position picked up randomly from the correct answer image data or gradation information generated from the correct answer image data may be input as the hint information together with the gradation processing target image data to a model to perform the learning of the model. In this case, it is preferable that the number of given hint information is randomly switched from 0 to a finite predetermined number to execute the learning, thereby performing the learning so that a learned model can perform the gradation processing in the same manner in the case in which the number of hint information is 0 and in the case in which a plurality of hint information is given.

In some embodiments, at the time of the learning, the light source position information may be given together with the gradation processing target image data to perform the learning, thereby learning gradation processing with the light source position information. The light source position information in the correct answer image data may be extracted, and the extracted light source position information may be input together with the gradation processing target image data to the model to perform learning of the model. The light source position information may be information such as tag information. In some embodiments, the light source position information may be designated based on predetermined divisions. Learning with designation for the light source position information from various directions may be sufficiently performed, such that it is possible to perform the gradation processing regardless of a designated direction. In some embodiments, the light source position information may be mechanically produced from 3D data or 2D data. In some embodiments, the light source position information need not necessarily designate an accurate position, and may be rough information such as backlight.

In addition, in the description of the learning, the description has been made as a method common to all types of gradation processing, but elements specific to each gradation processing may also exist. The shadow color processing and the highlight processing may be processing for realizing expression of a shadow or highlight by making gradation information (e.g; brightness information) different from that of the surroundings in the final gradation-processed image data of the gradation processing target image data. On the other hand, the tone processing may be processing for adding a tone by determining a place at which the processing is performed and determining which of plural types of tones that exist are to be used to perform the processing. That is, in the case of the tone processing, a set of effective classes of which the number corresponds to the number of types of tones may be set, and the processing may be performed by the neural network as a classification problem of effective classes on the gradation-processed image data, such that processing of a tone corresponding to a class having the highest possibility is performed on each place of the gradation-processed image data. In some embodiments, in order to simultaneously perform processing by plural types of tones, a mask channel may be prepared for each class, and a plurality of tones processed using a plurality of mask channels may be finally overlapped with the gradation-processed image data, thereby obtaining the gradation-processed image data. In other gradation processing including the shadow color processing, the highlight processing, the oblique line drawing processing and the like, as well as the tone processing, a set of a plurality of effective classes depending on the number of types of processing methods may be set, and the processing may be performed by the neural network as a classification problem of the effective classes, thereby freely using the processing methods.

In some embodiments, in generation of the learned model, the learning may be performed for each type of gradation processing such as the shadow color processing, the highlight processing, the oblique line drawing processing, and the tone processing to obtain the learned model. In a step in which designation for at least one type of gradation processing contents is given at the time of performing the gradation processing, a learned model for the gradation processing contents may be used. The automatic gradation processing device 10 may be an automatic gradation processing device specialized for one type of gradation processing, and in this case, only a learned model relating to one type of gradation processing may be generated and stored.

The storing unit 15 has a function of storing data necessary for various types of processing performed in the automatic gradation processing device 10 including the gradation processing target image data acquiring unit 11, the hint information acquiring unit 12, the light source position information acquiring unit 13, and the gradation processing unit 14 and the like, and data obtained as a processing result. In addition, a learned model for each type of gradation processing obtained by performing the learning in advance may be stored in the storing unit 15.

Next, a flow of gradation processing in the automatic gradation processing device 10 according to the first embodiment will be described. FIG. 6 is a flowchart showing a flow of gradation processing in the automatic gradation processing device 10 according to the first embodiment. The gradation processing in the automatic gradation processing device 10 according to the first embodiment is started by acquiring the gradation processing target image data (step S101). For example, the user can select the gradation processing target image data, such that the acquisition of the gradation processing target image data is performed. A gradation processing content that the user desires to apply to the gradation processing target image data may be determined before the gradation processing target image data is acquired or may be determined after the gradation processing target image data is acquired.

After the gradation processing target image data is acquired, the hint information on the gradation processing for the gradation processing target image data may be acquired (step S102). The hint information may be information including designation for a position of the gradation processing target image data to which the determined gradation processing content is to be applied. In some embodiments, the designation is not only performed for one type of gradation processing, but may also performed at the same timing for two or more types of gradation processing. In this case, the hint information may include information relating to determination for a type of gradation processing and designation for a place at which the gradation processing is performed. In some embodiments, the gradation processing can be executed even though the hint information is not input.

Next, the light source position information on the gradation processing target image data may be acquired (step S103). The light source position information may be information designated by the user, or may be information acquired by automatically discriminating a light source position in the gradation processing target image data based on a learned model for discriminating a light source position. In some embodiments, the gradation processing can be executed even though the light source position information is not input.

Then, the gradation processing may be executed on the gradation processing target image data (step S104). The gradation processing may be executed by inputting the gradation processing target image data, the hint information, and the light source position information to a learned model corresponding to the determined gradation processing content. Then, the automatic gradation processing device 10 may output at least one or more mask channels for performing the gradation processing as the gradation processing result from the learned model (step S105), and may end the gradation processing.

As described above, with the automatic gradation processing device 10 according to the first embodiment, it is possible to properly perform the gradation processing after giving the hint information on the position at which the gradation processing is performed on the acquired gradation processing target image data or the light source position information for designating the light source position. It is possible to perform the gradation processing in which the hint information or the light source position information is designated for the gradation processing target image data by advancing the learning in a form in which designation for the hint information or the light source position information is included in a learning process of the learned model used for the gradation processing.

Second Embodiment

Hereinafter, an information processing device according to a second embodiment will be described with reference to the drawings. In the first embodiment, the automatic gradation processing device 10 has been described, but in the second embodiment, an information processing device for providing a graphical user interface used at the time of using the automatic gradation processing device 10 according to the first embodiment will be described. The information processing device according to the second embodiment will be described as providing a graphical user interface as an automatic gradation processing tool. For example, a method of providing a server device functioning as the automatic gradation processing device 10 according to the first embodiment and providing an automatic gradation processing tool to a user who has accessed the server device from a terminal device through a communication network can be considered. In such a case, the automatic gradation processing tool is not limited to being provided to the terminal device by software of a package, and can also be provided by being read and functioning by a browser or the like, causing a graphical user interface (GUI) stored in the server device to be displayed on a display of the terminal device. The automatic gradation processing tool refers to a tool used when a user uses the automatic gradation processing device 10 according to the first embodiment, and various providing patterns of the automatic gradation processing tool such as a case in which the automatic gradation processing tool is provided as an independent program, a case in which the automatic gradation processing tool is provided as a Web browser, and a case in which the automatic gradation processing tool is included as some functions of software of a package such as image editing software can be considered.

As described below, according to the second embodiment, at least one or more learned models are stored in the server device functioning as the automatic gradation processing device 10 and the automatic gradation processing tool is used by accessing the server device from the terminal device through the communication network, and a terminal device in which all of the learned models are stored can also use the same GUI.

FIGS. 7A and 7B are explanatory diagrams showing an example of a display screen displayed by a GUI as an automatic gradation processing tool of the second embodiment. FIG. 7A shows a display screen 700 when gradation processing target image data is input, and FIG. 7B shows a display screen 800 as an example of hint information input tools, a gradation processing target image display region 880 for displaying gradation processing target image data, and a gradation-processed image display region 890 for displaying gradation-processed image data obtained by overlapping a mask channel of a gradation processing result with the gradation processing target image data. When the automatic gradation processing tool is provided from the server device to the terminal device (e.g., the automatic gradation processing tool is provided to a user who has accessed the server device from a terminal device through a communication network), first, as shown in FIG. 7A, a gradation processing target image data input form 3010, which is a form region for the user to input the gradation processing target image data, may be displayed on a display of the terminal device through, for example, a Web browser or the like. This gradation processing target image data input form may be a form in which the gradation processing target image data is inputted by a method of designating a path of a file in FIG. 7A, but is not limited thereto. For example, a method of selecting line drawing data by drag and drop may also be used. According to the second embodiment, the display screen can be displayed to the user when the server device provides a GUI as an automatic gradation processing tool by a program for a graphical user interface, a Web browser or the like. In some embodiments, the display screen may include both of a display screen generated by the server device and a display screen generated by the terminal device.

When the gradation processing target image data is designated, an image represented by the gradation processing target image data may be displayed on a display screen 800 in the gradation processing target image display region 880 shown in FIG. 7B. In some embodiments, as shown in FIG. 7B, hint Information input tools 810, 820, 830, 840, 850, 860 for designating a place at which the gradation processing should be performed on the gradation processing target image data displayed in the gradation processing target image display region 880 may be displayed on the display screen. The hint Information input tools may include a “return one work” tool 810, an “advance one work” tool 820, a “selection of pen for inputting hint information” tool 830, a “delete input hint information (eraser)” tool 840, a “selection of type of gradation processing” tool 850 and an “input of light source position information” tool 860 which can also called a light source position information input tool in an example shown in FIG. 7B, but is not limited thereto. In addition, FIG. 7B shows a display screen after a type of gradation processing that the user desires to perform is determined. In some embodiments, in order to input the hint information while switching plural types of gradation processing, a “switching of types of gradation processing” tool (not shown) may be provided as the hint information input tool. As a method of using the hint information input tools, for example, in the case of tone processing, hint information on a position at which the tone processing is performed may be given by actually performing designation for the position by a mouse operation by the user. For example, the user can perform a mouse operation to designate the position by adding a dot, wiring a line segment, or painting out a region, with a pointer on a place at which the user desires to perform the tone processing. When an execution button 870 displayed on the same screen is clicked by a mouse operation or the like, the gradation processing may be executed in a state in which the hint information is input, and the gradation-processed image data in which the hint information is reflected may be displayed in the gradation-processed image display region 890.

In some embodiments, as a method of using the “input of light source position information” tool 860 of the hint information input tools shown in FIG. 7B, when the “input of light source position information” tool 860 is selected (e.g., by a mouse operation or a keyboard operation), designation for a light source position by the user may be received by displaying an animation for designating the light source position as shown in FIG. 5A or FIG. 5B in the gradation processing target image display region 880 in an overlap manner. In some embodiments, designation for a light source position may be received by moving to another screen for designating the light source position as shown in FIG. 5A or FIG. 5B. When an execution button of the display screen is clicked in a state in which the light source position information is input, the gradation processing may be executed in a state in which the light source position information is input, and gradation-processed image data in which the light source position information is reflected may be displayed in the gradation-processed image display region 890.

FIG. 8 is a flowchart showing a flow of processing in the case of providing an automatic gradation processing tool based on a GUI according to the second embodiment. A flow of processing for providing an automatic gradation processing tool may be started by displaying the gradation processing target image data input form (e.g., the form 710) on the display screen (e.g., the display screen 700 in FIG. 7A) of the display of the terminal device from the server device and receiving the input of the gradation processing target image data (step S201), as shown in FIG. 8. When the user inputs the gradation processing target image data, the gradation processing target image data may be transmitted to the server device. In the server device receiving the gradation processing target image data, the gradation processing target image data may be displayed in the gradation processing target image display region 880 of the display screen (step S202). Next, the input of the hint information using the hint information input tools (e.g., the tools 810-870 in FIG. 7B) for the gradation processing target image data displayed on the display screen may be received (step S203). In some embodiments, designation for the light source position information may be received in step S203. In some embodiments, a type of gradation processing has been determined up to a step before the hint information is received. Then, it may be determined whether or not an execution button (e.g., the button 870 in FIG. 7B) for the gradation processing is pressed (step S204). When the execution button for the gradation processing is not pressed (S204-N), a flow may return to step S203 to further receive the input of the hint information. When the execution button for the gradation processing is pressed (S204-Y), the gradation processing in which the hint information is reflected may be executed (step S205). The execution of the gradation processing in step S205 may be the same as a flow of the gradation processing according to the first embodiment described with reference to the flowchart of FIG. 6, and the same processing as steps S101 to S105 in FIG. 6 may be executed in step S205. Finally, the gradation-processed image data obtained by the gradation processing may be displayed in the gradation-processed image display region (e.g., the display region 890 in FIG. 7B) of the display screen (step S206), and the processing ends.

As described above, in some embodiments, the GUI may be provided from the server device to the display screen of the display of the terminal device, the automatic gradation processing tool may be provided to the user by the GUI, and the gradation processing target image display region and the gradation-processed image display region may be provided on the same display screen as a function of the GUI. Therefore, the user can compare and observe the original gradation processing target image data and the gradation-processed image data, and thus, it is possible to directly compare a gradation characteristic of an image before the gradation processing and a gradation characteristic of an image changed after the gradation processing with each other. In addition, the hint information for designating the place at which the gradation processing should be performed on the gradation processing target image indicated by the gradation processing target image data displayed in the gradation processing target image display region can be input, such that the gradation processing can be executed in a state in which the hint information is added. Therefore, it is possible to easily and effectively give the hint for the place at which the user desires to perform the gradation processing on the gradation processing target image to execute the gradation processing. Since the addition of the hint information does not determine the execution of the gradation processing for the place, but causes the learned model to execute the gradation processing in a state in which the hint information is input, the gradation processing is not necessarily performed on the designated place. Since learning is performed in a learning process of a learned model used by an automatic gradation processing program in the state in which the hint information is input, it can be determined how the designated hint information is adopted by a tendency of training data and the hint information used for the learning. Such adoption of the designated hint information by a learning process using training data to execute gradation processing at the designated place according to embodiments of the present disclosure is a feature completely different from gradation processing in the conventional image editing software or the like.

Third Embodiment

In the second embodiment, the information processing device for providing the graphical user interface used at the time of using the automatic gradation processing device 10 has been described, but, in some embodiments, the automatic gradation processing device 10 and the automatic gradation processing program according to the present disclosure can also be incorporated into drawing creation software, image editing software, a WEB service (hereinafter, also referred to as editing software) for providing a service equivalent to these software from a server device to a terminal device, and the like. In some embodiments, the editing software itself may include all components of the automatic gradation processing device 10. For example, the editing software may include a learned model. In some embodiments, the editing software in the terminal device may be connected to the automatic gradation processing device 10 including a learned model provided in the server device through the communication network to use components of the automatic gradation processing device 10.

The editing software including the drawing creation software or the image editing software may have a function of creating the gradation processing target image data or a function of adding the gradation processing target image data. In some embodiments, the editing software has a configuration in which the creation of the gradation processing target image data or the addition of the gradation processing target image data are managed by a plurality of layers, such that the gradation processing target image data is created or added in a specific layer. In some embodiments, in the editing software, there is a function of forming a closed region within a layer and performing specific processing only within the closed region, for example, a painting-out function, a hatching processing function or the like. The automatic gradation processing program according to some embodiments of the present disclosure may be incorporated into the editing software having the configuration and the functions as described above, such that more multifunctional editing software can be provided.

For example, it is possible to execute the gradation processing by the automatic gradation processing program on gradation processing target image data created based on the function of the editing software or added gradation processing target image data. Through the configuration as described above, it is possible to execute the gradation processing on the gradation processing target image data created by the editing software or the gradation processing target image data fetched into the editing software, and it is possible to further edit the gradation-processed image data obtained based on various functions of the editing software.

In some embodiments, when it is possible to manage the functions of the editing software using a plurality of layers, it is possible to output mask channels obtained as the gradation processing result based on the automatic gradation processing program to a specific layer. That is, since it is possible to output the mask channel to a layer different from a layer in which original gradation processing target image data is disposed, it can be determined whether or not to apply the gradation processing by switching turn-on/off of a display of the layer. In some embodiments, when the gradation processing result is output as a plurality of mask channels, it is possible to divide and output all of the mask channels to different layers. In some embodiments, when information of different textures is included for each applied tone in tone processing, after the gradation processing result is output to the layer of the editing software, it is possible to edit for changing a tone into a tone of a texture different from a type of a tone designated when the gradation processing result is output as the mask channel.

In addition, it is possible to execute the gradation processing by the automatic gradation processing program only on a specific layer by using the functions managed by the plurality of layers in the editing software. By using the functions as described above, the gradation processing target image data can be separately written into different layers for each part of the gradation processing target image data, and the gradation processing can be executed in a unit of a part, such that it is possible to finally obtain edited image data in which all the layers are superimposed. For example, a layer relating to gradation processing target image data of a person and a layer relating to gradation processing target image data of a background are provided, and each gradation processing is performed and then superimposed, such that it is possible to finally obtain one piece of image data.

In some embodiments, it is possible to execute the gradation processing by the automatic gradation processing program only in the region closed in the layer using the function of forming the closed region within the layer and performing the specific processing only within the closed region. When an image drawn in the closed region is recognized as one gradation processing target image data, the gradation processing can be applied particularly within the closed region without technical difficulty. It is possible to execute the gradation processing only in a region selected by the user by the function as described above.

As described above, by applying the automatic gradation processing program according to the present disclosure to the existing editing software, it is possible to further perform editing on the gradation-processed image indicated by the gradation-processed image data subjected to the automatic gradation processing based on the editing software. In some embodiments, it is possible to execute the automatic gradation processing in a unit of the layer or in a unit of the closed region within the layer, such that it is possible to provide editing software having an automatic gradation processing function that further improves user's convenience. The editing software having the automatic gradation processing function as described above can have the possibility of contributing to efficiency of the gradation processing work as a whole by allowing a part of the gradation processing work to be executed based on the automatic gradation processing program in a production site of computer graphics and animation.

In the first to third embodiments, it has been described that the learned model is generated and obtained by performing the learning for each type of gradation processing such as the shadow color processing, the highlight processing, the oblique line drawing processing, and the tone processing, but the number of learned models prepared for each type of gradation processing need not be one. A plurality of learned models (preferably, a plurality of learned models in which tendencies of training data used for learning are different from each other) may be prepared for each type of gradation processing to allow the user to be able to select which of the learned models is used to perform the gradation processing. In addition, in the first to third embodiments, it has been described that the learned model is generated and obtained by performing the learning for each type of gradation processing such as the shadow color processing, the highlight processing, the oblique line drawing processing, and the tone processing, but one learned model that can deal with any gradation processing may be obtained by learning one model for all types of gradation processing.

In the first to third embodiments, it has been described that the automatic gradation processing device 10 includes the hint information acquiring unit 12 and the light source position information acquiring unit 13, but the present disclosure is not limited thereto. In some embodiments, the automatic gradation processing device 10 may include only any one of the hint information acquiring unit 12 and the light source position information acquiring unit 13. In some embodiments, the automatic gradation processing device 10 may not include either of the hint information acquiring unit 12 or the light source position information acquiring unit 13. Even in the case in which the automatic gradation processing device 10 does not include either of the hint information acquiring unit 12 or the light source position information acquiring unit 13, when the learning is performed so that the gradation processing can be appropriately executed in a state in which the hint is not given at the time of the learning of the learned model, it is possible to appropriately execute the gradation processing.

In the first to third embodiments, it has been described that at least one or more mask channels for performing the gradation processing are output as the gradation processing result from the learned model, the present disclosure is not limited thereto, and the gradation-processed image data may be obtained by directly synthesizing the gradation processing result with the gradation processing target image data. In this case, the learning of the learned model is performed so that the gradation processing result can be directly synthesized with the gradation processing target image data.

Claims

1. An information processing device comprising:

at least one memory; and
at least one processor configured to:
acquire gradation processing target image data,
acquire light source position information for designating a position of a virtual light source to be applied to the gradation processing target image data, the light source position information being designated by a user,
acquire, from a neural network using at least both the gradation processing target image data and the light source position information, one or more channels to be applied to the gradation processing target image data in gradation processing that modifies gradation information included in the gradation processing target image data so as to depict a light effect of the virtual light source on the gradation processing target image data, the one or more channels being based on at least a feature of the gradation processing target image data, and the gradation processing including at least one of a process for adding a shadow or a process for highlighting an object in the gradation processing target image data, and
perform the gradation processing of modifying the gradation information included in the gradation processing target image data by applying the one or more channels to the gradation processing target image data, to generate gradation-processed image data.

2. The information processing device according to claim 1, wherein the one or more channels are overlapped with the gradation processing target image data in order to generate the gradation-processed image data.

3. The information processing device according to claim 1, wherein the at least one processor is further configured to:

estimate the light source position information of the acquired gradation processing target image data.

4. The information processing device according to claim 1, wherein the neural network includes both an encoder and a decoder.

5. The information processing device according to claim 1, wherein the light source position information is three-dimensionally provided for the acquired gradation processing target image data.

6. The information processing device according to claim 1, wherein the light source position information is given by designating any position on a surface of a sphere.

7. The information processing device according to claim 1, wherein the feature of the gradation processing target image data is acquired from the gradation processing target image data using a neural network.

8. The information processing device according to claim 1, wherein the light source position information is acquired via a tool for designating the position of the virtual light source, the tool being displayed in a manner such that the tool is overlapped onto an area that is displaying the gradation processing target image data.

9. The information processing device according to claim 1, wherein the at least one processor configured to:

display the gradation processing target image data in a predetermined display region;
display a tool for designating the position of the virtual light source to be applied to the gradation processing target image data, in the predetermined display region in an overlap manner; and
acquire the light source position information, via the tool displayed in the predetermined display region where the gradation processing target image data is displayed.

10. An information processing method comprising:

acquiring, by at least one processor, gradation processing target image data,
acquiring, by the at least one processor, light source position information for designating a position of a virtual light source to be applied to the gradation processing target image data, the light source position information being designated by a user,
acquiring, by the at least one processor, from a neural network using at least both the gradation processing target image data and the light source position information, one or more channels to be applied the gradation processing target image data in gradation processing that modifies gradation information included in the gradation processing target image data so as to depict a light effect of the virtual light source on the gradation processing target image data, the one or more channels being based on at least a feature of the gradation processing target image data, and the gradation processing including at least one of a process for adding a shadow or a process for highlighting an object in the gradation processing target image data, and
performing, by the at least one processor, the gradation processing on the gradation processing target image data of modifying the gradation information included in the gradation processing target image data by applying one or more channels to the gradation processing target image data, to generate gradation-processed image data.

11. The information processing method according to claim 10, wherein the one or more channels are overlapped with the gradation processing target image data in order to generate the gradation-processed image data.

12. The information processing method according to claim 10, further comprising:

estimating the light source position information of the acquired gradation processing target image data.

13. The information processing method according to claim 10, wherein the neural network includes both an encoder and a decoder.

14. The information processing method according to claim 10, wherein the light source position information is three-dimensionally provided for the acquired gradation processing target image data.

15. The information processing method according to claim 10, wherein the light source position information is given by designating any position on a surface of a sphere.

16. A non-transitory computer readable medium storing program instructions for causing at least one processor to:

acquire gradation processing target image data,
acquire light source position information for designating a position of a virtual light source to be applied to the gradation processing target image data, the light source position information being designated by a user,
acquire, from a neural network using at least both the gradation processing target image data and the light source position information, one or more channels to be applied to the gradation processing target image data in gradation processing that modifies gradation information included in the gradation processing target image data so as to depict a light effect of the virtual light source on the gradation processing target image data, the one or more channels being based on at least a feature of the gradation processing target image data, and the gradation processing including at least one of a process for adding a shadow or a process for highlighting an object in the gradation processing target image data, and
perform the gradation processing of modifying the gradation information included in the gradation processing target image data by applying the one or more channels to the gradation processing target image data, to generate gradation-processed image data.

17. The non-transitory computer readable medium according to claim 16, wherein the one or more channels are overlapped with the gradation processing target image data in order to generate the gradation-processed image data.

18. The non-transitory computer readable medium according to claim 16, wherein the at least one processor is further caused to:

estimate the light source position information of the acquired gradation processing target image data.

19. The non-transitory computer readable medium according to claim 16, wherein the neural network includes both an encoder and a decoder.

20. The non-transitory computer readable medium according to claim 16, wherein the light source position information is three-dimensionally provided for the acquired gradation processing target image data.

21. The non-transitory computer readable medium according to claim 16, wherein the light source position information is given by designating any position on a surface of a sphere.

Referenced Cited
U.S. Patent Documents
5134396 July 28, 1992 Sirat et al.
10706351 July 7, 2020 Rainwater
20100094696 April 15, 2010 Molinelli
20100119153 May 13, 2010 Rai
20160127630 May 5, 2016 Kitajima
20170076438 March 16, 2017 Kottenstette
20180122090 May 3, 2018 Ha
20180308281 October 25, 2018 Okoyama
20190051039 February 14, 2019 Tsuru
20190318534 October 17, 2019 Mory
Foreign Patent Documents
H02-303220 December 1990 JP
H09-237355 September 1997 JP
Other references
  • Over Gu et al. (“Image Shadow Removal Using Pulse Coupled Neural Network”) IEEE Transactions on Neural Networks, vol. 16, No. 3, May 2005 (Year: 2005).
  • “Cel shading” Wikipedia, (retrieved 2019) available on the Internet, <URL: https://en.wikipedia.org/wiki/Cel_shading>.
  • Archive of “Cell Shading” Wikipedia, (capture Dec. 2, 2017) Available on the Internet <URL: https://web.archive.org/web/20171202210839/https://en.wikipedia.org/wiki/Cel_shading>.
  • Iizuka et al., “Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification,” ACM Transaction on Graphics (Proc. of SIGGRAPH), vol. 35, No. 4, Section 110, (2016) Available on the Internet <URL: http://hi.cs.waseda.ac.jp/˜iizu ka/projects/colorization/ja/>.
  • Iizuka et al., “Neural Network-based Automatic Image Colorization,” Available on the Internet <URL: http://hi.cs.waseda.ac.jp:8082/>.
  • Sangkloy et al., “Scribbler: Controlling Deep Image Synthesis with Sketch and Color,” arXiv:1612.00835v2 [cs.CV] (Dec. 5, 2016), Available on the Internet <URL: https://arxiv.org/pdf/1612.00835.pdf>.
  • Kawanaka et al., “Color Photometric Stereo and Virtual Image Rendering Using Neural Network,” IEICE Transactions on Information and Systems, Japan Institute of Electronics, Information and Communication Engineers, vol. J89-D No. 2, pp. 381-392 (2006).
  • Yuki Odagiri et al., “Automatic color of cartoons by convolutional neural network using color features as input,” DEIM Forum 2017 B3-4, (2017) Available online, URL: https://db-event.jpn.org/deim2017/proceedings.html.
Patent History
Patent number: 11321584
Type: Grant
Filed: Dec 26, 2018
Date of Patent: May 3, 2022
Patent Publication Number: 20190197353
Assignee: PREFERRED NETWORKS, INC. (Tokyo)
Inventor: Taizan Yonetsuji (Tokyo)
Primary Examiner: Yanna Wu
Application Number: 16/232,495
Classifications
Current U.S. Class: Referral Award System (705/14.16)
International Classification: H04N 1/60 (20060101); G06K 9/62 (20220101);