SYSTEM

Abstract

A system includes a memory that stores a trained model and a processor. The processor acquires a captured image in which at least one energy device and at least one biological tissue are imaged. The processor performs a process based on the trained model stored in the memory to estimate a heat diffusion region and a specific tissue region from the captured image. The processor determines a risk for heat damage on a specific tissue by energy output from the energy device from the estimated heat diffusion region and the estimated specific tissue region.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Patent Application No. PCT/JP2022/009693, having an international filing date of Mar. 7, 2022, which designated the United States, the entirety of which is incorporated herein by reference. U.S. Provisional Patent Application No. 63/221,128 filed on Jul. 13, 2021 and U.S. Provisional Patent Application No. 63/222,252 filed on Jul. 15, 2021 are also incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

U.S. Patent Application Publication No. 2017/0252095 discloses a surgery system that determines a type of a tissue being gripped by an energy device based on energy output data of the energy device, a position of the tissue, and a patient condition or optical tissue sensor information. For example, whether or not it is a vascular or non-vascular tissue, or the presence or absence of nerves therein, and the like, are recognized as the type of the tissue. This surgery system stops energy output and warns a user when treatment is inappropriate for the recognized tissue type.

SUMMARY OF THE INVENTION

In accordance with one of some aspect, there is provided a system comprising:

• • a memory configured to store a trained model that is trained to estimate a heat diffusion region and a specific tissue region from a training device tissue image or a training tissue image, the training device tissue image being an image in which at least one energy device that receives energy supply to output energy and that is outputting energy and at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged, the heat diffusion region being a region in which heat diffusion from the at least one energy device to the at least one biological tissue is caused by energy output from the at least one energy device, and the specific tissue region being a region in the at least one biological tissue; and
  • a processor,
  • wherein the processor is configured to
  • acquire a captured image that is an image during the energy output and in which the at least one energy device and the at least one biological tissue are imaged,
  • perform a process based on the trained model stored in the memory to estimate the heat diffusion region and the specific tissue region from the captured image, and
  • determine a risk for heat damage on a specific tissue by the energy output from the at least one energy device from the estimated heat diffusion region and the estimated specific tissue region.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a configuration example of a system.

FIG. 2 shows a configuration example of a controller.

FIG. 3 shows a flowchart for explaining processing performed by the system.

FIG. 4 shows a configuration example of a monopolar device.

FIG. 5 shows a configuration example of a bipolar device.

FIG. 6 shows a configuration example of an ultrasonic device.

FIG. 7 shows a processing example according to a first embodiment.

FIG. 8 is an explanatory diagram when a control section detects a specific tissue.

FIG. 9 is an explanatory diagram when the control section detects a heat diffusion region.

FIG. 10 is an explanatory diagram when the control section determines a risk for heat damage.

FIG. 11 shows an example of output adjustment when the control section adjusts energy output.

FIG. 12 shows a configuration example of a training device.

FIG. 13 is a diagram for explaining a training phase regarding estimation of an important tissue.

FIG. 14 shows an example of an image used in the training phase regarding estimation of the heat diffusion region.

FIG. 15 is an explanatory diagram according to a second embodiment.

FIG. 16 is a diagram for explaining input information used in a processing example according to the present embodiment.

FIG. 17 is a diagram showing dependence of a distance between a specific tissue region and the heat diffusion region on energy output time.

FIG. 18 is a diagram for explaining input information according to a third embodiment.

FIG. 19 is a diagram for explaining a training phase regarding estimation of the heat diffusion region.

FIG. 20 is an example of an endoscope image when special light is used.

FIG. 21 is an explanatory diagram according to a fourth embodiment.

FIG. 22 is a diagram for explaining a training phase regarding estimation of the heat diffusion region.

FIG. 23 is an explanatory diagram according to a fifth embodiment.

FIG. 24 is an explanatory diagram according to a sixth embodiment.

FIG. 25 is an explanatory diagram according to a seventh embodiment.

DETAILED DESCRIPTION

The following disclosure provides many different embodiments, or examples, for implementing different features of the provided subject matter. These are, of course, merely examples and are not intended to be limiting. In addition, the disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. Further, when a first element is described as being “connected” or “coupled” to a second element, such description includes embodiments in which the first and second elements are directly connected or coupled to each other, and also includes embodiments in which the first and second elements are indirectly connected or coupled to each other with one or more other intervening elements in between.

1. System

FIG. 1 shows a configuration example of a system 10 according to the present embodiment. FIG. 1 shows a configuration example of the system for capturing images of a surgical field using an endoscope. The system 10 shown in FIG. 1 includes a controller 100, an endoscope system 200, a generator 300, and an energy device 310. The system 10 is a surgery system for performing surgery using at least one energy device under an endoscope. Although an example in which the system 10 includes a single energy device 310 is shown, the system 10 may include a plurality of energy devices.

The endoscope system 200 is a system that performs imaging by an endoscope, image processing of an endoscope image, and display of the endoscope image in a monitor. The endoscope system 200 includes an endoscope 210, a main body device 220, and a display 230. Herein, a rigid mirror for surgical operation is described as an example.

The endoscope 210 includes an insertion section to be inserted into a body cavity, an operation section to be connected to a base end of the insertion section, a universal cord connected to the base end of the operation section, and a connector section to be connected to a base end of the universal cord. The insertion section includes a rigid tube, an objective optical system, an imaging sensor, an illumination optical system, a transmission cable, and a light guide. The objective optical system and the imaging sensor for capturing images inside the body cavity and the illumination optical system for illuminating the inside of the body cavity are installed in a distal end section of the rigid tube having an elongated cylindrical shape. The distal end section of the rigid tube may be configured to be bendable. The transmission cable that transmits image signals acquired by the imaging sensor, and the light guide that guides illumination light to the illumination optical system are provided inside the rigid tube. The operation section is held by a user and accepts operations from the user. The operation section has buttons to which various functions are assigned. When the distal end of the insertion section is bendable, an angle operation lever is provided in the operation section. The connector section includes a video connector that detachably connects the transmission cable to the main body device 220, and a light guide connector that detachably connects the light guide to the main body device 220.

The main body device 220 includes a processing device that controls the endoscope, performs image processing of endoscope images, and displays the endoscope images, and a light source device that generates and controls illumination light. The main body device 220 is also called a video system center. The processing device is constituted of a processor such as a CPU, and performs image processing of the image signals transmitted from the endoscope 210 to generate endoscope images and then outputs the endoscope images to the display 230 and the controller 100. The illumination light emitted from the light source device is guided by the light guide to the illumination optical system and is emitted from the illumination optical system into the body cavity.

The energy device 310 is a device that outputs energy in a form of high-frequency power, ultrasonic waves, or the like from its distal end section to perform treatments including coagulation, sealing, hemostasis, incision, division, dissection, or the like, with respect to tissues in contact with its distal end section. The energy device 310 is also referred to as an energy treatment tool. The energy device 310 may be a monopolar device in which high-frequency power is energized between an electrode at the distal end of the device and an electrode outside the body, a bipolar device in which high-frequency power is energized between two jaws, an ultrasonic device, which has a probe and a jaw and emits ultrasonic waves from the probe, a combination device in which high-frequency power is energized between the probe and the jaw and also emits ultrasonic waves from the probe, or the like.

The generator 300 supplies energy to the energy device 310, controls the energy supply, and acquires electrical information from the energy device 310. When the energy device 310 outputs high-frequency energy, the generator 300 provides high-frequency power, and the energy device 310 outputs the high-frequency power from the electrode or the jaw. When the energy device 310 outputs ultrasonic energy, the generator 300 provides electric power, and the probe of the energy device 310 converts the electric power into ultrasonic waves and outputs the ultrasonic waves.

The electrical information refers to electrical information of the tissue that comes in contact with the electrode, probe, or jaw of the energy device 310; more specifically, the electrical information is information obtained as a response to the output of the high-frequency power to the tissue by the energy device 310. The electrical information is, for example, impedance information of the tissue to be treated by the energy device 310. However, as described later, the electrical information is not limited to the impedance information.

The generator 300 performs control of time-based change in the energy output from the energy device 310 according to an output sequence. The generator 300 may vary the energy output according to a time-based change in the impedance information. In this case, the output sequence may specify how the energy output is changed in response to the change in the impedance information. The generator 300 may also automatically turn off the energy output according to the time-based change in the impedance information. For example, the generator 300 may determine that the treatment is completed when the impedance rises to a certain level or higher, and may turn off the energy output.

The controller 100 recognizes a biological tissue and the energy device 310 from the endoscope image through an image recognition process using machine learning or other methods, and outputs an energy output adjustment instruction to the generator 300 based on the recognized information. Information regarding the biological tissue, information regarding the energy device 310, each of which is recognized from the endoscope image, or a combination of the information regarding the biological tissue and the information regarding the energy device 310 is also referred to as image recognition information. Specifically, these information items relate to matters that affect heat diffusion by the energy device 310.

The information regarding the biological tissue includes, not only information regarding a specific organ, but also information regarding a portion associated with an organ such as a tissue that connect organs. The biological tissue is referred to as a specific tissue or an important tissue as appropriate in the following description. In addition, a region including the specific tissue is referred to as a specific tissue region. The energy device 310 is, as described later with reference to FIGS. 4 to 6, an energy device such as a monopolar device 320, a bipolar device 330, and an ultrasonic device 340, but may be a device other than these devices.

The generator 300 adjusts the energy output of the energy device 310 according to the energy output adjustment instruction. Specifically, the system 10 of the present embodiment is a system that automatically adjusts the energy output from the energy device 310 based on an endoscope image. The generator 300 supplies energy to the energy device 310 in an energy supply amount directed by the energy output adjustment instruction. As the energy device 310 receives the energy supply and performs energy output accordingly, the energy output is adjusted according to the energy output adjustment instruction.

The energy output adjustment instruction includes an instruction to increase or decrease the output as the overall waveform of the output sequence, an instruction to set an output sequence from among a plurality of selectable output sequences, and the like. For example, when the energy output from the energy device 310 is adjustable according to a magnification factor that increases/decreases in a stepwise manner, the energy output adjustment instruction is an instruction indicating the energy output's magnification factor that increases/decreases in the step wise manner. The generator 300 increases or decreases high-frequency output or ultrasound output according to the magnification factor according to the instruction. The magnification factor may be continuously adjustable. In another case where a plurality of output sequences are provided, the energy output adjustment instruction is an instruction to specify one of these plural output sequences. The generator 300 performs energy output from the energy device 310 according to the output sequence thus instructed. The energy output adjustment instruction may include both of the instruction to increase or decrease the energy output and an instruction to change the output sequence.

2. Controller

FIG. 2 shows a configuration example of the controller 100. The controller 100 includes a control section 110, a storage section 120, an I/O device 180, and an I/O device 190. FIGS. 1 and 2 each show an example in which the controller 100 is constituted of a device separated from the generator 300. In this case, the controller 100 is constituted of an information processing device, such as a PC, a server device, or the like. Alternatively, the controller 100 may be implemented by a cloud or the like that performs processing with one or a plurality of information processing devices connected via a network.

The I/O device 180 receives image data of the endoscope image from the main body device 220 of the endoscope system 200. The I/O device 180 is a connector to which an image transmission cable is connected, or an interface circuit connected to the connector to perform communication with the main body device 220.

The control section 110 estimates the specific tissue and the heat diffusion region from the endoscope image through an image recognition process using a trained model 121, and outputs the energy output adjustment instruction based on a result of the estimation. The control section 110 includes one or a plurality of processors serving as hardware. The processor is a general-purpose processor such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), a DSP (Digital Signal Processor). Alternatively, the processor may be a dedicated processor such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

The storage section 120 stores the trained model 121 used for the image recognition process. For example, when the image recognition process is performed by a general-purpose processor, the storage section 120 stores, as the trained model 121, a program that describes an inference algorithm and parameters used for the inference algorithm. The trained model 121 includes a first trained model 122 and a second trained model 123. The first trained model 122 is the trained model 121 regarding estimation of the specific tissue region described later with reference to FIG. 12, and the second trained model 123 is the trained model 121 regarding estimation of the heat diffusion region. When the image recognition process is performed by a dedicated processor with a hardware inference algorithm, the storage section 120 stores the parameters used for the inference algorithm as the trained model 121.

The storage section 120 is a storage device, such as a semiconductor memory, a hard disc drive, and an optical disc drive. The semiconductor memory is, for example, a RAM, a ROM, a nonvolatile memory or the like.

For example, a neural network may be used as the inference algorithm of the image recognition process. Weight coefficients and a bias of inter-node connections in the neural network correspond to the parameters. The neural network includes an input layer to which image data is entered, an intermediate layer for performing a calculation process with respect to the data input via the input layer, and an output layer for outputting recognition results based on the calculation result output from the intermediate layer. For example, a CNN (Convolutional Neural Network) may be used as the neural network to be used for the image recognition process.

The control section 110 also includes a heat diffusion detection section 111, an important tissue detection section 112, a risk for heat damage determination section 114, and an output setting section 113. The storage section 120 stores a program describing functions of each of the heat diffusion detection section 111, the important tissue detection section 112, the risk for heat damage determination section 114, and the output setting section 113. One or more processors in the control section 110 read out a program from the storage section 120 and executes the program, thereby implementing the functions of each of the heat diffusion detection section 111, the important tissue detection section 112, the risk for heat damage determination section 114, and the output setting section 113. The program describing the functions of each of these sections may be stored in a non-transitory information storage medium, which is a computer-readable medium. The information storage medium can be implemented by, for example, an optical disc, a memory card, an HDD, a semiconductor memory, or the like. The semiconductor memory is, for example, a ROM or a nonvolatile memory.

The I/O device 190 transmits a signal of the energy output adjustment instruction to the generator 300. The I/O device 190 is a connector to which a signal transmission cable is connected, or an interface circuit connected to the connector to perform communication with the generator 300.

FIG. 3 is a flowchart for explaining processing performed by the controller 100 and the system 10. In the step S1, the control section 110 acquires an endoscope image from the main body device 220 of the endoscope system 200 via the I/O device 180. Subsequently, the control section 110 executes the steps S2A and S2B. In the step S2A, the important tissue detection section 112 performs an image recognition process on the endoscope image using the first trained model 122 to estimate an important tissue from the image, and thereby detects the important tissue. In the step S2B, the heat diffusion detection section 111 performs an image recognition process on the endoscope image using the second trained model 123 to estimate a heat diffusion region around the jaw of the energy device 310, and thereby detects the heat diffusion region. The heat diffusion region is a region in which heat diffusion is caused by energy output from the energy device 310. For example, when a region of a biological tissue region is at a predetermined temperature or higher, the corresponding portion can be assumed as the heat diffusion region. Subsequently, in step S3, the risk for heat damage determination section 114 determines a risk that heat diffuses into the important tissue and heat damage is caused on the important tissue based on a result of the estimation in the steps S2A and S2B, and transmits a signal to the output setting section 113. Examples of the heat damage include denaturation of proteins and deactivation of intracellular enzymes. The risk is hereinafter referred to as a risk for heat damage. In the step S4, the output setting section 113 adjusts the energy output from the energy device 310 based on the signal.

3. Energy Device

In the following, the monopolar device 320, the bipolar device 330, the ultrasonic device 340, and the combination device are described as examples of the energy device 310.

FIG. 4 shows a configuration example of the monopolar device 320. The monopolar device 320 includes an insertion section 322 having an elongated cylindrical shape, an electrode 321 provided at the distal end of the insertion section 322, an operation section 323 connected to the base end of the insertion section 322, and a cable 325 connecting the operation section 323 and a connector (not shown). The connector is detachably connected to the generator 300.

The high-frequency power output by the generator 300 is transmitted by the cable 325 and output from the electrode 321. A counter electrode plate is provided outside the patient's body, and energization occurs between the electrode 321 and the counter electrode plate. This applies high-frequency energy to the tissue in contact with the electrode 321, and Joule heat is generated in the tissue. Electrodes having various shapes are used for the electrode 321 depending on a type of treatment. The monopolar device 320 is capable of adjusting a degree of coagulation and incision by changing an energization pattern. An object to be treated by the monopolar device 320 is typically a tissue in contact with the electrode 321, and heat diffused around this tissue in contact with the electrode 321 may affect a surrounding tissue.

FIG. 5 shows a configuration example of the bipolar device 330. The bipolar device 330 includes an insertion section 332 having an elongated cylindrical shape, two jaws 337 and 338 provided at a distal end section 331 of the insertion section 332, an operation section 333 connected to the base end of the insertion section 332, and a cable 335 connecting the operation section 333 and a connector (not shown). The connector may be detachably connected to the generator 300. The jaws 337 and 338 are grip sections for gripping a tissue and also applying energy to the gripped tissue, and are structured to be openable/closable around an axis provided at the base end 336. The operation section 333 has a grip section for operating the opening and closing of the jaws 337 and 338. When the doctor tightly holds the grip section, the jaws 337 and 338 are closed to grip the tissue.

The high-frequency power output by the generator 300 is transmitted by the cable 335, and, when the jaws 337 and 338 grip the tissue, energization occurs between the two jaws 337 and 338. As a result, high-frequency energy is applied to the tissue sandwiched between the two jaws 337 and 338, Joule heat is generated in the tissue, and the tissue is coagulated. The generator 300 may measure the impedance information of the tissue gripped by the jaws 337 and 338, detect completion of the treatment based on the impedance information, and may automatically stop the energy output. Further, the generator 300 may also automatically adjust the energy applied to the tissue based on the impedance information. With regard to a device temperature of the bipolar device 330, for example, although the device rises only to about 100 degrees Celsius, there is a possibility that a sneak current is generated around a portion gripped by the jaws 337 and 338, and heat diffusion may be caused by the sneak current.

A vessel sealing device is a derivative device of the bipolar device. The vessel sealing device is a bipolar device provided with a cutter on its jaw, and separates the tissue by running the cutter after coagulating the tissue by energization.

FIG. 6 shows a configuration example of the ultrasonic device 340. The ultrasonic device 340 includes an insertion section 342 having an elongated cylindrical shape, a jaw 347 and a probe 348 provided at a distal end section 341 of the insertion section 342, an operation section 343 connected to the base end of the insertion section 342, and a cable 345 connecting the operation section 343 and a connector (not shown). The connector is detachably connected to the generator 300. The jaw 347 is movable around an axis provided at a base end 346, and is structured to be openable/closable with respect to the non-movable probe 348. The operation section 343 has a grip section for operating the opening and closing of the jaw 347. When the doctor tightly holds the grip section, the jaw 347 is closed, and the jaw 347 and the probe 348 grip the tissue. For example, as shown in FIG. 6, the operation section 343 is provided with an operation button 344a to which a first output mode is assigned, and an operation button 344b to which a second output mode is assigned. The output mode is selected according to what treatment is to be performed. When the operation button for each output mode is pressed, ultrasonic energy is output in the output sequence for the corresponding mode. The number of operation buttons arranged in the operation section 343 is not limited to that in the configuration in FIG. 6.

The power output by the generator 300 is transmitted by the cable 335, and when the operation button 344a or the operation button 344b is pressed, the probe 348 converts the power into ultrasonic waves and outputs the ultrasonic waves. As a result, frictional heat is generated in the tissue sandwiched between the jaw 347 and the probe 348, and the tissue is coagulated or incised.

The combination device that uses both high-frequency power and ultrasonic waves has a configuration similar to that of the ultrasonic device shown in FIG. 6, for example. However, the combination device is capable of energizing high-frequency power between the jaw and the probe to generate Joule heat in the tissue gripped by the jaw and the probe, thus coagulating the tissue. Similarly to the ultrasonic device, the combination device is also capable of incising a tissue gripped by the jaw and the probe by outputting ultrasonic waves from the probe. A high-frequency mode is assigned to one of the two operation buttons provided on the operation section, and a seal-and-cut mode is assigned to the other one of the two operation buttons. The high-frequency mode is a mode in which coagulation and other treatments are performed using only high-frequency energy output. The seal-and-cut mode is a mode in which a combination of high-frequency energy and ultrasonic energy is used, and the tissue is coagulated and separated by high-frequency energy output. With regard to the heat diffusion of the combination device, for example, heat diffusion similar to either or both of those of the bipolar device and the ultrasonic device may occur.

In the following embodiment, an exemplary case where the bipolar device 330 is mainly used as the energy device 310 is described. However, it should be noted that the present embodiment is applicable to any cases of using various energy devices mentioned above that may cause heat diffusion.

4. First Embodiment

FIG. 7 is a processing example according to a first embodiment. First, as show in the step S21, an endoscope image is input to the control section 110. Specifically, each frame image of moving images captured by the endoscope is sequentially input to the control section 110. In the endoscope image input to the control section 110, one or more energy devices that are performing energy output and one or more biological tissues are seen.

Subsequently, as described in the step S22A, the control section 110 executes an estimation program adjusted by machine learning to detect the important tissue from the endoscope image. Specifically, the control section 110 detects the important tissue by inputting the endoscope image captured during surgery to a network having the estimation program that has been trained by addition of an annotation of the important tissue to the endoscope image. The important tissue mentioned herein is, for example, a great vessel, the pancreas, the duodenum, or the like, but is not limited thereto. FIG. 8 is a diagram for explaining detection of the important tissue described in the step S22A in FIG. 7. As shown in FIG. 8, the control section 110 estimates an important tissue region among objects shown in the endoscope image based on the estimation program, and gives coloring to the important tissue region to perform labeling. The control section 110 then outputs the endoscope image subjected to the labeling to the risk for heat damage determination section 114.

In addition, as described in the step S22B in FIG. 7, the control section 110 executes the above-mentioned estimation program to detect a heat diffusion region from the endoscope image. Similarly to the step S22A, the control section 110 detects the heat diffusion region by inputting the endoscope image captured during surgery to the network having an estimation program that has been trained by addition of an annotation of a temperature range of a threshold or higher to the endoscope image. FIG. 9 is a diagram for explaining detection of the heat diffusion region described in the step S22B in FIG. 7. As shown in FIG. 9, the control section 110, based on the estimation program, gives coloring to the region that is estimated to be at a predetermined temperature or higher in an organ seen in the endoscope image to perform labeling of the heat diffusion range. The control section 110 then outputs the endoscope image subjected to the labeling to the risk for heat damage determination section 114. It should be noted that the labeling of the important tissue and the heat diffusion region may be performed using a method other than coloring.

Subsequently, as described in the step S23, the risk for heat damage determination section 114 of the control section 110 receives information of a result of the estimation about the important tissue and the heat diffusion region, and determines the risk for heat damage based on the result of the estimation. FIG. 10 is a diagram for explaining a method of determining the risk for heat damage described in the step S23 in FIG. 7. The risk for heat damage determination section 114 of the control section 110 generates an image displayed in a manner that the labeling of the important tissue and the heat diffusion region is superimposed thereon as shown in FIG. 10, based on respective endoscope images output from the important tissue detection section 112 and the heat diffusion detection section 111. The risk for heat damage determination section 114 then calculates a distance between the region estimated to be the important tissue and the region estimated to be at the predetermined temperature or higher. In FIG. 10, for example, when the stomach is estimated to be the important tissue, illustrated is a distance between the stomach and the region estimated to be at the predetermined temperature or higher. For example, when the energy device 310 is the bipolar device, the distance mentioned herein is, for example, an actual distance between the heat diffusion region indicated in gray around the jaw and a portion of the stomach that is the closest to the heat diffusion region. The stomach serves as the important issue. The distance can be calculated by, for example, an image analysis process on the image shown on the display 230. The control section 110 then uses the distance as a margin M to determine the risk for heat damage. The margin M is data or the like that can be used as a determination material when the control section 110 determines the risk for heat damage, and is, for example, the above-mentioned distance. When the margin M is a preliminarily set threshold or smaller, the control section 110 determines that there is the risk for heat damage on the important tissue. When the margin M is larger than the threshold, the control section 110 determines that there is no risk for heat damage on the important tissue. The risk for heat damage determination section 114 outputs a signal indicating a result of the determination about presence/absence of the above-mentioned risk for heat damage to the output setting section 113.

As described in the step S24 in FIG. 7, the output setting section 113 then gives an output change instruction based on the signal output by the risk for heat damage determination section 114. Specifically, the storage section 120 stores table data in which an energy output adjustment instruction is associated with the presence/absence of the risk for heat damage, and the control section 110 refers to the table data to output the energy output adjustment instruction associated with the presence/absence of the risk for heat damage to the generator 300 via the I/O device 190. The generator 300 then adjusts the output sequence of the bipolar device according to the energy output adjustment instruction output by the control section 110. It should be noted that an algorithm for outputting the energy output adjustment instruction according to the presence or absence of the risk for heat damage is not limited to the above. FIG. 11 shows an example of output adjustment when the output setting section 113 of the control section 110 gives the energy output adjustment instruction based on the result of the determination about the risk for heat damage. When it is determined that there is the risk for heat damage on the important tissue, the output setting section 113 performs output adjustment so as to decrease energy output. Specifically, the output setting section 113 decreases or stops a voltage, power, or the like. When it is determined that there is no risk for heat damage on the important tissue, the output setting section 113 output adjustment so as to maintain or increase energy output. The output setting section 113 outputs signals indicating these output adjustment instructions to the generator 300.

In this manner, the control section 110 recognizes, from the endoscope image, the important tissue and the heat diffusion region in which heat diffusion is caused by the energy output, and automatically adjusts the energy output from the generator 300 when determining that there is the risk for heat damage on the important tissue.

In the determination of the risk for heat damage described in the step S23 in FIG. 7, the threshold for the margin M used when the presence/absence of the risk for heat damage is determined may be different depending on a tissue type of the specific tissue detected in the step S22A. Alternatively, the threshold may be variable. This can avert occurrence of heat damage on an organ for which the occurrence of heat damage is particularly problematic. It is possible to avert the occurrence of heat damage particularly on the important tissue having a high degree of seriousness particularly at the time of occurrence of heat damage by increasing the margin M, for example, in a case of a biological tissue such as the pancreas and the duodenum, and by decreasing the margin M, for example, in a case of a biological tissue such as the stomach and the liver.

The energy output adjustment instruction is an instruction to increase, decrease, or maintain the energy output based on reference energy output. The generator 300 has an operation section for accepting an energy output setting operation, and the energy output can be set by the operation section to one of, for example, five intensity levels (1 to 5). The intensity 1 represents the lowest energy output and the intensity 5 represents the highest energy output. The reference energy output is, for example, predetermined energy output such as “intensity 3”. In this case, the instruction to increase the energy output to be greater than the reference energy output is an instruction to set the intensity to “intensity 4” or “intensity 5”, and the instruction to decrease the energy output to be lower than the reference energy output is an instruction to set the intensity to “intensity 2” or “intensity 1”. Alternatively, the reference energy output may be the energy output currently set by the operation section of the generator 300. In this case, the instruction to increase the energy output to be greater than the reference energy output is an instruction to set the energy output to be higher than the currently set energy output, and the instruction to decrease the energy output to be lower than the reference energy output is an instruction to set the energy output to be lower than the currently-set energy output. Alternatively, the reference energy output may be within an output range of intensity 1 to intensity 5 that can be set for the generator 300. In this case, the instruction to increase the energy output to be greater than the reference energy output is an instruction to set the energy output to be higher than intensity 5, and the instruction to decrease the energy output to be lower than the reference energy output is an instruction to set the energy output to be lower than intensity 1.

Meanwhile, one of points to perform energy treatment in surgery is to suppress heat diffusion from the energy device 310 and avert thermal damage on surrounding organs. Because the tissues to be treated are not uniform, time required for treatment, such as division, varies depending on a difference in tissue type, a difference in tissue condition, individual differences of patients, or the like, and a degree of heat diffusion also varies. To cope with these issues and suppress heat diffusion, doctors have been adjusting energy output from the energy device; however, such an operation requires experience and appropriate adjustment may be difficult in some cases, particularly for non-experts.

In this manner, generally, heat diffusion to the surrounding region is often problematic in treatments using energy devices, and therefore the doctors perform the treatments while estimating a degree of diffusion. The surgery system disclosed in U.S. Patent Application Publication No. 2017/0252095 described above determines whether or not heat denaturation of a treatment target tissue occurs based on an impedance value or the like, and switches, when the heat denaturation has already occurred, an energy output mode to a mode of outputting low energy. However, the surgery system merely determines whether or not the heat denaturation of the treatment target tissue occurs by the energy device, and does not take into consideration of the heat diffusion to the periphery of the treatment tissue. Thus, there is an issue that the surgery system is unable to perform appropriate adjustment of the energy output depending on the state of the treatment tissue or the state of the energy device. For example, when an energy treatment tool for surgery is used, there is a case where an inexperienced doctor in particular pays less attention to heat diffusion in surrounding tissues. In such a case, in a tissue to which heat diffusion occurs extensively and rapidly, there is a possibility that heat propagates to the important tissue before the treatment target tissue is heat-denatured and heat damage is caused on the important tissue.

In this regard, according to the present embodiment, the system 10 acquires the captured image in which the energy device 310 and the biological tissue are imaged, performs the process based on the trained model to estimate the heat diffusion region and the specific tissue region, and determines the risk for heat damage on the specific tissue by the energy device. The system 10 is capable of adjusting energy to be applied to the treatment target tissue to an appropriate amount based on the risk for heat damage, and thus is capable of reducing the risk for occurrence of heat damage on the important tissue without noticing heat diffusion. In this manner, since the system 10 performs adjustment of energy output from the energy device, which has been previously performed by the doctors, it is possible to reduce the burden on the doctors. Furthermore, autonomous adjustment of the output by the system 10 allows even non-experts to perform stable treatments. With the procedures described above, it is possible to improve the stability of the surgery or equalize the manipulation regardless of the experiences of the doctors.

FIG. 12 shows a configuration example of a training device 500 that performs machine learning of the estimation process of estimating the important tissue and the heat diffusion region. The training device 500 is implemented by an information processing device, such as a PC and a server device, and includes a processing section 510 and a storage section 520.

The processing section 510 is a processor such as a CPU, and the storage section 520 is a storage device such as a semiconductor memory and a hard disc drive. The storage section 520 stores a training model 522 and training data 521. The training data 521 mentioned herein includes first training data 521A and second training data 521B. Further, the training model 522 includes a first training model 522A and a second training model 522B. The processing section 510 then uses the training data 521 to train the training model 522, and thereby generates a trained model 121. The training data 521 includes image data of a plurality of training images and correct answer data added to each training image. Herein, the first training data 521A is data regarding the important tissue, and the second training data 521B is data regarding the heat diffusion region. Further, the first training model 522A is a training model regarding the important tissue, and the second training model 522B is a training model regarding the heat diffusion region. The plurality of training images include an endoscope image in which one or more biological tissues and one or more energy devices 310 are imaged. Such an endoscope image is also referred to as a training device tissue image. In addition, the plurality of training images may also include an endoscope image in which one or more biological tissues are imaged and no energy device 310 is imaged. Such an endoscope image is also referred to as a training tissue image. The correct answer data serves as an annotation in segmentation (region detection), an annotation in detection (position detection), a correct answer label in classification (classification), or a correct answer label in regression (regression analysis). The processing section 510 inputs training images in the inference process by the training model 522, and provides feedback to the training model 522 based on an error between a result of the inference process and the correct answer data. The processing section 510 repeats this process with a large amount of training data to generate the trained model 121. The trained model 121 thus generated is transferred to the storage section 120 of the controller 100.

FIG. 13 is a diagram for explaining a training phase for estimating the important tissue. As shown in FIG. 13, the training device 500 (not shown) feeds back the training device tissue image or the training tissue image as the first training data 521A to the first training model 522A. The training device tissue image mentioned herein is an image in which at least one energy device 310 that receives energy supply to output energy and that is outputting energy, and at least one biological tissue are imaged. The training tissue image mentioned herein is an image in which at least one biological tissue is imaged. The training device tissue image or the training tissue image is labelled thereon with the important tissue region. The labeling is performed by, for example, the doctors. The feedback of a large amount of such training device tissue images or training tissue images as the first training data 521A to the first training model 522A in the training device 500 increases the accuracy in estimation of the important tissue by the control section 110.

The training device 500 performs the above-mentioned annotation of the estimation program to estimate the heat diffusion region by creating the second training data 521B based on, for example, an image captured by a thermostat camera or the like. FIG. 14 shows an example of an endoscope image in which a heat diffusion range in a biological tissue of various types is colored using temperature information that is imaged by the thermostat camera. In the example shown in FIG. 14, a heat diffusion state is displayed with three temperature ranges of 59 degrees Celsius or lower, from 60 to 79 degrees Celsius, and a threshold temperature or higher. If a temperature is the threshold temperature or higher, there is a possibility for occurrence of heat damage. In the case of FIG. 14, the threshold temperature is 80 degrees Celsius. In this case, the image shown in FIG. 14 serves as the training image of the second training data 521B, and information regarding the heat diffusion region colored in the image serves as the correct answer data. The risk for heat damage caused by heat diffusion is evaluated using, for example, denaturation of proteins or deactivation of intracellular enzymes as a criterion. As alternative indexes, known are a risk evaluation using a simple temperature range, a risk evaluation using a history of products of temperature and time, and a risk evaluation using an amount of heating, and a risk evaluation based on activation energy calculated using the Arrhenius equation. In the case of the risk evaluation using the simple temperature range, for example, 60 degrees Celsius or 50 degrees Celsius serves as the criterion for the risk evaluation. In the case of the risk evaluation using the history of the products of temperature and time, for example, 0.1 seconds at 60 degrees Celsius or 0. 1 seconds at 50 degrees Celsius serves as the criterion for the risk evaluation. The risk evaluation using the amount of heat is used for ablation of the liver or the like, and the risk evaluation based on the activation energy calculated using the Arrhenius equation is mainly used for burn injury.

5. Second Embodiment

In a second embodiment, history information is used in estimation of the heat diffusion region. Parts different from the first embodiment are mainly described hereinbelow. FIG. 15 is a diagram for explaining a training phase for estimating the heat diffusion region. As shown in FIG. 15, the training device 500 (not shown) feeds back history images captured by the above-mentioned thermostat camera or the like as the second training data 521B to the second training model 522B. Each history image is an endoscope image in which the heat diffusion region is indicated by a color or the like based on, for example, temperature information obtained by the above-mentioned thermostat camera. In this manner, the latest history images can be used as the second training data 521B in training of a network used in a heat diffusion detection function.

In addition, for example, history information of energy output from the energy device 310, together with the above-mentioned history images, can also be used as the second training data 521B in the training phase for estimating the heat diffusion region. Specifically, as shown in FIG. 15, information regarding dependence of energy output from the energy device 310 on time can be used. The history images and the history information of the output energy described above include a large amount of data in such a manner as a scene 1, a scene 2, . . . , a scene n, and are used as the second training data 521B. The second training data 521B is fed back to the second training model 522B, whereby the second trained model 123 is generated. This also enables feedback of the history information of the output energy, in addition to the latest endoscope images, to the second training model 522B, and allows the control section 110 to estimate the heat diffusion region with higher accuracy.

FIG. 16 shows an example of input information used in the step S21 in the processing example shown in FIG. 7. As shown in FIG. 16, the control section 110 can use a plurality of endoscope images that are different in imaging timing as the input information in the step S21. Each of the endoscope images different in imaging timing is, for example, a frame image of moving images. Further, the control section 110 can also use information regarding a history of energy output from the energy device 310, together with the plurality of endoscope images, as the input information. The detection of the heat diffusion region in the step S22B in FIG. 7 is performed based on the plurality of endoscope images and the information regarding the history of energy output.

In this manner, by using not only images captured in real time, but also the latest history images and the history of energy output as the input information to estimate the heat diffusion region, it is possible to estimate the heat diffusion region including dynamic information including, for example, tissue contraction. Thus, the control section 110 is capable of estimating the heat diffusion region with high accuracy.

FIG. 17 is a diagram showing the above-mentioned distance between the important tissue region and the heat diffusion region, that is, dependence of the margin M on energy output time T. A time t in an abscissa axis shown in FIG. 17 is a present time, a time t0 is a time at which heat diffusion is estimated to reach the important tissue. Thus, a time t0-n that is predefined time n earlier than the time t0 serves as a timing at which it is determined that there is the risk for heat damage on the important tissue. The behavior of the margin M during time from a time 0 to the time t in the graph in FIG. 17 can be estimated by the control section 110 from the plurality of endoscope images that are different in imaging timing and that are the latest history images, as described with reference to FIG. 16. As shown in FIG. 17, the margin M decreases over time from the time 0 to the time t. The estimated time t0 at which the heat diffusion reaches the important tissue can be predicted from the behavior of the margin M from the time 0 to the time t with respect to the energy output time T. For example, the control section 110 is capable of predicting the time t0 by extrapolating the dependence of the margin M on the energy output time T from the behavior of the margin M from the time 0 to the time t and calculating an intersection point with the abscissa axis indicating the energy output time T. The time t0-n that is obtained in this manner and that is predefined time n earlier than the time t0 serves as a timing at which it is determined that there is the risk for heat damage on the important tissue.

In this manner, the decrease of the margin M that is the distance between the important tissue and the heat diffusion region is acquired on a time-series basis, whereby it is possible to determine that there is the risk for heat damage the predefined time n earlier than the time at which the heat diffusion is predicted to reach the important tissue. That is, the control section 110 is capable of outputting the time at which the heat diffusion reaches the specific tissue region as a result of prediction, determining the risk for heat damage, and outputting a result of the determination. This enables reduction of the risk for heat damage on the important tissue and safe execution of surgery.

6. Third Embodiment

In addition, the input information in the step S21 in the first embodiment shown in FIG. 7 may also include history information of multiple wavelength images. For example, as shown in FIG. 18, in addition to the history of energy output and the endoscope images captured with normal light, a real time image and history images that are captured with special light can be included as the input information. Each image captured with special light is, for example, a multiple wavelength image captured with illumination light using narrow band light with a narrower wavelength band than that of visible light, or a plurality of types of narrow band light. Examples of such a multiple wavelength image include an image captured by Auto Florescence Imaging (AFI) and an image captured by Narrow Band Imaging (NBI). The normal light is also called white light. In this manner, with use of the multiple wavelength images at the time of detection of the heat diffusion region, it is possible to also input information that is focused on the specific tissue such as a connective tissue around a blood vessel. This facilitates obtaining of dynamic information such as aggregation of the connective tissue, and increases accuracy of estimation of the important tissue and estimation of the heat diffusion region by utilizing the dynamic information. The training phase for estimating the heat diffusion region is described with reference to FIGS. 14 and 15. FIG. 19 is an explanatory diagram when the history information of the multiple wavelength images is included in the second training data 521B. In comparison with the case shown in FIG. 15, a real-time multiple wavelength image and history images thereof are included in the second training data 521B. The training device 500 (not shown) feeds back the second training data 521B including the multiple wavelength images to the second training model 522B, whereby the second trained model 123 is generated.

FIG. 20 shows an example of an image of a result of observation of an isolated blood vessel by the AFI. As shown in FIG. 20, with use of the AFI, the control section 110 is capable of clearly discriminating between the blood vessel and the connective tissue to observe the isolated blood vessel. In this manner, with use of the multiple wavelength images captured by the AFI, it is possible to isolate and observe, for example, a tissue that is susceptible to heat contraction, and increase accuracy of image recognition.

7. Fourth Embodiment

In the first embodiment shown in FIG. 7, an image including white burns due to heat denaturation can also be used as the captured image acquired by the control section 110. The white burns mentioned herein are a state where the biological tissue incurs heat damage by heat, becomes white in comparison with other portions where no heat damage is caused, and appears to be different from the other regions. In this manner, the region with the risk for heat damage can be estimated from a range in which white burns occur. FIG. 21 is a diagram for describing a case where the image including the white burns due to heat denaturation is used as the captured image acquired by the control section 110. In the step S31, the control section 110 inputs the endoscope image during surgery into the network with the present function and performs an image process to estimate the range in which the white burns occur. Specifically, the control section 110 uses a network that has been trained by addition of an annotation of the range of the white burns to the endoscope image. Further, there is a case where heat damage occurs even through there is no apparent abnormality. To address this, in the step S32, the control section 110 assumes a range obtained by adding a predefined range to the extracted range of the white burns as the heat diffusion region, and uses the heat diffusion region as the input information for determination about the risk for heat damage.

This allows acquisition of information regarding the range of the white burns from the endoscope image, and eliminates the need for using a device such as the thermostat camera at the time of creation of the training data 521. This can also reduce the number of networks used by the control section 110, and can increase the speed of processing of the control section 110.

FIG. 22 is a diagram for explaining a training phase when the input information includes images labelled with the region in which the white burns occur. The training device 500 (not shown) feeds back the training device tissue image or the training tissue image, in which the biological tissue labelled with the range of the white burns is imaged, as the second training data 521B, to the second training model 522B, and creates the second trained model 123.

8. Fifth Embodiment

A fifth embodiment shown in FIG. 23 is different from the first embodiment shown in FIG. 7 in the estimation process in the step S22A in which the important tissue is detected and the step S22B in which the heat diffusion region is detected. Specifically, in the estimation process according to the fifth embodiment, the step S42C in which a thermal capacity and thermal resistance of a heat-transfer path are recognized is added. For example, even if the distance between the important tissue and the heat diffusion region is estimated, time until heat diffuses to the important tissue varies depending on tissue heat-transfer characteristics of the biological tissue between the important tissue and the heat diffusion region. Thus, since the influence of the heat-transfer characteristics is not taken into consideration in the first embodiment shown in FIG. 7, the accuracy of the result of prediction of the heat diffusion remains within a certain level.

In this regard, in the fifth embodiment, the trained model 121 is trained to estimate the tissue heat-transfer characteristics from training energy output information of the energy device 310, the training device tissue image, or the training tissue image. The control section 110 then performs a process based on the trained model 121 stored in the storage section 120 to estimate the tissue heat-transfer characteristics from the energy output information of the energy device 310 or the captured image. Hence, the control section 110 is capable of predicting the decrease of the distance between the heat diffusion region and the important tissue, that is, the decrease of the margin M, from the characteristics of the heat-transfer path. Thus, it is possible to reduce the risk for occurrence of heat damage on the important tissue and perform surgery safely. A change in decrease rate of the margin M can be predicted, for example, by utilizing machine learning or the like.

9. Sixth Embodiment

A sixth embodiment shown in FIG. 24 is different from the first embodiment shown in FIG. 7 in the process after the step S23 in which the risk for heat damage is determined. In the sixth embodiment, after determining the risk for heat damage, the control section 110 determines whether or not presentation of an alert is necessary in the step S54. Specifically, the control section 110 determines the presentation of the alert is necessary when determining that there is the risk for heat damage in the determination in the step S53, and determines the presentation of the alert is not necessary when determining that there is no risk for heat damage in the determination in the step S53. The control section 110 then performs the presentation of the alert in the step S55 when determining that there is the risk for heat damage, and does not perform the presentation of the alert in the step S56 when determining that there is no risk for heat damage. The presentation of the alert poses the risk for heat damage on, for example, a monitor screen that is watched by the doctor who performs surgery and thus is implemented by presentation of recommendation to decrease the energy output from the present energy output or recommendation to stop the energy output. This allows the doctor to make determination about whether to decrease or stop the energy output after being informed of the risk for heat damage.

10. Seventh Embodiment

A seventh embodiment shown in FIG. 25 is different from the first embodiment shown in FIG. 7 in the step S22A in which the important tissue is detected. In the seventh embodiment, the important tissue is not detected from the endoscope image, but is estimated by 3D matching with an image captured before surgery. The important tissue is detected by 3D matching with Computed Tomography (CT) or Magnetic Resonance Imaging (MRI). This enables detection of the important tissue that cannot be seen under the endoscope.

The system according to the present embodiment can be implemented also as a program. That is, the program according to the present embodiment is capable of causing a computer to execute: real-time acquisition of the captured image in which at least one energy device that receives energy supply to output energy and that is outputting energy and at least one biological tissue are imaged; the process based on the trained model that is trained to output the image recognition information, which is at least one of the information regarding the heat diffusion region in which heat diffusion from the energy device to the biological tissue is caused by energy output from the energy device or the information regarding the specific tissue region in the biological tissue, from the training device tissue image in which at least one energy device that is outputting energy and at least one biological tissue are imaged or from the training tissue image in which at least one biological tissue is imaged to estimate the image recognition information from the captured image; and determination about the risk for heat damage on the specific tissue by the energy output from the energy device from the estimated image recognition information. As the computer mentioned herein, a network terminal or the like such as a personal computer is assumed. The computer may be a smartphone, a tablet terminal, a wearable terminal such as a smart watch. Accordingly, it is possible to obtain an effect that is similar to the above-mentioned effect.

The system 10 according to the present embodiment described above includes the storage section 120 that stores the trained model 121 and the control section 110. The trained model 121 is trained to estimate, from the training device tissue image or the training tissue image, the heat diffusion region in which heat diffusion from the energy device 310 to the biological tissue is caused by energy output from the energy device 310 and the specific tissue region in the biological tissue. The training device tissue image is an image in which at least one energy device 310 that receives energy supply to output energy and at least one biological tissue are imaged. The training tissue image is an image in which at least one biological tissue is imaged. The control section 110 acquires the captured image that is an image during energy output and in which at least one energy device 310 and at least one biological tissue are imaged. The control section 110 performs the process based on the trained model 121 stored in the storage section 120 to estimate the heat diffusion region and the specific tissue region from the captured image. The control section 110 determines, from the estimated heat diffusion region and the estimated specific tissue region, the risk for heat damage on the specific tissue by the energy output from the energy device 310.

According to the present embodiment, the system 10 acquires the captured image in which the energy device 310 and the biological tissue are imaged, performs the process based on the trained model to estimate the heat diffusion region and the specific tissue region, and determines the risk for heat damage on the specific tissue by the energy device. The system 10 is capable of adjusting energy to be applied to the treatment target tissue based on the risk for heat damage to an appropriate amount, and thus is capable of reducing the risk for occurrence of heat damage on the important tissue without noticing heat diffusion. In this manner, since the system 10 performs adjustment of energy output from the energy device, which has been previously performed by the doctors, it is possible to reduce the burden on the doctors. Furthermore, autonomous adjustment of output by the system 10 allows even non-experts to perform stable treatments. With the procedures described above, it is possible to improve the stability of the surgery or equalize the manipulation regardless of the experiences of the doctors. The training device tissue image, the training tissue image, the specific tissue, and the important tissue are described in the section “1. System”.

In the present embodiment, the control section 110 may determine that there is the risk for heat damage when the distance between the specific tissue region and the heat diffusion region is the preliminarily set threshold or smaller.

According to the present embodiment, it is possible to evaluate the distance between the specific tissue region and the heat diffusion region based on the result of estimation from the image acquired by the controller 100, and use the distance as the criterion for determination of the presence/absence of the risk for heat damage. The method of determination of the risk for heat damage using the threshold is described in the section “4. First Embodiment”.

In the present embodiment, the threshold may be different depending on a tissue type of the specific tissue.

According to the present embodiment, it is possible to avert the risk for occurrence of heat damage on the specific tissue for which heat damage is particularly problematic. Details are described in the section “4. First Embodiment”.

Further, in the present embodiment, the captured image may be a plurality of endoscope images that are different in timing to capture respective endoscope images.

According to the present embodiment, the heat diffusion region is generated. Thus, it is possible to estimate the heat diffusion region by giving consideration also to dynamic information, and increase accuracy of estimation of the heat diffusion region. The plurality of endoscope images that are different in imaging timing are described in FIG. 16 in the section “5. Second Embodiment”.

In the present embodiment, the captured image is the plurality of endoscope images that are different in timing to capture respective endoscope images, and the control section 110 may estimate the heat diffusion region and the specific tissue region from each image, and output time until heat diffusion reaches the specific tissue region as a result of prediction based on a plurality of heat diffusion regions and a plurality of specific tissue regions that are estimated from each image.

According to the present embodiment, the controller 100 acquires the decrease of the distance between the heat diffusion region and the specific tissue on a time-series basis, and can thereby predict time at which heat diffusion reaches the specific tissue region. This allows a person who performs surgery to preliminarily adjust output from the energy device based on a result of the prediction. Thus, it is possible to reduce the risk for heat damage on the specific tissue region and perform surgery safely. The prediction of the time at which heat diffusion reaches the specific tissue region is described in FIG. 17 in the section “5. Second Embodiment”.

In the present embodiment, the control section 110 may determine the risk for heat damage before the time, which is the result of the prediction, and output a result of the determination.

According to the present embodiment, the result of the determination about the risk for heat damage is output before heat diffusion reaches the specific tissue region. Thus, it is possible to adjust the energy output before a predetermined time at which heat damage is caused on the specific tissue, reduce the risk for heat damage on the specific tissue region, and perform surgery safely. The prediction of the time at which the heat diffusion reaches the specific tissue region is described in FIG. 17 in the section “5. Second Embodiment”.

In the present embodiment, the control section 110 may output, based on the result of the determination, the energy output adjustment instruction, which is the instruction to decrease the energy output from the present energy output or the instruction to stop the energy output, to the generator 300 that controls an amount of energy supply to the energy device 310 based on the energy output adjustment instruction.

According to the present embodiment, the system 10 is capable of performing autonomous energy output adjustment depending on the risk for occurrence of heat damage on the specific tissue region. Thus, it is possible for even non-experts to avert the risk for occurrence of heat damage on the specific tissue region. The energy output adjustment instruction is described in FIG. 11 in the section “4. First Embodiment”.

In the present embodiment, the control section 110 may present recommendation to decrease the energy output from the present energy output or recommendation to stop the energy output based on the result of the determination.

According to the present embodiment, the recommendation to decrease or stop the energy output can be presented. Hence, it becomes possible for a person who performs surgery to adjust the energy output based on the recommendation. Details are described in FIG. 24 in the section “9. Sixth Embodiment”.

In the present embodiment, the trained model 121 is the trained model that is trained to estimate the heat diffusion region in which heat diffusion from the energy device 310 to the tissue is caused and the specific tissue region from the training energy output information of the energy device 310, the training device tissue image, or the training tissue image, and the control section 110 may perform the process based on the trained model 121 stored in the storage section 120 to estimate the heat diffusion region and the specific tissue region from the energy output information of the energy device 310 and the captured image.

According to the present embodiment, it is possible to feed back the history information of the output energy to the training model 522 in the training phase. Thus, the control section 110 is capable of estimating the heat diffusion region with higher accuracy. The training energy output information is described in FIG. 15 in the section “5. Second Embodiment”.

Further, in the present embodiment, the energy device 310 may be the device that includes the two jaws 337 and 338 capable of gripping a tissue and that receives energy supply from the generator 300 to perform energy output from the two jaws 337 and 338.

That is, the energy device 310 may be the bipolar device 330. The bipolar device is described, for example, in FIG. 5 in the section “3. Energy Device”.

In the present embodiment, the energy device 310 may be the ultrasonic device 340.

The ultrasonic device 340 mentioned herein may be the combination device of the ultrasonic device 340 and the bipolar device 330. The ultrasonic device 340 and the combination device are described, for example, in FIG. 6 in the section “3. Energy Device”.

Further, in the present embodiment, the captured image may include white burns due to heat denaturation.

According to the present embodiment, since the information regarding the range in which white burns occur can be acquired from the endoscope image, there is no need for using another device at the time of creating the training data 521. In addition, it is also possible to reduce the number of networks used by the control section 110, and increase the speed of processing of the control section 110. The white burns are described, for example, in the section “7. Fourth Embodiment”.

Further, in the present embodiment, the captured image may include the endoscope image captured with special light that is different from normal light.

According to the present embodiment, with use of also the multiple wavelength images as time-series data at the time of training of the network used for the heat diffusion detection function and estimation of the heat diffusion region, it is possible to additionally input the information that is focused on the specific tissue such as the connective tissue around the blood vessel. This facilitates obtaining of dynamic information such as aggregation of the connective tissue and increases accuracy of estimation of the specific tissue region, the heat diffusion region, and the like by utilizing the dynamic information. Each of the normal light and the special light is described in the section “6. Third Embodiment”.

Further, in the present embodiment, the trained model 121 is the trained model that is trained to estimate the tissue heat-transfer characteristics from the training energy output information of the energy device 310, the training device tissue image, or the training tissue image, and the control section 110 may perform the process based on the trained model 121 stored in the storage section 120 to estimate the tissue heat-transfer characteristics from the energy output information of the energy device 310 or the captured image.

According to the present embodiment, the control section 110 is capable of predicting the decrease of the distance between the heat diffusion region and the specific tissue region from the characteristics of the heat-transfer path. Thus, it is possible to reduce the risk for heat damage on the specific tissue region and perform surgery safely. The tissue heat-transfer characteristics are described in the section “8. Fifth Embodiment”.

Further, the above processing may also be written as a program. That is, the program of the present embodiment causes the controller 100 to acquire the captured image, perform the process based on the trained model 121 to estimate, from the captured image, the image recognition information, which is at least one of the information regarding the heat diffusion region in which heat diffusion from the energy device to the biological tissue is caused or the information regarding the specific tissue region in the biological tissue, and determine, from the estimated image recognition information, the risk for heat damage on the specific tissue by the energy output from the energy device 310.

Further, the above processing may also be written as a method. That is, the method according to the present embodiment includes acquiring in real time the captured image in which at least one energy device that receives energy supply to output energy and that is outputting energy and at least one biological tissue are imaged, performing the process based on the trained model that is trained to output the image recognition information, which is at least one of the information regarding the heat diffusion region in which heat diffusion from the energy device to the biological tissue is caused by energy output from the energy device or the information regarding the specific tissue region in the biological tissue, from the training device tissue image in which at least one energy device that is outputting energy and at least one biological tissue are imaged or from the training tissue image in which at least one biological tissue is imaged to estimate the image recognition information from the captured image, and determining, from the estimated image recognition information. the risk for heat damage on the specific tissue by the energy output from the energy device.

Although the embodiments to which the present disclosure is applied and the modifications thereof have been described in detail above, the present disclosure is not limited to the embodiments and the modifications thereof, and various modifications and variations in components may be made in implementation without departing from the spirit and scope of the present disclosure. The plurality of elements disclosed in the embodiments and the modifications described above may be combined as appropriate to implement the present disclosure in various ways. For example, some of all the elements described in the embodiments and the modifications may be deleted. Furthermore, elements described in different embodiments and modifications may be combined as appropriate. Thus, various modifications and applications can be made without departing from the spirit and scope of the present disclosure. Any term cited with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced by the different term in any place in the specification or the drawings.

Claims

1. A system comprising:

a memory configured to store a trained model that is trained to estimate a heat diffusion region and a specific tissue region from a training device tissue image or a training tissue image, the training device tissue image being an image in which at least one energy device that receives energy supply to output energy and that is outputting energy and at least one biological tissue are imaged, the training tissue image being an image in which the at least one biological tissue is imaged, the heat diffusion region being a region in which heat diffusion from the at least one energy device to the at least one biological tissue is caused by energy output from the at least one energy device, and the specific tissue region being a region in the at least one biological tissue; and

a processor,

wherein the processor is configured to

acquire a captured image that is an image during the energy output and in which the at least one energy device and the at least one biological tissue are imaged,

perform a process based on the trained model stored in the memory to estimate the heat diffusion region and the specific tissue region from the captured image, and

determine a risk for heat damage on a specific tissue by the energy output from the at least one energy device from the estimated heat diffusion region and the estimated specific tissue region.

2. The system as defined in claim 1, wherein the processor determines that there is the risk for heat damage in a case where a distance between the specific tissue region and the heat diffusion region is a threshold or smaller, the threshold being preliminarily set.

3. The system as defined in claim 2, wherein the threshold is different depending on a tissue type of the specific tissue.

4. The system as defined in claim 1, wherein the captured image is a plurality of endoscope images that are different in timing to capture respective images.

5. The system as defined in claim 4, wherein

the processor

estimates the heat diffusion region and the specific tissue region from each of the plurality of endoscope images, and

outputs time at which heat diffusion reaches the specific tissue region as a result of prediction based on a plurality of the heat diffusion regions and a plurality of the specific tissue regions that are estimated from each of the plurality of endoscope images.

6. The system as defined in claim 5, wherein the processor determines the risk for heat damage before the time as the result of the prediction, and outputs a result of the determination.

7. The system as defined in claim 1, wherein the processor outputs, based on a result of the determination, an energy output adjustment instruction, which is an instruction to decrease the energy output from present energy output or an instruction to stop the energy output, to a generator that controls an amount of energy supply to the at least one energy device based on the energy output adjustment instruction.

8. The system as defined in claim 1, wherein the processor presents recommendation to decrease the energy output from present energy output or recommendation to stop the energy output based on a result of the determination.

9. The system as defined in claim 1, wherein

the trained model is trained to estimate, from training energy output information of the at least one energy device, the training device tissue image, or the training tissue image, the heat diffusion region in which the heat diffusion from the at least one energy device to the at least one biological tissue is caused and the specific tissue region, and

the processor performs a process based on the trained model stored in the memory to estimate the heat diffusion region and the specific tissue region from the energy output information of the at least one energy device and the captured image.

10. The system as defined in claim 1, wherein the at least one energy device is a device that includes two jaws capable of gripping a tissue and that receives the energy supply from a generator to perform the energy output from the two jaws.

11. The system as defined in claim 1, wherein the at least one energy device is an ultrasonic device.

12. The system as defined in claim 1, wherein the captured image includes white burns due to heat denaturation.

13. The system as defined in claim 1, wherein the captured image includes an endoscope image captured with special light that is different from normal light.

14. The system as defined in claim 1, wherein

the trained model is trained to estimate tissue heat-transfer characteristics from training energy output information of the at least one energy device, the training device tissue image, or the training device tissue image, and

the processor performs a process based on the trained model stored in the memory to estimate the tissue heat-transfer characteristics from the energy output information of the at least one energy device or the captured image.