Abstract: A method including, for each of a plurality of job nodes, corresponding to a job request of a scheduler node, distributing an application container and a sidecar container of a corresponding job node of the plurality of job nodes, and storing, by the sidecar container, information about a respective state of the application container in a memory through a communication between the sidecar container and at least one sidecar container of another job node of the plurality of job nodes.

Abstract: Disclosed is a multi-device based inference method and apparatus, where the multi-device based inference method includes receiving information related to operation devices performing an operation included in a neural network and a graph corresponding to the neural network, obtaining a size of an output of the operation in a forward direction of the graph based on the information and the graph, dividing an input of the operation in a backward direction of the graph based on the information, the graph, and the size of the output, and performing an inference based on the divided input.

Abstract: An accelerator resource management method and apparatus are disclosed. The accelerator resource management method includes receiving a task request for a neural network-related task and a resource scheduling policy for the neural network-related task, obtaining information on a current resource utilization status of an accelerator cluster comprising a plurality of accelerators, in response to the task request, and allocating an accelerator resource for performing the task based on a utility of a resource allocation that is based on the resource scheduling policy and the information.

Abstract: A method includes predicting, for sets of input data, an input data number of a subsequent interval of a first interval using an input data number of the first interval and an input data number of a previous interval of the first interval set in a neural network inference optimization, determining the predicted input data number to be a batch size of the subsequent interval, determining whether pipelining is to be performed in a target device based on a resource state of the target device, and applying, to the target device, an inference policy including the determined batch size and a result of the determining of whether the pipelining is to be performed.

Abstract: An operation method includes: dividing a model to be executed in an accelerator into a plurality of stages; determining, for each of the stages, a maximum batch size processible in an on-chip memory of the accelerator; determining the determined maximum batch sizes to each be a candidate batch size to be applied to the model; and determining, to be a final batch size to be applied to the model, one of the determined candidate batch sizes that minimizes a sum of a computation cost of executing the model in the accelerator and a memory access cost.

Abstract: Disclosed is a multi-device based inference method and apparatus, where the multi-device based inference method includes receiving information related to operation devices performing an operation included in a neural network and a graph corresponding to the neural network, obtaining a size of an output of the operation in a forward direction of the graph based on the information and the graph, dividing an input of the operation in a backward direction of the graph based on the information, the graph, and the size of the output, and performing an inference based on the divided input.