Abstract: A neural processing unit (NPU) configured to perform neural network inference is disclosed. The NPU includes a processing core configured to process matrix operations in parallel based on integer data, and a special function unit (SFU) configured to process special operations including nonlinear functions based on floating-point data. The NPU further includes a controller configured to control the processing core and the special function unit based on compiled execution code. The controller is further configured to perform speculative decoding using a first language model having a first number of parameters and a second language model having a second number of parameters less than the first number, by verifying an inference result generated by the second language model using the first language model.

Abstract: According to an example of the present disclosure, a system is provided. The system includes a plurality of heterogeneous processors configured to perform inference using a neural network model including a plurality of layers, and a memory configured to store the neural network model, each layer including a plurality of parameters having respective bit-widths. A controller is configured to assign an operation for an arbitrary layer of the neural network model to at least one target processor among the plurality of heterogeneous processors. The controller calculates an objective function for each of the heterogeneous processors, the objective function accounting for a performance loss of the neural network model relative to a hardware constraint of each processor, selects a first processor that minimizes the objective function as a main target processor for a first portion of the arbitrary layer, and assigns an operation for the first portion to the selected first processor.

Abstract: A movable apparatus is provided. The apparatus may comprise a sensor, a plurality of cameras, a neural processor configured to detect and track an object by receiving images from the plurality of cameras, and a controller circuit configured to determine a moving direction by the sensor, increase a first frame per second (FPS) of a selected camera among the plurality of cameras corresponding to the moving direction, and decrease a second FPS of at least one other camera among the plurality of cameras corresponding to a different direction.

Abstract: A neural processing unit (NPU) is disclosed. The NPU includes a processing core configured to process matrix operations based on integer data, and a special function unit (SFU) configured to process special operations including nonlinear function operations based on floating-point data. The NPU further includes a controller configured to determine validity of an operation to be processed in the processing core or the SFU, and, when the operation is determined to be an unnecessary operation, to perform at least one of clock gating and power gating on at least one of the processing core and the SFU related to the unnecessary operation.

Abstract: An artificial intelligence (AI) module is disclosed. The AI module includes at least one neural processing unit (NPU) configured to process AI operations. The at least one NPU includes a processing core configured to process tensor operations based on integer data, and a special function unit configured to approximate nonlinear activation functions using programmable parameters. The special function unit includes a programmable function approximation circuit configured to determine an interval to which an input value belongs, select a programmable parameter corresponding to the determined interval, and perform a polynomial operation using the selected programmable parameter.

Abstract: An electronic device mounted on a fixed or a movable apparatus is provided. The electronic device may comprise a neural processing unit (NPU), including a plurality of processing elements (PEs), configured to process an operation of an artificial neural network model trained to detect or track at least one object and output an inference result based on at least one image acquired from at least one camera; and a signal generator generating a signal applicable to the at least one camera.

Abstract: A method for controlling a data allocatable neural processing unit is provided. The method may comprise: determining a dependency determination between data blocks of a specific layer of at least one neural network model; allocating memory banks to store the data blocks based on the dependency determination; and storing each data block in the allocated memory bank.

Abstract: An image processing method performed by a neural processing unit is disclosed. The method includes receiving an input image including at least one object and processing the input image using a first model via the neural processing unit to detect a particular object among the at least one object, the first model being an artificial neural network-based object detector trained to detect the particular object in an image. The method further includes processing the input image using a second model via the neural processing unit to blur the particular object in the input image, the second model being an artificial neural network trained to blur a region corresponding to the particular object. An output image including the blurred particular object is generated.

Abstract: Methods and devices of a function approximation unit configured to approximate a nonlinear function within a neural processing unit are described. According to one embodiment, the method includes storing an input value through an input register of the function approximation unit, transmitting the input value to a selected one of a plurality of preprocessing circuits of the unit according to a control signal, generating a preprocessing result corresponding to the input value by the selected one of the preprocessing circuits, transmitting the preprocessing result to a programmable function approximation circuit and a selected one of a plurality of post-processing circuits of the unit, generating an approximated function output based on the preprocessing result in the programmable function approximation circuit, and generating a final output value by post-processing the preprocessing result or the approximated function output in the selected one of the post-processing circuits.

Abstract: A neural processing apparatus is disclosed. The apparatus includes a memory array including a plurality of physically distinct memory banks, configurable interconnect circuitry coupled to the plurality of memory banks, and a controller. The controller is configured to dynamically reallocate the plurality of memory banks into variable-sized memory regions corresponding to different data types required for an inference operation of a neural network model. The controller adjusts a number of memory banks allocated to each data type for a current layer of the neural network model based on configuration information derived from a structure of the neural network model. The controller is further configured to enable data access between the reallocated memory banks and a processing engine via the configurable interconnect circuitry.

Abstract: A neural processing unit for performing inference operations of a large-scale language model based on an artificial neural network is disclosed.

Abstract: A scalable AI system includes a plurality of slots on a main board, at least one AI module mounted in the slots to perform AI operations, and a main processor that controls the entire system. A user can flexibly scale the overall AI computation performance of the system by adjusting the number of AI modules as needed. Each AI module is equipped with a low-power NPU specialized for AI operations and adopts standard form factors (e.g., M.2, E1.S) to improve economy and scalability. This can solve the problems of high-power consumption and cost, which are limitations of GPU-based systems, and can efficiently support various AI applications.

Abstract: A mobile electronic device may include a main memory for storing weight values of a compiled and quantized large language model, a processing core for performing an inference operation based on an attention mechanism on text based on the large language model, a controller for controlling the operation of the processing core and the main memory and for scheduling the inference operation, and a special function unit for performing vector operations and scalar operations necessary for the inference operation. Quantization and dequantization are performed in the inference operation of a large language model, enabling on-device inference operations with minimal performance degradation.

Abstract: Provided is a method for performing an aging test on a neural processing unit (NPU) with a capability of a runtime test. The method may comprise: performing an aging test on the NPU which comprises a plurality of functional components. The plurality of functional components may comprise at least one memory and plural processing elements. The performing of the aging test may include: performing a scan test on the NPU to verify whether at least one functional component in the NPU is defective or not; and performing a memory test on the at least one memory. At least one of the scan test and the memory test may be repeatedly performed to put a stress on the NPU for the aging test. The aging test may be repeated by a predetermined number.

Abstract: A neural processing unit (NPU) for decoding video or feature map is provided. The NPU may comprise at least one processing element (PE) to perform an inference using an artificial neural network. The at least one PE may be configured to receive and decode data included in a bitstream. The data included in the bitstream may comprise data of a base layer. Alternatively, the data included in the bitstream may comprise data of the base layer and data of at least one enhancement layer. The data of the base layer included in the bitstream may include a first feature map. The data of the at least one enhancement layer included in the bitstream may include a second feature map.

Abstract: An image processing method is disclosed. The method includes receiving an input image including at least one object, and classifying the at least one object in the input image using a first model based on an artificial neural network trained to classify objects into one of a plurality of predetermined categories. At least one second model corresponding to the classified category of the at least one object is determined from among a plurality of second models, each of which is based on an artificial neural network trained to output a specialized processing applied image specific to a respective category. An output image is obtained by inputting the input image, or a region thereof corresponding to the at least one object, into the determined at least one second model.

Abstract: A neural processing unit (NPU) is capable of testing a component of the NPU in a running system, i.e., during runtime. The NPU includes a plurality of functional components, each of which includes an electronic circuit; at least one wrapper connected to at least one of the functional components; and an in-system component tester (ICT). The ICT performs a selection of one of the at least one functional component, in an idle state, as a component under test (CUT) and performs a test, via the at least one wrapper, of the selected functional component. The ICT may monitor states of the plurality of the functional components via the at least one wrapper, stop the test based on a detection of a collision due to an access to the selected functional component, and return a connection of the selected functional component to the at least one wrapper according to the stop.

Abstract: A function approximation unit includes a programmable function approximation circuit configured to directly compute a piecewise function according to a pre-stored coefficient; one or more dedicated function circuits configured to process at least one specific function among a reciprocal, a reciprocal square root, and a negative exponential function through a fixed hard-wired operation pipeline; and control logic configured to selectively activate either the programmable function approximation circuit or the dedicated function circuit according to the type of the nonlinear function to be processed.

Abstract: An electronic device according to various examples is provided. The electronic device may include a decoder for decoding a compressed video; and a neural processing unit (NPU) for outputting a result frame comprising a first target data in response to searching for the first target data in one or more frames of a first compressed video while the decoder is decoding the first compressed video comprising the one or more frames.

Abstract: According to one example of the present disclosure, the neural processing unit may comprise a processing element array configured to perform operations of a neural network model and a post-processing unit configured to process data output from the processing element array. The post-processing unit includes a first computation circuit that extracts a subset of classes for each bounding box by comparing class scores of classes and a second computation circuit configured to extract one or more bounding boxes by comparing a class confidence score of each bounding box with a threshold confidence score.