Abstract: Fixed-shaped large language models (LLMs) are generated to perform dual-mode inference. First, a base model of dynamic shapes is quantized. A first LLM and a second LLM are generated from the base model after the quantizing. The first LLM has a fixed input shape greater than one, and the second LLM has a fixed input shape of one. The first LLM and the second LLM have the same network structure and the same weights. The layers of the network structure are grouped into multiple chunks. The number of layers in each chunk is determined by a size limit of a target device memory that loads the chunks during inference. The first LLM and the second LLM are compiled for inference time execution of an LLM task on a target device. The compiling incorporates information of the chunks.

Abstract: A device operates to perform acoustic echo cancellation. The device includes a speaker to output a far-end signal at the device, a microphone to receive at least a near-end signal and the far-end signal from the speaker to produce a microphone output, and an AI accelerator operative to perform neural network operations according to a first neural network model and a second neural network model to output an echo-suppressed signal. The device further includes a digital signal processing (DSP) unit. The DSP unit is operative to perform adaptive filtering to remove at least a portion of the far-end signal from the microphone output to generate a filtered near-end signal, and perform Fast Fourier Transform (FFT) and inverse FFT (IFFT) to generate input to the first neural network model and the second neural network model, respectively.

Abstract: A bit-widths determination method selects bit-widths for mixed-precision neural network computing on a target hardware platform. An activation quantization sensitivity (AQS) value is calculated for each convolution layer in a neural network. The AQS value indicates the sensitivity of convolution output to quantized convolution input. One or more convolution layers are grouped into a quantization group, which is to be executed by a corresponding set of target hardware. A group AQS value is calculated for each quantization group based on the AQS values of the convolution layers in the quantization group. Then bit-widths supported by the target hardware platform are selected for the corresponding quantization groups. The bit-widths are selected to optimize, under a given constraint, a sensitivity metric that is calculated based on each quantization group's group AQS value.

Abstract: A device operates to perform acoustic echo cancellation. The device includes a speaker to output a far-end signal at the device, a microphone to receive at least a near-end signal and the far-end signal from the speaker to produce a microphone output, and an AI accelerator operative to perform neural network operations according to a first neural network model and a second neural network model to output an echo-suppressed signal. The device further includes a digital signal processing (DSP) unit. The DSP unit is operative to perform adaptive filtering to remove at least a portion of the far-end signal from the microphone output to generate a filtered near-end signal, and perform Fast Fourier Transform (FFT) and inverse FFT (IFFT) to generate input to the first neural network model and the second neural network model, respectively.