Abstract: Integrated cells may perform matrix multiplication (MatMul) operations. An integrated cell may include a random-access memory (RAM) cell, dot product unit(s), multiplexer(s), adder, route-in unit, control unit, and vector machine. The RAM cell may store weights and activations. The dot product unit(s) may compute dot products from the weights and activations. The adder may accumulate the dot products. The route-in unit may facilitate data transfer from the RAM cell to the dot product unit(s) or data transfer from another integrated cell to the integrated cell. The control unit may manage memory operations and detect and repair errors in memory operations. The vector machine may provide instructions to the dot product unit(s) and multiplexers to direct the flow of multiply-accumulate operations. Counters may be used to control weight fetching from RAM cells. A MatMul operation may be decomposed, and the integrated cells may perform the MatMul operation through multiple clock cycles.

Abstract: An integrated circuit (IC) device may implement a deep neural network (DNN). The IC device may be a three-dimensional (3D) integrated system that includes a memory die and logic die. The memory die may include memory blocks, such as sequential random-access memory blocks or a sequential read-only memory blocks. The logic die may include an interface unit, a vector operation unit, compute units (e.g., multiply-accumulate units), and an interconnect fabric with adders. The interface unit may receive the input of the DNN and transfer the input to the vector operation unit. The vector operation unit may perform one or more vector operations of the DNN based on the input. The compute units and adders may perform matrix multiplication operations of the DNN based on the vector operation unit's output. Each memory block may be coupled with a compute unit through a via.

Abstract: An integrated circuit (IC) device may implement a deep neural network (DNN). The IC device may include an activator unit that implements a nonlinear activation function in the DNN. The nonlinear activation function may be decomposed into a rectified linear unit (ReLU) function and a symmetric function. After receiving an input value, the activator unit may apply the ReLU function on the input value to compute a first value. The input range of the nonlinear activation function may be partitioned into segments. The activator unit may determine which segment the input value falls into. The activator unit may apply a linear function, which approximates the symmetric function within the segment, on the input value to compute a second value. The activator unit may correct an error in the second value and compute an output of the nonlinear activation function based on the first value and the second value.

Abstract: A state space model with selective updates, also referred to as a Mamba-based block, in a Mamba-based model can be embedded onto a silicon chip. Specialized hardware modules in a models-on-silicon chip, such as an optimized selective scan unit and an optimized 1D convolution unit, can perform the operations of the selective state space model of the Mamba-based model. These modules individually and collectively enhance processing speed, power efficiency, and overall performance. The parameters such as weights of the Mamba-based model are arranged in a sequential order in one or more sequential read memories according to a predetermined timing sequence. By embedding the selective state space model onto the models-on-silicon architecture, which excels in managing larger input context sizes, this solution transforms the Mamba-based model into a highly viable and efficient option for AI tasks being performed on resource-constrained devices.

Abstract: An integrated circuit (IC) device may implement a neural network model. The IC device may include stacked embedding dies, stacked attention dies, and a base die. The embedding dies may perform embedding computations in the model. Each embedding die may have an embedding dot unit that includes memories for storing precomputed embedding vectors, multiply units for performing multiplication operations on embeddings, add units for summing the results of the multiplication operations. The attention dies may perform attention computations in the model. Each attention die may have an attention dot unit that includes memories for storing intermediate values, multiply units for performing multiplication operations for attention mechanisms, add units for summing the results of the multiplication operations. The base die may coordinate the overall operation of the model and perform preprocessing, embedding, normalization, activation, and final output generation.

Abstract: A speculative decoding system may include integrated circuits (ICs), a router, and a processing unit. The ICs may implement different models that can perform different types of tasks. The router may route an input prompt, which may include one or more input tokens, to an IC based on the task to be performed using the input prompt. The IC may include hardware implementations of operators in a model. The IC may generate speculative token(s) from the input prompt by running the operators in the model. The speculative token(s) may be drafted to the processing unit. The processing unit may validate the speculative token(s) and generate output token(s) by executing another model, which may be larger than the model executed by the IC. The processing unit may validate multiple speculative tokens in parallel. Key-value pairs generated by the IC may be used by the processing unit for executing the other model.

Abstract: An integrated circuit (IC) device may implement a contextual embedding model. The IC device may include a tokenizer unit, embedder unit, layer normalizer unit, dot unit, activator units, and flow control unit. The tokenizer unit may implement a tokenizer in the model and convert text to tokens using the vocabulary of the model. The embedder unit may implement embedders in the model and generate embeddings from the tokens. The layer normalizer unit may implement one or more layer normalizers in the model and compute embedding vectors. The dot unit may implement matrix multiplication and add operations in the encoders and pooler of the model. The activator units may implement activation functions, including tanh function, in the model. The flow control unit may orchestrate the other components of the IC device based on a timing sequence of neural network operations in the model.

Abstract: An agent chip in a multi-chip architecture orchestrates multiple specialized AI models embedded and/or etched on different chips. Implementing the agent chip effectively solves the problem of deploying multiple specialized AI models in a cost-effective and scalable manner by training and utilizing the agent chip to orchestrate multiple specialized AI models embedded on different models-on-silicon chips. Each models-on-silicon chip is optimized for a specific task or goal, and the agent chip coordinates and/or routes their activities to perform complex, multi-faceted tasks efficiently. Accordingly, the multi-chip architecture allows for efficient, scalable, and cost-effective machine learning inference, significantly reducing power consumption and latency.

Abstract: A convolutional neural network may be embedded onto an integrated circuit (IC) device, which includes an embedder unit, a flow control unit, and etched mind unit(s). The embedder unit may generate a feature map from an input image. The etched mind unit(s) may be a hardware implementation of the CNN and execute neural network operations of the CNN using the feature map. An etched mind unit may include a convolution unit implementing convolution, a batch-norm unit implementing batch normalization, an activator unit implementing an activation function operation, a max pooling unit implementing max pooling, and an average pooling unit implementing average pooling, and a MatMul unit implementing matrix multiplication, each of which may has its own memory that stores weights or other data for performing a neural network operation. The flow contour unit may orchestrate the other components of the IC device based on a timing sequence of the network.

Abstract: Building on the models-on-silicon (model-on-chip or model-on-die) architecture and design, multiple models-on-silicon chips/dies can be arranged in a stacked formation to form a single cube, referred to herein as AI cube. Each of these chips or dies can embed one or more transformer blocks, such as one or more consecutive transformer blocks of a transformer-based neural network. This stacked configuration enables processing of data in a feedforward manner, effectively performing processing for an inference task of a transformer-based neural network, e.g., an entire large language model, within one compact semiconductor integrated circuit package. For example, a 70 billion parameter LLM can be arranged and implemented onto an AI cube, where different groups of transformer blocks are distributed to different chips in the AI cube in a feedforward manner.

Abstract: An integrated circuit (IC) device may implement a speech recognition model with a transformer-based architecture. The IC device may include an embedder unit, etched mind unit(s), a layer normalizer unit, a sampler unit, and a flow control unit. The embedder unit may be a hardware implementation of an embedder in the model. The etched mind unit(s) may be a hardware implementation of matrix multiplications and additions in the model. The layer normalizer unit may implement a layer normalizer in the model. The sampler unit may implement a sampler in the model. The sampler unit may use comparators to find the largest value of a vector received from the etched mind unit(s). The sampler unit may determine the index of the largest value and output a predicted token. The flow contour unit may orchestrate the other components of the IC device based on a timing sequence of the model.