Abstract: Systems, apparatus, articles of manufacture, and methods are disclosed to evict tokens from a key value cache. An example apparatus includes interface circuitry, machine readable instructions, and programmable circuitry to at least one of instantiate or execute the machine readable instructions to: determine score history values for tokens based on attention scores associated with the tokens, wherein a token is a numerical representation of text, after a number of tokens present in the key value cache exceeds a threshold number of tokens, compute group importance scores for groups of tokens based on score history values of the tokens in the groups of tokens, identify low-ranked groups of tokens having lowest group importance scores, the low-ranked groups of tokens associated with an eviction range in the key value cache, and remove an identified low-ranked group of tokens from the eviction range of the key value cache.

Abstract: Systems, apparatuses and methods may provide for technology that accesses a pre-trained artificial intelligence (AI) model, quantizes a plurality of weights of the pre-trained AI model to generate a compressed AI model, and applies normalization correction to the compressed AI model to generate an output AI model.

Abstract: Systems, apparatuses and methods may provide for technology that selects a subset of linear layers from a plurality of linear layers in a pre-trained artificial intelligence (AI) model, wherein a quantization error of the subset of linear layers exceeds an error threshold. For each linear layer in the subset of linear layers, the technology solves a singular value decomposition (SVD) approximation, generates a first adapter layer and a second adapter layer based on the SVD decomposition, wherein the first adapter layer and the second adapter layer include weight matrices having a first dimension that is less than a first rank threshold and a second dimension that is greater than a second rank threshold, and determines an inference output based on the linear layer, the first adapter layer and the second adapter layer.