Abstract: Implementations relate to reducing latency in generating and/or rendering natural language (NL) output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, and generate the NL based output utilizing the LLM. The NL based output can be a stream of NL based output in that it includes a plurality of segments, and is generated on a segment-by-segment basis. In some implementations, a first segment of the stream of NL based output is selected for inclusion in the stream of NL based output as a second segment (and any subsequent segment) is being generated to reduce latency in evaluating the NL based output as a whole prior to rendering thereof. In some versions of those implementations, the first segment is rendered as the second segment (and any subsequent segment) is being generated to further reduce latency in rendering thereof.

Abstract: Implementations relate to reducing latency in generating and/or rendering natural language (NL) output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, and generate the NL based output utilizing the LLM. The NL based output can be a stream of NL based output in that it includes a plurality of segments, and is generated on a segment-by-segment basis. In some implementations, a first segment of the stream of NL based output is selected for inclusion in the stream of NL based output as a second segment (and any subsequent segment) is being generated to reduce latency in evaluating the NL based output as a whole prior to rendering thereof. In some versions of those implementations, the first segment is rendered as the second segment (and any subsequent segment) is being generated to further reduce latency in rendering thereof.

Abstract: Implementations relate to reducing latency in generating and/or rendering natural language (NL) output generated using a large language model (LLM). Processor(s) of a system can: receive NL based input associated with a client device, and generate the NL based output utilizing the LLM. The NL based output can be a stream of NL based output in that it includes a plurality of segments, and is generated on a segment-by-segment basis. In some implementations, a first segment of the stream of NL based output is selected for inclusion in the stream of NL based output as a second segment (and any subsequent segment) is being generated to reduce latency in evaluating the NL based output as a whole prior to rendering thereof. In some versions of those implementations, the first segment is rendered as the second segment (and any subsequent segment) is being generated to further reduce latency in rendering thereof.

Abstract: A method and system for processing a bit-packed array using one or more processors, including determining a data element size of the bit-packed array, determining a lane configuration of a single-instruction multiple-data (SIMD) unit for processing the bit-packed array based at least in part on the determined data element size, the lane configuration being determined from among a plurality of candidate lane configurations, each candidate lane configuration having a different number of vector register lanes and a corresponding bit capacity per vector register lane, configuring the SIMD unit according to the determined lane configuration, and loading one or more data elements into each vector register lane of the SIMD unit. SIMD instructions may be executed on the loaded one or more data elements of each vector register lane in parallel, and a result of the SIMD instruction may be stored in memory.

Abstract: A method and system for processing a bit-packed array using one or more processors, including determining a data element size of the bit-packed array, determining a lane configuration of a single-instruction multiple-data (SIMD) unit for processing the bit-packed array based at least in part on the determined data element size, the lane configuration being determined from among a plurality of candidate lane configurations, each candidate lane configuration having a different number of vector register lanes and a corresponding bit capacity per vector register lane, configuring the SIMD unit according to the determined lane configuration, and loading one or more data elements into each vector register lane of the SIMD unit. SIMD instructions may be executed on the loaded one or more data elements of each vector register lane in parallel, and a result of the SIMD instruction may be stored in memory.

Abstract: A computing apparatus may process an operation. The computing apparatus may output information regarding an aggregation operation and an operand corresponding to a variable stored in a memory, store information regarding an operator and the aggregation operands regarding the aggregation operation, perform a first partial operation with respect to the aggregation operands and store a result value of the first partial operation, and process the aggregation operation based on storing the variable, performing a second partial operation with respect to the result value of the first partial operation stored in the cache and the operand corresponding to the variable, and storing a result value of the second partial operation.

Abstract: A method of managing a cache includes storing first data of an upper level cache in a lower level cache, predicting a reuse distance level of second data having a same signature as the first data based on access information about the first data, and storing the second data in one of the lower level cache and a main memory based on the predicted reuse distance level of the second data.

Abstract: A computing apparatus may process an operation. The computing apparatus may output information regarding an aggregation operation and an operand corresponding to a variable stored in a memory, store information regarding an operator and the aggregation operands regarding the aggregation operation, perform a first partial operation with respect to the aggregation operands and store a result value of the first partial operation, and process the aggregation operation based on storing the variable, performing a second partial operation with respect to the result value of the first partial operation stored in the cache and the operand corresponding to the variable, and storing a result value of the second partial operation.

Abstract: A method of managing a cache includes storing first data of an upper level cache in a lower level cache, predicting a reuse distance level of second data having a same signature as the first data based on access information about the first data, and storing the second data in one of the lower level cache and a main memory based on the predicted reuse distance level of the second data.

Abstract: In one example embodiment of the inventive concepts, a cache memory system includes a main cache memory including a nonvolatile random access memory, the main cache memory configured to exchange data with an external device and store the exchange data, each exchanged data includes less significant bit (LSB) data and more significant bit (MSB) data. The cache memory system further includes a sub-cache memory including a random access memory, the sub-cache memory configured to store LSB data of at least a portion of data stored at the main cache memory, wherein the main cache memory and the sub-cache memory are formed of a single-level cache memory.