Abstract: A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.

Abstract: In one embodiment, a method includes accessing a map of a building floor plan with locations of access points within the floor plan, the access points being capable of performing wireless communications with wireless devices. Determining a pose of a wireless device within the map using images captured by one or more cameras of the wireless device. Selecting a preferred access point based on the pose of the wireless device, the floor plan, and the locations of the plurality of access points within the floor plan. Configuring wireless communication settings of the wireless device to communicate with the preferred access point based on the pose of the wireless device and the location of the preferred access point within the floor plan.

Abstract: A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.

Abstract: A compute engine including an input buffer and a compute-in-memory (CIM) hardware module is described. The input buffer is coupled to the CIM hardware module and provides an input vector to the CIM hardware module. The CIM hardware module includes an array of storage cells and compute logic. The array of storage cells is configured to store weights corresponding to a matrix. The compute logic is configured to perform a vector-matrix multiplication (VMM) for the matrix and the input vector. The array of storage cells includes storage blocks. Each storage block includes rows and a particular number of columns corresponding to a portion of the matrix. The compute logic includes compute logic blocks. Each compute logic block corresponds to a storage block of the storage blocks. The compute logic block performs a portion of the VMM for the portion of the matrix.

Abstract: A compute engine (CE) is described. The CE includes a compute-in-memory (CIM) module and an input buffer coupled with the CIM module. The CIM module includes storage cells and compute logic coupled with the storage cells. The storage cells are arranged in rows and columns. The input buffer is configured to receive data, reshape the data, and provide reshaped data to the CIM module.

Abstract: A method includes accessing map data of an area of a real environment, the map data comprising three-dimensional feature descriptors describing features visible in the real environment. A plurality of map packages are generated based on the map data, wherein each of the map packages (1) corresponds to a two-dimensional sub-area within the area of the real environment, and (2) comprises a subset of the three-dimensional feature descriptors describing features visible in the sub-area. A first sequence of the plurality of map packages are broadcast through one or more base stations, wherein the first sequence is based on the two-dimensional sub-area of each of the map packages, wherein each of the map packages is configured to be received and used by an artificial-reality device to determine a pose of the artificial-reality device in the associated sub-area based on the associated subset of the three-dimensional feature descriptors.

Abstract: A method for performing a convolution is described. The method includes providing an activation to a general purpose (GP) processor. The GP processor is coupled with compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module stores weights corresponding to a kernel and is configured to perform vector-matrix multiplications (VMMs) for the kernel. The method also includes performing, by the GP processor or at least one of the compute engines, a quantization of the activation to provide a quantized activation. The compute engine(s) perform the VMMs for the quantized activation and the kernel to provide a product. Dequantization of the product is performed by the GP processor or the compute engine(s).

Abstract: In one embodiment, a computing system accesses a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose. The computing system determines a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera. The computing system projects the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera. The computing system generates, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated. The computing system instructs, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment. The computing system reads pixel values of the new image.

Abstract: A compute tile is described. The compute tile includes compute engines and a general-purpose (GP processor coupled with the compute engines. Each of the compute engines includes a compute-in-memory (CIM) hardware module. The CIM hardware module is configured to store weights corresponding to a matrix and to perform a vector-matrix multiplication (VMM) for the matrix. The GP processor is configured to control the compute engines, to receive output of the VMM for the matrix from the compute engines, and to perform a nonlinear operation on the output. The compute engines are addressable by data movement initiators. Data may be moved to and/or from the compute engines in data paths that bypass the GP processor.

Abstract: A method is described. A general-purpose (GP) processor configured to communicate with a single co-processor identifies a first compute engine. Each compute engine includes a compute-in-memory (CIM) hardware module. The CIM hardware module stores weights corresponding to a matrix and performs a vector-matrix multiplication (VMM) of a vector and the matrix. First data is written to or loaded from the first compute engine by the GP processor. The method also includes identifying, by the GP processor, a second compute engine after the first data is written to and/or loaded from the first compute engine by the GP processor. Second data is written to and/or loaded from the second compute engine by the GP processor. The first data and the second data are for the weights, the vector, and/or a VMM for the first compute engine. The GP processor provides control and data movement for the compute engines.

Abstract: In one embodiment, a method includes accessing a map of a building floor plan with locations of access points within the floor plan, the access points being capable of performing wireless communications with wireless devices. Determining a pose of a wireless device within the map using images captured by one or more cameras of the wireless device. Selecting a preferred access point based on the pose of the wireless device, the floor plan, and the locations of the plurality of access points within the floor plan. Configuring wireless communication settings of the wireless device to communicate with the preferred access point based on the pose of the wireless device and the location of the preferred access point within the floor plan.

Abstract: A method includes accessing map data of an area of a real environment, the map data comprising three-dimensional feature descriptors describing features visible in the real environment. A plurality of map packages are generated based on the map data, wherein each of the map packages (1) corresponds to a two-dimensional sub-area within the area of the real environment, and (2) comprises a subset of the three-dimensional feature descriptors describing features visible in the sub-area. A first sequence of the plurality of map packages are broadcast through one or more base stations, wherein the first sequence is based on the two-dimensional sub-area of each of the map packages, wherein each of the map packages is configured to be received and used by an artificial-reality device to determine a pose of the artificial-reality device in the associated sub-area based on the associated subset of the three-dimensional feature descriptors.

Abstract: In one embodiment, a computing system accesses a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose. The computing system determines a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera. The computing system projects the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera. The computing system generates, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated. The computing system instructs, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment. The computing system reads pixel values of the new image.

Abstract: In one embodiment, a method for tracking includes capturing a first frame of the environment using a first camera, identifying, in the first frame, a first patch that corresponds to the first feature, accessing a first local memory of the first camera that stores reference patches identified in one or more previous frames captured by the first camera, and determining that none of the reference patches stored in the first local memory corresponds to the first feature. The method further includes receiving, from a second camera through a data link connecting the second camera with the first camera, a reference patch corresponding to the first feature. The reference patch is identified in a previous frame captured by the second camera and of the second camera. The method may then determine correspondence data between the first patch and the reference patch, and tracks the first feature in the environment based on the determined correspondence data.

Abstract: In one embodiment, a method for tracking includes capturing a first frame of the environment using a first camera, identifying, in the first frame, a first patch that corresponds to the first feature, accessing a first local memory of the first camera that stores reference patches identified in one or more previous frames captured by the first camera, and determining that none of the reference patches stored in the first local memory corresponds to the first feature. The method further includes receiving, from a second camera through a data link connecting the second camera with the first camera, a reference patch corresponding to the first feature. The reference patch is identified in a previous frame captured by the second camera and of the second camera. The method may then determine correspondence data between the first patch and the reference patch, and tracks the first feature in the environment based on the determined correspondence data.

Abstract: A system and method for visual inertial navigation are described. In some embodiments, a device comprises an inertial measurement unit (IMU) sensor, a camera, a radio-based sensor, and a processor. The IMU sensor generates IMU data of the device. The camera generates a plurality of video frames. The radio-based sensor generates radio-based sensor data based on an absolute reference frame relative to the device. The processor is configured to synchronize the plurality of video frames with the IMU data, compute a first estimated spatial state of the device based on the synchronized plurality of video frames with the IMU data, compute a second estimated spatial state of the device based on the radio-based sensor data, and determine a spatial state of the device based on a combination of the first and second estimated spatial states of the device.