Abstract: Provided are improved end-to-end self-supervised pre-training frameworks that leverage a combination of contrastive and masked modeling loss terms. In particular, the present disclosure provides framework that combines contrastive learning and masked modeling, where the former trains the model to discretize input data (e.g., continuous signals such as continuous speech signals) into a finite set of discriminative tokens, and the latter trains the model to learn contextualized representations via solving a masked prediction task consuming the discretized tokens. In contrast to certain existing masked modeling-based pre-training frameworks which rely on an iterative re-clustering and re-training process or other existing frameworks which concatenate two separately trained modules, the proposed framework can enable a model to be optimized in an end-to-end fashion by solving the two self-supervised tasks (the contrastive task and masked modeling) simultaneously.

Abstract: Systems and methods can utilize a conformer model to process a data set for various data processing tasks, including, but not limited to, speech recognition, sound separation, protein synthesis determination, video or other image set analysis, and natural language processing. The conformer model can use feed-forward blocks, a self-attention block, and a convolution block to process data to learn global interactions and relative-offset-based local correlations of the input data.

Abstract: Transportation plan adjustments based on recommended shipping packages may utilize a shipment data model for a facility that is updated with cubic volumes of recommended packages for received orders. The shipment data model may also be updated with the cubic volumes of the packages that are actually used to pack the orders and transportation utilization data such as assignment of packages to particular transportation resources. A transportation plan may be updated based on the updated shipment data model and transportation utilization data such as which transportation resources have left the facility and how much of the available cubic volume of the resource was used when the resource left the facility. The transportation plan may be updated at times associated with transportation resource change deadlines from one or more transportation resource providers. Adjustments to previously scheduled transportation resources may be made based on the updated transportation plan.