Abstract: Provided are systems and methods which more efficiency train embedding models through the use of a cache of item embeddings for candidate items over a number of training iterations. The cached item embeddings can be “stale” embeddings that were generated by a previous version of the model at a previous training iteration. Specifically, at each iteration, the (potentially stale) item embeddings included in the cache can be used when generating similarity scores that are the basis for sampling a number of items to use as negatives in the current training iteration. For example, a Gumbel-Max sampling approach can be used to sample negative items that will enable an approximation of a true gradient. New embeddings can be generated for the sampled negative items and can be used to train the model at the current iteration.

Abstract: Implementations disclose selecting, in response to receiving a request and from among multiple candidate generative models (e.g., multiple candidate large language models (LLMs)) with differing computational efficiencies, a particular generative model to utilize in generating a response to the request. Those implementations reduce latency and/or conserve computational resource(s) through selection, for various requests, of a more computationally efficient generative model for utilization in lieu of a less computationally efficient generative model. Further, those implementations seek to achieve such benefits, through utilization of more computationally efficient generative models, while also still selectively utilizing less computationally efficient generative models for certain requests to mitigate occurrences of a generated response being inaccurate and/or under-specified.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a data item conditioned on a conditioning input. For example, the data item can be audio, video, or an image. The data item is generated across a plurality of denoising steps, with at least some of the denoising steps being performed using a conditional latent random field model.

Abstract: Provided are systems and methods which more efficiency train embedding models through the use of a cache of item embeddings for candidate items over a number of training iterations. The cached item embeddings can be “stale” embeddings that were generated by a previous version of the model at a previous training iteration. Specifically, at each iteration, the (potentially stale) item embeddings included in the cache can be used when generating similarity scores that are the basis for sampling a number of items to use as negatives in the current training iteration. For example, a Gumbel-Max sampling approach can be used to sample negative items that will enable an approximation of a true gradient. New embeddings can be generated for the sampled negative items and can be used to train the model at the current iteration.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for estimation techniques for efficient neural network inference processing. In some implementations, parameter values for a trained neural network comprising multiple layers are stored, including (i) a matrix of parameter values for at least one layer and (ii) an approximate matrix of values corresponding to the at least one layer. Input is processed using the trained neural network, including determining an input for the at least one layer, computing approximate outputs corresponding to elements in a set using the approximate matrix, and computing intermediate outputs for only a proper subset of the elements in the set using the matrix of parameter values for the at least one layer. The proper subset is determined based on the approximate outputs.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for increasing sparsity to improve neural network efficiency. In some implementations, a system stores parameter values of parameter matrices of one or more layers of a neural network. The parameter values of the parameter matrices include (i) weight values of the one or more layers of the neural network, and (ii) predictor values that have been trained to predict levels of importance of items processed by the neural network. The system generates an output, including: determining a value for each of multiple items using the predictor values, selecting a proper subset of the items based on the values in the vector based on a threshold, and generating output of the one or more layers limiting computation based on the selected proper subset.

Abstract: A method includes receiving, by a computing device, training data to train a neural network, wherein the training data comprises a plurality of inputs and a plurality of corresponding labels. The method also includes mapping, by a representation learner of the neural network, the plurality of inputs to a plurality of feature vectors. The method additionally includes training a kernelized classification layer of the neural network to perform nonlinear classification of an input feature vector into one of a plurality of classes, wherein the kernelized classification layer is based on a kernel which enables the nonlinear classification, and wherein the kernel is selected from a space of positive definite kernels based on application of a nonlinear softmax loss function to the plurality of feature vectors and the plurality of corresponding labels. The method further includes outputting a trained neural network comprising the representation learner and the trained kernelized classification layer.

Abstract: Generally, the present disclosure provides systems and methods for performing machine learning in hyperbolic space. Specifically, techniques are provided which enable the learning of a classifier (e.g., large-margin classifier) for data defined within a hyperbolic space (e.g., which may be particularly beneficial for data possessing a hierarchical structure).

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for privacy preserving training of a machine learning model.

Abstract: A computing system can obtain a machine-learned model comprising one or more parameters having a four-bit binary format. The four-bit binary format can correlate a plurality of sixteen respective binary values to a plurality of sixteen corresponding numerical values represented by the binary values. The sixteen numerical values can be symmetric about a median. A plurality of step sizes between the sixteen numerical values can be non-uniform. The computing system can obtain one or more input values for one or more layers of the machine-learned model. The computing system can process, based at least in part on the one or more parameters having the four-bit binary format, the one or more input values to generate one or more output values.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for accounting for long-tail training data.

Abstract: Generally, the present disclosure is directed to systems and methods that perform spreadout regularization to enable learning of a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a limited number of classes (e.g., a single class). Examples of such settings include decentralized training of face recognition models or speaker identification models, where in addition to the user specific facial images and voice samples, the class embeddings for the users also constitute sensitive information that cannot be shared with other users.

Abstract: Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for processing a network input using a neural network to generate a network output for the network input. That is, by using a neural network that includes a sequence of layer blocks that, for each layer block, processes a block input for the particular layer block through a learned non-linear transformation to generate an initial block output for the particular layer block and combines the initial block output for the particular layer block with at least the block input in accordance with one or more learned parameters to generate the block output for the particular layer block, the described techniques maximize the neural network performance for a given neural network footprint.

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

Abstract: An apparatus includes a pocket portion disposed on production tubing, the pocket portion defining a volume therewithin, wherein the volume is in fluid communication with an interior of the production tubing. The pocket portion includes a valve arrangement that permits unidirectional fluid flow between a tubing-casing annulus and the volume. The apparatus also includes a pressure bleeder configured to be selectively mounted within the pocket portion. The pressure bleeder is configured to permit fluid flow between the valve arrangement and the interior of the production tubing only above a predetermined threshold pressure. A related method includes: providing the pocket portion and valve arrangement; mounting the pressure bleeder within the pocket portion; and, with the pressure bleeder, permitting fluid flow between the valve arrangement and the interior of the production tubing only above a predetermined threshold pressure.

Abstract: Generally, the present disclosure is directed to systems and methods of quantizing a database with respect to a novel loss or quantization error function which applies a weight to an error measurement of quantized elements respectively corresponding to the datapoints in the database. The weight is determined based on the magnitude of an inner product between the respective datapoints and a query compared therewith. In contrast to previous work, embodiments of the proposed loss function are responsive to the expected magnitude of an inner product between the respective datapoints and a query compared therewith and can prioritize error reduction for higher-ranked pairings of the query and the datapoints. Thus, the systems and methods of the present disclosure provide solutions to some of the problems with traditional quantization approaches, which regard all error as equally impactful.

Abstract: A system includes an equipment platform, an electro-magnetic defectoscopy tool, a signal emitter, a signal receiver, and a computer processor. The equipment platform is formed in a ring-like shape having an orifice of a size large enough to fit a circumference of the wellhead or the casing string within the orifice. The electro-magnetic defectoscopy tool is mounted to the equipment platform. The signal emitter is located in the electro-magnetic defectoscopy tool and is configured to emit a signal towards the orifice of the equipment platform. The signal receiver is located in the electro-magnetic defectoscopy tool and is configured to receive a reflected signal from the wellhead or the casing string. The computer processor is electronically connected to the signal receiver and is configured to receive and use the reflected signal to determine a condition of the wellhead or the casing string.

Abstract: A system includes coiled tubing configured to be lowered into a well using a reel. A bottom-hole assembly is connected to a downhole-most end of the coiled tubing. A fail-safe valve is installed within the coiled tubing at a location up-hole from the bottom-hole assembly. A fluid is configured to flow within the coiled tubing. The fail-safe valve is configured to prevent the fluid from flowing in an up-hole direction and permit the fluid from flowing in a downhole direction.

Abstract: A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.

Abstract: A system includes coiled tubing configured to be lowered into a well using a reel. A bottom-hole assembly is connected to a downhole-most end of the coiled tubing. A fail-safe valve is installed within the coiled tubing at a location up-hole from the bottom-hole assembly. A fluid is configured to flow within the coiled tubing. The fail-safe valve is configured to prevent the fluid from flowing in an up-hole direction and permit the fluid from flowing in a downhole direction.