Patents by Inventor Xinnan Yu
Xinnan Yu has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240098138Abstract: The present disclosure provides systems and methods for communication efficient distributed mean estimation. In particular, aspects of the present disclosure can be implemented by a system in which a number of vectors reside on a number of different clients, and a centralized server device seeks to estimate the mean of such vectors. According to one aspect of the present disclosure, a client computing device can rotate a vector by a random rotation matrix and then subsequently perform probabilistic quantization on the rotated vector. According to another aspect of the present disclosure, subsequent to quantization but prior to transmission, the client computing can encode the quantized vector according to a variable length coding scheme (e.g., by computing variable length codes).Type: ApplicationFiled: August 31, 2023Publication date: March 21, 2024Inventors: Ananda Theertha Suresh, Sanjiv Kumar, Hugh Brendan McMahan, Xinnan Yu
-
Patent number: 11874866Abstract: The present disclosure provides systems and methods that include or otherwise leverage use of a multiscale quantization model that is configured to provide a quantized dataset. In particular, the multiscale quantization model can receive and perform vector quantization of a first dataset. The multiscale quantization model can generate a residual dataset based at least in part on a result of the vector quantization. The multiscale quantization model can apply a rotation matrix to the residual dataset to generate a rotated residual dataset that includes a plurality of rotated residuals. The multiscale quantization model can perform reparameterization of each rotated residual in the rotated residual dataset into a direction component and a scale component. The multiscale quantization model can perform product quantization of the direction components of the plurality of rotated residuals, and perform scalar quantization of the scale components of the plurality of rotated residuals.Type: GrantFiled: December 14, 2022Date of Patent: January 16, 2024Assignee: GOOGLE LLCInventors: Xiang Wu, David Simcha, Daniel Holtmann-Rice, Sanjiv Kumar, Ananda Theertha Suresh, Ruiqi Guo, Xinnan Yu
-
Publication number: 20230376856Abstract: The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.Type: ApplicationFiled: August 4, 2023Publication date: November 23, 2023Inventors: Hugh Brendan McMahan, Dave Morris Bacon, Jakub Konecny, Xinnan Yu
-
Patent number: 11785073Abstract: The present disclosure provides systems and methods for communication efficient distributed mean estimation. In particular, aspects of the present disclosure can be implemented by a system in which a number of vectors reside on a number of different clients, and a centralized server device seeks to estimate the mean of such vectors. According to one aspect of the present disclosure, a client computing device can rotate a vector by a random rotation matrix and then subsequently perform probabilistic quantization on the rotated vector. According to another aspect of the present disclosure, subsequent to quantization but prior to transmission, the client computing can encode the quantized vector according to a variable length coding scheme (e.g., by computing variable length codes).Type: GrantFiled: October 15, 2021Date of Patent: October 10, 2023Assignee: GOOGLE LLCInventors: Ananda Theertha Suresh, Sanjiv Kumar, Hugh Brendan McMahan, Xinnan Yu
-
Patent number: 11763197Abstract: The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.Type: GrantFiled: April 16, 2020Date of Patent: September 19, 2023Assignee: GOOGLE LLCInventors: Hugh Brendan McMahan, Dave Morris Bacon, Jakub Konecny, Xinnan Yu
-
Publication number: 20230130021Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing privacy-preserving machine learning models (e.g., neural networks) in secure multi-part computing environments. Methods can include computing an output of a particular layer of a neural network deployed in a two computing system environment using a cosine activation function.Type: ApplicationFiled: October 26, 2022Publication date: April 27, 2023Inventors: Wittawat Jitkrittum, Michal Mateusz Lukasik, Ananda Theertha Suresh, Xinnan Yu, Gang Wang
-
Publication number: 20230123941Abstract: The present disclosure provides systems and methods that include or otherwise leverage use of a multiscale quantization model that is configured to provide a quantized dataset. In particular, the multiscale quantization model can receive and perform vector quantization of a first dataset. The multiscale quantization model can generate a residual dataset based at least in part on a result of the vector quantization. The multiscale quantization model can apply a rotation matrix to the residual dataset to generate a rotated residual dataset that includes a plurality of rotated residuals. The multiscale quantization model can perform reparameterization of each rotated residual in the rotated residual dataset into a direction component and a scale component. The multiscale quantization model can perform product quantization of the direction components of the plurality of rotated residuals, and perform scalar quantization of the scale components of the plurality of rotated residuals.Type: ApplicationFiled: December 14, 2022Publication date: April 20, 2023Inventors: Xiang Wu, David Simcha, Daniel Holtmann-Rice, Sanjiv Kumar, Ananda Theertha Suresh, Ruiqi Guo, Xinnan Yu
-
Patent number: 11531695Abstract: The present disclosure provides systems and methods that include or otherwise leverage use of a multiscale quantization model that is configured to provide a quantized dataset. In particular, the multiscale quantization model can receive and perform vector quantization of a first dataset. The multiscale quantization model can generate a residual dataset based at least in part on a result of the vector quantization. The multiscale quantization model can apply a rotation matrix to the residual dataset to generate a rotated residual dataset that includes a plurality of rotated residuals. The multiscale quantization model can perform reparameterization of each rotated residual in the rotated residual dataset into a direction component and a scale component. The multiscale quantization model can perform product quantization of the direction components of the plurality of rotated residuals, and perform scalar quantization of the scale components of the plurality of rotated residuals.Type: GrantFiled: May 14, 2018Date of Patent: December 20, 2022Assignee: GOOGLE LLCInventors: Xiang Wu, David Simcha, Daniel Holtmann-Rice, Sanjiv Kumar, Ananda Theertha Suresh, Ruiqi Guo, Xinnan Yu
-
Publication number: 20220046082Abstract: The present disclosure provides systems and methods for communication efficient distributed mean estimation. In particular, aspects of the present disclosure can be implemented by a system in which a number of vectors reside on a number of different clients, and a centralized server device seeks to estimate the mean of such vectors. According to one aspect of the present disclosure, a client computing device can rotate a vector by a random rotation matrix and then subsequently perform probabilistic quantization on the rotated vector. According to another aspect of the present disclosure, subsequent to quantization but prior to transmission, the client computing can encode the quantized vector according to a variable length coding scheme (e.g., by computing variable length codes).Type: ApplicationFiled: October 15, 2021Publication date: February 10, 2022Inventors: Ananda Theertha Suresh, Sanjiv Kumar, Hugh Brendan McMahan, Xinnan Yu
-
Patent number: 11196800Abstract: The present disclosure provides systems and methods for communication efficient distributed mean estimation. In particular, aspects of the present disclosure can be implemented by a system in which a number of vectors reside on a number of different clients, and a centralized server device seeks to estimate the mean of such vectors. According to one aspect of the present disclosure, a client computing device can rotate a vector by a random rotation matrix and then subsequently perform probabilistic quantization on the rotated vector. According to another aspect of the present disclosure, subsequent to quantization but prior to transmission, the client computing can encode the quantized vector according to a variable length coding scheme (e.g., by computing variable length codes).Type: GrantFiled: September 19, 2017Date of Patent: December 7, 2021Assignee: Google LLCInventors: Ananda Theertha Suresh, Sanjiv Kumar, Hugh Brendan McMahan, Xinnan Yu
-
Publication number: 20210326757Abstract: Generally, the present disclosure is directed to systems and methods that perform spreadout regularization to enable learning of a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a limited number of classes (e.g., a single class). Examples of such settings include decentralized training of face recognition models or speaker identification models, where in addition to the user specific facial images and voice samples, the class embeddings for the users also constitute sensitive information that cannot be shared with other users.Type: ApplicationFiled: April 12, 2021Publication date: October 21, 2021Inventors: Ankit Singh Rawat, Xinnan Yu, Aditya Krishna Menon, Sanjiv Kumar
-
Publication number: 20210049298Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for privacy preserving training of a machine learning model.Type: ApplicationFiled: August 14, 2020Publication date: February 18, 2021Inventors: Ananda Theertha Suresh, Xinnan Yu, Sanjiv Kumar, Sashank Jakkam Reddi, Venkatadheeraj Pichapati
-
Publication number: 20210019654Abstract: Systems and methods for low bias negative sampling of classes according to the sampled softmax method are described herein. The systems and methods can include training a machine-learned model for classifying inputs into one or more classes of a plurality of classes, each of the plurality of classes having an associated class embedding in a plurality of class embeddings. The systems and methods can include selecting, by the one or more computing devices, one or more negative classes from the plurality of classes based at least in part on a probability distribution approximating a softmax distribution, wherein the probability distribution is determined based at least in part on a Random Fourier Features map.Type: ApplicationFiled: July 17, 2020Publication date: January 21, 2021Inventors: Xinnan Yu, Ankit Singh Rawat, Jiecao Chen, Ananda Theertha Suresh, Sanjiv Kumar
-
Publication number: 20200242514Abstract: The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.Type: ApplicationFiled: April 16, 2020Publication date: July 30, 2020Inventors: Hugh Brendan McMahan, Dave Morris Bacon, Jakub Konecny, Xinnan Yu
-
Patent number: 10719509Abstract: Implementations provide an efficient system for calculating inner products between high-dimensionality vectors. An example method includes clustering database items represented as vectors, selecting a cluster center for each cluster, and storing the cluster center as an entry in a first layer codebook. The method also includes, for each database item, calculating a residual based on the cluster center for the cluster the database item is assigned to and projecting the residual into subspaces. The method also includes determining, for each of the subspaces, an entry in a second layer codebook for the subspace, and storing the entry in the first layer codebook and the respective entry in the second layer codebook for each of the subspaces as a quantized vector for the database item. The entry can be used to categorize an item represented by a query vector or to provide database items responsive to a query vector.Type: GrantFiled: October 11, 2016Date of Patent: July 21, 2020Assignee: GOOGLE LLCInventors: Sanjiv Kumar, David Morris Simcha, Ananda Theertha Suresh, Ruiqi Guo, Xinnan Yu, Daniel Holtmann-Rice
-
Publication number: 20200183964Abstract: The present disclosure provides systems and methods that include or otherwise leverage use of a multiscale quantization model that is configured to provide a quantized dataset. In particular, the multiscale quantization model can receive and perform vector quantization of a first dataset. The multiscale quantization model can generate a residual dataset based at least in part on a result of the vector quantization. The multiscale quantization model can apply a rotation matrix to the residual dataset to generate a rotated residual dataset that includes a plurality of rotated residuals. The multiscale quantization model can perform reparameterization of each rotated residual in the rotated residual dataset into a direction component and a scale component. The multiscale quantization model can perform product quantization of the direction components of the plurality of rotated residuals, and perform scalar quantization of the scale components of the plurality of rotated residuals.Type: ApplicationFiled: May 14, 2018Publication date: June 11, 2020Inventors: Xiang Wu, David Simcha, Daniel Holtmann-Rice, Sanjiv Kumar, Ananda Theertha Suresh, Ruiqi Guo, Xinnan Yu
-
Patent number: 10657461Abstract: The present disclosure provides efficient communication techniques for transmission of model updates within a machine learning framework, such as, for example, a federated learning framework in which a high-quality centralized model is trained on training data distributed overt a large number of clients each with unreliable network connections and low computational power. In an example federated learning setting, in each of a plurality of rounds, each client independently updates the model based on its local data and communicates the updated model back to the server, where all the client-side updates are used to update a global model. The present disclosure provides systems and methods that reduce communication costs. In particular, the present disclosure provides at least: structured update approaches in which the model update is restricted to be small and sketched update approaches in which the model update is compressed before sending to the server.Type: GrantFiled: September 7, 2017Date of Patent: May 19, 2020Assignee: Google LLCInventors: Hugh Brendan McMahan, Dave Morris Bacon, Jakub Konecny, Xinnan Yu
-
Publication number: 20190385063Abstract: A sparse dataset is encoded using a data-driven learned sensing matrix. For example, an example method includes receiving a dataset of sparse vectors with dimension d from a requesting process, initializing an encoding matrix of dimension k×d, selecting a subset of sparse vectors from the dataset, and updating the encoding matrix via machine learning. Updating the encoding matrix includes using a linear encoder to generate an encoded vector of dimension k for each vector in the subset, the linear encoder using the encoding matrix, using a non-linear decoder to decode each of the encoded vectors, the non-linear decoder using a transpose of the encoding matrix in a projected subgradient, and adjusting the encoding matrix using back propagation. The method also includes returning an embedding of each sparse vector in the dataset of sparse vectors, the embedding being generated with the updated encoding matrix.Type: ApplicationFiled: June 14, 2019Publication date: December 19, 2019Inventors: Xinnan Yu, Shanshan Wu, Daniel Holtmann-Rice, Dmitry Storcheus, Sanjiv Kumar, Afshin Rostamizadeh
-
Patent number: 10510021Abstract: Systems and methods for evaluating a loss function or a gradient of the loss function. In one example embodiment, a computer-implemented method includes partitioning a weight matrix into a plurality of blocks. The method includes identifying a first set of labels for each of the plurality of blocks with a score greater than a first threshold value. The method includes constructing a sparse approximation of a scoring vector for each of the plurality of blocks based on the first set of labels. The method includes determining a correction value for each sparse approximation of the scoring vector. The method includes determining an approximation of a loss or a gradient of a loss associated with the scoring function based on each sparse approximation of the scoring vector and the correction value associated with the sparse approximation of the scoring vector.Type: GrantFiled: June 7, 2019Date of Patent: December 17, 2019Assignee: Google LLCInventors: Satyen Chandrakant Kale, Daniel Holtmann-Rice, Sanjiv Kumar, Enxu Yan, Xinnan Yu
-
Publication number: 20190378037Abstract: Systems and methods for evaluating a loss function or a gradient of the loss function. In one example embodiment, a computer-implemented method includes partitioning a weight matrix into a plurality of blocks. The method includes identifying a first set of labels for each of the plurality of blocks with a score greater than a first threshold value. The method includes constructing a sparse approximation of a scoring vector for each of the plurality of blocks based on the first set of labels. The method includes determining a correction value for each sparse approximation of the scoring vector. The method includes determining an approximation of a loss or a gradient of a loss associated with the scoring function based on each sparse approximation of the scoring vector and the correction value associated with the sparse approximation of the scoring vector.Type: ApplicationFiled: June 7, 2019Publication date: December 12, 2019Inventors: Satyen Chandrakant Kale, Daniel Holtmann-Rice, Sanjiv Kumar, Enxu Yan, Xinnan Yu