Patents by Inventor Xiaojin Shi
Xiaojin Shi has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240112303Abstract: In some implementations, a method includes: obtaining image data associated with a physical environment; obtaining first contextual information including at least one of first user information associated with a current state of a user of the computing system, first application information associated with a first application being executed by the computing system, and first environment information associated with a current state of the physical environment; selecting a first set of perspective correction operations based at least in part on the first contextual information; generating first corrected image data by performing the first set of perspective correction operations on the image data; and causing presentation of the first corrected image data.Type: ApplicationFiled: September 22, 2023Publication date: April 4, 2024Inventors: Vincent Chapdelaine-Couture, Emmanuel Piuze-Phaneuf, Julien Monat Rodier, Hermannus J. Damveld, Xiaojin Shi, Sebastian Gaweda
-
Publication number: 20240098232Abstract: In one implementation, a method of performing perspective correction is performed by a device having a three-dimensional device coordinate system and including a first image sensor, a first display, one or more processors, and non-transitory memory. The method includes capturing, using the first image sensor, a first image of a physical environment. The method includes transforming the first image from a first perspective of the first image sensor to a second perspective based on a difference between the first perspective and the second perspective, wherein the second perspective is a first distance away from a location corresponding to a first eye of a user less than a second distance between the first perspective and the location corresponding to the first eye of the user. The method includes displaying, on the first display, the transformed first image of the physical environment.Type: ApplicationFiled: September 18, 2023Publication date: March 21, 2024Inventors: Emmanuel Piuze-Phaneuf, Hermannus J. Damveld, Jean-Nicola F. Blanchet, Mohamed Selim Ben Himane, Vincent Chapdelaine-Couture, Xiaojin Shi
-
Patent number: 11699097Abstract: A method includes receiving input data at a trained machine learning model that includes a common part and task-specific parts, receiving an execution instruction that identifies one or more processing tasks to be performed, processing the input data using the common part of the trained machine learning model to generate intermediate data, and processing the intermediate data using one or more of the task-specific parts of the trained machine learning model based on the execution instruction to generate one or more outputs.Type: GrantFiled: May 19, 2020Date of Patent: July 11, 2023Assignee: APPLE INC.Inventors: Francesco Rossi, Vignesh Jagadeesh, Vinay Sharma, Marco Zuliani, Xiaojin Shi, Benjamin Poulain
-
Publication number: 20230177350Abstract: The subject technology provides receiving a neural network (NN) model to be executed on a target platform, the NN model including multiple layers that include operations and some of the operations being executable on multiple processors of the target platform. The subject technology further sorts the operations from the multiple layers in a particular order based at least in part on grouping the operations that are executable by a particular processor of the multiple processors. The subject technology determines, based at least in part on a cost of transferring the operations between the multiple processors, an assignment of one of the multiple processors for each of the sorted operations of each of the layers in a manner that minimizes a total cost of executing the operations. Further, for each layer of the NN model, the subject technology includes an annotation to indicate the processor assigned for each of the operations.Type: ApplicationFiled: September 6, 2022Publication date: June 8, 2023Inventors: Gaurav KAPOOR, Cecile M. FORET, Francesco ROSSI, Kit-Man WAN, Umesh S. VAISHAMPAYAN, Etienne BELANGER, Albert ANTONY, Alexey MARINICHEV, Marco ZULIANI, Xiaojin SHI
-
Patent number: 11561621Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.Type: GrantFiled: October 14, 2019Date of Patent: January 24, 2023Assignee: Apple Inc.Inventors: Feng Tang, Chong Chen, Haitao Guo, Xiaojin Shi, Thorsten Gernoth
-
Patent number: 11468338Abstract: The subject technology provides receiving a neural network (NN) model to be executed on a target platform, the NN model including multiple layers that include operations and some of the operations being executable on multiple processors of the target platform. The subject technology further sorts the operations from the multiple layers in a particular order based at least in part on grouping the operations that are executable by a particular processor of the multiple processors. The subject technology determines, based at least in part on a cost of transferring the operations between the multiple processors, an assignment of one of the multiple processors for each of the sorted operations of each of the layers in a manner that minimizes a total cost of executing the operations. Further, for each layer of the NN model, the subject technology includes an annotation to indicate the processor assigned for each of the operations.Type: GrantFiled: January 30, 2019Date of Patent: October 11, 2022Assignee: Apple Inc.Inventors: Francesco Rossi, Cecile M. Foret, Gaurav Kapoor, Kit-Man Wan, Umesh S. Vaishampayan, Etienne Belanger, Albert Antony, Alexey Marinichev, Marco Zuliani, Xiaojin Shi
-
Patent number: 11367163Abstract: Artistic styles extracted from source images may be applied to target images to generate stylized images and/or video sequences. The extracted artistic styles may be stored as a plurality of layers in one or more neural networks, which neural networks may be further optimized, e.g., via the fusion of various elements of the networks' architectures. The artistic style may be applied to the target images and/or video sequences using various optimization methods, such as the use of a first version of the neural network by a first processing device at a first resolution to generate one or more sets of parameters (e.g., scaling and/or biasing parameters), which parameters may then be mapped for use by a second version of the neural network by a second processing device at a second resolution. Analogous multi-processing device and/or multi-network solutions may also be applied to other complex image processing tasks for increased efficiency.Type: GrantFiled: February 19, 2020Date of Patent: June 21, 2022Assignee: Apple Inc.Inventors: Francesco Rossi, Marco Zuliani, Bartlomiej W. Rymkowski, Albert Antony, Brian P. Keene, Xiaojin Shi
-
Publication number: 20210397596Abstract: The subject technology provides a framework for evaluating activation functions of a neural network using lookup tables. In order to provide lookup table based activation functions with a desired precision within hardware constraints for the lookup tables, multiple lookup tables for each activation function can be provided. Each of the multiple lookup tables may correspond to a respective subrange of input values, within a full range of input values for the activation function.Type: ApplicationFiled: September 22, 2020Publication date: December 23, 2021Inventors: Albert ANTONY, Francesco ROSSI, Guillaume TARTAVEL, Xiaojin SHI, Marco ZULIANI
-
Patent number: 11175898Abstract: The subject technology receives a neural network model in a model format, the model format including information for a set of layers of the neural network model, each layer of the set of layers including a set of respective operations. The subject technology generates neural network (NN) code from the neural network model, the NN code being in a programming language distinct from the model format, and the NN code comprising a respective memory allocation for each respective layer of the set of layers of the neural network model, where the generating comprises determining the respective memory allocation for each respective layer based at least in part on a resource constraint of a target device. The subject technology compiles the NN code into a binary format. The subject technology generates a package for deploying the compiled NN code on the target device.Type: GrantFiled: September 25, 2019Date of Patent: November 16, 2021Assignee: Apple Inc.Inventors: Timothy S. Paek, Francesco Rossi, Jamil Dhanani, Keith P. Avery, Minwoo Jeong, Xiaojin Shi, Harveen Kaur, Brandt M. Westing
-
Patent number: 10909657Abstract: Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.Type: GrantFiled: July 11, 2018Date of Patent: February 2, 2021Assignee: APPLE INC.Inventors: Francesco Rossi, Xiaohuan C. Wang, Brian E. Walsh, Bartlomiej W. Rymkowski, Xiaojin Shi, Marco Zuliani, Alexey Marinichev, Benjamin Poulain, Omid Khalili
-
Publication number: 20200380639Abstract: Artistic styles extracted from source images may be applied to target images to generate stylized images and/or video sequences. The extracted artistic styles may be stored as a plurality of layers in one or more neural networks, which neural networks may be further optimized, e.g., via the fusion of various elements of the networks' architectures. The artistic style may be applied to the target images and/or video sequences using various optimization methods, such as the use of a first version of the neural network by a first processing device at a first resolution to generate one or more sets of parameters (e.g., scaling and/or biasing parameters), which parameters may then be mapped for use by a second version of the neural network by a second processing device at a second resolution. Analogous multi-processing device and/or multi-network solutions may also be applied to other complex image processing tasks for increased efficiency.Type: ApplicationFiled: February 19, 2020Publication date: December 3, 2020Inventors: Francesco Rossi, Marco Zuliani, Bartlomiej W. Rymkowski, Albert Antony, Brian P. Keene, Xiaojin Shi
-
Publication number: 20200379740Abstract: The subject technology receives a neural network model in a model format, the model format including information for a set of layers of the neural network model, each layer of the set of layers including a set of respective operations. The subject technology generates neural network (NN) code from the neural network model, the NN code being in a programming language distinct from the model format, and the NN code comprising a respective memory allocation for each respective layer of the set of layers of the neural network model, where the generating comprises determining the respective memory allocation for each respective layer based at least in part on a resource constraint of a target device. The subject technology compiles the NN code into a binary format. The subject technology generates a package for deploying the compiled NN code on the target device.Type: ApplicationFiled: September 25, 2019Publication date: December 3, 2020Inventors: Timothy S. PAEK, Francesco ROSSI, Jamil DHANANI, Keith P. AVERY, Minwoo JEONG, Xiaojin SHI, Harveen KAUR, Brandt M. WESTING
-
Publication number: 20200372408Abstract: A method includes receiving input data at a trained machine learning model that includes a common part and task-specific parts, receiving an execution instruction that identifies one or more processing tasks to be performed, processing the input data using the common part of the trained machine learning model to generate intermediate data, and processing the intermediate data using one or more of the task-specific parts of the trained machine learning model based on the execution instruction to generate one or more outputs.Type: ApplicationFiled: May 19, 2020Publication date: November 26, 2020Inventors: Francesco Rossi, Vignesh Jagadeesh, Vinay Sharma, Marco Zuliani, Xiaojin Shi, Benjamin Poulain
-
Patent number: 10664963Abstract: Artistic styles extracted from one or more source images may be applied to one or more target images, e.g., in the form of stylized images and/or stylized video sequences. The extracted artistic style may be stored as a plurality of layers in a neural network, which neural network may be further optimized, e.g., via the fusion of various elements of the network's architectures. An optimized network architecture may be determined for each processing environment in which the network will be applied. The artistic style may be applied to the obtained images and/or video sequence of images using various optimization methods, such as the use of scalars to control the resolution of the unstylized and stylized images, temporal consistency constraints, as well as the use of dynamically adjustable or selectable versions of Deep Neural Networks (DNN) that are responsive to system performance parameters, such as available processing resources and thermal capacity.Type: GrantFiled: July 11, 2018Date of Patent: May 26, 2020Assignee: Apple Inc.Inventors: Francesco Rossi, Xiaohuan C. Wang, Bartlomiej W. Rymkowski, Xiaojin Shi, Marco Zuliani, Alexey Marinichev
-
Publication number: 20200082274Abstract: The subject technology provides receiving a neural network (NN) model to be executed on a target platform, the NN model including multiple layers that include operations and some of the operations being executable on multiple processors of the target platform. The subject technology further sorts the operations from the multiple layers in a particular order based at least in part on grouping the operations that are executable by a particular processor of the multiple processors. The subject technology determines, based at least in part on a cost of transferring the operations between the multiple processors, an assignment of one of the multiple processors for each of the sorted operations of each of the layers in a manner that minimizes a total cost of executing the operations. Further, for each layer of the NN model, the subject technology includes an annotation to indicate the processor assigned for each of the operations.Type: ApplicationFiled: January 30, 2019Publication date: March 12, 2020Inventors: Francesco ROSSI, Cecile M. FORET, Gaurav KAPOOR, Kit-Man WAN, Umesh S. VAISHAMPAYAN, Etienne BELANGER, Albert ANTONY, Alexey MARINICHEV, Marco ZULIANI, Xiaojin SHI
-
Publication number: 20200042096Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.Type: ApplicationFiled: October 14, 2019Publication date: February 6, 2020Inventors: Feng Tang, Chong Chen, Haitao Guo, Xiaojin Shi, Thorsten Gernoth
-
Patent number: 10477249Abstract: A video decoder system includes a video decoding engine, noise database, artifact estimator and post-processing unit. The video coder may generate recovered video from a data stream of coded video data, which may have visually-perceptible artifacts introduced as a byproduct of compression. The noise database may store a plurality of previously developed noise patches. The artifact estimator may estimate the location of coding artifacts present in the recovered video and select noise patches from the database to mask the artifacts and the post-processing unit may integrate the selected noise patches into the recovered video. In this manner, the video decoder may generate post-processed noise which may mask artifacts that otherwise would be generated by a video coding process.Type: GrantFiled: June 5, 2009Date of Patent: November 12, 2019Assignee: APPLE INC.Inventors: Yuxin Liu, Hsi-Jung Wu, Xiaojin Shi, Chris Yoochang Chung
-
Patent number: 10444854Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.Type: GrantFiled: August 6, 2018Date of Patent: October 15, 2019Assignee: Apple Inc.Inventors: Feng Tang, Chong Chen, Haitao Guo, Xiaojin Shi, Thorsten Gernoth
-
Publication number: 20180348885Abstract: Intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene may be monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, e.g., expressed through hand gesture movements. In some embodiments, such gesture movements may be interpreted based on real-time depth information obtained from, e.g., optical or non-optical type depth sensors.Type: ApplicationFiled: August 6, 2018Publication date: December 6, 2018Inventors: Feng Tang, Chong Chen, Haitao Guo, Xiaojin Shi, Thorsten Gernoth
-
Patent number: 10048765Abstract: Varying embodiments of intelligent systems are disclosed that respond to user intent and desires based upon activity that may or may not be expressly directed at the intelligent system. In some embodiments, the intelligent system acquires a depth image of a scene surrounding the system. A scene geometry may be extracted from the depth image and elements of the scene, such as walls, furniture, and humans may be evaluated and monitored. In certain embodiments, user activity in the scene is monitored and analyzed to infer user desires or intent with respect to the system. The interpretation of the user's intent or desire as well as the system's response may be affected by the scene geometry surrounding the user and/or the system. In some embodiments, techniques and systems are disclosed for interpreting express user communication, for example, expressed through fine hand gesture movements.Type: GrantFiled: September 25, 2015Date of Patent: August 14, 2018Assignee: Apple Inc.Inventors: Feng Tang, Chong Chen, Haitao Guo, Xiaojin Shi, Thorsten Gernoth