Abstract: A training framework includes a transparent and unbiased dataset generation pipeline to generate unbiased multi-modal training data for training a multi-modal LLM. Specifically, one or more images may be annotated using image recognition models, e.g., for object detection, attributes, relations, segmentation, etc. Then, the generated annotations are used to generate scene graph. A scene graph is a data structure that represents objects with attributes in an image as nodes and relationships between objects in an image as edges. Several computer programs are then used to systematically generate question-answer pairs. Because the computer programs include a set of known rules, how the data is generated is known explicitly and thus transparent. The question-answer pairs may be of several types, e.g., a type associated with each of the types of image recognition models. The question-answer pairs may be included in a training dataset and used to train a multimodal LLM.

Abstract: The present application discloses a server system, a job execution method and apparatus, a device, and a medium. The server system includes a server and an extended computing domain; the server includes a processor control domain and a local computing domain, the local computing domain includes a plurality of local computing units, the processor control domain is connected to the local computing units and the local computing units are configured to execute local computing tasks; the extended computing domain includes an extended controller and a plurality of extended computing units connected to the extended controller, the server is connected to the extended controller through an extended cable compliant with a peripheral component interconnect express protocol and/or an external communication interface, the extended controller is configured to communicate with the server to acquire extended computing tasks, and the extended computing units are configured to execute the extended computing tasks.

Abstract: Embodiments described herein provide a mechanism for replacing existing text encoders in text-to-image generation models with more powerful pre-trained language models. Specifically, a translation network is trained to map features from the pre-trained language model output into the space of the target text encoder. The training preserves the rich structure of the pre-trained language model while allowing it to operate within the text-to-image generation model. The resulting modularized text-to-image model receives prompt and generates an image representing the features contained in the prompt.

Abstract: Embodiments described herein provide a vision-language neural network framework that outputs a text response to a user text query relating to the media content of the video input. Specifically, the vision-language neural network may comprise (1) a vision encoder (ViT) transforming each frame input from the video input into a set of tokens, (2) a frame-level tokenizer to reduce the number of tokens, (3) a temporal encoder to build video-level token representations, and (4) an autoregressive LLM generating a text output based on such video tokens and text prompt tokens.

Abstract: A system may receive video information. The system may extract light weight features from the video information. The system may select a combination of light-weight features and heavy weight feature types, where the light-weight features are extracted from the video information. The system may forecast, based on a combination of the light-weight features and the heavy weight feature types, accuracy and latency metrics for performing the object detection and tracking using a plurality of candidate branch configurations, respectively. The system may select a branch configuration from the plurality of candidate branch configurations in response to satisfaction of an optimization criterion. The system may perform object detection and tracking based on the selected branch configuration. Performing object detection and tracking may include extracting heavy weight features according to the branch configuration.

Abstract: Embodiments described herein provide a method of performing a vision-language task by a neural network multimodal model in response to multiple input images, the method comprising: receiving, via a data interface, a text input and an image input; generating text tokens based on the text input; generating a plurality of image patches, wherein each image patch of the plurality of image patches includes a portion of the image input at substantially a same resolution as the image input; generating a downsized image based on a downsizing of the image input; generating vision tokens based on the plurality of image patches and the downsized image; generating, via a neural network based language model, an output based on the text tokens and the vision tokens; and updating parameters of the neural network based language model based on a loss objective based on the output.

Abstract: Provided is a system that includes at least one processor programmed or configured to receive an XML data file, wherein the XML data file includes data associated with one or more input parameters of a machine learning model, generate a code generation template based on the data associated with one or more input parameters of the machine learning model included in the XML file, where the code generation template includes one or more keys associated with one or more parameters of a transaction aggregate for an account of a user, and generate a file of executable code based on the code generation template, wherein the file of executable code includes instructions that, when executed by at least one processor, causes at least one processor to retrieve transaction aggregate data associated with the transaction aggregate for the account of the user. A method and computer program product are also provided.

Abstract: Embodiments described herein provide a generation model comprising a video-specific variational auto-encoder (VAE) for effective compression of video pixel information with reduced spatial and temporal dimensions and a video diffusion transformer (vDiT) to generate latent representations of frames. Specifically, the VAE may, instead of encoding each frame independently, incorporate both temporal and spatial compression. This significantly decreases the token length, improves the computational cost of training and inference, and facilitates the generation of long videos. The encoded training video, in the form of latent representations from a VAE encoder may then be passed to the vDiT to reconstruct the latent representations during training. The trained vDiT may then generate latent representations of a video in response to a text input, and the latent representations may be converted to a video output by a VAE decoder.

Abstract: Embodiments described a field extraction system that does not require field-level annotations for training. Specifically, the training process is bootstrapped by mining pseudo-labels from unlabeled forms using simple rules. Then, a transformer-based structure is used to model interactions between text tokens in the input form and predict a field tag for each token accordingly. The pseudo-labels are used to supervise the transformer training. As the pseudo-labels are noisy, a refinement module that contains a sequence of branches is used to refine the pseudo-labels. Each of the refinement branches conducts field tagging and generates refined labels. At each stage, a branch is optimized by the labels ensembled from all previous branches to reduce label noise.

Abstract: Embodiments described herein provide a method of image generation. The method includes a fixed diffusion model, and a trainable diffusion model. The fixed diffusion model may be pretrained on a large training corpus. The trainable diffusion model may be used to control the image generation of the fixed diffusion model by modifying internal representations of the fixed diffusion model. A task instruction may be provided in addition to a text prompt, and the task instruction may guide the trainable diffusion model together with the visual conditions. The visual conditions may be adapted according to the task instruction. During training, a fixed number of task instructions may be used. At inference, unseen task instructions may be used by combining convolutional kernels of the visual condition adapter.

Abstract: Embodiments described herein provide systems and methods for multimodal layout generations for digital publications. The system may receive as inputs, a background image, one or more foreground texts, and one or more foreground images. Feature representations of the background image may be generated. The foreground inputs may be input to a layout generator which has cross attention to the background image feature representations in order to generate a layout comprising of bounding box parameters for each input item. A composite layout may be generated based on the inputs and generated bounding boxes. The resulting composite layout may then be displayed on a user interface.

Abstract: A method of determining a property of a substrate surface and a processing chamber configured to determine a property of a substrate are disclosed herein. The method includes flowing a liquid onto a rotating substrate, stopping the flow, and measuring a time for a trailing edge of a liquid layer to move across the rotating substrate to determine the property.

Abstract: Embodiments described herein provide a method of building an artificial intelligence (AI) agent to respond to a task request from a user. The method includes: receiving a set of single-modal data samples of a plurality of modalities; selecting a first single-modal data sample of a first modality and a second single-modal data sample of a second modality; generating a question associated with the first single-modal data sample and the second single-modal data sample; generating an answer with a reasoning to the question based on a second input prompt; training, a second neural network based language model, using a dataset comprising the question and the answer to generate a candidate answer in response to a training query; building the AI conversation bot through an application programming interface to the trained second neural network language model; and generating, using the AI conversation bot, a response to the task request.

Abstract: Embodiments described herein provide a feedback based instructional image editing framework that employs a diffusion process to follow user instruction for image editing. A diffusion model is fine-tuned using a reward model, which may be trained via human annotation. The training of the reward model may be done by having the image editing model output a number of images, which a human annotator ranks based on their alignment with the original image and a given instruction.

Abstract: A system may receive video information. The system may select a combination of light-weight features and heavy weight features. The light-weight features may be extracted from the video information and the heavy weight features not extracted. The system may forecast, based on the light-weight features and the heavy weight features, accuracy, and latency metrics for performing the object detection and tracking using a plurality of candidate branch configurations, respectively. The system may select a branch configuration from the plurality of candidate branch configurations in response to satisfaction of an optimization criterion. The system may perform object detection and tracking based on the selected branch configuration.

Abstract: The present disclosure provides a method, system, and apparatus for controlling X-rays. The method of the present disclosure includes: obtaining control parameters in response to an X-ray control request from a CT control unit, wherein the control parameters include at least one tank identifier, at least one voltage parameter, at least one current parameter, and at least one exposure timing; controlling, based on the tank identifier and the voltage parameter, at least one high frequency inverter of a high frequency inverter assembly to output a high frequency voltage to a corresponding tank; and controlling, based on the tank identifier and the current parameter, at least one filament power supply of a filament power supply assembly to output a filament current to the corresponding tank, thereby controlling the tank to perform an X-ray exposure task according to the exposure timing.

Abstract: The invention relates to methods of increasing plant yield, and in particular grain or seed number by introducing at least one mutation into at least one UPL2 gene. Also described are genetically altered plants characterised by the above phenotype.

Abstract: A method of training a neural network based three-dimensional (3D) encoder is provided. A first plurality of samples of a training dataset are generated using a first 3D model. An image generator with multi-view rendering is used to generate a plurality of two-dimensional (2D) images having different viewpoints of the first 3D model. A first language model is used to generate a plurality of texts corresponding to the plurality of 2D images respectively. A first text for a first image is generated by using one or more text descriptions generated by the first language model. A point cloud is generated by randomly sampling points in the 3D model. The first plurality of samples are generated using the plurality of 2D images, the corresponding plurality of texts, and the point cloud. The neural network based 3D encoder is trained using the training dataset including the first plurality of samples.

Abstract: Systems and methods for training a neural network based three-dimensional (3D) encoder for 3D classification are provided. A training dataset including a plurality of samples is received, wherein a first sample includes an image, a text, and a point cloud. An image encoder of a pretrained vision and language model is used to generate image representations for the image of the first sample. A text encoder of the pretrained vision and language model is used to generate text representations for the text of the first sample. The neural network based 3D encoder is used to generate 3D representations for the point cloud of the first sample. A loss objective is computed based on the image representations, text representations, and 3D representations. Parameters of the neural network based 3D encoder are updated based on the computed loss objective via backpropagation.

Abstract: A method of training a neural network based three-dimensional (3D) encoder is provided. A training dataset is generated using a plurality of 3D models of a 3D model dataset. To generate a first sample of the training dataset, an image generator with multi-view rendering is used to generate a plurality of image candidates of a first 3D model. A word is chosen from metadata associated with the first 3D model. A language model is used to generate one or more text descriptions using the selected word and a plurality of prompts. A point cloud is generated by randomly sampling points in the 3D model. The first sample is generated to include a first image randomly selected from the plurality of image candidates, one or more text descriptions, and the point cloud is generated. The 3D encoder is trained using the training dataset including the first sample.