Abstract: Systems and methods are provided for encoding and decoding images using differentiable JPEG compression, including converting images from RGB color space to YCbCr color space to obtain a luminance and chrominance channels, and applying chroma subsampling to the chrominance channels to reduce resolution. The YCbCr image is divided into pixel blocks and a DCT is performed on the pixel blocks to obtain DCT coefficients. DCT coefficients are quantized using a scaled quantization table to reduce precision, and quantized DCT coefficients are encoded using lossless entropy coding, forming a compressed JPEG file decoded by reversing the lossless entropy coding to obtain quantized DCT coefficients, which are dequantized using the scaled quantization table to restore the precision. The dequantized DCT coefficients are converted back to a spatial domain using an IDCT, the chrominance channels are upsampled to original resolution, and the YCbCr image is converted back to the RGB color space.

Abstract: Systems and methods are provided for dynamically optimizing microservice placement in a distributed edge and cloud computing environment, including receiving application specifications that include telemetry data collection methods, placement rules, and modes of operation, validating the received application specifications to ensure completeness and correctness, and composing an application graph where vertices represent microservices and edges represent connections between the microservices. Availability of resources specified in the application graph is checked, and the microservices are deployed according to initial placement rules. Telemetry data from the deployed microservices and underlying infrastructure is collected and evaluated against the placement rules, and the placement of microservices is dynamically adjusted responsive to a determination that current microservice placement is suboptimal based on the evaluating of the collected telemetry data.

Abstract: A method for performing resource orchestration for microservices-based 5G applications in a dynamic, heterogenous, multi-tiered compute and network environment is presented.

Abstract: Systems and methods are provided for dynamically adjusting computing resources allocated to tasks within a stream processing application, including initiating monitoring of application-specific characteristics for each task, the characteristics including processor (CPU) usage and processing time, assessing resource allocation needs for each task based on the monitored characteristics to determine discrepancies between current resource allocation and optimal performance requirements, and implementing exploratory resource adjustments by incrementally modifying CPU resources allocated to a subset of tasks and analyzing an impact of the exploratory resource adjustments on task performance metrics. Optimal resource allocations are determined for each task using a regression model that incorporates historical and real-time performance data, and the optimal resource allocations are applied to the tasks to minimize processing time and maximize resource use efficiency.

Abstract: A method for employing a semi-supervised learning approach to improve accuracy of a small model on an edge device is presented. The method includes collecting a plurality of frames from a plurality of video streams generated from a plurality of cameras, each camera associated with a respective small model, each small model deployed in the edge device, sampling the plurality of frames to define sampled frames, performing inference to the sampled frames by using a big model, the big model shared by all of the plurality of cameras and deployed in a cloud or cloud edge, using the big model to generate labels for each of the sampled frames to generate training data, and training each of the small models with the training data to generate updated small models on the edge device.

Abstract: Methods and systems for executing an application include extending a container orchestration system application programming interface (API) to handle objects that specify components of an application. An application representation is executed using the extended container orchestration system API, including the instantiation of one or more services that define a data stream path from a sensor to a device.

Abstract: Systems and methods are provided for deploying applications within a wireless network infrastructure, including initiating, by a centralized control module in a pre-configured hardware unit having a 5G wireless communication module, edge computing device, centralized control module, and data processing module with access to cloud resources, a setup procedure upon receiving a deployment command, the setup procedure including activating the 5G wireless communication module to establish a network connection. User equipment for communication with sensors and cameras is deployed using an edge device through the network connection. Application deployment is managed using a centralized control module including an edge cloud optimizer for allocating resources between an edge computing device and the cloud resources based on real-time analysis of network conditions and application requirements.

Abstract: Systems and methods are provided for optimizing video compression using end-to-end learning, including capturing, using an edge device, raw video frames from a video clip and determining maximum network bandwidth. Predicting, using a control network implemented on the edge device, optimal codec parameters, based on dynamic network conditions and content of the video clip, encoding, using a differentiable surrogate model of a video codec, the video clip using the predicted codec parameters and to propagate gradients from a server-side vision model to adjust the codec parameters. Decoding, using a server, the video clip and analyzing the video clip with a deep vision model located on the server, transmitting, using a feedback mechanism, analysis from the deep vision model back to the control network to facilitate end-to-end training of the system. Adjusting the encoding parameters based on the analysis from the deep vision model received from the feedback mechanism.

Abstract: Systems and methods are provided for optimizing video compression for remote vehicle control, including capturing, capturing video and sensor data from a vehicle using a plurality of sensors and high-resolution cameras, analyzing the captured video to identify critical regions within frames of the video using an attention-based module. Current network bandwidth is assessed and future bandwidth availability is predicted. Video compression parameters are predicted based on an analysis of the video and an assessment of the current network bandwidth using a control network, and the video is compressed based on the predicted parameters with an adaptive video compression module. The compressed video and sensor data is transmitted to a remote-control center, and received video and sensor data is decoded at the remote-control center. The vehicle is autonomously or remotely controlled from the remote-control center based on the decoded video and sensor data.

Abstract: A method for real-time cross-spectral object association and depth estimation is presented. The method includes synthesizing, by a cross-spectral generative adversarial network (CS-GAN), visual images from different data streams obtained from a plurality of different types of sensors, applying a feature-preserving loss function resulting in real-time pairing of corresponding cross-spectral objects, and applying dual bottleneck residual layers with skip connections to accelerate real-time inference and to accelerate convergence during model training.

Abstract: A pull-based communication method for microservices-based real-time streaming video analytics pipelines is provided. The method includes receiving a plurality of frames from a plurality of cameras, each camera including a camera sidecar, arranging a plurality of detectors in layers such that a first detector layer includes detectors with detector sidecars and detector business logic, and the second detector layer includes detectors with only sidecars, arranging a plurality of extractors in layers such that a first extractor layer includes extractors with extractor sidecars and extractor business logic, and the second extractor layer includes extractors with only sidecars, and enabling a mesh controller, during registration, to selectively assign inputs to one or more of the detector sidecars of the first detector layer and one or more of the extractor sidecars of the first extractor layer to pull data items for processing.

Abstract: A method for implementing a self-optimized video analytics pipeline is presented. The method includes decoding video files into a sequence of frames, extracting features of objects from one or more frames of the sequence of frames of the video files, employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline, employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices, applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames, and applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.

Abstract: Systems and methods for scaling in a container orchestration platform are described that include configuring an autoscaler in a control plane of the container orchestration platform to receive stream data from a data exchange system that is measuring stream processing of a pipeline of microservices for an application. The systems and methods further include controlling a number of deployment pods in at least one node of the container orchestration platform to meet requirements for the application provided by the pipeline of microservices.

Abstract: Methods and systems for camera configuration include configuring an image capture configuration parameter of a camera according to a multi-objective reinforcement learning aggregated reward function. Respective quality estimates for analytics are determined after configuring the image capture parameters. The aggregated reward function is updated based on the quality estimates.

Abstract: A computer implemented method is provided for resource management of stream analytics at each individual node that includes computing a mean of output processing rate of microservices in a pipeline; and evaluating a state of each microservice of the microservices in the pipeline. The computer implemented method also includes selecting a single microservice from the pipeline for updating resources for an action that changes the state in single the microservice that is selected; and performing resource allocation update for the selected microservice. The computer implemented method may also include updating the state of the selected microservice.

Abstract: Systems and methods are provided for dynamically tuning camera parameters in a video analytics system to optimize analytics accuracy. A camera captures a current scene, and optimal camera parameter settings are learned and identified for the current scene using a Reinforcement Learning (RL) engine. The learning includes defining a state within the RL engine as a tuple of two vectors: a first representing current camera parameter values and a second representing measured values of frames of the current scene. Quality of frames is estimated using a quality estimator, and camera parameters are adjusted based on the quality estimator and the RL engine for optimization. Effectiveness of tuning is determined using perceptual Image Quality Assessment (IQA) to quantify a quality measure. Camera parameters are adaptively tuned in real-time based on learned optimal camera parameter settings, state, quality measure, and set of actions, to optimize the analytics accuracy for video analytics tasks.

Abstract: Methods and systems for transmission over a heterogeneous network include determining a path through a first network, including collecting quality of service (QoS) parameters for network devices in the path. The QoS parameters of the network devices are configured to provide a predetermined QoS assurance across the path. A network flow is transmitted from a network slice of a second network through the first network, along the path.

Abstract: Systems and methods are provided for increasing accuracy of video analytics tasks in real-time by acquiring a video using video cameras, and identifying fluctuations in the accuracy of video analytics applications across consecutive frames of the video. The identified fluctuations are quantified based on an average relative difference of true-positive detection counts across consecutive frames. Fluctuations in accuracy are reduced by applying transfer learning to a deep learning model initially trained using images, and retraining the deep learning model using video frames. A quality of object detections is determined based on an amount of track-ids assigned by a tracker across different video frames. Optimization of the reduction of fluctuations includes iteratively repeating the identifying, the quantifying, the reducing, and the determining the quality of object detections until a threshold is reached. Model predictions for each frame in the video are generated using the retrained deep learning model.

Abstract: A method for implementing application self-optimization in serverless edge computing environments is presented. The method includes requesting deployment of an application pipeline on data received from a plurality of sensors, the application pipeline including a plurality of microservices, enabling communication between a plurality of pods and a plurality of analytics units (AUs), each pod of the plurality of pods including a sidecar, determining whether each of the plurality of AUs maintains any state to differentiate between stateful AUs and stateless AUs, scaling the stateful AUs and the stateless AUs, enabling communication directly between the sidecars of the plurality of pods, and reusing and resharing common AUs of the plurality of AUs across different applications.

Abstract: A pull-based communication method for microservices-based real-time streaming video analytics pipelines is provided. The method includes receiving a plurality of frames from a plurality of cameras, each camera including a camera sidecar, arranging a plurality of detectors in layers such that a first detector layer includes detectors with detector sidecars and detector business logic, and the second detector layer includes detectors with only sidecars, arranging a plurality of extractors in layers such that a first extractor layer includes extractors with extractor sidecars and extractor business logic, and the second extractor layer includes extractors with only sidecars, and enabling a mesh controller, during registration, to selectively assign inputs to one or more of the detector sidecars of the first detector layer and one or more of the extractor sidecars of the first extractor layer to pull data items for processing.