Abstract: Hierarchical structured sparse parameter pruning and processing improves runtime performance and energy efficiency of neural networks. In contrast with conventional (non-structured) pruning which allows for any distribution of the non-zero values within a matrix that achieves the desired sparsity degree (e.g., 50%) and is consequently difficult to accelerate, structured hierarchical sparsity requires each multi-element unit at the coarsest granularity of the hierarchy to be pruned to the desired sparsity degree. The global desired sparsity degree is a function of the per-level sparsity degrees. Distribution of non-zero values within each multi-element unit is constrained according to the per-level sparsity degree at the particular level of the hierarchy. Each level of the hierarchy may be associated with a hardware (e.g., logic or circuit) structure that can be enabled or disabled according to the per-level sparsity.

Abstract: A method includes, with aid of one or more processors individually or collectively, analyzing stereoscopic video data of an environment to determine environmental information, generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information, and controlling an unmanned aerial vehicle (UAV) to avoid an obstacle on a motion path of the UAV according to the augmented stereoscopic video data of the environment.

Abstract: A method for sensing an environment in which an unmanned aerial vehicle (UAV) is configured to operate includes, with aid of one or more processors onboard the UAV individually or collectively, obtaining video data of the environment that is collected using a binocular video camera mounted in a forward-looking direction of the UAV, encoding the video data to generate stereoscopic video data, and transmitting the stereoscopic video data to a terminal remote to the movable object.

Abstract: A method for generating a first person view (FPV) of an environment includes, with aid of one or more processors individually or collectively, analyzing stereoscopic video data of the environment to determine environmental information and generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information.

Abstract: A method for processing video data of an environment includes, with aid of one or more processors individually or collectively, obtaining in or near real-time a reference position of an imaging device located on a movable object based on one or more previously traversed positions of the imaging device, and modifying an image frame in the video data to obtain a modified image frame based on the reference position of the imaging device and an actual position of the imaging device at which the image frame is taken. The one or more previously traversed positions are obtained using at least one sensor on the movable object. The video data is acquired by the imaging device.

Abstract: A computer-implemented method for controlling bit rate includes determining an expected adjustment for a first coding parameter that encodes a first slice based on a difference between a cumulative number of bits used to encode one or more slices of a frame up to and including the first slice and a maximum number of bits allowed to encode the one or more slices, updating a coding parameter threshold based on an indicator that indicates a relationship between the coding parameter threshold and one or more coding parameters used to encode the one or more slices, and determining an actual adjustment for the first coding parameter based on the expected adjustment and the updated coding parameter threshold.

Abstract: Methods and systems of determining a quantization step for encoding video based on motion data are provided. Video captured by an image capture device is received. The video comprises a video frame component. Additionally, motion data associated with the video frame component is received. Further, a quantization step for encoding the video frame component is determined based on the motion data.

Abstract: A method for constructing an optical flow field includes classifying a plurality of scenarios according to motions of a mobile platform carrying an imaging device and statuses of the imaging device. The plurality of scenarios include at least one of elementary scenarios or combined scenarios. The method further includes constructing a plurality of optical flow fields each corresponding to one of the plurality of scenarios, acquiring a motion of the mobile platform and a status of the imaging device relative to the mobile platform, and selecting a corresponding optical flow field from the constructed optical flow fields corresponding to the plurality of scenarios based upon the motion of the mobile platform and the status of the imaging device for the imaging device to capture a frame at a shooting direction.

Abstract: A computer-implemented method for controlling bit rate includes determining a difference between a cumulative number of bits used to encode one or more slices of a frame up to and including a first slice encoded using a first coding parameter and a maximum number of bits allowed to encode the one or more slices of the frame, updating a coding parameter threshold based at least in part on a counter that indicates a number of times when one or more coding parameters used to encode the one or more slices reach or exceed the coding parameter threshold, and determining a second coding parameter used to encode a second slice of the frame based at least in part on the difference and the updated coding parameter threshold.

Abstract: Methods and systems of determining a quantization step for encoding video based on motion data are provided. Video captured by an image capture device is received. The video comprises a video frame component. Additionally, motion data associated with the video frame component is received. Further, a quantization step for encoding the video frame component is determined based on the motion data.

Abstract: A video encoding method includes receiving a video captured by an image capture device on-board a movable object, where the video includes a video frame component; receiving sensor data from a plurality of sensors on-board the movable object; generating, according to the sensor data, an optical flow field associated with the video frame component; and evaluating motion of the video for video encoding based on the optical flow field.

Abstract: Methods and systems for evaluating a search area for encoding video are provided. The method comprises receiving video captured by an image capture device, the video comprising video frame components. Additionally, the method comprises receiving optical flow field data associated with the video frame component, wherein at least a portion of the optical flow field data is captured by sensors. The method also comprises determining a search area based on the optical flow field data.

Abstract: A method for constructing an optical flow field includes classifying a plurality of scenarios according to motions of a mobile platform carrying an imaging device and statuses of the imaging device. The plurality of scenarios include at least one of elementary scenarios or combined scenarios. The method further includes constructing a plurality of optical flow fields each corresponding to one of the plurality of scenarios, acquiring a motion of the mobile platform and a status of the imaging device relative to the mobile platform, and selecting a corresponding optical flow field from the constructed optical flow fields corresponding to the plurality of scenarios based upon the motion of the mobile platform and the status of the imaging device for the imaging device to capture a frame at a shooting direction.

Abstract: A method for generating a first person view (FPV) of an environment includes, with aid of one or more processors individually or collectively, analyzing stereoscopic video data of the environment to determine environmental information and generating augmented stereoscopic video data of the environment by fusing the stereoscopic video data and the environmental information.

Abstract: A system constructs an optical flow field that corresponds with a selected video frame. The optical flow field is constructed based on a first motion of a mobile platform having an imaging device and a status of the imaging device. The first motion and the status are determined with measurements of sensors installed on the mobile platform and/or the imaging device installed on the mobile platform. The first motion includes at least one of a first rotation, a horizontal movement, or a vertical movement of the mobile platform. The status includes a rotation of the imaging device and/or an orientation of the imaging device relative to the mobile platform.

Abstract: A computer-implemented method for controlling bit rate includes determining a difference between a cumulative number of bits used to encode one or more slices of a frame up to and including a first slice encoded using a first coding parameter and a maximum number of bits allowed to encode the one or more slices of the frame, updating a coding parameter threshold based at least in part on a counter that indicates a number of times when one or more coding parameters used to encode the one or more slices reach or exceed the coding parameter threshold, and determining a second coding parameter used to encode a second slice of the frame based at least in part on the difference and the updated coding parameter threshold.

Abstract: A method for processing video data of an environment includes, with aid of one or more processors individually or collectively, obtaining in or near real-time a reference position of an imaging device located on a movable object based on one or more previously traversed positions of the imaging device, and modifying an image frame in the video data to obtain a modified image frame based on the reference position of the imaging device and an actual position of the imaging device at which the image frame is taken. The one or more previously traversed positions are obtained using at least one sensor on the movable object.

Abstract: A system constructs an optical flow field that corresponds with a selected video frame. The optical flow field is constructed based on a first motion of a mobile platform having an imaging device and a status of the imaging device. The first motion and the status are determined with measurements of sensors installed on the mobile platform and/or the imaging device installed on the mobile platform. The first motion includes at least one of a first rotation, a horizontal movement, or a vertical movement of the mobile platform. The status includes a rotation of the imaging device and/or an orientation of the imaging device relative to the mobile platform.

Abstract: Methods and systems of determining a quantization step for encoding video based on motion data are provided. Video captured by an image capture device is received. The video comprises a video frame component. Additionally, motion data associated with the video frame component is received. Further, a quantization step for encoding the video frame component is determined based on the motion data.

Abstract: Methods and systems for evaluating a search area for encoding video are provided. The method comprises receiving video captured by an image capture device, the video comprising video frame components. Additionally, the method comprises receiving optical flow field data associated with the video frame component, wherein at least a portion of the optical flow field data is captured by sensors. The method also comprises determining a search area based on the optical flow field data.