Abstract: An encoder and a video processing system including the encoder are provided. The encoder may include a quantization operation selection circuit configured to generate first data by performing a first quantization operation based on input data including image data of a frequency domain, to generate a feature value based on the first data, and to generate a control signal for selecting a second quantization operation or a rate-distortion optimized quantization (RDOQ) operation based on a machine learning model trained based on training data that includes the feature value. Additionally, the encoder may include a quantizer configured to perform the second quantization operation or the RDOQ operation based on the control signal and to generate output data (e.g., after performing the second quantization operation or the RDOQ operation).