Abstract: An optical transmission system provides digital signals which have added bits for error correction by an encoder which are converted into 16-QAM signals with a predetermined carrier frequency by a modulator, and then frequency division muitiplexed with an AM signal so as to be inputted to an optical-electric converter. When the total sum of the light modulation indices of signals to be transmitted is over 1, a light modulation index of an M-QAM signal is set by calculating the light modulation index that marks the onset of diverging from the error rate given by a ratio between receiver noise power and the M-QAM signal power while considering a light modulation index corresponding to a usable transmission margin. The multiplexed signals are transmitted via optical fiber and converted into electric signals by a photo receiver, demodulated by a demodulator, and then outputted after having their errors corrected by an error corrector.