Performance and complexity analysis of linear mean square error, decision feedback equalizer, and reduced-state MLSE equalizer for PAM4 communication systems

Abstract

The research compares three digital equalization methods, which are Linear Minimum Mean Square Error Equalizer (LE), Decision Feedback Equalizer (DFE), and Reduced-State Maximum Likelihood Sequence Estimator (RS-MLSE), for evaluating Pulse-Amplitude Modulation (PAM4) systems that operate under additive white Gaussian noise (AWGN) and mild intersymbol interference conditions. The authors created a MATLAB simulator to assess Bit-Error Rate (BER) performance across all Eb/N0 values spanning from 0 to 18 dB while measuring the system's operational complexity through its operations per symbol metric. The RS-MLSE method demonstrates superior performance through its lowest BER results across all SNR levels, which reach error-free operation at 16 dB (BER = 0) when compared to LE (BER = 2.27 x 10-4 at 12 dB) and DFE (BER = 8.049 x 10-4 at 12 dB). The computational needs of all three equalizers show equal requirements when testing the one-symbol memory setup, which shows that LE needs 21 multiplications/symbol and DFE needs 21 multiplications/symbol, while RS-MLSE needs 16 metric calculations/symbol. The findings confirm that RS-MLSE provides the best performance-complexity balance for PAM4 systems in AWGN-dominated environments with mild ISI, while LE remains a viable low-complexity alternative, and DFE offers limited advantage under these specific channel conditions. The results provide equalizer selection guidance for high-speed communication links that require channel characteristics and implementation constraints to achieve a balanced system design.

Keywords: PAM4 modulation; Linear MMSE equalizer; Decision Feedback Equalizer (DFE); Reduced-State MLSE (RS-MLSE); Bit-error rate (BER); Complexity-Analysis