Maximum SINR prefiltering for reduced-state trellis-based equalization