I suspect that's true. A long time ago when I was researching building my own, I remember a discussion about a unit with MOSFET's that was pretty bad. I don't now recall the details of the circuitry though.The commercial ones that degraded the signal were made to pass RF or video, not speaker power.
My thinking I believe was to employ dpdt relays in lieu of a complicated circuit, but system changes meant I no longer needed a device. The mechanical switches I was using were pretty simple, but were prone to corrosion and needed cleaning about once a year.