First, play a few true 4.0 (no silent channels) source, without DSU. Does your Denon show 'stereo' or 'Mch In'? (or you can press the 'info' button on the remote, and see which input channels are active).
Please let me know what 4.0 tracks you played, so I can try to replicate
The Denon shows "Mch In"
It shows only the four channels input active: FL, FR, SL, SR
I'm playing a FLAC (4 channels), from Suzanne Ciani Quadraphonic Live. using Oppo 203 via hdmi.
You can 'build' mch WAV files, with the channels you want, using Audacity or better with the MMH tool here in the forum (menu 'Remix Channel Layout'). And then convert to mch FLAC if you wish.
Last, can you tell me what model # your DenonAVR is, and/or how old it is?
It is shown in my signature: Denon AVC-X8500H (2018). AVC is the European version (without radio tuner) instead of AVR
I'm not sure what you mean here. In my experience 4.0 audio in a 5.1 container (i.e., having only silence in C and LFE tracks) will simply play silence from the C and LFE (unless bass management is on, in which case 4 channels of bass is routed to the 'subwoofer' out). It can only be 'upmixed' by DSU to added back/side speakers, e.g. to 7.0
What I *do* observe is that my current Denon sees 4.0 as 2.0**, in which case turning DSU 'on' produces 'upmixed' output from all 5 main channels, just as it would for any stereo source. Doesn't sound good, though.
None of the 3.0 tracks I own (Mercury Living Presence) will play as true 3.0. There are seen/played as 'stereo' unless I add 'silent' LFE/back channels.
I'm using a AVRX3300W, all tracks are ripped from discs, and played via foobar2k/WASAPI/HDMI.
**unless the two rear channels are formatted as 'SL and SR' rather than 'BL' and 'BL'. Then the Denon does see a 4 channel 'Mch In' signal, and DSU can be activated on top of it. Unfortunately I have so far only found a single, unofficial, quad release that uses SL/SR rather than BL/BR formatting.
What I mean is that if the 4.0 source is formatted 'into' a DTS or AC3 format (with silent channels), then the AVR will see it just as 5.1 DTS or AC3, and the DSU upmixer will work as usual for those formats.
Some AVR's or players could have restrictions to 'understand' mch if there are a weird number of channels different from 5.1. And then play it just as stereo.
My Oppo 203 does play mch 4.0 and 5.1, but cannot play 7.1. I use DUNE instead, to play mch 7.1
I think is better to 'build' mch files with a 'standard' number of channels, like 5.1. With silence in the channels with no content. Full compatibility for all systems.
Some players/AVR could not process 5.0 (a file with only five channels), for instance. 3.0 could be even more 'weird'.
In my system, and it may be the 'standard', in a 7.1 system, 4.0 or 5.1 files direct the rear channels to the Side Surrounds (not to the surround backs)
If I want to play Quad using the Surround Backs (rears instead of sides) in my 7.1 (without reconfiguring custom speakers assignment), I have to build a 7.1 file with silence in the sides and move the sides to the rears. It can be easily done with MMH, there is an option exactly for doing that.