Unfortunately, it’s not that specific. The conventional wisdom is that when a switch buzzes, it’s the wiring, when lights buzz, it’s the bulb compatibility. But conventional wisdom doesn’t help you narrow the issue, so I’m not saying it’s specifically on the 3-wire.
Your wiring description seems correct to me. At this point, we’re chasing gremlins, but here are a couple things to try.
1 - Test the 3-wire. Remove all four conductors from the switches and bundles so that you just have 4 loose conductors in each box.
1a - Using a meter, go to one end and test for continuity among all of the conductors, including the bare ground. There should not be any. Any continuity between any of the four indicates a fault.
1b - If you have some extra Romex or THHN to extend the conductors on one side back to the other box, connect them at one box and then go to the other box and test for continuity for each of the conductors. Obviously, you should have continuity for each of the four. A lack of continuity indicates a fault.
2 - Swap out the dumb switch. Odds are pretty low here but who knows . .
3 - In the back of my mind, that long run keeps making me wonder, although I can’t point to anything that should cause the issue. There isn’t a line in the dumb switch box by chance, is there? I’m wondering what would happen if you located the Inovelli in the other box, still keeping it as a 3-way. You can’t do that without a line there, though. (And you’ve already proven that a 2-way works.)
4 - Do you have an Aux? You could try an Aux in place of the dumb switch. I can’t say why an Aux would work any better, but it’s worth a shot.
At this point I’m just throwing out anything I can think of . . .
EDIT: That dumb switch isn’t lighted, is it? I would have thought I would have asked, but I read back and couldn’t find it.