I have almost the exact same setup and have the two switches slaved to each other via group association. In the HA z-wave configuration tool (Configuration->Z-Wave) I selected the dummy/aux switch (the one without load) and grouped both group 2 and group 4 to the load switch. This association is one way (Aux switch sends to load switch), otherwise, I think you would get the result you are talking about.
This is working well for me.
As a kicker, I wrote this node red subflow to keep the LED bars in sync and your post inspired me to share it: Home Assistant Node Red Sync Multiple Dimmer LEDs Subflow Code
Here is my config:
