Hmm, I guess I never read it carefully before, but the setting is explicitly color temperature for “cold power on”, which I guess means when power is cut/restored, not when a Zigbee off/on message is sent. So my initial advice was totally irrelevant to your question.
At the device level, I don’t think the Inovelli switches can send the Hue bulbs any color messages through Zigbee binding. So you might need to update the color with a HA automation.
(If you decide to go down that route, check out this thread, where some of us figured out how to change Hue bulb color temperatures even when they’re off, to fix a jarring color change when Adaptive Lighting takes over a light’s color)