Smart Bulb mode means that the switch continually applies power to the load so that the smart bulb can stay connected to its network. It does not provide control signals to the bulb. To do that, you have to write an automation in Home Assistant that detects changes in the switch and then controls the bulb.
Did you do that?
If you need help with that, I have a Home Assistant blueprint you could use. Here: Copy Light State to Multiple Controlled Devices · GitHub
If you are not familiar with installing and using Home Assistant blueprints, you should be able to find some online tutorials. Using this blueprint, you simply pick the Controlling Light (in this case, your Z-wave switch), and which devices you want to control (i.e., the Nanoleaf bulb).
