Yes, it should be cutting power to the Load terminal. While we often think of the air gap as a way to reboot a smart switch, the original purpose I believe is to provide a safety mechanism for when the light bulb is changed. Not every smart switch uses a relay where such a gap is automatically provided, as some of them are switched via solid state.
Presuming that your switch is wired correctly, I think what you are seeing is a bit of leakage combined with LED bulbs that illuminate at very low thresholds.