Oh my god I just figured it out! I have a smart water valve from LinkTap (https://www.link-tap.com/) that I honestly forgot about and there is a little hub thing that it uses to communicate with the valves outside via some sort of RF signal. I unplugged that and suddenly the switch is working as designed!
So basically, interference was the issue. Still a strange failure mode but that was definitely the cause!