Curious if anyone else might be having an issue here too. Home Assistant offered an Open Thread Border Router update (from version 2.16.8 to 3.0.0). It says it’s enabled Thread 1.4, which I suspect might be the root cause.
Anyhow, I have a bunch of the Inovelli White On/Off switches, and after the Open Thread Border Router (OTBR) update in Home Assistant, I was getting utterly horrible delays from the switches. Talking a minute or more here and often signals just got lost. In Home Assistant, I went to a Device page, and saw the delays were someplace between me physically pressing the switch, versus when the Event showed-up in the Device page.
I restored my OTBR version (2.16.8) and I initially thought things went back to how they were before, but now testing a light again, I’m getting huge delays and drops again.
Curious if anyone else might have been having issues too? I’ve always found my Thread network a bit brittle/flaky (weird with 3 routers: 2 Apple HomePods and 1 ZBT-2 Router connected to Home Assistant via USB). But every time I’ve gone through any recommended debugging/checking configs, have never found anything wrong.
Perhaps a clue, or at least something for others to find/try if you have this problem…
In Home Assistant, I went to the “Matter Server” (Settings->Apps->Matter Server->Open Web UI). Then in the upper right area, click on '“Thread” (doesn’t look like a button, but it is). Gives a spinning mess of mesh lines (you can pause it). I noticed it showed 4 routers. Three were blue (2 HomePods, 1 ZBT-2 device). The 4th was Yellow, named like the ZBT-2, and said “External Network”, something about it not being part of the mesh. I thought maybe this is why signals getting lost.
Googling a bit, I found the suggestion to go to each node that was connected to it via the Wire, click “Node Details” and then Re-Interview the Node. Go back to the “Thread” button and repeat for each (but let it interview between each one, don’t jump and queue the work). That yellow router got fewer and fewer lines, and then went away completely.
Lights are now being responsive (I’m still on the OTBR version 2.16.8). Gonna stay there for a few days and see if things stay stable, then try again with the update. Non-intuitive to figure this, so hoping some step-by-steps can help someone else if you find yourself in the same situation. I’ll post an update again after I see if it’s stable and going to 3.0.0.
Maybe worthy of note, but I did have 2 changes today as well, I upgraded the switches to Inovelli Firmware 1.0.5 (I was already 1.0.4r4). But I did those before doing the Open Thread Border Router update. Dunno if that’s related, I suspect not. But worth the footnote.
Circle back… Had the Inovelli Switches falling offline randomly a few at a time. I found something that suggested making the Home Assistant’s ZBT-2 antenna the “Primary” in the thread mesh. I allowed the Open Thread Border Router (OTBR) App on Home Assistant to update to version 3.0.0 and rebooted the entire HA box (not just the HA software).
Unplugged both of my HomePods, air gapped all of the Inovelli switches (simultaneously) for 5 minutes, then turned all the switches back on with only the ZBT-2 as a router in the mesh network. After all switches reported Online, I was getting some phantom connections to the HomePods (which were still off), so clearly they were artifacts. Re-interviewed each node connecting to a phantom HomePod until they all disappeared. And then finally plugged back in the HomePods.
After a few days, happy to report everything has been stable, with the ZBT-2 antenna still keeping its Primary role in the Thread mesh network. Who knows how much of this dance was really needed, but hard to argue with success.
AFAIK the “primary” thread node is allocated automatically by “agreement” between the available router capable nodes and will periodically change. This means that the Thread master may relocate itself over time. I don’t know what impact this will have on your network after an HAOS upgrade (which resets the complete instance) for example.
I don’t use the HA based OTBR (my HA instance is connected to my Ethernet/Wifi backbone) and I have 2 x AppleTVs and 2 x HomePods. I have not seen any issues at all with Phantom nodes etc. One of my Apple OTBRs is typically the Thread master (but not always) and one of my AppleTVs is always the primary OTBR (I presume since it is Ethernet attached).
Anyway… hopefully everything is quiet for you moving forward.
Gotcha.. So with yours and not using the OTBR on the Home Assistant box… Are your routers exclusively the Apple TVs and HomePods? Dunno why I kinda assumed that a Thread antenna was necessary to connect to the Home Assistant for it to have the protocol capability for Thread. (If it’s possible without I’d prob have just bought an extra HomePod or two instead of the ZBT-2).
And thanks, the system is great when it’s smooth. But the hiccups are time traps for me!
That is correct, all four of my OTBRs are Apple. HA just connects via Ethernet to the AppleTVs and hence via 5G Wifi to the HomePods (it has no Thread dongle plugged in at all). This has an additional advantage that the location of the box running HA does not matter (pun intended) since it has no wireless interfaces.
Matter runs transparently over Ethernet/Wifi to the OTBRs and then on via Thread to Thread devices. The Apple OTBRs support TREL (I believe the HA OTBR does also) so any Matter traffic that can pass between OTBRs goes over Ethernet/Wifi reducing the traffic on the Thread Network.
In my network just about all of the Thread Capable Routers (all Inovelli switches) have a direct route to one or more of the Apple OTBRs.
The issue you have seen (zombie routers) has been around since the beginnings of IP routers (just like me). It happens when a failure occurs and a router or a route exiting is no longer connected but its peers are still advertising the “dead” route. Packets get sent to the dead route and then have to time out and then be resent by the originating device.
Got it thanks for the info! Definitely don’t know a ton about the Thread protocols and such. I figured it was only able to see the Apple devices as Routers because I had the ZBT2 antenna on the HA box. Interesting to know that was totally unneeded. Other than being another router closer to half the switches. But could’ve solved that with another HomePod Mini or two and get extra advantages there for more features other than just a router.