Poor zigbee binding reliability: switches sending broadcast messages = flooded network to blame?

tl;dr because this is going to be a long one… my zigbee bindings to Hue bulbs aren’t as reliable as I had hoped- sniffed the zigbee traffic and can see odd/wasteful(?) broadcast messages being sent, potentially flooding the network

Prefacing this with the fact that I’m not a zigbee expert by any means and am learning a lot in the process of my troubleshooting.

I’ve had ~25 Blue series dimmers installed for a month or so now (I got lucky and only had a few in the bad batches) and have had the chance to really understand their behavior. One disappointing thing I’ve found is that binding from the switches to my Hue bulbs is not as reliable as I hoped: I was experiencing failures (Hue bulbs not responding) maybe 10-20% of the time I would try to use the switch from the wall.

My first thought was interference on the 2.4ghz channel causing the binding messages to be dropped. I optimized my zigbee channels vs my wireless access point 2.4ghz channels so that no overlap should be occurring and scanned the channels to ensure nothing else was crowding the bands I was using. No improvement in reliability after this, but in my further testing I noticed that reliability was much worse when I would press multiple switches at nearly the same time e.g. if I was trying to turn off all the lights in the basement across 3 switches: I’d press one after the other quickly- I could get the binding to fail on at least one of those presses nearly every time.

So then I went deep…

I flashed a spare sonoff 3.0 dongle with sniffing firmware (using the nice instructions on this reddit thread) and started testing things while monitoring the traffic.

The first thing I noticed was the INCREDIBLE amount of traffic that the energy reporting on these switches produces: I set the min rep interval and min rep change to 1000 in Z2M (as I think there is another bug that keeps it from being fully disabled?) and that cut the overall traffic way down, but didn’t solve my binding problems.

Next, I did a simple test comparing the messages sent from a GE zigbee switch vs Blue series when both were given a bind to the same single Hue bulb. Here is a screenshot of the traffic sent when I pressed down on the GE switch. I highlighted the 4 messages that were produced within a second of the press: the first being the OFF event being sent from the switch to the Hue bulb, the next 2 to the coordinator for something I’m not sure of, and the last one telling the coordinator the new status of the switch (OFF). 4 messages- feels about right.

Here is a screenshot of the same test done on the Inovelli switch. We can see the OFF message being sent from the switch the the bulb and the coordinator just as we see with the GE switch, but then the inovelli switch propagates a [seemingly unnecessary] broadcast across the entire zigbee network resulting in 60+ messages across the network in the matter of a few hundred milliseconds.

Again, I’m not a zigbee engineer or anything so I can’t say if this is concerning behavior, but it does seem to be in line with my binding problems: when these switches hit the network with the broadcast, it seems to break the reliability of the network for the next ~500ms or so. My research also seems to confirm that broadcasting can flood networks and should be used sparingly (pdf source, source).

Anyhow, hoping that somebody on the Inovelli side can take a look at this! The switches are great and I want to love them, but keeping very reliable control of my lights at the switch is a P0 requirement in my house and it’s not great at the moment. Very invested in solving this (if it wasn’t clear from my troubleshooting up to this point :slightly_smiling_face:).

Notes on my setup:
I have ~100 devices (almost exclusively Hue bulbs and now these switches) split across 2 networks (the one I did this testing on was my smaller network of ~40 devices- the larger the network: the worse the broadcast performance hit will be) using tubeszb network coordinators with Zigbee2MQTT. I recently split into the two networks because I was seeing overall poor network behavior after installing the Blue switches bringing me to >100 devices- but now am wondering if the Blue switches and all of their power reporting + these types of broadcasts were perhaps partially to blame?

6 Likes

Ayy, it’s my thread! Glad you found it useful :slight_smile: (I don’t know anything about the binding stuff though, sorry)

1 Like

Ah thank you! I’m certain I would not have figured all of that out on my own :slightly_smiling_face:

1 Like

Could you upload the pcap? I’d want to see if this matches a similar issue I’m having

I just installed around 20 blue series on my network and it’s now become nearly unusable. I’m at 150 devices, 137 are routers. Very similar setup with most devices being hue lights. Also using a tubeszb 2652 on zigbee2mqtt. Please keep us posted if you hear anything or learn anything new. Has this impacted just binding or overall network performance as well? It seems to be both for me

3 Likes

Came here to say I’m experiencing very similar behavior and poor zigbee network reliability.

I’m using an SMLight SLZB-06 as a coordinator.

I’ve migrated from the MQTT broker add-on in Home Assistant to a dockerized EMQX MQTT Broker. Home Assistant accessing the MQTT broker via websockets and Z2M accessing it via TLS… But I’m still having problems with reliability.

In hopes of making my network better, I’m now splitting up my 140+ device network in two (by separating it to upstairs and downstairs) with another SMLight SLZB-06.

Has anyone found any good solutions in quieting down the amount of traffic from the switches and improving zigbee network reliability?

You might wish to follow the Firmware Change Log Thread. Inovelli is testing a to-be-released firmware that addresses a number of issues, including excess broadcasts. Reportedly, 2.13, or whatever winds up being released, has reduced unnecessary broadcasts. See Eric’s comments.

1 Like

Thanks @Bry@karaminder, very sorry about the network issues. I think this latest firmware will fix this and we plan on releasing it once I’m done testing it (all has gone well for the past week, so it should be very soon).

I’m following this one closely as well. I have over 50 blue 2-in-1 switches in my new home which were working great yesterday, but which are totally unresponsive to SmartThings today. Changes I made recently were to (1) link some other accounts (Ring, Hue, Lutron) and (2) creation of about a half-dozen new Zigbee bindings between pairs of switches.

Are all of them unresponsive? I’m thinking this may be an undisclosed SmartThings network issue as I’ve had a heck of a time trying to add/remove things and some of my Osram cabinet lights are simply, “Offline”, which they’ve never been before.

Then any new device I add says, “This device hasn’t updated its status yet, please wait” (or something close to that).

Yeah, it’s all of them. No info to or from the app. Status indication of on/off doesn’t work either.

An undisclosed ST could be the problem, in terms of symptoms I’m seeing. Samsung Frame TV won’t power up via the app either so that could be it.

Looks like I can control the temp of my nest thermostat though. So none of the Zigbee switches, Zigbee Hue bulbs, nor the LAN-connected TV are working but the wifi connected thermostat is.

Edit: cleaned up some potentially confusing info about the connections that aren’t working

Very interesting. Are the Zigbee Hue bulbs connected directly to ST or are they connected via the Hue integration?

If they are connected directly, then I can definitely see this being a ST issue (as the Nest is connected via API).

I know this is dumb, but have you tried pulling the power and ethernet out of the back of your hub and then waiting 30 seconds or so and then plugging them both back in?

1 Like

The Hue hub (v3 I think) is LANed to the ST hub in the same rack in my storage room, but the bulbs are a bit further away. We just moved in so I’ve only added Hue to my front porch. It was reaching some of them reliably; some not last night when I first got the Hue app setup, but today it’s not reaching any of them. One hypothesis might be that the lost connection to the Inovelli switches between the rack and the front porch now places the Hue bulbs too far from the two hubs. I’ll pop down there and plug a bulb in a can right outside the door and see if I can get that to be recognized and controlled.

Were you able to get your devices added? Everything is still working on the wall for me (including zigbee bindings that I setup prior to last night), but I don’t seem to be able to add or change anything, nor read status / control anything via ST (neither the mobile nor desktop app).

Looks like the nearby Hue bulb was able to be controlled by the ST app. The Inovelli blue switch in the same room as the rack cycles or (occasionally) shows “Network or server error” when I try to power it on or off with the app though.

This is precisely my issue too. I had bindings setup from my testing of the new firmware this week and they all work great still, even though I can’t control anything remotely. Or, if I do, it takes forever and then just times out.

My multi-taps are no longer working and nothing showed up in the history, which led me to believe something wonky was going on with ST. I was just planning on giving it a day or two to work itself out before I start to panic lol.

If the Hue is integrated via LAN, then it wouldn’t be affected by the Inovelli switch because, while they are on the same LAN network, the Zigbee network is different (since the Hue bulbs are connected directly to the Hue bridge and the Inovelli is connected directly to the ST Hub).

Yup. My experience as well. I’ll give it another day to work itself out. I’m having the same issues with other Zigbee and my Z-Wave devices as well – do you have any other non-Inovelli products connected yet?

2 Likes

Not yet. Just a few:

  • nearby Hue bulb works fine
  • Nest thermostat is controllable in the app
  • Samsung the frame tv (connected by LAN to the sand switch) is interestingly NOT controllable in ST

The lack of control is true on PC desktop app too. Not just mobile.

One thing that may validate the ST theory is that the ‘…’ in the upper right corner on IOS now has a drop down that says “Shows status information” that (at least for me) wasn’t there two days ago. It now displays the battery status of two hue motion sensors I have at the top of the main dashboard too (also new vs. 2 days ago)

The issue seems to be the connection between the Driver and ST. Specifically, I can delete the switches and then manually re-add them and they are discovered by the ST app so communication is happening on some level. However, when I click into them, the ST app and the Switch’s Driver no longer seem to be speaking the same language. Here’s a screenshot of a switch I just re-added to ST, which was discovered but is not communicating.

I’m also attaching a pic of the Edge Driver I’m using just in case the fault is on my end and I’m using an outdated or the wrong driver or something.

All of my switches are Blue 2-in-1, are confirmed not to have been from the initial bad batch, are working well in the wall, and were working in ST until about 4 days ago when ST issued a new app version to the App Store (1.6.98)


@EricM_Inovelli - did you have any luck?

I’m still not able to use the SmartThings hub so I’m excluding everything, unenrolling from Beta and the Edge Drivers and factory resetting today. Wish I knew what had caused it, though, to avoid going through the same thing again.