Null states in z2m?

Smog_Privacy · January 16, 2025, 4:37am

Hello. I’m building out a new house and am giving z2m a try instead of zha. I added 16 blue dimmers and 1 blue fan switch, and on/off/level works great, as do updates, but most states are showing as “Unknown” in HA, and when I look at the “State” tab for a device in z2m I see the following. Is this normal? I’m not sure if I’ve left out a step, if there’s a problem in z2m, or a problem on the switches. Anyone have an idea why so much shows “null” in z2m?

{
    "brightness": 254,
    "energy": 0.4,
    "linkquality": 18,
    "power": 30.1,
    "state": "ON",
    "update": {
        "installed_version": 16908818,
        "latest_version": 16908818,
        "state": "idle"
    },
    "activeEnergyReports": null,
    "activePowerReports": null,
    "autoTimerOff": null,
    "auxSwitchUniqueScenes": null,
    "bindingOffToOnSyncLevel": null,
    "brightnessLevelForDoubleTapDown": null,
    "brightnessLevelForDoubleTapUp": null,
    "buttonDelay": null,
    "defaultLed1ColorWhenOff": null,
    "defaultLed1ColorWhenOn": null,
    "defaultLed1IntensityWhenOff": null,
    "defaultLed1IntensityWhenOn": null,
    "defaultLed2ColorWhenOff": null,
    "defaultLed2ColorWhenOn": null,
    "defaultLed2IntensityWhenOff": null,
    "defaultLed2IntensityWhenOn": null,
    "defaultLed3ColorWhenOff": null,
    "defaultLed3ColorWhenOn": null,
    "defaultLed3IntensityWhenOff": null,
    "defaultLed3IntensityWhenOn": null,
    "defaultLed4ColorWhenOff": null,
    "defaultLed4ColorWhenOn": null,
    "defaultLed4IntensityWhenOff": null,
    "defaultLed4IntensityWhenOn": null,
    "defaultLed5ColorWhenOff": null,
    "defaultLed5ColorWhenOn": null,
    "defaultLed5IntensityWhenOff": null,
    "defaultLed5IntensityWhenOn": null,
    "defaultLed6ColorWhenOff": null,
    "defaultLed6ColorWhenOn": null,
    "defaultLed6IntensityWhenOff": null,
    "defaultLed6IntensityWhenOn": null,
    "defaultLed7ColorWhenOff": null,
    "defaultLed7ColorWhenOn": null,
    "defaultLed7IntensityWhenOff": null,
    "defaultLed7IntensityWhenOn": null,
    "defaultLevelLocal": null,
    "defaultLevelRemote": null,
    "deviceBindNumber": null,
    "dimmingSpeedDownLocal": null,
    "dimmingSpeedDownRemote": null,
    "dimmingSpeedUpLocal": null,
    "dimmingSpeedUpRemote": null,
    "doubleTapClearNotifications": null,
    "doubleTapDownToParam56": null,
    "doubleTapUpToParam55": null,
    "fanControlMode": null,
    "fanLedLevelType": null,
    "firmwareUpdateInProgressIndicator": null,
    "highLevelForFanControlMode": null,
    "higherOutputInNonNeutral": null,
    "identify": null,
    "individual_led_effect": null,
    "internalTemperature": null,
    "invertSwitch": null,
    "ledBarScaling": null,
    "ledColorForFanControlMode": null,
    "ledColorWhenOff": null,
    "ledColorWhenOn": null,
    "ledIntensityWhenOff": null,
    "ledIntensityWhenOn": null,
    "led_effect": null,
    "loadLevelIndicatorTimeout": null,
    "localProtection": null,
    "lowLevelForFanControlMode": null,
    "maximumLevel": null,
    "mediumLevelForFanControlMode": null,
    "minimumLevel": null,
    "onOffLedMode": null,
    "outputMode": null,
    "overheat": null,
    "periodicPowerAndEnergyReports": null,
    "powerType": null,
    "quickStartLevel": null,
    "quickStartTime": null,
    "rampRateOffToOnLocal": null,
    "rampRateOffToOnRemote": null,
    "rampRateOnToOffLocal": null,
    "rampRateOnToOffRemote": null,
    "relayClick": null,
    "remoteProtection": null,
    "singleTapBehavior": null,
    "smartBulbMode": null,
    "stateAfterPowerRestored": null,
    "switchType": null
}

rohan · January 16, 2025, 4:55am

Are you running the latest firmware?

Sometimes in Zigbee2mqtt you have to reinterview the whole switch or individually refresh each of the parameters in the “expose” tab.

Smog_Privacy · January 16, 2025, 6:32pm

Thank you for the reply. I think it’s the latest firmware (2.18 on the dimmers). I tried the Interview button on one of the devices just now, but it didn’t seem to do anything.

Clicking the refresh button for an individual item does seem to load it (and show in the “State” tab and in HA correctly). Do I really need to manually click refresh for every item in exposes for every device? And does it then keep them in sync?

rohan · January 16, 2025, 6:45pm

I’ve needed to do it the first time I pair a device (for some devices), I have no idea why (or why not). Once it’s done, they stay in sync.

Smog_Privacy · January 16, 2025, 6:47pm

Thank you. This is definitely going to be painful (20 devices so far, and another 40 Inovelli and 50+ Hue to go). Time to search for a way to see if I can easily script it.

Smog_Privacy · January 16, 2025, 6:58pm

Just noticed the Reporting section. Doesn’t seem to have a lot of details, and it looks like its on a per-entity basis, but is there something there at needs to be setup for the Inovelli cluster?

rohan · January 16, 2025, 10:07pm

Nope, the reporting section should be fine. It’s setup automatically when the device is first configured.

Smog_Privacy · January 16, 2025, 10:28pm

Hmm, if it should have set something up then maybe that’s the problem then. This is a new install with z2m 2.0.0, where all the devices have been paired using 2.0.0. Here’s a screenshot of the Reporting section for one of the devices. Only a few items in.

rohan · January 16, 2025, 10:36pm

That is setup correctly.

I believe the issue is that during the initial interview it sometimes doesn’t get all of the parameters loaded in correctly (possibly a timing / race condition). Since the parameters are not changing (at all), there is no update that happens from the switch to Z2M to pick it up until you hit the refresh button and do it manually. From there all of the parameters are changed from Z2M to the switch and should stay in sync.

Smog_Privacy · January 17, 2025, 5:13am

OK, I wrote up a quick and dirty script to send a message to mqtt to refresh each attribute for a device. It’s very brute force at the moment (it knows the attributes for the dimmer and the fan controller), and it doesn’t try to be smart about it, it just does all the attributes even if they are already populated. It seemed to work and got everything in.

I still have more switches to install, so I’m going to work on the script more so it can only fetch what is needed. Once I’m okay with it I’ll post it as I have a feeling it will be useful generally.

rohan · January 17, 2025, 7:16am

Let me think about it a little over the weekend. We may be add something that does the equivalent to the end of the interview or reconfigure process.

Either way, an external script would be useful!

Smog_Privacy · January 20, 2025, 11:31pm

Here’s my current script for refreshing the states for a single device. I’m still not happy with the one that does all the devices, so I’ll probably post that later this week. I think the only states this won’t refresh are the main fan/light entities themselves, and the update entity.

sequence:
  - variables:
      attrs:
        - activeEnergyReports
        - activePowerReports
        - autoTimerOff
        - auxSwitchUniqueScenes
        - bindingOffToOnSyncLevel
        - breezeMode
        - brightness
        - brightnessLevelForDoubleTapDown
        - brightnessLevelForDoubleTapUp
        - buttonDelay
        - defaultLed1ColorWhenOff
        - defaultLed1ColorWhenOn
        - defaultLed1IntensityWhenOff
        - defaultLed1IntensityWhenOn
        - defaultLed2ColorWhenOff
        - defaultLed2ColorWhenOn
        - defaultLed2IntensityWhenOff
        - defaultLed2IntensityWhenOn
        - defaultLed3ColorWhenOff
        - defaultLed3ColorWhenOn
        - defaultLed3IntensityWhenOff
        - defaultLed3IntensityWhenOn
        - defaultLed4ColorWhenOff
        - defaultLed4ColorWhenOn
        - defaultLed4IntensityWhenOff
        - defaultLed4IntensityWhenOn
        - defaultLed5ColorWhenOff
        - defaultLed5ColorWhenOn
        - defaultLed5IntensityWhenOff
        - defaultLed5IntensityWhenOn
        - defaultLed6ColorWhenOff
        - defaultLed6ColorWhenOn
        - defaultLed6IntensityWhenOff
        - defaultLed6IntensityWhenOn
        - defaultLed7ColorWhenOff
        - defaultLed7ColorWhenOn
        - defaultLed7IntensityWhenOff
        - defaultLed7IntensityWhenOn
        - defaultLevelLocal
        - defaultLevelRemote
        - deviceBindNumber
        - dimmingSpeedDownLocal
        - dimmingSpeedDownRemote
        - dimmingSpeedUpLocal
        - dimmingSpeedUpRemote
        - doubleTapClearNotifications
        - doubleTapDownToParam56
        - doubleTapUpToParam55
        - fan_mode
        - fan_state
        - fanControlMode
        - fanLedLevelType
        - fanTimerMode
        - firmwareUpdateInProgressIndicator
        - higherOutputInNonNeutral
        - highLevelForFanControlMode
        - internalTemperature
        - invertSwitch
        - ledBarScaling
        - ledColorForFanControlMode
        - ledColorWhenOff
        - ledColorWhenOn
        - ledIntensityWhenOff
        - ledIntensityWhenOn
        - loadLevelIndicatorTimeout
        - localProtection
        - lowLevelForFanControlMode
        - maximumLevel
        - mediumLevelForFanControlMode
        - minimumLevel
        - nonNeutralAuxLowGear
        - nonNeutralAuxMediumGear
        - onOffLedMode
        - outputMode
        - overheat
        - periodicPowerAndEnergyReports
        - powerType
        - quickStartLevel
        - quickStartTime
        - rampRateOffToOnLocal
        - rampRateOffToOnRemote
        - rampRateOnToOffLocal
        - rampRateOnToOffRemote
        - relayClick
        - remoteProtection
        - singleTapBehavior
        - smartBulbMode
        - stateAfterPowerRestored
        - switchType
      manufacturer: "{{ device_attr(device, 'manufacturer') }}"
      topic: "{{ 'zigbee2mqtt/' + device_attr(device, 'name') + '/get' }}"
      payloads: >
        {% set ns = namespace(entities=[], payloads=[]) %}
        {% set entity = expand(device_entities(device)) | selectattr('domain', 'in', ['light', 'fan']) | first %}
        {% set expr = '^' + entity.object_id + '_' %}
        {% set ns.entities = expand(device_entities(device)) | rejectattr('domain', 'eq', 'button') | list %}
        {% if unknown %}
        {% set ns.entities = ns.entities | selectattr('state', 'eq', 'unknown') | list %}
        {%- endif %}
        {% for state in ns.entities %}
        {% set attr = "^" + state.object_id | regex_replace(find=expr, replace='', ignorecase=True) + "$" %}
        {% set found = attrs | select('match', attr, ignorecase=true) | list %}
        {% if found | length > 0 %}
        {% set ns.payloads = ns.payloads + ['{"' + found | first + '": ""}'] %}
        {%- endif %}
        {%- endfor %}
        {{ ns.payloads }}
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ manufacturer == 'Inovelli' }}"
        sequence:
          - repeat:
              for_each: "{{ payloads }}"
              sequence:
                - action: logbook.log
                  data:
                    name: Z2M_fix
                    message: >-
                      manf={{ manufacturer }}, topic={{ topic }}, payload={{ repeat.item }}
                - action: mqtt.publish
                  data:
                    qos: 0
                    retain: false
                    topic: "{{ topic }}"
                    payload: "{{ repeat.item }}"
                - delay:
                    milliseconds: 100
fields:
  device:
    selector:
      device: {}
    name: Device ID
    description: Device to refresh (must be Inovelli Blue series on MQTT)
    required: true
  unknown:
    selector:
      boolean: {}
    name: Unknown Only
    description: Only refresh entities with unknown state
    default: true
    required: false
alias: Inovelli Z2M Device Refresh
description: ""

Smog_Privacy · January 28, 2025, 4:46pm

And a script to update all Inovelli Z2M devices, which just calls the script above.

sequence:
  - repeat:
      for_each: "{{ integration_entities('mqtt') | select('is_device_attr', 'manufacturer', 'Inovelli') | map('device_id') | sort | unique | list }}"
      sequence:
        - action: script.inovelli_z2m_device_refresh
          metadata: {}
          data:
            device: "{{ repeat.item }}"
            unknown: "{{ unknown }}"
fields:
  unknown:
    selector:
      boolean: {}
    name: Unknown Only
    description: Only refresh entities with unknown state
    default: true
    required: false
alias: Inovelli Z2M All Devices Refresh
description: ""

rohan · January 30, 2025, 9:22am

I just put in a PR for Z2M that should automatically populate all of the attributes when configuring a switch (happens automatically during pairing and can also be triggered from the reconfigure button). Add additional improvements for Inovelli devices by rohankapoorcom · Pull Request #8700 · Koenkk/zigbee-herdsman-converters · GitHub

Smog_Privacy · January 30, 2025, 5:59pm

Thank you, that’s great! Hopefully it will get included into the next release (and hoping that release happens before I do the next round of device adds).

EricM_Inovelli · January 30, 2025, 9:35pm

Tested and seems to be working. Thanks @rohan! Hopefully it will make it into the next prod update of z2m.

rohan · January 30, 2025, 10:30pm

Nice! If not February’s release, then next one for sure.

rohan · February 9, 2025, 5:22am

Z2M 2.1.1 includes this fix. Hopefully you haven’t added those new devices yet @Smog_Privacy

Smog_Privacy · February 9, 2025, 5:38am

Thank you! Not yet, will be a few more weeks before that happens, but that patch will definitely make it easier!