Tag Archives: esp32

Read EVU smart meters with ESP32 and ESPhome and use them in Homeassistant

Loading

edit 7.11.24
In the meantime, I have also layouted an interface board with a USB type B socket for the 5V supply. (see layout below). Because as small and fine as the micro USB plugs are, I need something more robust.

new board version with USB type B socket for power supply

As I am asked more and more often for the production data, I am making the Gerber data of the circuit boards available for download:

ESP32_interface_2023-05-12

interface_usbB_2024-06-27


In the article entitled: “Reading energy supply company smart meters with ESP32 and sending data via MQTT” (link), I described how the energy supply companies’ smart meters can be read out via the customer interface. The measurement data is then available as topics via the mqtt broker and can be further processed in various home automation systems (HomeMatic, Homeassistant, etc.). All you need is an ESP32 board and a few small parts to establish the connection to the smart meter. As a small update, I have now embellished the structure (back then with pin headers on a breadboard) a little and made a circuit board.

Layout preview in designtool

The associated circuit diagram essentially corresponds to the sketch in the previous article. To make things a little more convenient with the new circuit board, the connection to the customer interface of the smart meter can be plugged in via an RJ socket. I have also implemented the power supply via a USB socket.

Once the ESP32 circuit board had been fitted and plugged in, the device was given a small housing and is now doing its job in the electrical distribution cabinet.

The hardware is therefore ready and functional. I have also considered changing something about the software. Until now, the ESP was running a program that decrypted the data from the smart meter and then sent it to the IP address of the broker via MQTT. However, as I am now also a user of the ESPHome integration in my HomeAssistant environment, I have flashed the ESP with an ESPHome base image. On GitHub there is the repository of Andre-Schuiki, where he publishes a version for ISKRA and SIEMENS Smartmeter for use with ESPHome. The installation instructions can be found under the following link: https://github.com/Andre-Schuiki/esphome_im350/tree/main/esp_home

The script for the ESPHome device looks like this:

 esphome:  
  name: kelagsmartmeter  
  friendly_name: KelagSmartmeter  
  libraries:  
  - "Crypto" # !IMPORTANT! we need this library for decryption!  
 esp32:  
  board: esp32dev  
  framework:  
   type: arduino  
 # Enable logging  
 logger:  
 # Enable Home Assistant API  
 api:  
  encryption:  
   key: "da kommt der key rein des neu angelegten ESPHome Gerätes rein"  
 ota:  
  password: "das automatisch generierte ota passwort"  
 wifi:  
  ssid: !secret wifi_ssid  
  password: !secret wifi_password  
  # Enable fallback hotspot (captive portal) in case wifi connection fails  
  ap:  
   ssid: "Kelagsmartmeter Fallback Hotspot"  
   password: "das automatisch generierte password"  
 captive_portal:  
 external_components:  
  - source:  
    type: local  
    path: custom_esphome  
 sensor:  
  - platform: siemens_im350  
   update_interval: 5s  
   trigger_pin: 26 # this pin goes to pin 2 of the customer interface and will be set to high before we try to read the data from the rx pin  
   rx_pin: 16 # this pin goes to pin 5 of the customer interface  
   tx_pin: 17 # not connected at the moment, i added it just in case we need it in the future..  
   decryption_key: "00AA01BB02CC03DD04EE05FF06AA07BB" # you get the key from your provider!  
   use_test_data: false # that was just for debugging, if you set it to true data are not read from serial and the test_data string is used  
   test_data: "7EA077CF022313BB45E6E700DB0849534B697460B6FA5F200005C8606F536D06C32A190761E80A97E895CECA358D0A0EFD7E9C47A005C0F65B810D37FB0DA2AD6AB95F7F372F2AB11560E2971B914A5F8BFF5E06D3AEFBCD95B244A373C5DBDA78592ED2C1731488D50C0EC295E9056B306F4394CDA7D0FC7E0000"  
   delay_before_reading_data: 1000 # this is needed because we have to wait for the interface to power up, you can try to lower this value but 1 sec was ok for me  
   max_wait_time_for_reading_data: 1100 # maximum time to read the 123 Bytes (just in case we get no data)  
   ntp_server: "pool.ntp.org" #if no ntp is specified pool.ntp.org is used  
   ntp_gmt_offset: 3600  
   ntp_daylight_offset: 3600  
   counter_reading_p_in:  
    name: reading_p_in  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kWh  
    accuracy_decimals: 3  
    device_class: energy  
   counter_reading_p_out:  
    name: reading_p_out  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kWh  
    accuracy_decimals: 3  
    device_class: energy  
   counter_reading_q_in:  
    name: reading_q_in  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kvarh  
    device_class: energy  
   counter_reading_q_out:  
    name: reading_q_out  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kvarh  
    device_class: energy  
   current_power_usage_in:  
    name: power_usage_in  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kW  
    accuracy_decimals: 3  
    device_class: energy  
   current_power_usage_out:  
    name: power_usage_out  
    filters:  
     - lambda: return x / 1000;  
    unit_of_measurement: kW  
    accuracy_decimals: 3  
    device_class: energy  
  # Extra sensor to keep track of uptime  
  - platform: uptime  
   name: IM350_Uptime Sensor  
 switch:  
  - platform: restart  
   name: IM350_Restart  

 

EVU Smartmeter read out with ESP32 and send data via MQTT

Loading

Little by little, I am bringing many of my smarthome components to a common standard. I have decided to bring all devices together via a NodeRed server. The HomeMatic system also communicates with NodeRed. Among other things, I also transfer the measured values of the EVU meter (I have a Siemens IM350 smart meter installed) to the HomeMatic CCU. This is done as mentioned in an earlier post, via the LED pulse interface (1000 pulses/kWh). For this purpose, a phototransistor is simply attached above the LED on the meter, which detects the flashing pulses of the LED and converts them into the instantaneous power in the meter sensor transmitter unit HM-ES-TX-WM and integrates them over time and then sends the data on to the CCU. This works quite well in itself. Only the update rate (in the minute range) is too long for me. Also, the phototransistor seems to react again and again to the stray light of the neighboring LED (which displays the reactive power in 1000 pulses/kvarh). This causes discrepancies between the count via the HomeMatic sensor and the values read directly from the meter.

This is definitely more accurate. If you look at the IM350 Smartmeter meter in detail, or read through the manual, you will quickly see that it has a so-called “customer interface”. This customer interface provides some measurement data via a galvanically isolated data line every second. This includes, among others, the momentary active power in both directions, as well as the meter readings of active and reactive power in the reference and feed-in direction. So perfect starting conditions to replace the HomeMatic meter sensor with my own design. After a little Internet research, I quickly realized that I am not the only one who deals with exactly this issue. The data of the customer interface tumbles out after request over a data request line with a speed of 115kbaud. However, they are encrypted, and not directly readable. To obtain the 16-byte decryption key, the utility must be consulted. The key is tied to the smart meter serial number and is unique to each smart meter. After some phone calls with my Carinthian energy provider, the key code was sent to me by mail. In the next step I tested with a USB-UART adapter on a PC, if data really come out of the meter when the interface is wired correctly. For this I crimped a RJ11 connector to a suitable 6pin cable and wired the open end of the cable according to the datasheet of the meter. Not much is needed for this. A 5V supply must activate the interface, likewise the Data Request line must be switched to 5V and already the data packets are available at the Data Out line. By the way, it also works with a 3V3 supply. With a terminal program on the PC (I usually use putty or hterm) you can visualize the encrypted data.

Now it was time to think about how to decode and process the data. For this, one finds two approaches with net:

* via a RaspberryPi, with a Python environment and a Python script. The scripts here take over the reception and decryption of the data and then make them available for further processing in different ways

* via an ESP32. The ESP is also able to decode a 128Bit AES encryption and still has plenty of resources to process the data and send it via WiFi. Furthermore, an ESP is available in sufficient quantities for little money. So I decided to use this solution. There is an open source project on GitHub from the user https://github.com/Andre-Schuiki/esphome_im350 in which he provides an ESP32 IM350 decoder as a basis for own projects. With his sources you get a decoder that reads the meter data every second and outputs it via the USB UART programming interface and also via Telnet over WiFi. I used this source as a basis.

My goal is to put the data obtained from the smart meter into MQTT messages and send them to my MQTT broker. From there it is then a simple matter to get them into NodeRed and the HomeMatic CCU and store them there. So I adapted the code. This involved setting the wifi connection to the router to a static IP. (are to be defined in settings.h). The readout readings, as well as the RSSI of the wifi connection, are now provided via MQTT Topics. (the IP address to the broker is also to be defined in settings.h). If you compile the code now and run it on the ESP, then it should log into the respective network. As long as the ESP is still connected to a PC, you can check what it is doing via the programming interface and a terminal. If you now connect the RJ11 plug to the customer interface of the meter, the triangle above the label “KU” should flash in the display of the meter every second. If this happens, the measured values should already be displayed in the terminal (provided that you have not forgotten to enter the KEY from the utility in secrets.h). If this also works, then a look at the MQTT broker (with e.g.: MQTT Explorer) makes sure that the messages arrive. Now the ESP can be removed from the PC.

Connection assignment
ESP32 in “free-flying” test setup

I chose a very simple solution and mounted the ESP on a breadboard. The 6pin cable to the smartmeter is soldered there. On the breadboard there is room for the pull-up resistors and a NPN transistor (BC547 etc.) for inverting the data pulses. I put the board in a small plastic box, which is now only connected with a cable to the customer interface and with a USB cable to a USB power supply.

The finished structure then (or currently) looks like this. The data ends up in the MQTT broker and NodeRed visualizes it and sends it to the HomeMatic CCU.

this is how the data arrives at the MQTT broker
and can be processed in NodeRed like this

if someone is interested in the customized scripts, I can send them to you. Regarding a publication on GitHub, I have to find out first which license conditions have to be fulfilled concerning the original repository. It will then be available here (public):

https://github.com/ingmarsretro/esphome_im350/tree/main/standalone_version_mqtt