Tag Archives: mqtt

electronics hobbyist, miscellaneous and news

LAN for Pylontech PV-batterystatus, OpenDTU and more in HomeAssistant

21. February 2025 ingmarsretro 6 Comments

In the article “Pylontech PV battery status in HomeAssistant”, I had improved the project “Pylontech battery monitoring” of the following GitHub links and drew a circuit board to make the whole construct a little more compact and professional.
https://github.com/irekzielinski/Pylontech-Battery-Monitoring
https://github.com/hidaba/PylontechMonitoring
All battery data of the Pylontech battery modules are displayed in the Homeassistant. Great! But when I take a look at the list of devices registered in my wifi networks, I almost feel sick – there are now far too many wireless devices, especially from the smart home sector, sharing the channel bandwidth. So my current plan is to bring some of the smart home devices onto the wired LAN network.

edit 06.Mar.2025: you find the project on
https://github.com/ingmarsretro/pylontech_Lan_Interface

The self-made devices based on the ESPs are ideal for this. These are devices such as the OpenDTU interface, the EVU smart home interface or, as here, the interface from the serial console of the Pylontech battery to the MQTT server in the Home Assistant.

I have done some tests with boards such as the OLIMEX ESP32-PoE and the WT32-ETH01. The Olimex board would have the great advantage of also being able to be supplied with power via PoE. However, the power supply for PoE operation is so “poor” that the required standards of external boards are not met. Here I can mention the NRF24L01 radio module, for example. I did some tests with it and decided to disregard the PoE functionality for the time being. This led to the plan to use the WT32-ETH01 with an ESP32 to design a universal board with several interfaces. It should be able to do the following:

communicate with the PV inverters using OpenDTU and NRF24L01
communicate with the Smarthome system via the Pylontech Console using MQTT
have an optional CAN interface
be able to communicate via RS422/RS485 in addition to the RS232 interface
receive the power supply via 5V USB
and to have everything packed nicely small and compact in one housing

So I designed a circuit and drew a circuit board. I got the boards manufactured by a Far East PCB manufacturer. The assembly is also done quickly.

Circuit diagram of the Universal Lan Interface

The picture below shows the PCB layout before production.

The WT32-ETH01 board does not have a USB port for programming the controller. It is programmed via an external USB-UART adapter. To activate the programming mode, an IO pin must also be connected to GND. To simplify this somewhat, there is now a “PROG” jumper on the board. If this jumper is plugged in, the WT32 can receive the firmware files. I have provided a pin header slot “TO-FTDI” as a connection option for the USB-UART adapter.
The board is now designed that it can be used to operate different devices. If you connect an NRF24L01 module to the “NRF24L01+” pin header and flash the ESP32-OpenDTU image to the controller, the inverter data can be received and transmitted via the LAN network. I have created a suitable IO-config jason-file for the use of the WT32.

Another application is the use of the board with the serial output of the battery data of the Pylontech PV batteries. The batteries provide a “Console” port which represents an RS232 interface. The data is transferred to the WT32 controller via this port and is then available via LAN in the local network.

I have adapted the ESP8266 script from hidaba and irekzielinski for the ESP32 controller. (see code at the end of the article)

Once the code has been compiled and uploaded, the status of the Pylontec batteries should be visible under the set IP address after all connections have been made.

The device setups shown in the picture are equipped with a housing. I have published the “.stl” files created with FreeCad on thingiverse.

Link to thingiverse files

Here is the adapted code for use with the WT32-ETH01 board:

The “TimerConfig.h” file must be created. The file must contain the following content:

// TimerConfig.h
#ifndef TIMERCONFIG_H
#define TIMERCONFIG_H
#define TIMER_BASE_CLK 80000000
#endif // TIMERCONFIG_H

Once the file has been created, it should be located in the directory of the main program. Here is the main program:

 //Pylontec2MQTT interface  
 //code by https://github.com/irekzielinski/Pylontech-Battery-Monitoring  
 //  and https://github.com/hidaba/PylontechMonitoring  
 // the original code used WEMOS Boards with ESP8266  
 // code changed to use with WT32-ETH01 Board for use with LAN connection  
 // changes by ingmarsretro 01/2025  
 #include <ETH.h>  
 #include <WiFi.h>  
 #include <ESPmDNS.h>  
 #include <ArduinoOTA.h>  
 #include <WebServer.h>  
 #include <circular_log.h>  
 #include <ArduinoJson.h>  
 #include <NTPClient.h>  
 #include "TimerConfig.h"  
 #include <ESP32TimerInterrupt.h>  
 //+++ START CONFIGURATION +++  
 #define LED 12  
 #define RXD2 5  
 #define TXD2 17  
 #define HOSTNAME "mppsolar-pylontec"  
 #define STATIC_IP  
 IPAddress local_IP(192, 168, xx, yy);  
 IPAddress subnet(255, 255, 255, 0);  
 IPAddress gateway(192, 168, yy, zz);  
 IPAddress primaryDNS(192, 168, yy, ww);  
 //Uncomment for authentication page  
 //#define AUTHENTICATION  
 const char* www_username = "admin";  
 const char* www_password = "password";  
 //IMPORTANT: Uncomment this line if you want to enable MQTT (and fill correct MQTT_ values below):  
 #define ENABLE_MQTT  
 // Set offset time in seconds to adjust for your timezone, for example:  
 // GMT +1 = 3600  
 // GMT 0 = 0  
 #define GMT 3600  
 //NOTE 1: if you want to change what is pushed via MQTT - edit function: pushBatteryDataToMqtt.  
 //NOTE 2: MQTT_TOPIC_ROOT is where battery will push MQTT topics. For example "soc" will be pushed to: "home/grid_battery/soc"  
 #define MQTT_SERVER    "192.168.aa.bb"  
 #define MQTT_PORT     1883  
 #define MQTT_USER     ""  
 #define MQTT_PASSWORD   ""  
 #define MQTT_TOPIC_ROOT  "ingmarsretro/pylontec/" //this is where mqtt data will be pushed  
 #define MQTT_PUSH_FREQ_SEC 2 //maximum mqtt update frequency in seconds  
 //+++  END CONFIGURATION +++  
 #ifdef ENABLE_MQTT  
 #include <PubSubClient.h>  
 WiFiClient ethClient;  
 PubSubClient mqttClient(ethClient);  
 #endif //ENABLE_MQTT  
 //text response  
 char g_szRecvBuff[7000];  
 const long utcOffsetInSeconds = GMT;  
 char daysOfTheWeek[7][12] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};  
 // Define NTP Client to get time  
 WiFiUDP ntpUDP;  
 NTPClient timeClient(ntpUDP, "pool.ntp.org", utcOffsetInSeconds);  
 //ESP8266WebServer server(80);  
 WebServer server(80);  
 circular_log<7000> g_log;  
 bool ntpTimeReceived = false;  
 int g_baudRate = 0;  
 void Log(const char* msg)  
 {  
  g_log.Log(msg);  
 }  
 //Define Interrupt Timer to Calculate Power meter every second (kWh)  
 #define USING_TIM_DIV1 true                       // for shortest and most accurate timer  
 //ESP8266Timer ITimer;  
 ESP32Timer ITimer(0); // Use timer 0  
 bool setInterval(unsigned long interval, timer_callback callback);   // interval (in microseconds)  
 #define TIMER_INTERVAL_MS 1000  
 //Global Variables for the Power Meter - accessible from the calculating interrupt und from main  
 unsigned long powerIN = 0;    //WS gone in to the BAttery  
 unsigned long powerOUT = 0;   //WS gone out of the Battery  
 //Global Variables for the Power Meter - Überlauf  
 unsigned long powerINWh = 0;    //WS gone in to the BAttery  
 unsigned long powerOUTWh = 0;   //WS gone out of the Battery  
 void setup() {  
  memset(g_szRecvBuff, 0, sizeof(g_szRecvBuff)); //clean variable  
  pinMode(LED, OUTPUT);   
  //digitalWrite(LED, HIGH);//high is off  
  digitalWrite(LED, LOW);//low is off  
  // put your setup code here, to run once:  
  //WiFi.mode(WIFI_STA);  
  //WiFi.persistent(false); //our credentialss are hardcoded, so we don't need ESP saving those each boot (will save on flash wear)  
  //WiFi.hostname(HOSTNAME);  
  ETH.begin();  
  ETH.setHostname(HOSTNAME);  
   #ifdef STATIC_IP  
    ETH.config(local_IP, gateway, subnet, primaryDNS);  
  #endif  
  for(int ix=0; ix<10; ix++)  
  {  
   Log("Wait for LAN Connection");  
   if (ETH.linkUp()) {  
     Serial2.println("Ethernet connected");  
     Serial2.print("IP Address: ");  
     Serial2.println(ETH.localIP());  
   } else {  
     Serial2.println("Failed to connect to Ethernet");  
   }  
   delay(1000);  
  }  
  ArduinoOTA.setHostname(HOSTNAME);  
  ArduinoOTA.begin();  
  server.on("/", handleRoot);  
  server.on("/log", handleLog);  
  server.on("/req", handleReq);  
  server.on("/jsonOut", handleJsonOut);  
  server.on("/reboot", [](){  
   #ifdef AUTHENTICATION  
   if (!server.authenticate(www_username, www_password)) {  
    return server.requestAuthentication();  
   }  
   #endif  
   ESP.restart();  
  });  
  server.begin();   
  timeClient.begin();  
 #ifdef ENABLE_MQTT  
  mqttClient.setServer(MQTT_SERVER, MQTT_PORT);  
 #endif  
  Log("Boot event");  
 }  
 void handleLog()  
 {  
  #ifdef AUTHENTICATION  
  if (!server.authenticate(www_username, www_password)) {  
   return server.requestAuthentication();  
  }   
  #endif  
  server.send(200, "text/html", g_log.c_str());  
 }  
 void switchBaud(int newRate)  
 {  
  if(g_baudRate == newRate)  
  {  
   return;  
  }  
  if(g_baudRate != 0)  
  {  
   Serial2.flush();  
   delay(20);  
   Serial2.end();  
   delay(20);  
  }  
  char szMsg[50];  
  snprintf(szMsg, sizeof(szMsg)-1, "New baud: %d", newRate);  
  Log(szMsg);  
  Serial2.begin(newRate,SERIAL_8N1, RXD2, TXD2);  
  g_baudRate = newRate;  
  delay(20);  
 }  
 void waitForSerial()  
 {  
  for(int ix=0; ix<150;ix++)  
  {  
   if(Serial2.available()) break;  
   delay(10);  
  }  
 }  
 int readFromSerial()  
 {  
  memset(g_szRecvBuff, 0, sizeof(g_szRecvBuff));  
  int recvBuffLen = 0;  
  bool foundTerminator = true;  
  waitForSerial();  
  while(Serial2.available())  
  {  
   char szResponse[256] = "";  
   const int readNow = Serial2.readBytesUntil('>', szResponse, sizeof(szResponse)-1); //all commands terminate with "$$\r\n\rpylon>" (no new line at the end)  
   if(readNow > 0 &&   
     szResponse[0] != '\0')  
   {  
    if(readNow + recvBuffLen + 1 >= (int)(sizeof(g_szRecvBuff)))  
    {  
     Log("WARNING: Read too much data on the console!");  
     break;  
    }  
    strcat(g_szRecvBuff, szResponse);  
    recvBuffLen += readNow;  
    if(strstr(g_szRecvBuff, "$$\r\n\rpylon"))  
    {  
     strcat(g_szRecvBuff, ">"); //readBytesUntil will skip this, so re-add  
     foundTerminator = true;  
     break; //found end of the string  
    }  
    if(strstr(g_szRecvBuff, "Press [Enter] to be continued,other key to exit"))  
    {  
     //we need to send new line character so battery continues the output  
     Serial2.write("\r");  
    }  
    waitForSerial();  
   }  
  }  
  if(recvBuffLen > 0 )  
  {  
   if(foundTerminator == false)  
   {  
    Log("Failed to find pylon> terminator");  
   }  
  }  
  return recvBuffLen;  
 }  
 bool readFromSerialAndSendResponse()  
 {  
  const int recvBuffLen = readFromSerial();  
  if(recvBuffLen > 0)  
  {  
   server.sendContent(g_szRecvBuff);  
   return true;  
  }  
  return false;  
 }  
 bool sendCommandAndReadSerialResponse(const char* pszCommand)  
 {  
  switchBaud(115200);  
  if(pszCommand[0] != '\0')  
  {  
   Serial2.write(pszCommand);  
  }  
  Serial2.write("\n");  
  const int recvBuffLen = readFromSerial();  
  if(recvBuffLen > 0)  
  {  
   return true;  
  }  
  //wake up console and try again:  
  wakeUpConsole();  
  if(pszCommand[0] != '\0')  
  {  
   Serial2.write(pszCommand);  
  }  
  Serial2.write("\n");  
  return readFromSerial() > 0;  
 }  
 void handleReq()  
 {  
  #ifdef AUTHENTICATION  
  if (!server.authenticate(www_username, www_password)) {  
   return server.requestAuthentication();  
  }  
  #endif  
  bool respOK;  
  if(server.hasArg("code") == false)  
  {  
   respOK = sendCommandAndReadSerialResponse("");  
  }  
  else  
  {  
   respOK = sendCommandAndReadSerialResponse(server.arg("code").c_str());  
  }  
  handleRoot();  
 }  
 void handleJsonOut()  
 {  
  #ifdef AUTHENTICATION  
  if (!server.authenticate(www_username, www_password)) {  
   return server.requestAuthentication();  
  }  
  #endif  
  if(sendCommandAndReadSerialResponse("pwr") == false)  
  {  
   server.send(500, "text/plain", "Failed to get response to 'pwr' command");  
   return;  
  }  
  parsePwrResponse(g_szRecvBuff);  
  prepareJsonOutput(g_szRecvBuff, sizeof(g_szRecvBuff));  
  server.send(200, "application/json", g_szRecvBuff);  
 }  
 void handleRoot() {  
  #ifdef AUTHENTICATION  
  if (!server.authenticate(www_username, www_password)) {  
   return server.requestAuthentication();  
  }  
  #endif  
  timeClient.update(); //get ntp datetime  
  unsigned long days = 0, hours = 0, minutes = 0;  
  unsigned long val = os_getCurrentTimeSec();  
  days = val / (3600*24);  
  val -= days * (3600*24);  
  hours = val / 3600;  
  val -= hours * 3600;  
  minutes = val / 60;  
  val -= minutes*60;  
  time_t epochTime = timeClient.getEpochTime();  
  String formattedTime = timeClient.getFormattedTime();  
  //Get a time structure  
  struct tm *ptm = gmtime ((time_t *)&epochTime);   
  int currentMonth = ptm->tm_mon+1;  
  static char szTmp[9500] = "";   
  long timezone= GMT / 3600;  
  snprintf(szTmp, sizeof(szTmp)-1, "<html><b>Pylontech Battery LAN Interface</b><br>Time GMT: %s (%s %d)<br>Uptime: %02d:%02d:%02d.%02d<br><br>free heap: %u<br>MQTT-Server: %s<br>MQTT-root topic: %s",   
       formattedTime, "GMT ", timezone,  
       (int)days, (int)hours, (int)minutes, (int)val,   
       ESP.getFreeHeap(),MQTT_SERVER,MQTT_TOPIC_ROOT);  
  strncat(szTmp, "<BR><a href='/log'>Runtime log</a><HR>", sizeof(szTmp)-1);  
  strncat(szTmp, "<form action='/req' method='get'>Command:<input type='text' name='code'/><input type='submit'> <a href='/req?code=pwr'>PWR</a> | <a href='/req?code=pwr%201'>Power 1</a> | <a href='/req?code=pwr%202'>Power 2</a> | <a href='/req?code=pwr%203'>Power 3</a> | <a href='/req?code=pwr%204'>Power 4</a> | <a href='/req?code=help'>Help</a> | <a href='/req?code=log'>Event Log</a> | <a href='/req?code=time'>Time</a><br>", sizeof(szTmp)-1);  
  //strncat(szTmp, "<form action='/req' method='get'>Command:<input type='text' name='code'/><input type='submit'><a href='/req?code=pwr'>Power</a> | <a href='/req?code=help'>Help</a> | <a href='/req?code=log'>Event Log</a> | <a href='/req?code=time'>Time</a><br>", sizeof(szTmp)-1);  
  strncat(szTmp, "<textarea rows='80' cols='180'>", sizeof(szTmp)-1);  
  //strncat(szTmp, "<textarea rows='45' cols='180'>", sizeof(szTmp)-1);  
  strncat(szTmp, g_szRecvBuff, sizeof(szTmp)-1);  
  strncat(szTmp, "</textarea></form>", sizeof(szTmp)-1);  
  strncat(szTmp, "</html>", sizeof(szTmp)-1);  
  //send page  
  server.send(200, "text/html", szTmp);  
 }  
 unsigned long os_getCurrentTimeSec()  
 {  
  static unsigned int wrapCnt = 0;  
  static unsigned long lastVal = 0;  
  unsigned long currentVal = millis();  
  if(currentVal < lastVal)  
  {  
   wrapCnt++;  
  }  
  lastVal = currentVal;  
  unsigned long seconds = currentVal/1000;  
  //millis will wrap each 50 days, as we are interested only in seconds, let's keep the wrap counter  
  return (wrapCnt*4294967) + seconds;  
 }  
 void wakeUpConsole()  
 {  
  switchBaud(1200);  
  //byte wakeUpBuff[] = {0x7E, 0x32, 0x30, 0x30, 0x31, 0x34, 0x36, 0x38, 0x32, 0x43, 0x30, 0x30, 0x34, 0x38, 0x35, 0x32, 0x30, 0x46, 0x43, 0x43, 0x33, 0x0D};  
  //Serial.write(wakeUpBuff, sizeof(wakeUpBuff));  
  Serial.write("~20014682C0048520FCC3\r");  
  delay(1000);  
  byte newLineBuff[] = {0x0E, 0x0A};  
  switchBaud(115200);  
  for(int ix=0; ix<10; ix++)  
  {  
   Serial.write(newLineBuff, sizeof(newLineBuff));  
   delay(1000);  
   if(Serial.available())  
   {  
    while(Serial.available())  
    {  
     Serial.read();  
    }  
    break;  
   }  
  }  
 }  
 #define MAX_PYLON_BATTERIES 8  
 struct pylonBattery  
 {  
  bool isPresent;  
  long soc;   //Coulomb in %  
  long voltage; //in mW  
  long current; //in mA, negative value is discharge  
  long tempr;  //temp of case or BMS?  
  long cellTempLow;  
  long cellTempHigh;  
  long cellVoltLow;  
  long cellVoltHigh;  
  char baseState[9];  //Charge | Dischg | Idle  
  char voltageState[9]; //Normal  
  char currentState[9]; //Normal  
  char tempState[9];  //Normal  
  char time[20];    //2019-06-08 04:00:29  
  char b_v_st[9];    //Normal (battery voltage?)  
  char b_t_st[9];    //Normal (battery temperature?)  
  bool isCharging()  const { return strcmp(baseState, "Charge")  == 0; }  
  bool isDischarging() const { return strcmp(baseState, "Dischg")  == 0; }  
  bool isIdle()    const { return strcmp(baseState, "Idle")   == 0; }  
  bool isBalancing()  const { return strcmp(baseState, "Balance") == 0; }  
  bool isNormal() const  
  {  
   if(isCharging()  == false &&  
     isDischarging() == false &&  
     isIdle()    == false &&  
     isBalancing()  == false)  
   {  
    return false; //base state looks wrong!  
   }  
   return strcmp(voltageState, "Normal") == 0 &&  
       strcmp(currentState, "Normal") == 0 &&  
       strcmp(tempState,  "Normal") == 0 &&  
       strcmp(b_v_st,    "Normal") == 0 &&  
       strcmp(b_t_st,    "Normal") == 0 ;  
  }  
 };  
 struct batteryStack  
 {  
  int batteryCount;  
  int soc; //in %, if charging: average SOC, otherwise: lowest SOC  
  int temp; //in mC, if highest temp is > 15C, this will show the highest temp, otherwise the lowest  
  long currentDC;  //mAh current going in or out of the battery  
  long avgVoltage;  //in mV  
  char baseState[9]; //Charge | Dischg | Idle | Balance | Alarm!  
  pylonBattery batts[MAX_PYLON_BATTERIES];  
  bool isNormal() const  
  {  
   for(int ix=0; ix<MAX_PYLON_BATTERIES; ix++)  
   {  
    if(batts[ix].isPresent &&   
      batts[ix].isNormal() == false)  
    {  
     return false;  
    }  
   }  
   return true;  
  }  
  //in Wh  
  long getPowerDC() const  
  {  
   return (long)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0));  
  }  
  // power in Wh in charge  
  float powerIN() const  
  {  
   if (currentDC > 0) {  
     return (float)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0));  
   } else {  
     return (float)(0);  
   }  
  }  
  // power in Wh in discharge  
  float powerOUT() const  
  {  
   if (currentDC < 0) {  
     return (float)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0)*-1);  
   } else {  
     return (float)(0);  
   }  
  }  
  //Wh estimated current on AC side (taking into account Sofar ME3000SP losses)  
  long getEstPowerAc() const  
  {  
   double powerDC = (double)getPowerDC();  
   if(powerDC == 0)  
   {  
    return 0;  
   }  
   else if(powerDC < 0)  
   {  
    //we are discharging, on AC side we will see less power due to losses  
    if(powerDC < -1000)  
    {  
     return (long)(powerDC*0.94);  
    }  
    else if(powerDC < -600)  
    {  
     return (long)(powerDC*0.90);  
    }  
    else  
    {  
     return (long)(powerDC*0.87);  
    }  
   }  
   else  
   {  
    //we are charging, on AC side we will have more power due to losses  
    if(powerDC > 1000)  
    {  
     return (long)(powerDC*1.06);  
    }  
    else if(powerDC > 600)  
    {  
     return (long)(powerDC*1.1);  
    }  
    else  
    {  
     return (long)(powerDC*1.13);  
    }  
   }  
  }  
 };  
 batteryStack g_stack;  
 long extractInt(const char* pStr, int pos)  
 {  
  return atol(pStr+pos);  
 }  
 void extractStr(const char* pStr, int pos, char* strOut, int strOutSize)  
 {  
  strOut[strOutSize-1] = '\0';  
  strncpy(strOut, pStr+pos, strOutSize-1);  
  strOutSize--;  
  //trim right  
  while(strOutSize > 0)  
  {  
   if(isspace(strOut[strOutSize-1]))  
   {  
    strOut[strOutSize-1] = '\0';  
   }  
   else  
   {  
    break;  
   }  
   strOutSize--;  
  }  
 }  
 /* Output has mixed \r and \r\n  
 pwr  
 @  
 Power Volt  Curr  Tempr Tlow  Thigh Vlow  Vhigh Base.St Volt.St Curr.St Temp.St Coulomb Time         B.V.St  B.T.St   
 1   49735 -1440 22000 19000 19000 3315  3317  Dischg  Normal  Normal  Normal  93%   2019-06-08 04:00:30 Normal  Normal   
 ....    
 8   -   -   -   -   -   -   -   Absent  -    -    -    -    -          -    -      
 Command completed successfully  
 $$  
 pylon  
 */  
 bool parsePwrResponse(const char* pStr)  
 {  
  if(strstr(pStr, "Command completed successfully") == NULL)  
  {  
   return false;  
  }  
  int chargeCnt  = 0;  
  int dischargeCnt = 0;  
  int idleCnt   = 0;  
  int alarmCnt   = 0;  
  int socAvg    = 0;  
  int socLow    = 0;  
  int tempHigh   = 0;  
  int tempLow   = 0;  
  memset(&g_stack, 0, sizeof(g_stack));  
  for(int ix=0; ix<MAX_PYLON_BATTERIES; ix++)  
  {  
   char szToFind[32] = "";  
   snprintf(szToFind, sizeof(szToFind)-1, "\r\r\n%d   ", ix+1);  
   const char* pLineStart = strstr(pStr, szToFind);  
   if(pLineStart == NULL)  
   {  
    return false;  
   }  
   pLineStart += 3; //move past \r\r\n  
   extractStr(pLineStart, 55, g_stack.batts[ix].baseState, sizeof(g_stack.batts[ix].baseState));  
   if(strcmp(g_stack.batts[ix].baseState, "Absent") == 0)  
   {  
    g_stack.batts[ix].isPresent = false;  
   }  
   else  
   {  
    g_stack.batts[ix].isPresent = true;  
    extractStr(pLineStart, 64, g_stack.batts[ix].voltageState, sizeof(g_stack.batts[ix].voltageState));  
    extractStr(pLineStart, 73, g_stack.batts[ix].currentState, sizeof(g_stack.batts[ix].currentState));  
    extractStr(pLineStart, 82, g_stack.batts[ix].tempState, sizeof(g_stack.batts[ix].tempState));  
    extractStr(pLineStart, 100, g_stack.batts[ix].time, sizeof(g_stack.batts[ix].time));  
    extractStr(pLineStart, 121, g_stack.batts[ix].b_v_st, sizeof(g_stack.batts[ix].b_v_st));  
    extractStr(pLineStart, 130, g_stack.batts[ix].b_t_st, sizeof(g_stack.batts[ix].b_t_st));  
    g_stack.batts[ix].voltage = extractInt(pLineStart, 6);  
    g_stack.batts[ix].current = extractInt(pLineStart, 13);  
    g_stack.batts[ix].tempr  = extractInt(pLineStart, 20);  
    g_stack.batts[ix].cellTempLow  = extractInt(pLineStart, 27);  
    g_stack.batts[ix].cellTempHigh  = extractInt(pLineStart, 34);  
    g_stack.batts[ix].cellVoltLow  = extractInt(pLineStart, 41);  
    g_stack.batts[ix].cellVoltHigh  = extractInt(pLineStart, 48);  
    g_stack.batts[ix].soc      = extractInt(pLineStart, 91);  
    //////////////////////////////// Post-process ////////////////////////  
    g_stack.batteryCount++;  
    g_stack.currentDC += g_stack.batts[ix].current;  
    g_stack.avgVoltage += g_stack.batts[ix].voltage;  
    socAvg += g_stack.batts[ix].soc;  
    if(g_stack.batts[ix].isNormal() == false){ alarmCnt++; }  
    else if(g_stack.batts[ix].isCharging()){chargeCnt++;}  
    else if(g_stack.batts[ix].isDischarging()){dischargeCnt++;}  
    else if(g_stack.batts[ix].isIdle()){idleCnt++;}  
    else{ alarmCnt++; } //should not really happen!  
    if(g_stack.batteryCount == 1)  
    {  
     socLow = g_stack.batts[ix].soc;  
     tempLow = g_stack.batts[ix].cellTempLow;  
     tempHigh = g_stack.batts[ix].cellTempHigh;  
    }  
    else  
    {  
     if(socLow > g_stack.batts[ix].soc){socLow = g_stack.batts[ix].soc;}  
     if(tempHigh < g_stack.batts[ix].cellTempHigh){tempHigh = g_stack.batts[ix].cellTempHigh;}  
     if(tempLow > g_stack.batts[ix].cellTempLow){tempLow = g_stack.batts[ix].cellTempLow;}  
    }  
   }  
  }  
  //now update stack state:  
  g_stack.avgVoltage /= g_stack.batteryCount;  
  g_stack.soc = socLow;  
  if(tempHigh > 15000) //15C  
  {  
   g_stack.temp = tempHigh; //in the summer we highlight the warmest cell  
  }  
  else  
  {  
   g_stack.temp = tempLow; //in the winter we focus on coldest cell  
  }  
  if(alarmCnt > 0)  
  {  
   strcpy(g_stack.baseState, "Alarm!");  
  }  
  else if(chargeCnt == g_stack.batteryCount)  
  {  
   strcpy(g_stack.baseState, "Charge");  
   g_stack.soc = (int)(socAvg / g_stack.batteryCount);  
  }  
  else if(dischargeCnt == g_stack.batteryCount)  
  {  
   strcpy(g_stack.baseState, "Dischg");  
  }  
  else if(idleCnt == g_stack.batteryCount)  
  {  
   strcpy(g_stack.baseState, "Idle");  
  }  
  else  
  {  
   strcpy(g_stack.baseState, "Balance");  
  }  
  return true;  
 }  
 void prepareJsonOutput(char* pBuff, int buffSize)  
 {  
  memset(pBuff, 0, buffSize);  
  snprintf(pBuff, buffSize-1, "{\"soc\": %d, \"temp\": %d, \"currentDC\": %ld, \"avgVoltage\": %ld, \"baseState\": \"%s\", \"batteryCount\": %d, \"powerDC\": %ld, \"estPowerAC\": %ld, \"isNormal\": %s}", g_stack.soc,   
                                                                                                       g_stack.temp,   
                                                                                                       g_stack.currentDC,   
                                                                                                       g_stack.avgVoltage,   
                                                                                                       g_stack.baseState,   
                                                                                                       g_stack.batteryCount,   
                                                                                                       g_stack.getPowerDC(),   
                                                                                                       g_stack.getEstPowerAc(),  
                                                                                                       g_stack.isNormal() ? "true" : "false");  
 }  
 void loop() {  
 #ifdef ENABLE_MQTT  
  mqttLoop();  
 #endif  
  ArduinoOTA.handle();  
  server.handleClient();  
  //if there are bytes availbe on serial here - it's unexpected  
  //when we send a command to battery, we read whole response  
  //if we get anything here anyways - we will log it  
  int bytesAv = Serial.available();  
  if(bytesAv > 0)  
  {  
   if(bytesAv > 63)  
   {  
    bytesAv = 63;  
   }  
   char buff[64+4] = "RCV:";  
   if(Serial.readBytes(buff+4, bytesAv) > 0)  
   {  
    //digitalWrite(LED, LOW);  
    digitalWrite(LED, HIGH);  
    delay(5);  
    //digitalWrite(LED, HIGH);//high is off  
    digitalWrite(LED, LOW);  
    Log(buff);  
   }  
  }  
 }  
 #ifdef ENABLE_MQTT  
 #define ABS_DIFF(a, b) (a > b ? a-b : b-a)  
 void mqtt_publish_f(const char* topic, float newValue, float oldValue, float minDiff, bool force)  
 {  
  char szTmp[16] = "";  
  snprintf(szTmp, 15, "%.2f", newValue);  
  if(force || ABS_DIFF(newValue, oldValue) > minDiff)  
  {  
   mqttClient.publish(topic, szTmp, false);  
  }  
 }  
 void mqtt_publish_i(const char* topic, int newValue, int oldValue, int minDiff, bool force)  
 {  
  char szTmp[16] = "";  
  snprintf(szTmp, 15, "%d", newValue);  
  if(force || ABS_DIFF(newValue, oldValue) > minDiff)  
  {  
   mqttClient.publish(topic, szTmp, false);  
  }  
 }  
 void mqtt_publish_s(const char* topic, const char* newValue, const char* oldValue, bool force)  
 {  
  if(force || strcmp(newValue, oldValue) != 0)  
  {  
   mqttClient.publish(topic, newValue, false);  
  }  
 }  
 void pushBatteryDataToMqtt(const batteryStack& lastSentData, bool forceUpdate /* if true - we will send all data regardless if it's the same */)  
 {  
  mqtt_publish_f(MQTT_TOPIC_ROOT "soc",     g_stack.soc,        lastSentData.soc,        0, forceUpdate);  
  mqtt_publish_f(MQTT_TOPIC_ROOT "temp",     (float)g_stack.temp/1000.0, (float)lastSentData.temp/1000.0, 0.1, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "currentDC",  g_stack.currentDC,     lastSentData.currentDC,     1, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "estPowerAC",  g_stack.getEstPowerAc(),  lastSentData.getEstPowerAc(),  10, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "battery_count",g_stack.batteryCount,    lastSentData.batteryCount,    0, forceUpdate);  
  mqtt_publish_s(MQTT_TOPIC_ROOT "base_state",  g_stack.baseState,     lastSentData.baseState      , forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "is_normal",  g_stack.isNormal() ? 1:0,  lastSentData.isNormal() ? 1:0,  0, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "getPowerDC",  g_stack.getPowerDC(),    lastSentData.getPowerDC(),    1, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "powerIN",   g_stack.powerIN(),     lastSentData.powerIN(),     1, forceUpdate);  
  mqtt_publish_i(MQTT_TOPIC_ROOT "powerOUT",   g_stack.powerOUT(),     lastSentData.powerOUT(),     1, forceUpdate);  
  // publishing details  
  for (int ix = 0; ix < g_stack.batteryCount; ix++) {  
   char ixBuff[50];  
   String ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/voltage";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_f(ixBuff, g_stack.batts[ix].voltage / 1000.0, lastSentData.batts[ix].voltage / 1000.0, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/current";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_f(ixBuff, g_stack.batts[ix].current / 1000.0, lastSentData.batts[ix].current / 1000.0, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/soc";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_i(ixBuff, g_stack.batts[ix].soc, lastSentData.batts[ix].soc, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/charging";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_i(ixBuff, g_stack.batts[ix].isCharging()?1:0, lastSentData.batts[ix].isCharging()?1:0, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/discharging";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_i(ixBuff, g_stack.batts[ix].isDischarging()?1:0, lastSentData.batts[ix].isDischarging()?1:0, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/idle";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_i(ixBuff, g_stack.batts[ix].isIdle()?1:0, lastSentData.batts[ix].isIdle()?1:0, 0, forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/state";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_s(ixBuff, g_stack.batts[ix].isIdle()?"Idle":g_stack.batts[ix].isCharging()?"Charging":g_stack.batts[ix].isDischarging()?"Discharging":"", lastSentData.batts[ix].isIdle()?"Idle":lastSentData.batts[ix].isCharging()?"Charging":lastSentData.batts[ix].isDischarging()?"Discharging":"", forceUpdate);  
   ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/temp";  
   ixBattStr.toCharArray(ixBuff, 50);  
   mqtt_publish_f(ixBuff, (float)g_stack.batts[ix].tempr/1000.0, (float)lastSentData.batts[ix].tempr/1000.0, 0.1, forceUpdate);  
  }  
 }   
 void mqttLoop()  
 {  
  //if we have problems with connecting to mqtt server, we will attempt to re-estabish connection each 1minute (not more than that)  
  static unsigned long g_lastConnectionAttempt = 0;  
  //first: let's make sure we are connected to mqtt  
  const char* topicLastWill = MQTT_TOPIC_ROOT "availability";  
  if (!mqttClient.connected() && (g_lastConnectionAttempt == 0 || os_getCurrentTimeSec() - g_lastConnectionAttempt > 60)) {  
   if(mqttClient.connect(HOSTNAME, MQTT_USER, MQTT_PASSWORD, topicLastWill, 1, true, "offline"))  
   {  
    Log("Connected to MQTT server: " MQTT_SERVER);  
    mqttClient.publish(topicLastWill, "online", true);  
   }  
   else  
   {  
    Log("Failed to connect to MQTT server.");  
   }  
   g_lastConnectionAttempt = os_getCurrentTimeSec();  
  }  
  //next: read data from battery and send via MQTT (but only once per MQTT_PUSH_FREQ_SEC seconds)  
  static unsigned long g_lastDataSent = 0;  
  if(mqttClient.connected() &&   
    os_getCurrentTimeSec() - g_lastDataSent > MQTT_PUSH_FREQ_SEC &&  
    sendCommandAndReadSerialResponse("pwr") == true)  
  {  
   static batteryStack lastSentData; //this is the last state we sent to MQTT, used to prevent sending the same data over and over again  
   static unsigned int callCnt = 0;  
   parsePwrResponse(g_szRecvBuff);  
   bool forceUpdate = (callCnt % 20 == 0); //push all the data every 20th call  
   pushBatteryDataToMqtt(lastSentData, forceUpdate);  
   callCnt++;  
   g_lastDataSent = os_getCurrentTimeSec();  
   memcpy(&lastSentData, &g_stack, sizeof(batteryStack));  
  }  
  mqttClient.loop();  
 }  
 #endif //ENABLE_MQTT

miscellaneous and news

Pylontech PV battery status in the HomeAssistant

12. November 2024 ingmarsretro 2 Comments

Anyone who has installed a photovoltaic system in their own home may even use an energy storage system. In this example, it is an off-grid system equipped with two modules from the manufacturer Pylontech. The Pylontech US3000C batteries have an output voltage of 48V. The nominal capacity is 3500Wh. The installed cells are LiFePO4 cells and the usable capacity is specified as 3374Wh according to the data sheet. The batteries are designed to be connected in parallel with other Pylontech batteries. The internally installed BMS (battery management system) communicates with the other Pylontech battery modules via a so-called “link” interface. A battery configured as a “master” handles the data exchange with the inverter. Here, Pylontech provides the CAN or RS485 bus as an interface. However, if you want information about the individual cells (voltages, currents, charges, temperatures, etc.), there is another interface on each module called “Console”. This is an RS232 interface via which you can communicate directly with the battery’s BMS. This port is also used to update the firmware of the BMS. However, I STRONGLY advise against playing around with firmware updates and flash software. This is reserved for the manufacturer or the liable party.

However, as this interface also provides a lot of information about the cells installed in the battery, this is an interesting approach. Initially, I had a laptop connected to a terminal and was able to discover and monitor the individual cell voltages and, above all, the possibly different charge status of the modules connected in parallel. So I thought it would be a good idea to have this information available in my home automation system, where it could be visualized and used for control purposes.

As we geeks and technology enthusiasts are quite familiar with terms such as Homeassistant, Docker, Proxmox, HomeMatic, NodeRed etc., I thought that this data should also become entities in the Homeassistant. So a small new project was quickly created. My plan was to read the data from the serial interface and send it to the Home Assistant via MQTT.

But before I start disassembling the data strings that come out via the serial port, I’ll have a look at the search engines. Perhaps someone else has already dealt with this topic. And that’s exactly what happened. I found what I was looking for on GitHub under the term “pylontec2mqtt”. A project is hosted at https://github.com/irekzielinski/Pylontech-Battery-Monitoring that uses ESP8266 to collect the serial data from the port and sends it to the Homeassistant server via MQTT and Wifi. A fork with a further development of this project can be found at https://github.com/hidaba/PylontechMonitoring.

Why am I publishing the project here on the blog despite the simple replica? I have optimized the circuit a little and packed it into a layout and adapted the code a little. I would like to share the result here. It was important to me to have a sensible structure on a circuit board that is connected with a USB A-B cable for the power supply and a LAN-RJ45 cable for the data connection. I wanted to use a “solid” USB connector (not the fragile mini or micro USB connectors)

On a breadboard and with the usual development boards, I quickly “knitted together” a functional model so that I could adapt the software to it.

So I first created a circuit diagram from the sketches in the Git project. There is a “real” RS232 level at the “Console” interface, which is converted to a 5V TTL via the MAX3232 IC. The BSS123 FET is used to realize a level converter to 3.3V for each of the RX and TX signals.

The ESP8266 processes this 3.3V TTL level in the form of the Wemos D1Mini or WemosD1Pro development board, which is plugged onto the circuit board. I then packed the entire construction into a small plastic housing, which can be conveniently connected to the Pylontec and a USB power source via the LAN and USB cables.

The layout design is shown in the picture above. The circuit board and the position of the components were checked again with the preview before production and then ordered from a trusted manufacturer.

Preview of the circuit board before production

After barely two weeks of waiting, I had the empty circuit boards in my hands and was able to fit them with the components.

The picture above shows the fully assembled board. The only thing missing here is the Wemos board with the ESP.

Comparison between functional model and first “production model”

In the end, I plugged in a WemosD1 Pro, as this offers the option of connecting an external WiFi antenna and thus getting a reasonable wireless range.

After flashing the software and commissioning, the Wemos web server can be accessed at the IP address specified in the code. Here you can also check whether the Pylontech battery is communicating with the Wemos. The result then looks like this.

Here you can see that both battery modules are recognized correctly. The next step is to check whether messages are being sent via the MQTT protocol. The IP address of the MQTT broker must also be specified in the Wemo code. In my setup, I have set up the MQTT Explorer in the Home Assistant to be able to check the MQTT functions quickly and easily.

The image above shows that the data also arrives correctly via MQTT. Now it is only necessary to create a sensor yaml file in the home assistant to make the topics available as entities. I have added the following code to configuration.yaml for this purpose:

mqtt:
  sensor:
#Pylontec Akku Serial Readout (ESP32 192.168.xxx.yyy)
    - state_topic: "ingmarsretro/pylontec/ESP_WiFi_RSSI"
      name: "Pylontec_RSSI"
      unit_of_measurement: dB
      
    - state_topic: "ingmarsretro/pylontec/availability"
      name: "Pylontec_Status"
      
    - state_topic: "ingmarsretro/pylontec/currentDC"
      name: "DC-Strom"
      unit_of_measurement: "mA"
      
    - state_topic: "ingmarsretro/pylontec/getPowerDC"  
      name: "getPower DC"
      unit_of_measurement: "W"
      
    - state_topic: "ingmarsretro/pylontec/powerIN"  
      name: "Power IN"
      unit_of_measurement: "W"  
      
    - state_topic: "ingmarsretro/pylontec/estPowerAC"
      name: "Pylontec_estPowerAC"
      unit_of_measurement: Watt  
      
    - state_topic: "ingmarsretro/pylontec/soc"
      name: "Pylontec_SOC"
      unit_of_measurement: "%"  
      
    - state_topic: "ingmarsretro/pylontec/temp"
      name: "Pylontec_temperature"
      unit_of_measurement: "°C"
      
    - state_topic: "ingmarsretro/pylontec/battery_count"
      name: "Pylontec_BatteryCount"
      unit_of_measurement: "pcs"      
      
    - state_topic: "ingmarsretro/pylontec/base_state"
      name: "Pylontec_BaseState"
      
    - state_topic: "ingmarsretro/pylontec/is_normal"  
      name: "Pylontec_is_normal"

    - state_topic: "ingmarsretro/pylontec/powerOUT"
      name: "Pylontec_powerOUT"
      unit_of_measurement: Watt
      
# Pylontech battery module 0      
      
    - state_topic: "ingmarsretro/pylontec/0/current"
      name: "Pylontec_Battery0_current"
      unit_of_measurement: "A"

    - state_topic: "ingmarsretro/pylontec/0/voltage"
      name: "Pylontec_Battery0_voltage"
      unit_of_measurement: "V"
      
    - state_topic: "ingmarsretro/pylontec/0/soc"
      name: "Pylontec_Battery0_soc"
      unit_of_measurement: "%"
      
    - state_topic: "ingmarsretro/pylontec/0/charging"
      name: "Pylontec_Battery0_charging"
      
    - state_topic: "ingmarsretro/pylontec/0/discharging"
      name: "Pylontec_Battery0_discharging"
     
    - state_topic: "ingmarsretro/pylontec/0/idle"
      name: "Pylontec_Battery0_idle"
      
    - state_topic: "ingmarsretro/pylontec/0/state"
      name: "Pylontec_Battery0_state"
      
    - state_topic: "ingmarsretro/pylontec/0/temp"
      name: "Pylontec_Battery0_temp"
      unit_of_measurement: "°C"
      
# Pylontech battery module 1      
      
    - state_topic: "ingmarsretro/pylontec/1/current"
      name: "Pylontec_Battery1_current"
      unit_of_measurement: "A"

    - state_topic: "ingmarsretro/pylontec/1/voltage"
      name: "Pylontec_Battery1_voltage"
      unit_of_measurement: "V"
      
    - state_topic: "ingmarsretro/pylontec/1/soc"
      name: "Pylontec_Battery1_soc"
      unit_of_measurement: "%"
      
    - state_topic: "ingmarsretro/pylontec/1/charging"
      name: "Pylontec_Battery1_charging"
      
    - state_topic: "ingmarsretro/pylontec/1/discharging"
      name: "Pylontec_Battery1_discharging"
     
    - state_topic: "ingmarsretro/pylontec/1/idle"
      name: "Pylontec_Battery1_idle"
      
    - state_topic: "ingmarsretro/pylontec/1/state"
      name: "Pylontec_Battery1_state"
      
    - state_topic: "ingmarsretro/pylontec/1/temp"
      name: "Pylontec_Battery1_temp"
      unit_of_measurement: "°C"

On the Homeassistant website, the visualization could then look like this, for example:

Last but not least, I am posting the customized code below. The libraries required for compilation and further information can be found in the GitHub links above.

#include <ESP8266WiFi.h>
#include <ESP8266mDNS.h>
#include <ArduinoOTA.h>
#include <ESP8266WebServer.h>
#include <circular_log.h>
#include <ArduinoJson.h>
#include <NTPClient.h>
#include <ESP8266TimerInterrupt.h>

//+++ START CONFIGURATION +++

//IMPORTANT: Specify your WIFI settings:
#define WIFI_SSID "wifiname"
#define WIFI_PASS deinpasswort1234"
#define WIFI_HOSTNAME "mppsolar-pylontec"

//Uncomment for static ip configuration
#define STATIC_IP
  IPAddress local_IP(192, 168, xxx, yyy);
  IPAddress subnet(255, 255, 255, 0);
  IPAddress gateway(192, 168, xxx, zzz);
  IPAddress primaryDNS(192, 168, xxx, zzz);

//Uncomment for authentication page
//#define AUTHENTICATION
//set http Authentication
const char* www_username = "admin";
const char* www_password = "password";


//IMPORTANT: Uncomment this line if you want to enable MQTT (and fill correct MQTT_ values below):
#define ENABLE_MQTT

// Set offset time in seconds to adjust for your timezone, for example:
// GMT +1 = 3600
// GMT +1 = 7200
// GMT +8 = 28800
// GMT -1 = -3600
// GMT 0 = 0
#define GMT 3600

//NOTE 1: if you want to change what is pushed via MQTT - edit function: pushBatteryDataToMqtt.
//NOTE 2: MQTT_TOPIC_ROOT is where battery will push MQTT topics. For example "soc" will be pushed to: "home/grid_battery/soc"
#define MQTT_SERVER        "192.168.xx.broker"
#define MQTT_PORT          1883
#define MQTT_USER          ""
#define MQTT_PASSWORD      ""
#define MQTT_TOPIC_ROOT    "ingmarsretro/pylontec/"  //this is where mqtt data will be pushed
#define MQTT_PUSH_FREQ_SEC 2  //maximum mqtt update frequency in seconds

//+++   END CONFIGURATION +++

#ifdef ENABLE_MQTT
#include <PubSubClient.h>
WiFiClient espClient;
PubSubClient mqttClient(espClient);
#endif //ENABLE_MQTT

//text response
char g_szRecvBuff[7000];

const long utcOffsetInSeconds = GMT;
char daysOfTheWeek[7][12] = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"};
// Define NTP Client to get time
WiFiUDP ntpUDP;
NTPClient timeClient(ntpUDP, "pool.ntp.org", utcOffsetInSeconds);


ESP8266WebServer server(80);
circular_log<7000> g_log;
bool ntpTimeReceived = false;
int g_baudRate = 0;

void Log(const char* msg)
{
  g_log.Log(msg);
}

//Define Interrupt Timer to Calculate Power meter every second (kWh)
#define USING_TIM_DIV1 true                                             // for shortest and most accurate timer
ESP8266Timer ITimer;
bool setInterval(unsigned long interval, timer_callback callback);      // interval (in microseconds)
#define TIMER_INTERVAL_MS 1000

//Global Variables for the Power Meter - accessible from the calculating interrupt und from main
unsigned long powerIN = 0;       //WS gone in to the BAttery
unsigned long powerOUT = 0;      //WS gone out of the Battery
//Global Variables for the Power Meter - Überlauf
unsigned long powerINWh = 0;       //WS gone in to the BAttery
unsigned long powerOUTWh = 0;      //WS gone out of the Battery

void setup() {
  
  memset(g_szRecvBuff, 0, sizeof(g_szRecvBuff)); //clean variable
  
  pinMode(LED_BUILTIN, OUTPUT); 
  digitalWrite(LED_BUILTIN, HIGH);//high is off
  
  // put your setup code here, to run once:
  WiFi.mode(WIFI_STA);
  WiFi.persistent(false); //our credentialss are hardcoded, so we don't need ESP saving those each boot (will save on flash wear)
  WiFi.hostname(WIFI_HOSTNAME);
  #ifdef STATIC_IP
     WiFi.config(local_IP, gateway, subnet, primaryDNS);
  #endif
  WiFi.begin(WIFI_SSID, WIFI_PASS);

  for(int ix=0; ix<10; ix++)
  {
    Log("Wait for WIFI Connection");
    if(WiFi.status() == WL_CONNECTED)
    {
      break;
    }

    delay(1000);
  }

  ArduinoOTA.setHostname(WIFI_HOSTNAME);
  ArduinoOTA.begin();
  server.on("/", handleRoot);
  server.on("/log", handleLog);
  server.on("/req", handleReq);
  server.on("/jsonOut", handleJsonOut);
  server.on("/reboot", [](){
    #ifdef AUTHENTICATION
    if (!server.authenticate(www_username, www_password)) {
      return server.requestAuthentication();
    }
    #endif
    ESP.restart();
  });
  
  server.begin(); 
  
  timeClient.begin();

  
#ifdef ENABLE_MQTT
  mqttClient.setServer(MQTT_SERVER, MQTT_PORT);
#endif

  Log("Boot event");
  
}

void handleLog()
{
  #ifdef AUTHENTICATION
  if (!server.authenticate(www_username, www_password)) {
    return server.requestAuthentication();
  } 
  #endif
  server.send(200, "text/html", g_log.c_str());
}

void switchBaud(int newRate)
{
  if(g_baudRate == newRate)
  {
    return;
  }
  
  if(g_baudRate != 0)
  {
    Serial.flush();
    delay(20);
    Serial.end();
    delay(20);
  }

  char szMsg[50];
  snprintf(szMsg, sizeof(szMsg)-1, "New baud: %d", newRate);
  Log(szMsg);
  
  Serial.begin(newRate);
  g_baudRate = newRate;

  delay(20);
}

void waitForSerial()
{
  for(int ix=0; ix<150;ix++)
  {
    if(Serial.available()) break;
    delay(10);
  }
}

int readFromSerial()
{
  memset(g_szRecvBuff, 0, sizeof(g_szRecvBuff));
  int recvBuffLen = 0;
  bool foundTerminator = true;
  
  waitForSerial();
  
  while(Serial.available())
  {
    char szResponse[256] = "";
    const int readNow = Serial.readBytesUntil('>', szResponse, sizeof(szResponse)-1); //all commands terminate with "$$\r\n\rpylon>" (no new line at the end)
    if(readNow > 0 && 
       szResponse[0] != '\0')
    {
      if(readNow + recvBuffLen + 1 >= (int)(sizeof(g_szRecvBuff)))
      {
        Log("WARNING: Read too much data on the console!");
        break;
      }
      
      strcat(g_szRecvBuff, szResponse);
      recvBuffLen += readNow;

      if(strstr(g_szRecvBuff, "$$\r\n\rpylon"))
      {
        strcat(g_szRecvBuff, ">"); //readBytesUntil will skip this, so re-add
        foundTerminator = true;
        break; //found end of the string
      }

      if(strstr(g_szRecvBuff, "Press [Enter] to be continued,other key to exit"))
      {
        //we need to send new line character so battery continues the output
        Serial.write("\r");
      }

      waitForSerial();
    }
  }

  if(recvBuffLen > 0 )
  {
    if(foundTerminator == false)
    {
      Log("Failed to find pylon> terminator");
    }
  }

  return recvBuffLen;
}

bool readFromSerialAndSendResponse()
{
  const int recvBuffLen = readFromSerial();
  if(recvBuffLen > 0)
  {
    server.sendContent(g_szRecvBuff);
    return true;
  }

  return false;
}

bool sendCommandAndReadSerialResponse(const char* pszCommand)
{
  switchBaud(115200);

  if(pszCommand[0] != '\0')
  {
    Serial.write(pszCommand);
  }
  Serial.write("\n");

  const int recvBuffLen = readFromSerial();
  if(recvBuffLen > 0)
  {
    return true;
  }

  //wake up console and try again:
  wakeUpConsole();

  if(pszCommand[0] != '\0')
  {
    Serial.write(pszCommand);
  }
  Serial.write("\n");

  return readFromSerial() > 0;
}

void handleReq()
{
  #ifdef AUTHENTICATION
  if (!server.authenticate(www_username, www_password)) {
    return server.requestAuthentication();
  }
  #endif
  bool respOK;
  if(server.hasArg("code") == false)
  {
    respOK = sendCommandAndReadSerialResponse("");
  }
  else
  {
    respOK = sendCommandAndReadSerialResponse(server.arg("code").c_str());
  }

  handleRoot();
}



void handleJsonOut()
{
  #ifdef AUTHENTICATION
  if (!server.authenticate(www_username, www_password)) {
    return server.requestAuthentication();
  }
  #endif
  if(sendCommandAndReadSerialResponse("pwr") == false)
  {
    server.send(500, "text/plain", "Failed to get response to 'pwr' command");
    return;
  }

  parsePwrResponse(g_szRecvBuff);
  prepareJsonOutput(g_szRecvBuff, sizeof(g_szRecvBuff));
  server.send(200, "application/json", g_szRecvBuff);
}

void handleRoot() {
  #ifdef AUTHENTICATION
  if (!server.authenticate(www_username, www_password)) {
    return server.requestAuthentication();
  }
  #endif
  timeClient.update(); //get ntp datetime
  unsigned long days = 0, hours = 0, minutes = 0;
  unsigned long val = os_getCurrentTimeSec();
  days = val / (3600*24);
  val -= days * (3600*24);
  hours = val / 3600;
  val -= hours * 3600;
  minutes = val / 60;
  val -= minutes*60;

  time_t epochTime = timeClient.getEpochTime();
  String formattedTime = timeClient.getFormattedTime();
  //Get a time structure
  struct tm *ptm = gmtime ((time_t *)&epochTime); 
  int currentMonth = ptm->tm_mon+1;

  static char szTmp[9500] = "";  
  long timezone= GMT / 3600;
  snprintf(szTmp, sizeof(szTmp)-1, "<html><b>Pylontech Battery</b><br>Time GMT: %s (%s %d)<br>Uptime: %02d:%02d:%02d.%02d<br><br>free heap: %u<br>Wifi RSSI: %d<BR>Wifi SSID: %s", 
            formattedTime, "GMT ", timezone,
            (int)days, (int)hours, (int)minutes, (int)val, 
            ESP.getFreeHeap(), WiFi.RSSI(), WiFi.SSID().c_str());

  strncat(szTmp, "<BR><a href='/log'>Runtime log</a><HR>", sizeof(szTmp)-1);
  strncat(szTmp, "<form action='/req' method='get'>Command:<input type='text' name='code'/><input type='submit'> <a href='/req?code=pwr'>PWR</a> | <a href='/req?code=pwr%201'>Power 1</a> |  <a href='/req?code=pwr%202'>Power 2</a> | <a href='/req?code=pwr%203'>Power 3</a> | <a href='/req?code=pwr%204'>Power 4</a> | <a href='/req?code=help'>Help</a> | <a href='/req?code=log'>Event Log</a> | <a href='/req?code=time'>Time</a><br>", sizeof(szTmp)-1);
  //strncat(szTmp, "<form action='/req' method='get'>Command:<input type='text' name='code'/><input type='submit'><a href='/req?code=pwr'>Power</a> | <a href='/req?code=help'>Help</a> | <a href='/req?code=log'>Event Log</a> | <a href='/req?code=time'>Time</a><br>", sizeof(szTmp)-1);
  strncat(szTmp, "<textarea rows='80' cols='180'>", sizeof(szTmp)-1);
  //strncat(szTmp, "<textarea rows='45' cols='180'>", sizeof(szTmp)-1);
  strncat(szTmp, g_szRecvBuff, sizeof(szTmp)-1);
  strncat(szTmp, "</textarea></form>", sizeof(szTmp)-1);
  strncat(szTmp, "</html>", sizeof(szTmp)-1);
  //send page
  server.send(200, "text/html", szTmp);
}

unsigned long os_getCurrentTimeSec()
{
  static unsigned int wrapCnt = 0;
  static unsigned long lastVal = 0;
  unsigned long currentVal = millis();

  if(currentVal < lastVal)
  {
    wrapCnt++;
  }

  lastVal = currentVal;
  unsigned long seconds = currentVal/1000;
  
  //millis will wrap each 50 days, as we are interested only in seconds, let's keep the wrap counter
  return (wrapCnt*4294967) + seconds;
}

void wakeUpConsole()
{
  switchBaud(1200);

  //byte wakeUpBuff[] = {0x7E, 0x32, 0x30, 0x30, 0x31, 0x34, 0x36, 0x38, 0x32, 0x43, 0x30, 0x30, 0x34, 0x38, 0x35, 0x32, 0x30, 0x46, 0x43, 0x43, 0x33, 0x0D};
  //Serial.write(wakeUpBuff, sizeof(wakeUpBuff));
  Serial.write("~20014682C0048520FCC3\r");
  delay(1000);

  byte newLineBuff[] = {0x0E, 0x0A};
  switchBaud(115200);
  
  for(int ix=0; ix<10; ix++)
  {
    Serial.write(newLineBuff, sizeof(newLineBuff));
    delay(1000);

    if(Serial.available())
    {
      while(Serial.available())
      {
        Serial.read();
      }
      
      break;
    }
  }
}

#define MAX_PYLON_BATTERIES 8

struct pylonBattery
{
  bool isPresent;
  long  soc;     //Coulomb in %
  long  voltage; //in mW
  long  current; //in mA, negative value is discharge
  long  tempr;   //temp of case or BMS?
  long  cellTempLow;
  long  cellTempHigh;
  long  cellVoltLow;
  long  cellVoltHigh;
  char baseState[9];    //Charge | Dischg | Idle
  char voltageState[9]; //Normal
  char currentState[9]; //Normal
  char tempState[9];    //Normal
  char time[20];        //2019-06-08 04:00:29
  char b_v_st[9];       //Normal  (battery voltage?)
  char b_t_st[9];       //Normal  (battery temperature?)

  bool isCharging()    const { return strcmp(baseState, "Charge")   == 0; }
  bool isDischarging() const { return strcmp(baseState, "Dischg")   == 0; }
  bool isIdle()        const { return strcmp(baseState, "Idle")     == 0; }
  bool isBalancing()   const { return strcmp(baseState, "Balance")  == 0; }
  

  bool isNormal() const
  {
    if(isCharging()    == false &&
       isDischarging() == false &&
       isIdle()        == false &&
       isBalancing()   == false)
    {
      return false; //base state looks wrong!
    }

    return  strcmp(voltageState, "Normal") == 0 &&
            strcmp(currentState, "Normal") == 0 &&
            strcmp(tempState,    "Normal") == 0 &&
            strcmp(b_v_st,       "Normal") == 0 &&
            strcmp(b_t_st,       "Normal") == 0 ;
  }
};

struct batteryStack
{
  int batteryCount;
  int soc;  //in %, if charging: average SOC, otherwise: lowest SOC
  int temp; //in mC, if highest temp is > 15C, this will show the highest temp, otherwise the lowest
  long currentDC;    //mAh current going in or out of the battery
  long avgVoltage;    //in mV
  char baseState[9];  //Charge | Dischg | Idle | Balance | Alarm!

  
  pylonBattery batts[MAX_PYLON_BATTERIES];

  bool isNormal() const
  {
    for(int ix=0; ix<MAX_PYLON_BATTERIES; ix++)
    {
      if(batts[ix].isPresent && 
         batts[ix].isNormal() == false)
      {
        return false;
      }
    }

    return true;
  }

  //in Wh
  long getPowerDC() const
  {
    return (long)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0));
  }

  // power in Wh in charge
  float powerIN() const
  {
    if (currentDC > 0) {
       return (float)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0));
    } else {
       return (float)(0);
    }
  }
  
  // power in Wh in discharge
  float powerOUT() const
  {
    if (currentDC < 0) {
       return (float)(((double)currentDC/1000.0)*((double)avgVoltage/1000.0)*-1);
    } else {
       return (float)(0);
    }
  }

  //Wh estimated current on AC side (taking into account Sofar ME3000SP losses)
  long getEstPowerAc() const
  {
    double powerDC = (double)getPowerDC();
    if(powerDC == 0)
    {
      return 0;
    }
    else if(powerDC < 0)
    {
      //we are discharging, on AC side we will see less power due to losses
      if(powerDC < -1000)
      {
        return (long)(powerDC*0.94);
      }
      else if(powerDC < -600)
      {
        return (long)(powerDC*0.90);
      }
      else
      {
        return (long)(powerDC*0.87);
      }
    }
    else
    {
      //we are charging, on AC side we will have more power due to losses
      if(powerDC > 1000)
      {
        return (long)(powerDC*1.06);
      }
      else if(powerDC > 600)
      {
        return (long)(powerDC*1.1);
      }
      else
      {
        return (long)(powerDC*1.13);
      }
    }
  }
};

batteryStack g_stack;


long extractInt(const char* pStr, int pos)
{
  return atol(pStr+pos);
}

void extractStr(const char* pStr, int pos, char* strOut, int strOutSize)
{
  strOut[strOutSize-1] = '\0';
  strncpy(strOut, pStr+pos, strOutSize-1);
  strOutSize--;
  
  
  //trim right
  while(strOutSize > 0)
  {
    if(isspace(strOut[strOutSize-1]))
    {
      strOut[strOutSize-1] = '\0';
    }
    else
    {
      break;
    }

    strOutSize--;
  }
}

/* Output has mixed \r and \r\n
pwr

@

Power Volt   Curr   Tempr  Tlow   Thigh  Vlow   Vhigh  Base.St  Volt.St  Curr.St  Temp.St  Coulomb  Time                 B.V.St   B.T.St  

1     49735  -1440  22000  19000  19000  3315   3317   Dischg   Normal   Normal   Normal   93%      2019-06-08 04:00:30  Normal   Normal  

....   

8     -      -      -      -      -      -      -      Absent   -        -        -        -        -                    -        -       

Command completed successfully

$$

pylon
*/
bool parsePwrResponse(const char* pStr)
{
  if(strstr(pStr, "Command completed successfully") == NULL)
  {
    return false;
  }
  
  int chargeCnt    = 0;
  int dischargeCnt = 0;
  int idleCnt      = 0;
  int alarmCnt     = 0;
  int socAvg       = 0;
  int socLow       = 0;
  int tempHigh     = 0;
  int tempLow      = 0;

  memset(&g_stack, 0, sizeof(g_stack));

  for(int ix=0; ix<MAX_PYLON_BATTERIES; ix++)
  {
    char szToFind[32] = "";
    snprintf(szToFind, sizeof(szToFind)-1, "\r\r\n%d     ", ix+1);

    const char* pLineStart = strstr(pStr, szToFind);
    if(pLineStart == NULL)
    {
      return false;
    }

    pLineStart += 3; //move past \r\r\n

    extractStr(pLineStart, 55, g_stack.batts[ix].baseState, sizeof(g_stack.batts[ix].baseState));
    if(strcmp(g_stack.batts[ix].baseState, "Absent") == 0)
    {
      g_stack.batts[ix].isPresent = false;
    }
    else
    {
      g_stack.batts[ix].isPresent = true;
      extractStr(pLineStart, 64, g_stack.batts[ix].voltageState, sizeof(g_stack.batts[ix].voltageState));
      extractStr(pLineStart, 73, g_stack.batts[ix].currentState, sizeof(g_stack.batts[ix].currentState));
      extractStr(pLineStart, 82, g_stack.batts[ix].tempState, sizeof(g_stack.batts[ix].tempState));
      extractStr(pLineStart, 100, g_stack.batts[ix].time, sizeof(g_stack.batts[ix].time));
      extractStr(pLineStart, 121, g_stack.batts[ix].b_v_st, sizeof(g_stack.batts[ix].b_v_st));
      extractStr(pLineStart, 130, g_stack.batts[ix].b_t_st, sizeof(g_stack.batts[ix].b_t_st));
      g_stack.batts[ix].voltage = extractInt(pLineStart, 6);
      g_stack.batts[ix].current = extractInt(pLineStart, 13);
      g_stack.batts[ix].tempr   = extractInt(pLineStart, 20);
      g_stack.batts[ix].cellTempLow    = extractInt(pLineStart, 27);
      g_stack.batts[ix].cellTempHigh   = extractInt(pLineStart, 34);
      g_stack.batts[ix].cellVoltLow    = extractInt(pLineStart, 41);
      g_stack.batts[ix].cellVoltHigh   = extractInt(pLineStart, 48);
      g_stack.batts[ix].soc            = extractInt(pLineStart, 91);

      //////////////////////////////// Post-process ////////////////////////
      g_stack.batteryCount++;
      g_stack.currentDC += g_stack.batts[ix].current;
      g_stack.avgVoltage += g_stack.batts[ix].voltage;
      socAvg += g_stack.batts[ix].soc;

      if(g_stack.batts[ix].isNormal() == false){ alarmCnt++; }
      else if(g_stack.batts[ix].isCharging()){chargeCnt++;}
      else if(g_stack.batts[ix].isDischarging()){dischargeCnt++;}
      else if(g_stack.batts[ix].isIdle()){idleCnt++;}
      else{ alarmCnt++; } //should not really happen!

      if(g_stack.batteryCount == 1)
      {
        socLow = g_stack.batts[ix].soc;
        tempLow  = g_stack.batts[ix].cellTempLow;
        tempHigh = g_stack.batts[ix].cellTempHigh;
      }
      else
      {
        if(socLow > g_stack.batts[ix].soc){socLow = g_stack.batts[ix].soc;}
        if(tempHigh < g_stack.batts[ix].cellTempHigh){tempHigh = g_stack.batts[ix].cellTempHigh;}
        if(tempLow > g_stack.batts[ix].cellTempLow){tempLow = g_stack.batts[ix].cellTempLow;}
      }
      
    }
  }

  //now update stack state:
  g_stack.avgVoltage /= g_stack.batteryCount;
  g_stack.soc = socLow;

  if(tempHigh > 15000) //15C
  {
    g_stack.temp = tempHigh; //in the summer we highlight the warmest cell
  }
  else
  {
    g_stack.temp = tempLow; //in the winter we focus on coldest cell
  }

  if(alarmCnt > 0)
  {
    strcpy(g_stack.baseState, "Alarm!");
  }
  else if(chargeCnt == g_stack.batteryCount)
  {
    strcpy(g_stack.baseState, "Charge");
    g_stack.soc = (int)(socAvg / g_stack.batteryCount);
  }
  else if(dischargeCnt == g_stack.batteryCount)
  {
    strcpy(g_stack.baseState, "Dischg");
  }
  else if(idleCnt == g_stack.batteryCount)
  {
    strcpy(g_stack.baseState, "Idle");
  }
  else
  {
    strcpy(g_stack.baseState, "Balance");
  }


  return true;
}

void prepareJsonOutput(char* pBuff, int buffSize)
{
  memset(pBuff, 0, buffSize);
  snprintf(pBuff, buffSize-1, "{\"soc\": %d, \"temp\": %d, \"currentDC\": %ld, \"avgVoltage\": %ld, \"baseState\": \"%s\", \"batteryCount\": %d, \"powerDC\": %ld, \"estPowerAC\": %ld, \"isNormal\": %s}", g_stack.soc, 
                                                                                                                                                                                                            g_stack.temp, 
                                                                                                                                                                                                            g_stack.currentDC, 
                                                                                                                                                                                                            g_stack.avgVoltage, 
                                                                                                                                                                                                            g_stack.baseState, 
                                                                                                                                                                                                            g_stack.batteryCount, 
                                                                                                                                                                                                            g_stack.getPowerDC(), 
                                                                                                                                                                                                            g_stack.getEstPowerAc(),
                                                                                                                                                                                                            g_stack.isNormal() ? "true" : "false");
}

void loop() {
#ifdef ENABLE_MQTT
  mqttLoop();
#endif
  
  ArduinoOTA.handle();
  server.handleClient();

  //if there are bytes availbe on serial here - it's unexpected
  //when we send a command to battery, we read whole response
  //if we get anything here anyways - we will log it
  int bytesAv = Serial.available();
  if(bytesAv > 0)
  {
    if(bytesAv > 63)
    {
      bytesAv = 63;
    }
    
    char buff[64+4] = "RCV:";
    if(Serial.readBytes(buff+4, bytesAv) > 0)
    {
      digitalWrite(LED_BUILTIN, LOW);
      delay(5);
      digitalWrite(LED_BUILTIN, HIGH);//high is off

      Log(buff);
    }
  }
}

#ifdef ENABLE_MQTT
#define ABS_DIFF(a, b) (a > b ? a-b : b-a)
void mqtt_publish_f(const char* topic, float newValue, float oldValue, float minDiff, bool force)
{
  char szTmp[16] = "";
  snprintf(szTmp, 15, "%.2f", newValue);
  if(force || ABS_DIFF(newValue, oldValue) > minDiff)
  {
    mqttClient.publish(topic, szTmp, false);
  }
}

void mqtt_publish_i(const char* topic, int newValue, int oldValue, int minDiff, bool force)
{
  char szTmp[16] = "";
  snprintf(szTmp, 15, "%d", newValue);
  if(force || ABS_DIFF(newValue, oldValue) > minDiff)
  {
    mqttClient.publish(topic, szTmp, false);
  }
}

void mqtt_publish_s(const char* topic, const char* newValue, const char* oldValue, bool force)
{
  if(force || strcmp(newValue, oldValue) != 0)
  {
    mqttClient.publish(topic, newValue, false);
  }
}

void pushBatteryDataToMqtt(const batteryStack& lastSentData, bool forceUpdate /* if true - we will send all data regardless if it's the same */)
{
  mqtt_publish_f(MQTT_TOPIC_ROOT "soc",          g_stack.soc,                lastSentData.soc,                0, forceUpdate);
  mqtt_publish_f(MQTT_TOPIC_ROOT "temp",         (float)g_stack.temp/1000.0, (float)lastSentData.temp/1000.0, 0.1, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "currentDC",    g_stack.currentDC,          lastSentData.currentDC,          1, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "estPowerAC",   g_stack.getEstPowerAc(),    lastSentData.getEstPowerAc(),   10, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "battery_count",g_stack.batteryCount,       lastSentData.batteryCount,       0, forceUpdate);
  mqtt_publish_s(MQTT_TOPIC_ROOT "base_state",   g_stack.baseState,          lastSentData.baseState            , forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "is_normal",    g_stack.isNormal() ? 1:0,   lastSentData.isNormal() ? 1:0,   0, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "getPowerDC",   g_stack.getPowerDC(),       lastSentData.getPowerDC(),       1, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "powerIN",      g_stack.powerIN(),          lastSentData.powerIN(),          1, forceUpdate);
  mqtt_publish_i(MQTT_TOPIC_ROOT "powerOUT",     g_stack.powerOUT(),         lastSentData.powerOUT(),         1, forceUpdate);

  // publishing details
  for (int ix = 0; ix < g_stack.batteryCount; ix++) {
    char ixBuff[50];
    String ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/voltage";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_f(ixBuff, g_stack.batts[ix].voltage / 1000.0, lastSentData.batts[ix].voltage / 1000.0, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/current";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_f(ixBuff, g_stack.batts[ix].current / 1000.0, lastSentData.batts[ix].current / 1000.0, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/soc";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_i(ixBuff, g_stack.batts[ix].soc, lastSentData.batts[ix].soc, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/charging";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_i(ixBuff, g_stack.batts[ix].isCharging()?1:0, lastSentData.batts[ix].isCharging()?1:0, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/discharging";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_i(ixBuff, g_stack.batts[ix].isDischarging()?1:0, lastSentData.batts[ix].isDischarging()?1:0, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/idle";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_i(ixBuff, g_stack.batts[ix].isIdle()?1:0, lastSentData.batts[ix].isIdle()?1:0, 0, forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/state";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_s(ixBuff, g_stack.batts[ix].isIdle()?"Idle":g_stack.batts[ix].isCharging()?"Charging":g_stack.batts[ix].isDischarging()?"Discharging":"", lastSentData.batts[ix].isIdle()?"Idle":lastSentData.batts[ix].isCharging()?"Charging":lastSentData.batts[ix].isDischarging()?"Discharging":"", forceUpdate);
    ixBattStr = MQTT_TOPIC_ROOT + String(ix) + "/temp";
    ixBattStr.toCharArray(ixBuff, 50);
    mqtt_publish_f(ixBuff, (float)g_stack.batts[ix].tempr/1000.0, (float)lastSentData.batts[ix].tempr/1000.0, 0.1, forceUpdate);
  }
} 

void mqttLoop()
{
  //if we have problems with connecting to mqtt server, we will attempt to re-estabish connection each 1minute (not more than that)
  static unsigned long g_lastConnectionAttempt = 0;

  //first: let's make sure we are connected to mqtt
  const char* topicLastWill = MQTT_TOPIC_ROOT "availability";
  if (!mqttClient.connected() && (g_lastConnectionAttempt == 0 || os_getCurrentTimeSec() - g_lastConnectionAttempt > 60)) {
    if(mqttClient.connect(WIFI_HOSTNAME, MQTT_USER, MQTT_PASSWORD, topicLastWill, 1, true, "offline"))
    {
      Log("Connected to MQTT server: " MQTT_SERVER);
      mqttClient.publish(topicLastWill, "online", true);
    }
    else
    {
      Log("Failed to connect to MQTT server.");
    }

    g_lastConnectionAttempt = os_getCurrentTimeSec();
  }

  //next: read data from battery and send via MQTT (but only once per MQTT_PUSH_FREQ_SEC seconds)
  static unsigned long g_lastDataSent = 0;
  if(mqttClient.connected() && 
     os_getCurrentTimeSec() - g_lastDataSent > MQTT_PUSH_FREQ_SEC &&
     sendCommandAndReadSerialResponse("pwr") == true)
  {
    static batteryStack lastSentData; //this is the last state we sent to MQTT, used to prevent sending the same data over and over again
    static unsigned int callCnt = 0;
    
    parsePwrResponse(g_szRecvBuff);

    bool forceUpdate = (callCnt % 20 == 0); //push all the data every 20th call
    pushBatteryDataToMqtt(lastSentData, forceUpdate);
    
    callCnt++;
    g_lastDataSent = os_getCurrentTimeSec();
    memcpy(&lastSentData, &g_stack, sizeof(batteryStack));
  }
  
  mqttClient.loop();
}

#endif //ENABLE_MQTT

miscellaneous and news

Integrating the heat pump (NEURA) into the Smarthome

10. March 2023 ingmarsretro 3 Comments

A smarthome is no longer a rarity today and is very widespread. There are countless systems on the market that make your own home “smart”. The digital voice assistants from Google, Amazon and co. in conjunction with smart light bulbs are among the systems that are easy and quick to install. But there are also complex smart home systems, in which actuators for every lamp and socket are installed in the house distributors. The windows and doors are equipped with signaling contacts and secure the home or report if once forgotten to close the windows after shock ventilation. It goes without saying that these systems also contribute to energy optimization when programmed sensibly. I also operate Smarthome components from various manufacturers.

For years, this has included the HomeMatic system, which communicates with its actuators and sensors both wired and via the Bidcos protocol. The HUE system from Phillips talks to its smart lamps and sockets via ZigBee. The gateways of these systems are connected to a LAN network and each system brings its own web server, through which it can then be controlled and set. An inverter of photovoltaic systems can provide its data via different interfaces (RS485, CAN, RS232). To bring all of them to a central display level, I decided to use the NodeRed system. The necessary NodeRed server runs on a Raspberry PI. (On the CCU3 with the Raspbian image is still enough space to run the NodeRed server – it is even available as a separate plugin for the CCU and is called “RedMatic”). With this configuration you can “slay” almost everything in the field of home automation. With ESP32 and Raspberry you can easily transfer status information via MQTT (Message Queueing Telemetry Transport). I use this for example with the small feed-in inverters of a balcony PV system, as well as with the PV inverters of an offgrid system. Here the data is received via different bus systems in the Raspberry or ESP32 and converted into the MQTT protocol. The MQTT broker collects the data from the individual devices and via NodeRed they can then be written to a database, visualized in the browser or on the smartphone and also easily processed in the HomeMatic system, as required.

Thus, it is possible to network almost all systems with each other smartly and, importantly for me, to visualize them on ONE platform. One single system was missing until now. That is my old Neura heating heat pump. The company Neura has not existed for several years and the web server “webidalog” developed by “b.i.t.” has never been updated. So the heat pump has a web server on a small with Linux computer onboard and builds the web application with an ancient Java version. For the operation a Java Runtime must be installed on the PC, which runs only with some tricks on a current Windows computer (keyword: virtualization). For the operation via a smartphone an html – version with limited functionality is available. My plan now was to find an interface, with which I can read out the data of the heat pump at least once, in order to have flow- return temperatures of the floor heating, boiler temperature, etc. also available in my NodeRed system. But since there is almost no documentation for the system and reverse engineering is a bit critical if the system should continue to run, I came up with the following idea:

With a “headles browser” it should be possible to parse the html version of the Neura WebDialog website and find the relevant data and turn it into MQTT topics via variables. And here I have to give a special thanks to my colleague Mario Wehr, who built the software structure to parse the website. The software is written in PHP and finally runs on a Raspberry PI. All you need is a php8-cli runtime and a few modules. The way the software works is that every time the heat pump website is called, a login is executed, then the data is parsed and sent to MQTT broker. The continuous calling of the php script I then simply solved with a cronjob that is executed every minute.