HA: Use values of any WebGUI as entities

After my Internet access lost connection from time to time, I wanted to monitor the signal quality of the cable router in Home Assistant. Due to lack of API, I use the router's management interface for the query and parse the output in Bash. in Bash. With this approach, the values from almost any web GUI can be made available in Home Assistant.

First, I take a look at the web GUI from which I want to query the data. In the simplest case, this makes the data available without a user login: Which was the case with the previous version of my current router:

The goal is to make the GUI values available in Home-Assistant:

Step by step

As a first step I tried to see if the data can be downloaded in general. For this I downloaded the whole HTML source code of the page using the bash command curl and the router URL. For testing and compiling the command you can use a Linux-PC, the Home-Assistant-Docker-Container or Linux on Windows: WSL can be used.

First try: Downloading the home page using bash command: curl

curl http://192.168.0.1

As a response I get the source code of the start page, this includes the layout of the GUI as well as the status values:

[+]

<td id="ch_snr" scope="col" class="stdbold" nowrap><script language="javascript" type="text/javascript">dw(vdsch_snr);</script></TD>
</tr>
<tr>
<td id="channel_1" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 1<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_1 ch_pwr" scope="row" class="stdbold" nowrap> 13.6 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_1 ch_snr" class="stdbold" nowrap>40.2 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_2" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 2<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_2 ch_pwr" scope="row" class="stdbold" nowrap> 13.1 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_2 ch_snr" class="stdbold" nowrap>39.9 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_3" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 3<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_3 ch_pwr" scope="row" class="stdbold" nowrap> 12.8 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_3 ch_snr" class="stdbold" nowrap>39.9 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_4" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 4<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_4 ch_pwr" scope="row" class="stdbold" nowrap> 11.8 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_4 ch_snr" class="stdbold" nowrap>40.0 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_5" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 5<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_5 ch_pwr" scope="row" class="stdbold" nowrap> 14.0 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_5 ch_snr" class="stdbold" nowrap>40.0 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_6" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 6<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_6 ch_pwr" scope="row" class="stdbold" nowrap> 13.8 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_6 ch_snr" class="stdbold" nowrap>40.4 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_7" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 7<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_7 ch_pwr" scope="row" class="stdbold" nowrap> 13.1 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_7 ch_snr" class="stdbold" nowrap>40.0 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
</tr>
<tr>
<td id="channel_8" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 8<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="channel_8 ch_pwr" scope="row" class="stdbold" nowrap> 12.2 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
<td headers="channel_8 ch_snr" class="stdbold" nowrap>39.9 <script language="javascript" type="text/javascript">dw(vdb);</script></td>
...
                    <td class="Item3">
                        <table class="std" summary="Upstream Channels">
                            <tr> 
<td id="up_empty" width="140" nowrap>&nbsp; </td>
<td id="up_pwr" scope="col" width="140" class="stdbold" nowrap><script language="javascript" type="text/javascript">dw(vch_pwr);</script></td>
</tr>
<tr> 
<td id="up_channel_1" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 1<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="up_channel_1 up_pwr" scope="row" class="stdbold" nowrap>38.3 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
</tr> 
<tr> 
<td id="up_channel_2" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 2<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="up_channel_2 up_pwr" scope="row" class="stdbold" nowrap> 0.0 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
</tr> 
<tr> 
<td id="up_channel_3" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 3<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="up_channel_3 up_pwr" scope="row" class="stdbold" nowrap> 0.0 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
</tr> 
<tr> 
<td id="up_channel_4" width="140" nowrap><script language="javascript" type="text/javascript">dw(vs_channel);</script> 4<script language="javascript" type="text/javascript">dw(vcolon);</script></td>
<td headers="up_channel_4 up_pwr" scope="row" class="stdbold" nowrap> 0.0 <script language="javascript" type="text/javascript">dw(vdbmv);</script></td>
</tr>

To extract the values from the source code, it can be passed to certain bash functions. As an example, "grep" can extract certain passages from the text using regex . It is also possible to pass the text blocks first roughly and then by means of a more detailed definition another time to "grep". For example, the values from the displayed HTML source code can be retrieved and extracted as follows:

curl - call and pass to grep to extract the desired text passages:

[+]

root@DESKTOP-I9DEFGN:~# curl http://192.168.0.1/Docsis_system.asp -s | grep -iE "ch_snr|up_pwr" | grep -Po "nowrap>(.*?)\ <"  | grep -Po "[0-9.]{1,4}" | xargs |  grep -Po "[0-9.]{1,4}"
40.3
39.9
39.9
39.9
40.1
40.4
39.9
39.9
38.3
40.0
39.5
40.5

To read multiple values with one call, these can first be retrieved via the command_line platform and later used as individual entities using template sensors.

Home Assistant command_line Platform

command_line:
  - sensor:
      name: cablelink_channels
      command: curl http://192.168.0.1/Docsis_system.asp -s | grep -iE "ch_snr|up_pwr" | grep -Po "nowrap>(.*?)\ <"  | grep -Po "[0-9.]{1,4}" | xargs
      scan_interval: 30

Die Entität cablelink_channels sammelt alle Werte und trennt diese mit einem Leerzeichen.

Template sensors, extract individual values

Templates can be used to access the individual values by transforming the string into an array via its blanks. The values of the array can then be used via the respective index [0], [1] ... can be used:

[+]

sensor:
  - platform: template
    sensors:      
      cablelink_channel_snr_1:
        friendly_name: 'cablelink_channel_snr_1'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[0] }}'
      cablelink_channel_snr_2:
        friendly_name: 'cablelink_channel_snr_2'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[1] }}'      
      cablelink_channel_snr_3:
        friendly_name: 'cablelink_channel_snr_3'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[2] }}'
      cablelink_channel_snr_4:
        friendly_name: 'cablelink_channel_snr_4'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[3] }}'
      cablelink_channel_snr_5:
        friendly_name: 'cablelink_channel_snr_5'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[4] }}'
      cablelink_channel_snr_6:
        friendly_name: 'cablelink_channel_snr_6'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[5] }}'
      cablelink_channel_snr_7:
        friendly_name: 'cablelink_channel_snr_7'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[6] }}'
      cablelink_channel_snr_8:
        friendly_name: 'cablelink_channel_snr_8'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[7] }}'
      cablelink_channel_up_1:
        friendly_name: 'cablelink_channel_up_1'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[8] }}'
      cablelink_channel_up_2:
        friendly_name: 'cablelink_channel_up_2'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[9] }}'      
      cablelink_channel_up_3:
        friendly_name: 'cablelink_channel_up_3'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[10] }}'
      cablelink_channel_up_4:
        friendly_name: 'cablelink_channel_up_4'        
        unit_of_measurement: 'dB'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[11] }}'

Average, minimum or maximum values

The min_max platform allows to provide the average, maximum or minimum values of certain entities in a separate entity.

[+]

sensor:
  - platform: min_max
    name: cablelink_channels_mean
    type: mean
    round_digits: 2
    entity_ids:
      - sensor.cablelink_channel_snr_1
      - sensor.cablelink_channel_snr_2
      - sensor.cablelink_channel_snr_3
      - sensor.cablelink_channel_snr_4
      - sensor.cablelink_channel_snr_5
      - sensor.cablelink_channel_snr_6
      - sensor.cablelink_channel_snr_7
      - sensor.cablelink_channel_snr_8
      - sensor.cablelink_channel_up_1
      - sensor.cablelink_channel_up_2
      - sensor.cablelink_channel_up_3
      - sensor.cablelink_channel_up_4

If the status page of the WebGui can only be accessed after a user login, we have to take a closer look at the login in the developer tools (F12) with a browser (e.g. Google Chrome):

The "Network" tab tells us how the data for the login process is transferred:

A look into the payload shows us the form data transmitted via "POST":

Converted to a curl command, the login would look like this:

curl -X POST http://192.168.0.1/goform/logon -d "username_login=admin&password_login=mysupersecretpassword&language_selector=en" -c cookies.txt;

Of course, in the command line the password must be adjusted accordingly. The parameter "-c" creates a file with the cookies used for the session, i.e. the information for the login that took place. If we use this parameter for another query, curl is logged in to the GUI.

Now we can get the details for our monitoring from the status page: http://192.168.0.1/st_docsis.html

The goal is again to provide the values of the status page in Home Assistant:

The following command line performs a login and downloads the complete HTML source code of the status page:

curl -X POST http://192.168.0.1/goform/logon -d "username_login=admin&password_login=???&language_selector=en" -c cookies.txt; curl http://192.168.0.1/st_docsis.html -b cookies.txt

The HTML source code can then be parsed again using Linux functions such as grep, awk and xargs. The goal is to extract the desired values:

In summary, the call including login and parsing of the status page in the homeassistant config looks like this:

File: configuration.yml

...
command_line:
  - sensor:
      name: cablelink_channels
      command: curl -X POST http://192.168.0.1/goform/logon -d "username_login=admin&password_login=??&language_selector=en" -c cookies.txt; curl http://192.168.0.1/st_docsis.html -b cookies.txt | grep -Po "(.*?)" | awk '{print substr($NF,0,30)}' | grep -Po "[\-0-9]{1,3}[.][0-9]" | xargs
      scan_interval: 60
...

The command collects all channel data in a sensor named cablelink_channels and separates them with a space, appropriate templates can then extract the individual values:

[+]

...
sensor:
  - platform: template
    sensors:
      cablelink_channel_down_1:
        friendly_name: 'cablelink_channel_down_1'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[0] if states("sensor.cablelink_channels").split(" ") | count > 1 else 0 }}'
      cablelink_channel_down_2:
        friendly_name: 'cablelink_channel_down_2'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[1] if states("sensor.cablelink_channels").split(" ") | count > 1 else 0 }}'      
      cablelink_channel_down_3:
        friendly_name: 'cablelink_channel_down_3'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[2] if states("sensor.cablelink_channels").split(" ") | count > 2 else 0}}'
      cablelink_channel_down_4:
        friendly_name: 'cablelink_channel_down_4'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[3] if states("sensor.cablelink_channels").split(" ") | count > 3 else 0}}'
      cablelink_channel_down_5:
        friendly_name: 'cablelink_channel_down_5'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[4] if states("sensor.cablelink_channels").split(" ") | count > 4 else 0 }}'
      cablelink_channel_down_6:
        friendly_name: 'cablelink_channel_down_6'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[5] if states("sensor.cablelink_channels").split(" ") | count > 5 else 0 }}'      
      cablelink_channel_down_7:
        friendly_name: 'cablelink_channel_down_7'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[6] if states("sensor.cablelink_channels").split(" ") | count > 6 else 0}}'
      cablelink_channel_down_8:
        friendly_name: 'cablelink_channel_down_8'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[7] if states("sensor.cablelink_channels").split(" ") | count > 7 else 0}}'
      cablelink_channel_down_9:
        friendly_name: 'cablelink_channel_down_9'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[8] if states("sensor.cablelink_channels").split(" ") | count > 8 else 0 }}'
      cablelink_channel_down_10:
        friendly_name: 'cablelink_channel_down_10'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[9] if states("sensor.cablelink_channels").split(" ") | count > 9 else 0 }}'      
      cablelink_channel_down_11:
        friendly_name: 'cablelink_channel_down_11'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[10] if states("sensor.cablelink_channels").split(" ") | count > 10 else 0}}'
      cablelink_channel_down_12:
        friendly_name: 'cablelink_channel_down_12'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[11] if states("sensor.cablelink_channels").split(" ") | count > 11 else 0}}'
      cablelink_channel_down_13:
        friendly_name: 'cablelink_channel_down_13'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[12] if states("sensor.cablelink_channels").split(" ") | count > 12 else 0 }}'
      cablelink_channel_down_14:
        friendly_name: 'cablelink_channel_down_14'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[13] if states("sensor.cablelink_channels").split(" ") | count > 13 else 0 }}'      
      cablelink_channel_down_15:
        friendly_name: 'cablelink_channel_down_15'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[14] if states("sensor.cablelink_channels").split(" ") | count > 14 else 0}}'
      cablelink_channel_down_16:
        friendly_name: 'cablelink_channel_down_16'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[15] if states("sensor.cablelink_channels").split(" ") | count > 15 else 0}}'



      cablelink_channel_up_1:
        friendly_name: 'cablelink_channel_up_1'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[36] if states("sensor.cablelink_channels").split(" ") | count > 36 else 0 }}'
      cablelink_channel_up_2:
        friendly_name: 'cablelink_channel_up_2'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[37] if states("sensor.cablelink_channels").split(" ") | count > 37 else 0 }}'      
      cablelink_channel_up_3:
        friendly_name: 'cablelink_channel_up_3'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[38] if states("sensor.cablelink_channels").split(" ") | count > 38 else 0}}'
      cablelink_channel_up_4:
        friendly_name: 'cablelink_channel_up_4'        
        unit_of_measurement: 'dBmV'
        value_template: '{{states("sensor.cablelink_channels").split(" ")[39] if states("sensor.cablelink_channels").split(" ") | count > 39 else 0}}'
...

states("sensor.cablelink_channels").split(" ") creates an array with the individual values. The respective values can be extracted with its index [0], [1].

Developer tools: Template testing

For testing and compiling the templates, i.e. extracting the individual values, the developer tools in Home-Assistant help.

Under "TEMPLATE" the "value_template" values used in the configuration.yaml can be tested:

Example 3: Performance data: Deye SUN800G3-EU-230

Another example of reading a webgui I tested with the Deye inverter, see: Deye PV Balcony Power Plant: Commissioning and HA Integration.

Example 4: OeMag - market price

Another example is the feed-in tariff for a PV system in Austria. The market price for feeding into the Ömag can be taken from the following website: https://www.oem-ag.at/de/marktpreis. A look into the developer tools of the browser reveals the HTML source code and helps to compile the bash command:

I was able to read the tariff with the following command:

curl https://www.oem-ag.at/de/marktpreis/ | grep -Po "px\">(.*?) Cent/kWh" | grep -Po "[0-9,]{3,7}"

Using an "echo" command and passing it to "bc", the value can be converted from cents to euros:

I have stored the following sensor in the configuration.yaml:

command_line:
  - sensor:
      name: oemag_marktpreis
      command: echo $(curl https://www.oem-ag.at/de/marktpreis/ | grep -Po "px\">(.*?) Cent/kWh" | grep -Po "[0-9,]{3,7}" | sed 's/,/./g') / 100 | bc -l
      unit_of_measurement: 'EUR/kWh'
      scan_interval: 21600

Alternatively, I am currently testing the values from E-Control on this page, see: current Oemag market price.

The value can now be used in the energy dashboard and shows the feed-in credit in the statistics:

If the sensor is used in an automation, a message can be sent to the cell phone in case of a value change:

alias: Alert OeMAG-Markpreis change
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.oemag_marktpreis
    for:
      hours: 12
      minutes: 0
      seconds: 0
condition: []
action:
  - service: notify.mobile_app_???
    data:
      title: OeMAG-Marktpreis-Change
      message: "{{states('sensor.oemag_marktpreis') | round(6)}}€/kWh"
mode: single

For more information on Automations, see: Home Assistant Automation - Possibilities & Basics

Integration: Scrape

For single values, the integration "Scrape" can be used as an alternative to the command_line sensor. Scrape works with CSS selectors and can thus extract individual values from the web pages:

The following input mask can be used to store the details for calling the web page:

Even though extracting via CSS selectors certainly offers added value, in my opinion Bash commands can still be tested much more easily in a Linux OS before they are used in Home-Assistant.

Conclusion

The command-line platform in Home Assistant can basically do everything that Home Assistant's operating system allows. Among other things, Linux commands can download certain web pages and extract their data, allowing them to be made available in Home Assistant without corresponding integration.

^{({{pro_count}})}

Rate Post:
{{percentage}} % positive

^{({{con_count}})}

THANK YOU for your review!

created by Bernhard | Updated: 2024-04-24 | Übersetzung Deutsch |🔔 | Comments:2

➨ OpenWRT & Home Assistant: evaluate, visualize, automate | ➦ Integrations | Node-RED in Docker and Home Assistant integration ➨

Questions / Comments

(sorted by rating / date) [all comments(newest first)]

✍anonym

2023-10-01 17:58

Thanks for this great Samples, especially the ÖMAG value, that I was searching for a long time. However the Ömag value is delyed on the named website. I used the following command:
echo $(curl https://www.e-control.at/de/marktteilnehmer/oeko-energie/marktpreis/ | grep -Po "demzufolge (.*?) Euro/MWh" | grep -Po "[0-9,]{3,7}" | sed 's/,/./g') / 1000 | bc -l
to grab it from a different site. The remaining problem is, that THIS price is updated 1-2 days BEFORE the effective change. Without matching the current quarter with the one on the website, the value is still wrong for a few days.

If you have a suggestion how to circumvent that, this would be a great addition.

↳

✍Bernhard

comment date 03.10.2023 10:37

Thank you very much for your Input.
I started testing some logic on this site. The test runs on this page: https://www.libe.net/oemag

created by Bernhard

Table of Contents

Example 1: GUI -> curl without a login and multiple values with one call

Step by step

Home Assistant command_line Platform
Template sensors, extract individual values
Average, minimum or maximum values

Example 2: Web GUI with user login: Technicolor Router (Salzburg AG Cablelink).

Developer tools: Template testing

Example 3: Performance data: Deye SUN800G3-EU-230
Example 4: OeMag - market price
Integration: Scrape
Conclusion

[Deutsch]

Comments:2

➦ Integrations ➨ Node-RED in Docker and Home Assistant integration