integration of TinkerForge into HomeAssistant - Architecture questions...

[CChris]

Hi,

in the past (a few years ago), I've tried to create some smaller project with Tinkerforge - but honestly speaking, I never really finished them.
I tried to wirte some software in C# - but in the end, it did work - but never as stable as it was required for the use case - so I quit the projects and realized them with other components and hardware.

Since then, my hardware was just laying around and collected dust...

Now, I've decided to start again - but with a different approach...

I am using HomeAssistant for getting information about our solar-production and energy usage - have my ev charging-station implemented and some other "smarthome" components... so now, I am thinking to create another project with my current tinkerforge hardware and integrate this into homeassistant, too.

I already learned that there are some MQTT Bindings available - And, I could use MQTT in HomeAssistant...
But, there are some questions left about the best "approach"...

Now, I am struggling a bit with the decission, how to realize this project...

for now, I have created a debian 11 lxc container in Proxmox - running brickd...
A first python script, connecting to one of my bricks and barometer bricklet over tcp-ip (wifi 2.0 extension) seems to work so far.

I have also installed the MQTT Bindings so far:

> systemctl --type=service
  UNIT                                 LOAD   ACTIVE SUB     DESCRIPTION
  brickd.service                       loaded active running Brick Daemon
  tinkerforge_mqtt.service             loaded active running Tinkerforge MQTT API Bindings

Now I am struggling a bit...
Honesty speaking, I don't really see the correct approach right now...

As far as I could get, the MQTT Bindings are based on the Python bindings.
The tinkerforge_mqtt which comes with the bindings is "just" a python script, which does - more or less collect data from the bricklets - and send it to an mqtt broker.
Also, it listens to topics from the broker and send this to the brick / Bricklets...

Correct so far?
In this case, it is pretty much what I had first in mind - I need to write a python script that reads the data - and send it to the Broker...

So... how can I use the MQTT Bindings now?
I have found the following under /etc

tinkerforge_mqtt.cmdline

sorry - if this may sound dumb... but honestly speaking the last time I had the Tinkerforge stuff in my hands was somewhere in 2016...

[CChris]

ok, I think, I got it somehow... but unfortunately, my tests aren't running.

I am trying to do the follwing:

/usr/local/bin# >python3 tinkerforge_mqtt --ipcon-host 123.456.78.9 --ipcon-port 4215 --broker-host 123.456.78.10 --broker-port 1883 --broker-username myusername --broker-password mypassword_1120 --debug --show-payload --init-file tinkerforge_mqtt_test.txt

I am pretty sure that my credentials are correct in the command, but the response from the Broker is:

MQTT bindings: Connected to MQTT broker at 123.456.78.10:1883
paho.mqtt.client: Sending CONNECT (u1, p1, wr0, wq0, wf1, c1, k60) client_id=b''
phao.mqtt.client: Received CONNACK (0,5)
MQTT bindings: Failed to connect to mqtt broker: Connection Refused: not authorised.

What I've also noticed:
When I hit the arrow-up key to bring the command back, the password that has been given in the arguments is missing some characters or has some irregular chars which I haven't typed in earlier...

I can connect to the broker with these credentials without any issues with MQTT Explorer... so, any idea what could be wrong here?

Just for reference:
That's the content of my tinkerforge_mqtt_test.txt

{
    "tinkerforge/request/barometer_bricklet/UID/get_air_pressure": {"register": true},
    "tinkerforge/request/barometer_bricklet/UID/set_air_pressure_callback_period": {"period": 1000},
    
    "tinkerforge/request/barometer_bricklet/UID/get_altitude": {"register": true},
    "tinkerforge/request/barometer_bricklet/UID/set_altitude_callback_period": {"period": 1000},

    "tinkerforge/callback/barometer_bricklet/UID/air_pressure": "",
    "tinkerforge/callback/barometer_bricklet/UID/altitude": ""
}

 

[CChris]

some additions to the above...

When I don't get the "Connection refused" - I am getting other error messages from the command:
It seems, that the script does throw the arguments somehow together...

This was the command I've typed:

> python3 /usr/local/bin/tinkerforge_mqtt --ipcon-host 192.168.52.55 --ipcon-port 4215 --broker-host 192.168.52.248 --broker-port 1883 --broker-username admin --broker-password <password> --broker-tls-insecure --debug --show-payload --init-file barometer_test.txt

This is the error I am getting:

I don't know, where this is comming from... but when I press the arrow-up key to get the command, it has changed and (as mentioned above) mixes arguments together.
Could this be a case, because my password does contain special characters like _ ! and ?

tinkerforge_mqtt: error: unrecognized arguments: /usr/local/bin/tinkerforge_mqtt /usr/local/bin/tinkerforge_mqtt --h?!

[CChris]

I think, somehow the script can't handle " !! " which is part of my broker-password.
This will always be replaced by one of the previous commands I've entered - for example, it will be replaced with "nano /usr/local/bin/tinkerforge_mqtt....

that was it.
After changing the password on my broker - it did work...

So, the tinkerforge_mqtt can't handle an --broker-password argument with !! in it

bearbeitet November 7, 2022 at 19:03 von CChris

[photron]

15 hours ago, CChris said:

So, the tinkerforge_mqtt can't handle an --broker-password argument with !! in it

Nothing to do with our MQTT bindings. The shell you run the command in replaces !! with the previous command. You need escape !! like this:

--broker-password 'mypassword!!'

[CChris]

vor 1 Minute schrieb photron:

Nothing to do with our MQTT bindings. The shell you run the command in replaces !! with the previous command. You need escape !! like this:

--broker-password 'mypassword!!'

yeah, sorry - I already figured this out :D
It was a bit strange, because I had somehow in mind that using the password did work in other cases without "escaping" it ... *head on table*

[CChris]

ok, I have made some first basic steps... this is still a very very basic python script for testing...
 

# connection parameter
HOST = "192.168.52.55"
PORT = 4215

# these needs to be used dynamically later...
# right now, they are only for testing
BARO_1_UID = "vNB"
BARO_2_UID = "vHa"

from tinkerforge.ip_connection import IPConnection
from tinkerforge.bricklet_barometer import BrickletBarometer

def cb_enumerate(uid, connected_uid, position, hardware_version, firmware_version, devic>
# removed output for shorter code here

def cb_air_pressure(air_pressure):
    # here, I would like to output from which bricklet the information came
	# since we do have two barometers connected, the output should somehow include the uid of the bricklet...?
    print("Air Pressure:         " + str(air_pressure/1000.0) +  " hPA")

def cb_altitude(altitude):
    # here, I would like to output from which bricklet the information came
	# since we do have two barometers connected, the output should somehow include the uid of the bricklet...?
    print("Altitude:             " + str(altitude/100.0) + " m")

if __name__ == "__main__":
    ipcon = IPConnection() # Create IP connection
    
    # connect to the barometer bricklet(s)
	# if multiple devices with the same device-id are found, it should be done dynamically
	# f.e. for each barometer in barometers... something like that.
    b1 = BrickletBarometer(BARO_1_UID, ipcon)
    b2 = BrickletBarometer(BARO_2_UID, ipcon)

    ipcon.connect(HOST, PORT) # Connect to brickd or to network attached device
    ipcon.register_callback(IPConnection.CALLBACK_ENUMERATE, cb_enumerate)
    ipcon.enumerate()

    # register the callback(s) and set the callback period(s) for the barometer bricklet(s)
	# if multiple devices with the same device-id are found it should be done dynamically
	# f.e. for each barometer in barometers... something like that
    b1.register_callback(b1.CALLBACK_AIR_PRESSURE, cb_air_pressure)
    b2.register_callback(b2.CALLBACK_AIR_PRESSURE, cb_air_pressure)
    b1.register_callback(b1.CALLBACK_ALTITUDE, cb_altitude)
    b2.register_callback(b2.CALLBACK_ALTITUDE, cb_altitude)
    b1.set_air_pressure_callback_period(1000)
    b2.set_air_pressure_callback_period(1000)
    b1.set_altitude_callback_period(1000)
    b2.set_altitude_callback_period(1000)

I would like to know, if it is somehow possible to be a bit more dynamic with the detection of the connected bricklets?
For example, I have two barometers connected.

I would like to output the UID of the bricklet within the cb_air_pressure and cb_altitude into something like that:

def cb_air_pressure(air_pressure):   
    # something like that:
    print("Air Pressure " + UID + ": .....")

def cb_altitude(altitude):  
    # something like that:
    print("Altitude " + UID + ": .....")
    

Also, I am pretty sure that this can be reduced and somehow be done for each device recognized:

    b1 = BrickletBarometer(BARO_1_UID, ipcon)
    b1.register_callback(b1.CALLBACK_AIR_PRESSURE, cb_air_pressure)
    b1.register_callback(b1.CALLBACK_ALTITUDE, cb_altitude)
    b1.set_air_pressure_callback_period(1000)
    b1.set_altitude_callback_period(1000)
    
    b2 = BrickletBarometer(BARO_2_UID, ipcon)
    b2.register_callback(b2.CALLBACK_AIR_PRESSURE, cb_air_pressure)
    b2.register_callback(b2.CALLBACK_ALTITUDE, cb_altitude)
    b2.set_air_pressure_callback_period(1000)
    b2.set_altitude_callback_period(1000)

probably the syntax is not correct - but maybe, something like that:

foreach(uid in barometer_uids)
{
  b = BrickletBarometer(uid, ipcon)
  b.register....
  b.set......
}

 

[photron]

from tinkerforge.ip_connection import IPConnection
from tinkerforge.bricklet_barometer import BrickletBarometer

HOST = "192.168.52.55"
PORT = 4215

ipcon = IPConnection()
devices = {}

def cb_air_pressure(uid, air_pressure):
    print(f"Air Pressure ({uid}):         {air_pressure / 1000} hPa")

def cb_altitude(uid, altitude):
    print(f"Altitude ({uid}):             {altitude / 100} m")

def cb_enumerate(uid, connected_uid, position, hardware_version, firmware_version,
                 device_identifier, enumeration_type):
    if enumeration_type == IPConnection.ENUMERATION_TYPE_DISCONNECTED:
        return
    
    if device_identifier == BrickletBarometer.DEVICE_IDENTIFIER:
        device = BrickletBarometer(uid, ipcon)

        device.register_callback(BrickletBarometer.CALLBACK_AIR_PRESSURE, lambda *args: cb_air_pressure(uid, *args))
        device.register_callback(BrickletBarometer.CALLBACK_ALTITUDE, lambda *args: cb_altitude(uid, *args))
        device.set_air_pressure_callback_period(1000)
        device.set_altitude_callback_period(1000)
        
        devices[uid] = device

def main():
    ipcon.register_callback(IPConnection.CALLBACK_ENUMERATE, cb_enumerate)
    ipcon.connect(HOST, PORT)
    ipcon.enumerate()

    input("Press Enter to exit\n")

if __name__ == "__main__":
    main()

 

[CChris]

thanks a lot :)
That's perfect and exactly what I need... much appreciated :)

[CChris]

a short progress - update :)

So far, the Software seems to work as expected...
It does enumerate through all devices connected to the master - and is using the callbacks for each sensor...
Since I want to recognize all possible devices, I need to implement the code for each brick and bricklet hardware - which is a lot of "copy & paste" work, since I haven't yet found another way to do this...

# every single device needs to be listed here... see imports for reference
    if enumeration_type == tf_main.IPConnection.ENUMERATION_TYPE_DISCONNECTED:
        return
    
    if device_identifier == tf_main.BrickDC.DEVICE_IDENTIFIER:                   #   11
        device = tf_main.BrickDC(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        # nothing to do, since I don't have a DC Brick for testing

    elif device_identifier == tf_main.BrickMaster.DEVICE_IDENTIFIER:             #   13
        device = tf_main.BrickMaster(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

        device.register_callback(device.CALLBACK_STACK_VOLTAGE, lambda *args: tf_callbacks.cb_stack_voltage(uid, *args))
        device.set_stack_voltage_callback_period(int(config.get('TINKERFORGE', 'CallbackPeriod')))

        device.register_callback(device.CALLBACK_STACK_CURRENT, lambda *args: tf_callbacks.cb_stack_current(uid, *args))
        device.set_stack_current_callback_period(int(config.get('TINKERFORGE', 'CallbackPeriod')))     

    elif device_identifier == tf_main.BrickServo.DEVICE_IDENTIFIER:              #   14
        device = tf_main.BrickServo(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        # nothing to do, since I don't have a DC Brick for testing

    elif device_identifier == tf_main.BrickStepper.DEVICE_IDENTIFIER:            #   15
        device = tf_main.BrickStepper(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        # nothing to do, since I don't have a DC Brick for testing

    elif device_identifier == tf_main.BrickRED.DEVICE_IDENTIFIER:                #   16
        device = tf_main.BrickRED(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        '''
         RED Brick is depricated, due to delivery issues for ICs.
         Replacements are HAT / HAT Zero as well as RaspberryPi compatible devices
         There are several options to controll the Software running on the RED Brick:
         example: start_program(program_id) -> to start a specific program on the RED Brick
         example: continue_program_schedule(program_id)
         example: get_program_schedule(program_id)
         example: set_program_schedule(program_id)
        '''

    elif device_identifier == tf_main.BrickIMU.DEVICE_IDENTIFIER:                #   17
        device = tf_main.BrickIMU(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        # nothing to do, since I don't have a DC Brick for testing

    elif device_identifier == tf_main.BrickIMUV2.DEVICE_IDENTIFIER:              #   18
        device = tf_main.BrickIMUV2(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        # replacement for the IMU 1.0...
        '''
        device.register_callback(device.CALLBACK_ALL_DATA)              # <-- getting all data periodically
        device.register_callback(device.CALLBACK_ACCELERATION)          # <-- getting only acceleration data
        device.register_callback(device.CALLBACK_ANGULAR_VELOCITY)      # <-- getting only angular velocity data
        device.register_callback(device.CALLBACK_GRAVITY_VECTOR)        # <-- getting only gravity data
        device.register_callback(device.CALLBACK_LINEAR_ACCELERATION)   # <-- getting only linear acceleration data
        device.register_callback(device.CALLBACK_MAGNETIC_FIELD)        # <-- getting only magnetic field data
        device.register_callback(device.CALLBACK_ORIENTATION)           # <-- getting only orientation data
        device.register_callback(device.CALLBACK_QUATERNION)            # <-- getting only quaternion data
        device.register_callback(device.CALLBACK_TEMPERATURE)           # <-- getting only temperature data

        device.get_acceleration()   # <-- getting acceleration data (could be required for init)
        device.get_orientation()    # <-- getting orientation data (could be required for init)
        device.get_all_data()       # <-- getting all data (could be required for init)
        '''
        
    elif device_identifier == tf_main.BrickSilentStepper.DEVICE_IDENTIFIER:      #   19
        device = tf_main.BrickSilentStepper(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
        '''
         replacement for the Stepper Brick...
         but do we need Stepper-Information in HomeAssistant?
         we could get some information such as position-reached but probably it won't be possible to controll the
         stepper motor with this software / through HomeAssistant (maybe, we can set some "modes" like)
         set target position / set current position (?) / configure the step-resolution... etc.
        '''

    elif device_identifier == tf_main.BrickletAmbientLight.DEVICE_IDENTIFIER:    #   21
        device = tf_main.BrickletAmbientLight(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

        init_illuminance = device.get_illuminance()
        print("Current Illumination (init): " + str(init_illuminance / 10) + " lx")

        device.register_callback(device.CALLBACK_ILLUMINANCE, lambda *args: tf_callbacks.cb_illumination(uid, *args))
        device.set_illuminance_callback_period(int(config.get('TINKERFORGE', 'CallbackPeriod')))

    elif device_identifier == tf_main.BrickletCurrent12.DEVICE_IDENTIFIER:       #   23
        device = tf_main.BrickletCurrent12(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

    elif device_identifier == tf_main.BrickletCurrent25.DEVICE_IDENTIFIER:       #   24
        device = tf_main.BrickletCurrent25(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

    elif device_identifier == tf_main.BrickletDistanceIR.DEVICE_IDENTIFIER:      #   25
        device = tf_main.BrickletDistanceIR(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

    elif device_identifier == tf_main.BrickletDualRelay.DEVICE_IDENTIFIER:       #   26
        device = tf_main.BrickletDualRelay(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)
		# nothing to do, since I don't have a DC Brick for testing

    elif device_identifier == tf_main.BrickletHumidity.DEVICE_IDENTIFIER:        #   27
        device = tf_main.BrickletHumidity(uid, tf_main.ipcon)
        device_work(device, device.DEVICE_DISPLAY_NAME, uid)

        init_humidity = device.get_humidity()
        print("Current Humidity (init): " + str(init_humidity / 10) + " %RH")

        device.register_callback(device.CALLBACK_HUMIDITY, lambda *args: tf_callbacks.cb_humidity(uid, *args))
        device.set_humidity_callback_period(int(config.get('TINKERFORGE', 'CallbackPeriod')))

	# and so on - for each device that COULD be connected...

def device_work(device, device_display_name, uid):
    identifier = device.get_identity()
    hw_version = identifier[3]
    sw_version = identifier[4]
    print(f"{Fore.GREEN}Connected to: {device_display_name}  {Fore.RED}{uid}{Style.RESET_ALL}")
    tf_log.log_output_info(f"connected to {device_display_name} - {identifier}")
    print("Hardware-Version (" + uid + "): " + write_version(hw_version))
    print("Software-Version (" + uid + "): " + write_version(sw_version))

def write_version(version):
    version_no = str(version).replace(" ", "").replace("(", "").replace(")", "").replace(",",".")
    return version_no

The Console-Output for this initialisation:


after initializing and getting the current state of the sensors,
I am calling the different callbacks for each device - so the states are being updated when the state does change:



So - right now, I am working on the implementation to send these information over MQTT.
Since I want to use the HomeAssistants MQTT Autodiscovery function, I can't use the MQTT Bindings directly...
I have to change the topics and payload...

a first static test was working so far... The MQTT integration has detected a new device - the Barometer Bricklet from tinkerforge.

 

This device is creating the sensor "AirPressure" - and delivers the Firmware and Hardware-Information until now.

I still need to implement the "State" Topic, so that the sensor will get the information ... and then, I need to change the code that this will be done dynamically for each device...
So, still some work to do... and I am still not sure if I can implement a way to receive information from HomeAssistant, since it would be required that homeassistant does know to which topic something needs to be sent... so this would still be a lot of "manual configuration work" in HA... until I could MAYBE build some kind of integration which does create the entities and send the topics to the software.

So, I think, this project can be somehow called a "brickviewer" for HomeAssistant (only viewer, atm)