Large Hat + RPI4 +ubuntu 20.01 LTS

[xsherlock]

Hi is there any known problem with the following setup as in the title?

This is the first time I see that, using the TF parts for years now. But I think it is first time I use RPI4.

brickd instals fine.  But there is constant stream of errors. brickv connects and sometimes after 15 min will maybe see one bricklet but it will timeout a second later.

when run in a foreground  the log looks the flowing

HELP!

sherlock@morele-sensor1:~$ sudo brickd --version
2.4.2
sherlock@morele-sensor1:~$
sherlock@morele-sensor1:~$
sherlock@morele-sensor1:~$
sherlock@morele-sensor1:~$ sudo brickd --debug brick.log
2020-11-19 18:07:15.686866 <I> <main_linux.c:367> Brick Daemon 2.4.2 started (pid: 2095, daemonized: 0)
2020-11-19 18:07:15.687000 <W> <log.c:115> Unexpected char 'b' in debug filter 'brick.log' at index 0
2020-11-19 18:07:15.693281 <I> <bricklet.c:270> Found supported HAT product_id 0x084e in device tree, using default HAT Brick config
2020-11-19 18:07:15.693374 <I> <bricklet.c:311> Found Bricklet port A (spidev: /dev/spidev0.0, driver: gpio, name: gpio23, num: 23)
2020-11-19 18:07:15.693463 <I> <bricklet_stack_linux.c:117> Using BCM2835 backend for Bricklets (Raspberry Pi detected)
2020-11-19 18:07:15.697455 <I> <bricklet.c:311> Found Bricklet port B (spidev: /dev/spidev0.0, driver: gpio, name: gpio22, num: 22)
2020-11-19 18:07:15.697624 <I> <bricklet.c:311> Found Bricklet port C (spidev: /dev/spidev0.0, driver: gpio, name: gpio25, num: 25)
2020-11-19 18:07:15.697772 <I> <bricklet.c:311> Found Bricklet port D (spidev: /dev/spidev0.0, driver: gpio, name: gpio26, num: 26)
2020-11-19 18:07:15.697905 <I> <bricklet.c:311> Found Bricklet port E (spidev: /dev/spidev0.0, driver: gpio, name: gpio27, num: 27)
2020-11-19 18:07:15.698071 <I> <bricklet.c:311> Found Bricklet port F (spidev: /dev/spidev0.0, driver: gpio, name: gpio24, num: 24)
2020-11-19 18:07:15.698211 <I> <bricklet.c:311> Found Bricklet port G (spidev: /dev/spidev0.0, driver: gpio, name: gpio7, num: 7)
2020-11-19 18:07:15.698361 <I> <bricklet.c:311> Found Bricklet port H (spidev: /dev/spidev0.0, driver: gpio, name: gpio6, num: 6)
2020-11-19 18:07:15.698504 <I> <bricklet.c:311> Found Bricklet port I (spidev: /dev/spidev0.0, driver: gpio, name: gpio5, num: 5)
2020-11-19 18:07:15.700835 <E> <bricklet_stack.c:396> Frame error (port: I, count: 1)
2020-11-19 18:07:15.705105 <E> <bricklet_stack.c:396> Frame error (port: I, count: 3)
2020-11-19 18:07:15.711404 <E> <bricklet_stack.c:396> Frame error (port: I, count: 4)
2020-11-19 18:07:15.717774 <E> <bricklet_stack.c:396> Frame error (port: I, count: 7)
2020-11-19 18:07:15.724422 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 1)
2020-11-19 18:07:15.726721 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 2)
2020-11-19 18:07:15.733134 <E> <bricklet_stack.c:396> Frame error (port: I, count: 10)
2020-11-19 18:07:15.739503 <E> <bricklet_stack.c:396> Frame error (port: I, count: 13)
2020-11-19 18:07:15.745844 <E> <bricklet_stack.c:396> Frame error (port: I, count: 14)
2020-11-19 18:07:15.750116 <E> <bricklet_stack.c:396> Frame error (port: I, count: 15)
2020-11-19 18:07:15.758759 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 3)
2020-11-19 18:07:15.763057 <E> <bricklet_stack.c:396> Frame error (port: I, count: 19)
2020-11-19 18:07:15.771756 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 4)
2020-11-19 18:07:15.773989 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 5)
2020-11-19 18:07:15.776196 <E> <bricklet_stack.c:396> Frame error (port: I, count: 21)
2020-11-19 18:07:15.782560 <E> <bricklet_stack.c:396> Frame error (port: I, count: 22)
2020-11-19 18:07:15.791091 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 6)
2020-11-19 18:07:15.793300 <E> <bricklet_stack.c:396> Frame error (port: I, count: 24)
2020-11-19 18:07:15.801987 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 7)
2020-11-19 18:07:15.806293 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 8)
2020-11-19 18:07:15.810655 <E> <bricklet_stack.c:396> Frame error (port: I, count: 26)
2020-11-19 18:07:15.817000 <E> <bricklet_stack.c:396> Frame error (port: I, count: 28)
2020-11-19 18:07:15.821322 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 9)
2020-11-19 18:07:15.823546 <E> <bricklet_stack.c:396> Frame error (port: I, count: 29)
2020-11-19 18:07:15.829948 <E> <bricklet_stack.c:396> Frame error (port: I, count: 31)
2020-11-19 18:07:15.836591 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 10)
2020-11-19 18:07:15.838819 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 11)
2020-11-19 18:07:15.841033 <E> <bricklet_stack.c:396> Frame error (port: I, count: 32)
2020-11-19 18:07:15.845333 <E> <bricklet_stack.c:396> Frame error (port: I, count: 33)
2020-11-19 18:07:15.853887 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 12)
2020-11-19 18:07:15.856091 <E> <bricklet_stack.c:396> Frame error (port: I, count: 36)
2020-11-19 18:07:15.864805 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 13)
2020-11-19 18:07:15.867110 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 14)
2020-11-19 18:07:15.875601 <E> <bricklet_stack.c:396> Frame error (port: I, count: 40)
2020-11-19 18:07:15.884162 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 15)

-----------------------

Solved by just installing it on the RPI OS instead, and everything is just normal...

bearbeitet November 19, 2020 at 21:03 von xsherlock

[photron]

Brick Daemon expects the Raspberry Pi core_freq to be 250, but with the Raspberry Pi 4 core_freq is 500 by default. The SPI clock is based on the core_freq. With a core_freq twice as fast as expected the SPI clock is also twice as fast. This is too fast for the Bricklets.

This is a somewhat known problem. But our documentation doesn't cover this well. You can try forcing the core_freq to 250 in Ubuntu by editing /boot/firmware/config.txt and adding, but the Raspberry Pi documentation warns about doing this on a Raspberry Pi 4, as it might result in boot problems:

core_freq=250

But this might affect other things that expect the core_freq to be 500.

I'll fix this in brickd, to not assume the core_freq to be 250 when calculating the SPI clock, but read the actual core_freq value.

[photron]

Okay, the whole core_freq situation is way more complex. That it works for you with Raspberry Pi Os is just luck I think. Sorry, for the mess, I working on improving this situation.

On RPi 4 core_freq can be 500, 550 or 360 depending on the TV-Out and HDMI config. The core_freq_min can be 250 or 275. The kernel will dynamically change the actual core frequency depending on the load.

On RPi 1 and 2 core_freq and core_freq_min are fixed 250, no problem there.

On RPi 0/W and 3A/B+ core_freq is 400 and core_freq_min is 250.

In brickd 2.4.2 we switched to a new backend for SPI communication with the HAT (Zero) Brick. It seems that the old backend dealt correctly with this dynamic and higher core frequency, but the new one just assumes it to be fixed at 250 and fails in cases where core_freq is higher than 250.

[photron]

Please test the attached brickd version with Ubuntu. The arm64 variant is for Ubuntu, the armhf variant is for Raspberry Pi OS.

This version fixes the Problem for me with Raspberry Pi 4 and Ubuntu. There is no need to modify core_freq or anything else. This should work with an unmodified OS image.

With this version you should not see these kinds of messages in the log file anymore:

2020-11-19 18:07:15.845333 <E> <bricklet_stack.c:396> Frame error (port: I, count: 33)
2020-11-19 18:07:15.853887 <E> <bricklet_stack.c:478> Message checksum error (port: I, count: 12)

brickd_2.4.2+snapshot~7af86f9_armhf.deb brickd_2.4.2+snapshot~7af86f9_arm64.deb

[photron]

I've found the difference between Raspberry Pi OS and Ubuntu. Raspberry Pi OS scales down the core frequency to 200 if there is not much load on the CPU. Ubuntu keeps the core frequency at 500 all the time, resulting the SPI clock being to fast all the time.

[xsherlock]

so you say that RPI OS can fail any time if I put load on the CPU?

[photron]

12 minutes ago, xsherlock said:

so you say that RPI OS can fail any time if I put load on the CPU?

Yes, with brickd 2.4.2 the SPI frequency setting is broken in brickd and it might work for you under certain conditions, but might just fail under other conditions.

Please test the attached snapshot versions of brickd in the post above. These have the SPI frequency setting logic fixed. In my tests here this works without problems with Ubuntu on Raspberry Pi 4. The original 2.4.2 version doesn't work there because the SPI frequency is way too high there because of a bug in brickd.

[photron]

I tested Raspberry Pi 4 with Raspberry Pi OS with brickd 2.4.2 and "stress -c 4". I can see that the SPI clock is 2x faster than expected and there are errors in the brickd.log, but the error rate is still low enough that the error recovery mechanisms in brickd can deal with it. But this is not a recommended setup, it's barely working and on the edge to failing.

[xsherlock]

ok installing, now,  I have a 3 way valve governing 2000l   of boiling hot water  on that one RPi to manage

[xsherlock]

Ok so I have it working with the new snapshot version but I had today a very strange case where the RPI booted and  all 4 x PTC sensors started returning -246.88 readout  all 4 the same readout while the remaining 2 bricklets (VC and Humidity2) on the same large HAT were fine. A reboot not helped but removing power from the HAT fixed it.

[photron]

Do you run Raspberry Pi OS or Ubuntu at the moment?

The -246.88 is the value you get if the sensor input on the PTC Bricklet 2.0 is shorted.

[xsherlock]

RPI OS with the snapshot brickd

pi@morele-sensor-1:~ $ uname -a
Linux morele-sensor-1 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

Nothing could be short as it quite fixed on the test stand, and has not been touched except of the removing the power to RPI that fixed problem. Suprisingly all 4 ptc behaved the same.

bearbeitet November 26, 2020 at 11:32 von xsherlock

[photron]

Did the PTC problem occur once or is it reproducible?

What kind of power supply do you use to feed the black power input on the HAT Brick? Could the system be underpowered?

[xsherlock]

The global source for the system is 100W Meanwell 12V  AC PSU with battery backup.

For the RPI's  I use 60W Meanwell  12V to 5V converter that feeds 5.1V into  sensor  RPI over HAT power port  and one other Openhab RPI over USB-C. 

I now see that HAT power port is indeed has spec from 5.3V volts up. That could be it. I will power it directly from 12V line  to be safe.

So far it happend only once.