GoPiGo3 on Raspberry Pi 5 - GPIO Tales

GoPiGo
1

I want to, but somehow rarely follow the tried and true. It seems I have “stepped into it” again.
First, it was the gpg3_power.py throwing

# pi@GoPi5Go:~/GoPi5Go/systests/gpg_power_service $ python3 gopigo3_power.py 
# Traceback (most recent call last):
#   File "/home/pi/GoPi5Go/systests/gpg_power_service/gopigo3_power.py", line 23, in <module>
#     GPIO.setup(22, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
# RuntimeError: Cannot determine SOC peripheral base address

due to the RPi Foundation shipping PiOS Bookworm with a non-working RPi.GPIO package.

I rewrote the gopigo3_power.py as gopi5go_power.py using lgpio but today I found out I could have simply uninstalled the non-working version and installed a working version:

echo -e "Removing non-working RPi.GPIO supplied with Bookworm"
sudo apt remove python3-rpi.gpio
echo-e "Installing working RPi.GPIO "
sudo pip3 install rpi-lgpio --break-system-packages

Next, it was the issue of the GoPiGo3 class needing pigpiod, (which is not available for Pi5), to setup the SPI pins on the Raspberry Pi GPIO connector for ALT0 mode. I commented out those six lines, and found that the PiOS Bookworm, (only PiOS version that runs on the Raspberry Pi 5), already sets the SPI pins to ALT0 mode. After building wiringPi from source on 64-bit PiOS Bookworm gpio readall shows the pins are indeed set to ALT0 mode without my doing:

pi@GoPi5Go:~/GoPi5Go/systests/wiringpi $ gpio readall
 +-----+-----+---------+------+---+---Pi 5---+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 |     |     |    3.3v |      |   |  1 || 2  |   |      | 5v      |     |     |
 ...
 |  10 |  12 |    MOSI | ALT0 | 0 | 19 || 20 |   |      | 0v      |     |     |
 |   9 |  13 |    MISO | ALT0 | 0 | 21 || 22 | 0 |  -   | GPIO. 6 | 6   | 25  |
 |  11 |  14 |    SCLK | ALT0 | 0 | 23 || 24 | 1 | OUT  | CE0     | 10  | 8   |
...
 +-----+-----+---------+------+---+----++----+---+------+---------+-----+-----+
 | BCM | wPi |   Name  | Mode | V | Physical | V | Mode | Name    | wPi | BCM |
 +-----+-----+---------+------+---+---Pi 5---+---+------+---------+-----+-----+

So not an issue (I still think).

BUT, hold on a second… during an overnight test gopigo3.spi_read_16() which is used by gopigo3.get_voltage_battery() threw: OSError: No SPI Response and quit that test program.

SPI is supposed to be thread safe, so what gives. Apparently Dexter Industries thought there could be problems and wrote a check, but in 7 years of multi-threaded SPI calls on Carl with a Pi 3B and Pi3B+ he never complained.

I found only one person reporting SPI on the Pi5 having an issue - "Pi 5 SPI transaction latency under PREEMPT_RT and heavy loads". Long transfers is different than returning without the result ready, but does point to SPI acting differently on Pi5 due to the new RP1 chip handling the GPIO interfaces.

I ran a GoPiGo3 SPI get_voltage_battery test of over 80 million battery checks by four simultaneous running threads using a special “SPI fault tolerant” gopigo3.py , there were no SPI errors reported (but my safetyShutdown.py program that monitors battery health quietly shutdown without a way for me to know why…).

Life on the edge is never dull.

2 Likes
2

I modified my battery.get_vbatt_vreading(egpg) function to catch all exceptions, log them, and continue without blowing up the calling program.

Last night safetyShutdown stayed alive and logged one “No SPI Response” exception at 5:43 AM. Also interesting because it was the only GoPiGo3 application running, so the problem is not related to multi-threading and not related to processor load.

2 Likes
3

Another night, another single exception:

2024-04-01 23:30|[battery.py.vBatt_vReading]Exception OSError: No SPI response vReading:0.00 continuing

I don’t know if this is a Pi5 hardware issue or the PiOS Bookworm spidev software issue, but at one soft failure a day, I am going to live with my work-around (log it and continue).

Further adventures watching electrons jump into the bit-bucket

Returning to “Characterizing Battery Life” I wrote pctRemaining_from_vBatt() based on the prior batteryLife test and resulting curve. I’m not happy with the resulting estimates, so I’m re-running the batteryLife test.

The whole concept of battery life remaining is a bit of a mystery to me - I think it really needs to be based on a some combination of a summation of the average power being used over the time-degraded total power available in the battery. I have an IN728 current sensor but don’t really want to complicate the power wiring for estimates that I am not actually going to use programmatically. (Create3-Wali had it and I found myself obsessed with checking it often to know how he was doing, but only the full and get on the dock percentages were used by his program.) The reality is I have the 15 minute warning and the safety shutdown working.

2 Likes
4
  • Performed another full charge to battery cutoff test
  • Plugged vBatt vs pct_time_till_cutoff numbers from this test into my battery.pctRemaining_from_vBatt
  • Performed a full charge to safety shutdown at 9.75v
********* GoPi5Go-Dave STATUS *****
2024-04-03  11:13:44 up 23:17,  4 users,  load average: 0.00, 0.00, 0.00
Current Battery 9.4v  2%  (Reading 8.69v)
WARNING - Battery Is Nearing Shutdown Voltage
5v Supply: 4.98
Processor Temp: 47.2'C
Clock Frequency: 1.50 GHz
throttled=0x0

Broadcast message from root@GoPi5Go (Wed 2024-04-03 11:13:47 EDT):

The system will power off now!

The shutdown at vBatt<9.75v occurred at 2% remaining which is about 5 minutes before the battery cutoff. (The 2% remaining is the same safety shutdown point iRobot chose for the Create3 platform.)

The 10.25v battery low flashing light and “Put me on the dock” announcement occurred 10 minutes before the safety shutdown and was reporting 8% battery remaining.

While the percentages are not used programmatically, they have more meaning to me than voltages. I watched the values to see how close they were estimating the time remaining using a 4h 19m to cutoff of the last test. The values tracked +/-2% to what I expected.

Time to move on to the physical docking assurance mechanism.

2 Likes
5

Characterizing the GoPiGo3 get_voltage_battery() precision and performance.

Results:

  • Taking readings more often than 0.002 seconds results in frequent bad data and SPI No Response errors
  • bad data usually 0.045v or 0.046v
  • precision appears to be around 0.05 volts
gpg3_volt_test.py results

Note:  bad data was always 0.045v or 0.046v 

- 0.001s:

We took 75973 measurements at 0.0010 seconds (based on reference voltage of 11.77)
min: -0.069  max: 0.250  mean: -0.015  mode: -0.009   sdev: 0.015    bad_data: 13  SPI_errors: 1

We took 24608 measurements at 0.0010 seconds (based on reference voltage of 11.701)
min: -0.120  max: 0.000  mean: -0.083  mode: -0.086   sdev: 0.013    bad_data: 8  SPI_errors: 1


- 0.002s:

We took 19053 measurements at 0.0020 seconds (based on reference voltage of 11.761)
min: -0.060  max: 0.034  mean: -0.024  mode: -0.026   sdev: 0.013    bad_data: 1  SPI_errors: 0


- 0.005s:

We took 22102 measurements at 0.0050 seconds (based on reference voltage of 11.736)
min: -0.103  max: 0.000  mean: -0.051  mode: -0.051   sdev: 0.018    bad_data: 1  SPI_errors: 0

We took 27507 measurements at 0.0050 seconds (based on reference voltage of 11.77)
min: -0.069  max: 0.043  mean: -0.015  mode: -0.009   sdev: 0.018    bad_data: 1  SPI_errors: 0


- 0.01s:

We took 11017 measurements at 0.0100 seconds (based on reference voltage of 11.761)
min: -0.069  max: 0.034  mean: -0.027  mode: -0.026   sdev: 0.013    bad_data: 2  SPI_errors: 0
2 Likes
6

Using the GoPiGo3 Python API on Raspberry Pi 5

It appears the Raspberry Pi 5 has a problem talking to the GoPiGo3 (red board) where occasionally the RPi 5 thinks an SPI transfer is complete, but the GoPiGo3 has not performed the data reply transfer.

The GoPiGo3 API throws an exception to alert users not to trust the “result of the transfer”. All the GoPiGo3 examples and programs I wrote in the last 7 years of using the Python GoPiGo3 API, do not handle a No SPI Response exception.

This is how EVERY CALL TO A GoPiGo3 API ON A RASPBERRY PI 5 MUST BE PROGRAMMED to keep programs running happily (replace “api_call” with the specific GoPiGo3 API used):

try:
     result = egpg.api_call()
except Exception as e:
     str_to_log= __file__ + ": egpg.api_call(): Exception "+type(e).__name__+": "+str(e)+" continuing"
     print(str_to_log)   
     # retry will probably succeed
     result = = egpg.api_call()
2 Likes
7

I have not found anyone discussing Pi 5 SPI issues, but did find a similar issue reported for the Pi 4. That report suggested that their SPI issue did not happen below 500KHz or above 2MHz clock speeds.

This caused me to recall the line in gopigo3.py:

GPG_SPI.max_speed_hz = 500000

I created mygopigo3.py with

GPG_SPI.max_speed_hz = 432000

( I have no clue what actual transfer rate that results in - something about spidev choosing the nearest speed from a 250MHz clock with a 16bit divider)

I have been testing with multiple EasyGoPiGo3 programs running for three hours now without seeing any “No SPI Response” exceptions yet.

Hopefully the morning will be exceptionally exception-free!

UPDATE at 10 hours: Still clean log - looking hopeful at this point.

2 Likes
8

[SOLVED - No More] ‘No SPI Response Exceptions’

Confirmed lowering the max SPI transfer bit rate eliminates the ‘No SPI Response exceptions’

At this rate, Dave can check encoder position for left and right motors at over 1500 times/second, so lowering the SPI rate does not appear it will cramp his operation:

pi@GoPi5Go:~/GoPi5Go/systests/spimax $ ./mymeasure_spirate.py 
Calling get_motor_encoder() - ctrl-c to end test
^C
Called get_motor_encoder() 4679324 times in 1380.439 sec
get_motor_encoder(): 0.0003 sec at 3390 Hz
Done

Also interesting: running one process banging the SPI bus performing get_motor_encoder(MOTOR_LEFT) as fast as possible (3390 Hz) does not interfere with two other processes checking battery voltage every few seconds. The SPI bus is totally thread-safe.

2 Likes
9

. . . and why does Dave need to check it so often?

2 Likes
10

He don’t. I was worried that halving the serial comm between the RPi and the GoPiGo3 board might degrade normal sensor readings. In ROS 2 GoPiGo3 node, I publish all the GoPiGo3 data at 30Hz:

  • Left Encoder
  • Right Encoder
  • Battery Status
  • Motor Status
  • Odometry
  • Joint Status

so that is 180 to 200 transfers per second - well below the 1500 per sec performance.

Some folks like robot encoders at 100 Hz - still doable with 1500 per sec SPI transfers

2 Likes
11

That sounds like a lot.  You’re not planning to turn Dave into a video game are you? :wink:

I had to do something like that with the remote camera robot project simply because JavaScript in a web page seems to be designed to use a game loop, (requesting a new video frame), as the main callback routine.  Otherwise I would have preferred an event-driven interface.

2 Likes