GoPiGo3 SPI Test Results - Expect / Handle "No SPI Response"

GoPiGo
,
1

The GoPiGo3 Red Board uses an SPI bus connection for all command and data transfers with the Raspberry Pi. When transferring data from the Red Board to the Raspberry Pi, the GoPiGo3 API checks the transfer status in the fourth byte of the reply array. If the value is not 0xA5, the GoPiGo3 API will print an error message “No SPI Response” and return a value of 0 for the requested data, effectively “swallowing the error”.

After testing 800,000 get_battery_voltage() calls between two calling processes, it appears the errors are not totally related to multi-process SPI collisions, indicating the GoPiGo3 SPI bus transfers are not well behaved.

Testing:

  • the stock GoPiGo3 API versus
  • using an SPI mutex versus
  • using an SPI mutex with various delays after acquire()

showed a mutex cut the number of errors to roughly a third the unprotected transfers (28 → 10), but could not eliminate all “No SPI Response” errors.

GOPIGO3 SPI BUS ERROR TEST

Two programs are launched:

  • spitest.py checks gopigo3.get_voltage_battery() every 0.001 seconds
  • spitest2.py checks gopigo3.get_voltage_battery() every 0.0013 seconds
  • use special instrumented spi_gopigo3.py that counts “No SPI RESPONSE” errors
    in spi_read_8(), spi_read_16(), and spi_read32()
    get_voltage_battery() uses spi_read_16()

Results

  • Test 1 with spitest2.py in ROS2/Ubuntu22 Docker container, spitest.py in PiOS Bookworm:
    21 SPI errors reported in 800,000 voltage checks between the two processes.

  • Test 2 with both spitest.py and spitest2.py in PiOS Bookworm
    29 SPI errors reported in 820,000 voltage checks between the two processes.

MUTEX PROTECTED TRANSER_ARRAY()

Two programs are launched:

  • spi_mutex_user.py checks gopigo3.get_voltage_battery() every 0.001 seconds
  • spi_mutex_user2.py checks gopigo3.get_voltage_battery() every 0.0013 seconds
  • use special instrumented spi_mutex_gopigo3.py that uses DI_Mutex(“SPI”) and
    counts “No SPI RESPONSE” errors in spi_read_8(), spi_read_16(), and spi_read32()
    get_voltage_battery() uses spi_read_16()

Results

  • Test 1

    • both spi_mutex_user.py and spi_mutex_user2.py in PiOS Bookworm
    • only acquire() and release() (no delay after acquire)
      11 SPI errors reported in 800,000 voltage checks between the two processes.

  • Test 2

    • both spi_mutex_user.py and spi_mutex_user2.py in PiOS Bookworm
    • 0.001s delay after acquire()
      9 SPI errors reported in 800,000 voltage checks between the two processes.

  • Test 3

    • both spi_mutex_user.py and spi_mutex_user2.py in PiOS Bookworm
    • 0.0001s delay after acquire()
      11 SPI errors reported in 800,000 voltage checks between the two processes.
1 Like
2

What’s the difference between tests 2 and 3?

Do you have anything else using the SPI buss?

1 Like
3

Test 2 had an “after acquire delay of one thousandth second”,
Test 3 had a smaller delay of one ten-thousandth second.

My hypothesis was that if the quite long 0.001s delay improved the error rate, then perhaps a shorter delay would be sufficient. Turns out the delay did not improve the error rate, only the presence of the mutex was needed.

Since the mutex did not eliminate the SPI issue, I feel it is the GoPiGo3 red board implementation of SPI causing some SPI transfers to fail to supply a valid reply. I don’t know if the reply is coming too late, or never coming, or just have a corrupt byte 4 in the reply. The GoPiGo3 xfer2(dataout) is not resulting in the reply array/list having valid data when the chip select is released.

No, I stopped all other GoPiGo3 objects that normally would be accessing the bot - (my safetyShutdown.py, and docking test program that are launched at boot). I also stopped the Docker system that mapped the SPI device.

1 Like
4

This is where a logic analyzer would come in handy.
:man_facepalming:

(Quite frankly, I’m thrilled to actually have some kind of a 'scope.  A logic analyzer would be gravy!)

5

This is where an educational bot requires too much education.

1 Like
6

This kind of stuff is the kind of challenge I’ve dreamed of.

I finally get to flex my mental muscles trying the kind of research I’ve only dreamed about in the past. . . .

(All my Heathkit catalogs ended up drenched in drool.  That was the stuff of dreams!)

1 Like
7

Software Guy: “Must Be A HW Problem”
Hardware Guy: “Must Be A Software Problem” Oh this is fun.
Software Guy: Time for pizza.

I’m still dreaming about my robot being aware of more than its belly button (battery level).

8

Hardware Guy: “Hold my beer. . . .”

1 Like
9

I am faced with a choice -

  • use my already modified for Bookworm/Pi5 GoPiGo3 API and always test for zero return values
  • further mod my Bookworm/Pi5 GoPiGo3 API to not eat the exception and always test for no exception
  • live with my already modified for Bookworm/Pi5 GoPiGo3 and only do something when this bites my robot.

I’m pretty sure SPI collisions and mystery “No SPI Response” situations are not affecting Dave. I got to thinking about this because I could not imagine that my I2C mutex was not working, so I went looking for some other cause for the bad INA219 readings when two processes wanted INA219 readings. Turned out my EasyINA219 with default “use mutex” was working for two processes inside Docker or two processes outside Docker, but I had failed to test the “one inside / one outside” case. Mapping the outside /var/lock/ folder into the Docker container filesystem fixed the issue. Crazy to have wondered about SPI since the INA219 never gets close to the SPI bus.

I just have to make sure Dave’s logic always uses the INA219 voltage and never the GoPiGo3.volt() readings. Old code, written before I added the INA219 voltage/current sensor, may still be making decisions on the EasyGoPiGo3 class’s volt() method.

1 Like
10

You’re asking for sentinance.

Unfortunately, even with the best supercomputers available and the best and brightest people working on the AI, sentinance is still a dream for the future.

You’re trying to out-do massive government funding and people with PhD’s hanging off of their PhD’s, working on systems that require a whole hydroelectric power source to run?  And you’re trying to do it using a hobby-level robot running a Pi-4/Pi-5?

Maybe you need to re-think your goals or re-refine your models?

IMHO, I’m glad that I actually get to play with a robot!  This kind of “open-ended” research/play is something I’ve dreamed about for decades and decades and decades, and never even imagined that I could even hope for this - it’s been forever out of my reach.

I haven’t even begun to scratch the surface of the things you’ve done.  However, when I look back to the time when I first began and I see what I have accomplished - as modest as it may have been - I’m glad.  I see progress, no matter how small, and it encourages me to try to do more; to “boldly go where I’ve never been before”, and the robots are helping to guide the way.

I remember reading an anecdote about a guy who made a deal with the Devil, asking for the ability to stop time at the point of his choosing, when he’s the happiest.

He never did.  That is, until the Devil came to claim his soul.

He was being helped onto the train that would take him to Hell when he suddenly clicked the pocket-watch that would stop time for all eternity.

The Devil shrieked in horror!
“Why did you do that?  Especially right now - do you know how much trouble you’ve caused?  How am I going to straighten this mess out?!!”

The man replied:
“I realized that it’s not the destination that’s important, it’s the trip.”

Eventually the man and the Devil reached a compromise and he became the conductor on the Devil’s train.

It’s not the “destination” with the robots that’s important, it’s the trip.  It’s what you learn, both success and failure, what you accomplished and where you fell short.

In other words, you should look at “the trip” instead of focusing on a particular “destination” that you may, or may not, achieve.

What say ye?

11

Yeah - they write ROS miracles and Dave only needs one or two miracles at a slower pace.

Exactly, except I’m tired of pushing the train, mostly by myself. I really enjoy sitting in the dining car with a few friends, but the damn train keeps rolling to a stop while I’m enjoying myself.

1 Like
12

Except that you’re not “pushing the train”, you’re pushing yourself - expecting things that might not happen, or might not be possible.  [with the tools you have]

The best you can hope for is to enjoy your robots.  You need to stop trying to redefine modern physics and robotics, otherwise you will run yourself into the ground.

Remember:  It’s a hobby, not a religion.  You won’t go to hell if Dave doesn’t become a chess Grand Master, or Carl can’t have an existential conversation about the future of mankind.

Ol’ Doc Harris’ prescription is a cup of coffee mixed with a liberal amount of your favorite alcoholic beverage with a generous dollop of whipped cream on top.

Of course, you can substitute an ice cream float for the coffee if you wish.  Sonic makes one heck of a float, you bring the booze.

Or. . . .

Maybe you should go flying for a while?

1 Like
13

No. More like fluctuating levels of obsession mixed with varying levels of compulsion.

Interesting - that is a great idea. Thanks for the inspiration.

1 Like
14

You do know there’s medication for that?
(laughing!)

I think the float is a great idea, except they don’t make diabetic ice cream! (rats!!)

15

No, No that stuff will kill ya.

When the pandemic hit, when the wife and I were exploring low-carb eating, I tried all the different artificial sweeteners - including an interesting erythritol ice cream. Nothing tolerable.

Now I see research saying erythritol may fix your diabetes with death from cancer.

If “Life is like a box of chocolates”, Life must be bad for ya.

1 Like
16

There are companies that make “no added sugar” ice cream.  I think Blue Bunny is a local brand for you that has it.

I don’t know if they add artificial sweeteners.

We use Stevia in the Raw instead of sugar.

A friend of mine had a massive heart attack years ago.  After the heart attack he was told by his cardiologist “If it tastes good, spit it out!”
:rofl:

1 Like
17

Possible that you will still see occasional failures

This was using an SPI mutex in two client programs (no mutex in the site package gopigo3.py)

The next result was with a modified gopigo3.py which added a mutex around spi_transfer_array():

...
from di_mutex import DI_Mutex
...
 self.spi_mutex = DI_Mutex(name="SPI")
...
def spi_transfer_array(self, data_out):
        """
        Conduct a SPI transaction

        Keyword arguments:
        data_out -- a list of bytes to send. The length of the list will determine how many bytes are transferred.

        Returns a list of the bytes read.
        """
        self.spi_mutex.acquire()
        time.sleep(0.0001)
        result = GPG_SPI.xfer2(data_out)
        self.spi_mutex.release()
        return result
...

Not sure if there is another path to using SPI in the gopigo3.py - I didn’t study it close enough, but somehow there were still some bad SPI replies in spi_read_16().

1 Like
18

Maybe the Raspberry Pi’s SPI routines were carefully crafted to the same high standard of quality that the native i2c routines were crafted to?
:man_facepalming:

1 Like
19

Or perhaps the GoPiGo3 red board controller is “busy” and does not notice when the Raspberry Pi writes something into the controller’s memory map, resulting in “No SPI Response” determination by the GoPiGo3 API code running on the Raspberry Pi.

Not familiar with this issue.

1 Like
20

Of course you are familiar - intimately familiar.

Remember the native i2c clock stretching bug in the hardware routines and how it drove you to absolute distraction with the Dexter IMU module?

Or, just maybe, it’s PTSD manifesting itself as selective amnesia. . . :man_facepalming:

The idea is that maybe there’s a hardware/firmware bug in the raspberry pi’s SPI routines too?

1 Like