The Real Skinny about the Raspberry Pi i2c clock stretching bug

Greetings!

While doing some searching for a different project, I discovered what might be the definitive documentation for the i2c clock-stretching bug.

This information can be found here:
https://www.advamation.com/knowhow/raspberrypi/rpi-i2c-bug.html

I found it interesting.  If I find anything else, I’ll share it here.

Here’s the Raspberry Pi forum thread where it’s discussed:
https://www.raspberrypi.org/forums/viewtopic.php?f=44&t=13771

One thing that browns up my biscuits about this is that this is a known issue all the way back to the time the Raspberry Pi was built out of TTL logic and discrete transistors.  :wink:  Despite the fact that this is such an old and well known issue - and that there have been innumerable hardware/silicon updates since then - I cannot imagine for the life of me why this hasn’t been fixed?

Since it’s a known fact that the error lies in the Broadcomm silicon, the only thing I can imagine is that they know about the error and it’s marked “Won’t Fix (because we don’t give a hoot)”.  Imagine if Intel had done that with the Pentium FDIV bug?

More when I know more.

2 Likes

This is why the GoPiGo3 requires the IMU to be connected to the GoPiGo3 Software I2C via AD1 or AD2. To my knowledge the only supported Dexter Industries / Modular Robotics I2C sensor that requires clock-stretching is the BNO055 based Inertial Measurement Unit, and the drivers specifically state the unit should be connected via AD1 (default) or via AD2 by passing the port explicitly.

To avoid occasional I2C lockups in a configuration that has the IMU and the Distance Sensor, and Python, I suggest having only one program, (only one thread), running that accesses both sensors via I2C.

(For folks wishing to use the popular, much less expensive MPU-9250 IMU, I was only able to get good readings by connecting the MPU-9250 direct to a Raspberry Pi, without a GoPiGo3 attached, and with reducing the I2C bus speed as recommended. )

In fact, I just checked I2C status on my GoPiGo3 robot Carl:

i2cdetect -y 1

I2C Device Detect
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- 08 -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                         

See a problem? No Distance Sensor.

Another I2C failure after 1 day 4 minutes.

This is what it looks like when I2C bus is running fine:

I2C Device Detect
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- 08 -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- 2a -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --                     

I don’t know the actual cause of these conflicts - three in the last week.

Previously I ran over a million IMU readings, and the system lived for 7 days with zero hard I2C failures.

(Raspbian For Robots Oct 17, 2020 Release Test Log For My GoPiGo3)

1 Like

What I found interesting is that the article mentioned that the company was developing an i2c interface “buffer” - like a RS232 level shifter or a more modern 3.3/5.0 logic level shifter - it would take a crappy signal on the Pi’s side and clean it up so the slaves have a clue.

Conversely, it would take the conforming signals from the slave and adapt a perfectly formatted signal from a slave device to the relatively “flaky” implementation on the Pi.

1 Like

What I found interesting was that the mere mention of I2C and Carl flipped out.

Actually there may be a processor load factor in the failures. When I tested the Oct. 2020 beta, there were no failures during the test, but I was only running the basic Carl and the I2C bangers. I saw failures when I was running my Braitenburg Vehicle that used the PiCam and loaded the system quite a bit, and now I have added the voice commander that sometimes loads the processor a bit.

My solution is to turn off the IMU logger. And only use the IMU to retrieve room temperature when asked for via the voice commander.

1 Like

. . . and all this time you’ve been telling me that Carl never runs higher than a few percent on one core!

Pbbbbbbt!

1 Like

Somehow that had escaped me. It’s been a while since I’ve run Finmark, since I’ve been working on some of the ROS courses on The Construct. But the last couple of times I did run it, it seemed a bit wonky. I had “tidied up” some of the wiring, and had shifted the IMU to AD2. I wonder if that was the problem. I’ll have to check.

Also didn’t know the i2cdetect command. Thanks for 2 tips…

Thanks,
/K

Addendum:
I did check.
First - the i2ccheck doesn’t detect the IMU at all. Not sure if that’s because it’s plugged into the AD port???

Running the IMU_reading.py program that’s part of the Chapter 2 files that accompany the Hands on ROS book, that is actually expecting the IMU to be plugged into AD2. However for Chapter 6, the IMU-sensor.py program expects AD1. Argh - why can’t they be consistent?

As I dig further into other files (e.g. gopigo3_driver.py), I’m realizing that the IMU never actually gets started except for the one demo.

So - it clearly wasn’t the IMU causing my problems, since it’s not actually called. But for future use I’ll use the IMU-sensor.py, so it’s good I’ve moved the IMU to AD1. Time well spent this evening looking into this.

PS - in looking at the code, I also learned you can use a Python lambda function in place of a separate callback in a ROS subscriber. I think that’s cool.

2 Likes

AD2 works just fine for the limited amount of IMU testing I’ve done. (I have Charlie’s front bumper switches wired to AD1)

In Bloxter I can read the IMU about once per second until Hell freezes solid w/o issues. (on AD2)

I don’t know how often Carl reads his IMU, but, (IMHO) once a second is fine for casual use.

Of course, if you want accuracy down to 0.00000000001, you might have to read more often. :wink:

2 Likes

I assume you’re explicitly telling Charlie which port the IMU is on?

Just curious - are you using this primarily to detect orientation? Or are you able to do dead-reckoning by calling once a second?

/K

2 Likes

Carl checks the IMU 20 times per second to detect start of motion, and then the end of motion. At the end of motion, he logs the new heading, how much rotation occurred, the duration of motion, and how many soft I2C errors occurred.

Right now, Carl is “up 2 days 11:58” with no hard I2C errors but the IMU is showing 242 soft I2C errors.

2 Likes

I searched the seven repositories on ros-gopigo and only saw the encoders being broadcast, but I couldn’t tell how often they are sent. I’m not very adept at reading ROS node code.

Have you created a Finmark repository yet? It can help you to have a place to look at your code when Finmark is sleeping deep.

1 Like

Do you have a moment to look at that to determine how many reads/sec of the IMU it is doing?

The imu code at ros-gopigo appears to have a read rate of 30 Hz but the IMU has a queue size of 10.

1 Like

Absolutely!

If I remember rightly, it defaults to AD1 if you don’t specify a port.

However, (IMHO), I think it makes the code easier to read, easier to remember what the heck is going on, and easier for third parties to understand - if I always specify the port, even if it’s the default.

No - nothing that sophisticated.

I was running a Bloxter program something like

"While" [true]  #Infinite loop
"Print" => "the IMU [aircraft heading]" on => "AD2" # heading in degrees
"Print" => "the IMU [compass heading]" on => "AD2" # values like "north", "south-east", etc.
# Implied "wend" statement because of the shape of the block

and then just let it run for a while, occasionally moving the robot around to see the readings change. If you don’t use a “sleep” block, it runs about every 0.5 or 0.25 seconds due to the built-in delay.

The closest I’ve gotten to “sophisticated” with Charlie is a program that moves Charlie forward until either:

  • He comes within “x” distance of an obstacle.
  • His front bumper hits something.
     

He then backs up for a second or two, randomly turns a number of degrees to either the right or left, and then continues.

He does no mapping. There is no instrumentation. No logs are kept, he just wanders around like a drunken robot until he gets too close to something, bangs into something or I kill his program at the web console.

It starts with a tight loop that tests the value of Charlie’s front bumper, waiting for it to go from “0” to “1”. Once it goes from zero to one, it jumps out of the loop and runs the rest of the program. This lets me start the program, place the robot on the floor, position it, and then tap the bumper to start the ball rolling.

Right now there are two random functions, one for angular rotation, (20 to 360) and the other for direction of travel. I’m thinking that I can do both in one random call by taking the integer value of the angular rotation and making it turn left if even and right if odd. Or, I just thought of this: constrain the angular rotation value to -360 to +360, and feed that directly into the command that makes Charlie turn.

One other thing I do is require a minimum amount of rotation, (I think ±20° to make sure he’s moved enough to avoid the obstacle by constraining the angular rotation to no less than 20.

This would be how I would do it if the rotation were constrained between -360 and +360:
(pseudo-code)

if (abs(degrees)) < 20
    if (degrees < 0)  # is the rotation angle negative?
        degrees = degrees - 20  # then add an additional -20 degrees
    else degrees = degrees + 20  # Otherwise the rotation is positive or zero
    endif
endif

 
This is fairly complex for Bloxter.

2 Likes

Actually quite sophisticated - Depending on the speed of that backing, is he backing sufficient for the clearance radius of a 180 degree evasion turn?

Carl’s clearance turning radius is around 140mm, 5.5 inches with all distance readings 75mm 3 inches from the turning center. At 150 degrees per second Carl backs roughly 85 mm per second which means a 1 second backing will allow safe 180 degree turns.

Charlie’s bumper might be a bit farther from the turning center, so Charlie is “ensuring safe evasion maneuvering” (If Charlie backs at the default 300 DPS, then for sure 1 second is “ensuring safe evasion maneuvering”)

1 Like

I haven’t. I’ve still been mostly running code from Hands on ROS, and I do have a copy of those libraries on my PC. But it probably would make sense to have an exact copy of my catkin_ws/src files. I’ll have to look at setting that up.

I looked at the GitHub versions of ros-gopigo. Looking at the commits, they had changed actually changed the default port for the IMU example (imu.py) from “GPG3_AD1” to “RPI_1”. I ran the “book version” of the program (which actually isn’t in the gopigo3_node but a different package called mygopigo) last night and it seemed to work fine. It still has GPG3_AD1 as the port (the book/mygopigo version is called IMU-sensor.py; other than the file name and the port it seems to be identical to the ros-gopigo version; another weird inconsistency). As I traced the files imported as modules for the IMU program through GitHub, it looks like di_sensors/inertial_measurement_unit.py actually changed the default port from “RPI_1” to “RPI_1SW” in the repository (and this is the version of the di_sensors file on Finmark - I checked). So now I’m thoroughly confused as to what the port should be called.

Yes - that’s right.

2 Likes

Good practice.

Sounds like a good “random walk” program.
/K

2 Likes

Right now, Charlie doesn’t care - the minimum turn is an enhancement I hope to program in.

Right now I have him back up a specified time or distance, (I don’t remember which), make the random turn, and try again. He either escapes or he doesn’t. If he does, golden! If he doesn’t, stinks being him and he tries again.

If he doesn’t turn far enough, or turns too far, he bumps into something again - or notices that he’s still too close - and tries again.

Even in Bloxter that’s a bit hairy and it is teaching/has taught me a lot about what makes robots tick in the real world.

Eventually I’ll be brave enough to try that in Python.

2 Likes

Nothing worthwhile was ever as simple as it seemed.

2 Likes

I don’t use Bloxter - I know some of these kind of block languages have an option to show what code is generated. Does Bloxter generate Python? Reviewing that would be a good way to start. And I suspect some things would actually be easier in Python because you’re not limited to the options available in Bloxter.

As for learning Python, there’s a lot of free online resources.

/K

1 Like

Yes it does and that’s one my sneaky tricks.

If I want to know if a particular piece of hardware works, I can throw something together in Bloxter and see it working - without having to worry about indentation levels or parentheses.

1 Like

Found this on the Adafruit site:
A visual i2c monitor that shows buss activity in real time.

2 Likes