[Solved] Again seeing [Errno 121] I2C transfer: Remote I/O error

GoPiGo
, , ,
1

UPDATED: SKIP TO SOLUTION:

I thought I had beaten the fatal I2C error 121 when multiple processes access the distance sensor, by instantiating the distance sensor to use “RPI_1” the hardware I2C in all programs:

egpg.ds = egpg.init_distance_sensor(port='RPI_1')

(The IMU must use software I2C but only one process accesses the IMU and only when the wheel encoders report motion has started or stopped, to minimize use of the much less reliable software I2C.)

In testing my Easy PiCamera Sensor with a “Braitenberg Vehicle with Obstacle Inhibition” program, I am again seeing the dreaded fatal I2C Errno 121 after a minute or two of 10 times per second distance sensor checks. Even a warm boot (sudo reboot) does not clear the error, only a total shutdown (sudo shutdown -h followed by a GoPiGo3 power-button press) and cold boot (GoPiGo3 power-button press from off state) can fix this issue.

I did try calling the imu.reconfig_bus() method, but it does not revive the I2C bus from the Errno 121 state.

I2C Error during Braitenberg Vehicle 10 per second access

2020-12-16 09:11:41 light -    left:  25.8 right:  33.2 dist: 300
2020-12-16 09:11:41 Stimulus - left:  18.3 right:  40.7 gain: 3.0


2020-12-16 09:11:42 light -    left:  25.5 right:  33.0 dist: 300
2020-12-16 09:11:42 Stimulus - left:  18.1 right:  40.4 gain: 3.0


2020-12-16 09:11:42 light -    left:  25.5 right:  33.0 dist: 300
2020-12-16 09:11:42 Stimulus - left:  18.1 right:  40.4 gain: 3.0


2020-12-16 09:11:42 light -    left:  25.5 right:  33.0 dist: 300
2020-12-16 09:11:42 Stimulus - left:  18.1 right:  40.4 gain: 3.0


2020-12-16 09:11:43 light -    left:  25.5 right:  33.0 dist: 300
2020-12-16 09:11:43 Stimulus - left:  18.1 right:  40.4 gain: 3.0


[Errno 121] I2C transfer: Remote I/O error
2020-12-16 09:11:43 Obstruction at 0 centimeters
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error
2020-12-16 09:11:45 Obstruction at 0 centimeters
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error
2020-12-16 09:11:47 Obstruction at 0 centimeters
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error
[Errno 121] I2C transfer: Remote I/O error

Reconfigure I2C bus failed to fix

pi@Carl:~/Carl/systests/I2C $ ./reconfigI2Cbus.py 

Reconfiguring I2C bus using EasyIMUSensor().reconfig_bus()
   on GoPiGo3 port AD1

pi@Carl:~/Carl/systests/I2C $ cd ../DistanceSensor/
pi@Carl:~/Carl/systests/DistanceSensor $ ./readDistSensor.py 
Traceback (most recent call last):
  File "./readDistSensor.py", line 21, in <module>
    ds = EasyDistanceSensor(use_mutex=True)
  File "build/bdist.linux-armv7l/egg/di_sensors/easy_distance_sensor.py", line 56, in __init__
  File "build/bdist.linux-armv7l/egg/di_sensors/distance_sensor.py", line 28, in __init__
  File "build/bdist.linux-armv7l/egg/di_sensors/VL53L0X.py", line 126, in __init__
  File "build/bdist.linux-armv7l/egg/di_i2c.py", line 220, in write_reg_8
  File "build/bdist.linux-armv7l/egg/di_i2c.py", line 185, in transfer
  File "build/bdist.linux-armv7l/egg/di_i2c.py", line 469, in transfer
IOError: [Errno 5] Input/output error

pi@Carl:~/Carl $ ./logMaintenance.py "Braitenburg2B.py reading dist sensor killed I2C bus"

'** Braitenburg2B.py reading dist sensor killed I2C bus **' added to life.log

To Do:

I am going to try again later with the IMU not running to see if the issue still appears. I suspect this will eliminate the problem.

1 Like
2

I have an idea for you to try.

Can you overload the i2c hardware clock to either a slower or faster value?

I am curious if there is a signal timing edge-case somewhere on the pcb, or something somewhere else in the code?

i2c clock timing, even at slower clock speeds can be critical.

Would you please try to bend the clock over a range of ±25% and see what happens?

Thanks!

3

you must think I’m Einstein, to be able to bend the clock?

1 Like
4

No.

Even hardware clocks are configured by software after initial boot.

I remember seeing something somewhere about the i2c clock. Maybe in the gpg libraries?

Since I am in the hospital, I can’t help you find it.

1 Like
5

Jim, thank you for thinking about this, but I have to limit “my job” to reporting the issues and trying “my darndest” to make sure the issue is not in my code. Simple elimination of possible causes like running the same test with and without the software I2C interfaced IMU running can provide additional information, but going deep into the guts of the ModRobotics I2C code is above my pay grade.

1 Like
6

You’re getting PAID?!!
:wink: :laughing: :+1:

I’m not so good at what you do. Since this is a persistent issue, and there are corollary issues with firmware updates, my first reaction is to look at signal timing.

I’ll look at this again once I get home and feel better.

It probably won’t be until after the holidays.

Maybe Santa will leave a 'scope and logic analyzer under the tree. :wink:

1 Like
7

Actually, we’d make a pretty boss team!

I can make hardware dance jigs and you can make software play The Great Gates of Kiev in 5.1 Dolby surround sound.

We’ll whip it.

1 Like
8

Maybe Santa will leave you a 3-D printer?
:clap::clap::+1::+1::wink:

9

No way! Those things eat time and money.

Carl has dibs on my time, and my (next week) 23rd wedding anniversary called dibs on my money already!

2 Likes
10

I’m not holding my breath about my tree either.

A good Saelae logic analyzer costs like oil-rights, and a 'scope with the precision I would need costs like a battleship too.

2 Likes
11

Congrats on #23.
3D printers have come down in price, and can be really handy for both prototyping and making real parts. It’s the Fusion360 (or whatever you use for design) that’s the real time suck. There’s a bit of a learning curve. But even with modest skills you can make useful things (even without a heated bed - I can only use PLA).
/K

2 Likes
12

Looking forward to hearing you are an the back side of your unplanned “vacation”

1 Like
13

I’ve toyed with a usb scope thing for several years now, but I think having a spare “stock” GoPiGo3 that I could test only stock R4R code on would get a lot more use than a scope for me. I’m watching to see if the new GoPiGo3 board will pass initial purchasers’ usages.

1 Like
14

Ditto.

There ought to be a “masterpiece of understatement” emoji for things like this. :wink:

1 Like
15

My two centavos:

The two big reasons I don’t have a 3-D printer are the cost:
First-cost and follow-on costs.

Anything that looks like it will be worth the time to take home and unpack/setup/calibrate is in the $300+ range, and I’d get better use from a decent high-bandwidth scope.

It would (IMHO, based on what I’ve read if I understand correctly), cost at least half again that for supplies and scrap to get it accurately set-up, adjusted, and calibrated.

I live in the European version of a “studio” apartment.

(In Europe, divide expectations for space by about 1/2.)

The entire downstairs of the “house” I live in is smaller than the kitchen and living room where I live in Worcester Ma. - and this is considered a “generously” sized place that people drool over.

I can’t give something the size of a 3-D printer its own home so it needs to be setup, recalibrated, one or two pieces printed, cooled, cleaned, and then put back out of the way after each use.

The actual learning curve is (IMHO) like climbing Mt. Everest to get a learner’s permit. I don’t see enough on-going use to justify the cost.

I’d really like a reasonably well calibrated 100mhz 'scope, (though nowadays it might need to be closer to 1ghz), but I don’t have a place to keep it either.

What say ye?

Or are there things I’m not understanding rightly?

I’ve looked at those too.

This seems to be one of those “buy nice or buy twice” things.

An inexpensive 'scope to use as a learning tool is one thing. Trying to use a $50/$100 USB scope to troubleshoot sub-millimeter impedance mis-matches is something else entirely!

I guess we all will just have to make do.

1 Like
16

Update on I2C testing:

  • tests without IMU / software I2C in use can still cause fatal error case
  • multi-process test to distance sensor with new (threaded) easy PiCam sensor can cause error case
  • multi-process software I2C distance sensor only accesses do not cause error case

Building multi-threaded distance-sensor-only test to see if di_i2c mutexes are perhaps not thread safe.

1 Like
17

Sounds like space would be an issue. And you’re right - you’d spend at least $300 to have something worthwhile. I’ve heard good things about the Prusa MINI+ (https://shop.prusa3d.com/en/3d-printers/994-original-prusa-mini.html), which isn’t too much of a space hog, but may still take more space than you have. And I don’t know how much filament is where you are. It’s reasonably cheap for me to get decent PLA, but that may not be the case where you are.

Also sounds like your focus is really on hardware testing, so a good scope probably would be a better investment. All depends on what you enjoy spending your time on.

Happy holidays. :evergreen_tree: :mask: :robot:
/K

2 Likes
18

I like software testing too. It’s fun! It’s like Chess as a Turing test. (Not that I’m even punter-level at Chess.)

I know hardware design and testing. With people like yourself and Alan who have so much more talant holding down the software side, I’m not so concerned about leaving the gnarly and nobby software glitches in your capable hands and concentrating on what I do well:

  1. Being an idiot noob user
  2. Forensic “Pi-tholohy” :wink:
  3. Being nosey.
1 Like
19

Further thinking:

Since the “Easy Pi Camera Sensor” does not use the I2C bus, I need to use pure Python mutex, rather than the di_i2c mutex. Guessing that will prevent the threaded camera sensor from killing the i2c distance sensor.

1 Like
20

SOLVED! (Don’t use di_i2c mutex unless protecting I2C bus.)

Here is Carl using my new EasyPiCamSensor.light_left_right() as cross connected steer(left_percent,right_percent) stimulus with no I2C error.

2 Likes