Raspbian For Robots Oct 17, 2020 Release Test Log For My GoPiGo3

[cyclicalobsessive]

Initially, I did not detect any issues with moving from Stretch based R4R to the “PiOS” based R4R

• Then: Abandoned after fatal I²C failure.

Retest after running OS update/upgrade:

=== 20Nov2020: All Tests Passed - Carl has “Blessed” this beta OS version ==

My Configuration:

• Raspberry Pi 3B
• SanDisk Ultra 32GB card
• GoPiGo3 (red card) with V1.0.0 firmware
• 8 NiMH AA cells in DI supplied holder

1. I downloaded the 10-17-2020 R4R PiOS version and

2. installed on a fresh SDcard with Raspberry PI Imager.

3. I followed my prior setup steps (including disabling ipv6 based on @jimrh experience)

4. Power On: The bot announced his onboard WiFi address

5. I remoted in with ssh, and continued configuration
   a few minor changes - updated setup doc - will post later if all goes well

6. git cloned Carl down to the bot

7. ran my ./plib/status.py

**pi@Carl** : **~/Carl $** ./plib/status.py 
Starting status.py at 8.74v
********* CARL Basic STATUS *****
2020-10-22 12:36:55 up 11 min, 3 users, load average: 1.47, 0.87, 0.51
Battery Voltage: 9.61
5v Supply: 5.08
Estimated Life Remaining: 6 h 2 m
Processor Temp: 53.2'C
Clock Frequency: 800.084 MHz
throttled=0x0
Distance Sensor: 16.1 inches

and the Raspbian For Robots Desktop appears to be working:

All GoPiGo3 Control Panel functions worked correctly.

Servos, Distance Sensor, IMU, and Camera are all working.

Both onboard and USB dongle WiFi are working as expected.

USB Microphone is working (slightly different home/pi/.asoundrc and /etc/asound.conf than prior)

Text-To-Speech espeak-ng working

NoVNC/RealVNC in Web Browser and Mac Remote Desktop working well with 1920x1080 display configuration in raspi-config.

Completed the “Carl-ification” successfully:

• setup cron for my docking and battery maintenance,
• setup life.log function
• setup encoder and imu logging

(Did eventually run into NoVNC complaining “cannot display” when reconnecting after not using “disconnect session” at end of web browser session. I need to investigate further.)

[cleoqc]

Thanks for the test @cyclicalobsessive

Please note that this version is not an official release. For once, the SD card does not resize on first boot.

@cyclicalobsessive

(Did eventually run into NoVNC complaining “cannot display” when reconnecting after not using “disconnect session” at end of web browser session. I need to investigate further.)

This is an issue with the VNC server. I’m using a straight VNC viewer and running into the same thing. Usually it fixes itself. It looks like the VNC server has a sleep mode.

Cleo

[cyclicalobsessive]

Update2: 95 hours into the latest session, Carl went crazy repeating “Unable to Undock…Unable to Undock”. He does that when the distance sensor claims there is something blocking forward movement, or some error checking the distance. In this case - I2C failure.

Distance Sensor uses HW I2C which was rock solid before, but IMU uses SW I2C with clock stretching and was never rock solid. This time SW I2C messed up about 26 hours prior and the HW I2C lingered on somehow eventually also failing.

2020-10-28 05:58|[imulog.py.logMotionStop]heading:     2.4  rotation:     2.5 motion:     2.8 sec errors: 59
2020-10-28 12:33|[imulog.py.logMotionStop]heading:   179.1  rotation:   176.8 motion:     2.4 sec errors: 66
2020-10-28 12:33|[imulog.py.logMotionStop]heading:   358.1  rotation:   178.8 motion:     2.3 sec errors: 66
2020-10-28 12:33|[imulog.py.logMotionStop]heading:   356.9  rotation:    -1.3 motion:     2.8 sec errors: 66
2020-10-28 12:33|[imulog.py.logMotionStop]heading:   356.9  rotation:     0.0 motion:     0.3 sec errors: 66
2020-10-28 12:33|[imulog.py.logMotionStop]heading:   356.9  rotation:     0.1 motion:     0.4 sec errors: 66
2020-10-28 12:33|[imulog.py.logCurrentIMU]heading:   359.9  rotation:     0.0 motion:     0.0 sec errors: 66  imu reset

Software I2C failed between 10-28 12:33 and 15:16 (IMU did not register undocking)

2020-10-28 12:33|[new_juicer.py.dock]---- Docking 1659 completed  at 8.1 v after 6.6 h playtime
2020-10-28 15:16|[new_juicer.py.undock]---- Dismount 1659 at 11.4 v after 2.7 h recharge

juicer.out showing HW I2C failed at 10-29 17:40:25

********* CARL Basic STATUS *****
2020-10-29  17:39:53 up 3 days, 21:55,  3 users,  load average: 0.21, 0.16, 0.10
Battery Voltage: 10.90
5v Supply: 5.08
Estimated Life Remaining: 7 h 0 m
Processor Temp: 44.5'C
Clock Frequency: 600.0 MHz
throttled=0x0
Distance Sensor: nothing within 90 inches

Juicer Values:
Juicer Run Time 3 days 21h 53m
lastReading 10.922 volts
shortPeakVolts 11.008 volts
shortMeanVolts 10.906 volts
shortMinVolts  10.219 volts
longPeakVolts  12.601 volts
longMeanVolts  10.918 volts
longMinVolts   10.202 volts
Charging Status:  Charging
Last Changed: 0 days 0h 54m 24s
Last Change Rule:  120
Docking Status:  Docked
Docking Count:  1662
Last Docking Change: 0h 56m 59s
Play Time Limit 8.1v

********* CARL Basic STATUS *****
2020-10-29  17:40:25 up 3 days, 21:55,  3 users,  load average: 0.13, 0.14, 0.10
Battery Voltage: 10.91
5v Supply: 5.08
Estimated Life Remaining: 7 h 0 m
Processor Temp: 45.1'C
Clock Frequency: 800.0 MHz
throttled=0x0
[Errno 121] I2C transfer: Remote I/O error
Distance Sensor: -0.2 inches

[cyclicalobsessive]

Update3: I2C didn’t last the night. Carl is shutdown until I can return him to his backup OS. (Have to eat some breakfast now.)

[cyclicalobsessive]

@cleoqc or @mitch.kremm

The beta GoPiGo OS / DexterOS has TensorFlowLite and OpenCV installed by default.

(I don’t have the R4R Oct 17 image on a card anymore - so can’t look for myself - sorry.)

Q) Does the / Will the next Raspbian For Robots also have TFLite and OpenCV packages already installed?

@cleoqc

Fresh Python Libraries

The Python libraries have been updated to newer versions. Please note that tensorflow lite is now installed, and a face recognition library is available (see https://pypi.org/project/face-recognition/)

[cleoqc]

@cyclicalobsessive

Does the / Will the next Raspbian For Robots also have TFLite and OpenCV packages already installed?

Probably not. Raspbian for Robots lets you install whatever you want. And having those libraries in makes the image so much bigger

[cyclicalobsessive]

Trying again without running the IMU this time. Carl will test just hardware I²C.

If Carl survives for a week or two, I’ll add an IMU reliability test.

[cyclicalobsessive]

Now we’re talkin’ fun!

Installed the TensorFlowLite image classification example and modified it to print to the command-line if an object classification probability is greater than 60%. It successfully classified a clock that I held about 6 inches from Carl’s face…but then it also thought it saw some items not likely to be in my home!

pi@Carl:~/Carl/Examples/TF/examples/lite/examples/image_classification/raspberry_pi $ ./run_it.sh
analog clock 0.71 310.1ms
analog clock 0.60 309.5ms
analog clock 0.62 309.8ms
sliding door 0.60 337.7ms
sliding door 0.61 312.8ms
sliding door 0.66 311.9ms
sliding door 0.69 312.2ms
sliding door 0.62 312.3ms
sliding door 0.74 309.8ms
sliding door 0.67 309.7ms
sliding door 0.70 309.8ms
sliding door 0.66 309.7ms
turnstile 0.69 309.9ms
turnstile 0.79 309.8ms
^C
Exiting

Carl was getting low on juice and the TFLite program ran the battery down pretty quick as he exclaimed “Turning around to Docking Approach Point”.

Have to test more when he gets juiced up.

[cleoqc]

Liking the new ‘toy’ ?

In GoPiGo OS, I have modified the same script to print to console too. And by experience, you need to handle a confidence level of at least 85% to avoid the non existing sliding door. It did manage to identify my Shih Tzu as an actual Shih Tzu too

[cyclicalobsessive]

@cleoqc

modified the same script

Be sure to look at my version on git, it has all the options plus a save annotated image to file

[cyclicalobsessive]

@cyclicalobsessive

Trying again without running the IMU this time. Carl will test just hardware I2C.

If Carl survives for a week or two, I’ll add an IMU reliability test.

HW and SW I2C “banging” continues - good news so far:

• over 1.1 million SW I2C reads
• concurrent with over 23k HW I2C reads

=== Nov 2020 HW and SW I2C Investigation ===

PiOS based Raspbian For Robots 17 Oct 2020 beta release
No Change to DI_Sensors from 17Feb2019 version

Test1: Regular Operation on beta OS release

• Distance Sensor in HW I2C mode
• IMU in SW I2C mode
• Running: juicer, imulog, wheellog, lifelog
• imulog (SW I2C) failed after 26 hours
• juicer detected fatal I2C failure on HW I2C after 95 hours operation

 
Test2: Regular Operation w/o using SW I2C/IMU on beta OS

• Distance Sensor in HW I2C mode
• imulog invocation prevented in crontab-e
• Running: juicer, wheellog, lifelog and numerous TensorFlow Lite gpg_classify_picamera.py
• (No USB WiFi dongle - giving 8.3h “playtime” vs 6.4h with)
• No fatal errors detected in 115 hours

 
Test3: Add HW I2C stress on Distance Sensor

• Distance Sensor in HW I2C mode
• imulog invocation prevented in crontab-e
• Running: juicer, wheellog, lifelog and numerous TensorFlow Lite gpg_classify_picamera.py
• Running: HWDistanceSensor.py
• (No USB WiFi dongle - giving 8.3h “playtime”)
• No fatal errors detected in 22 hrs, 328k extra HW I2C readings of distance sensor

 
Test4: Add SW I2C Stress

• Distance Sensor in HW I2C mode
• imulog invocation prevented in crontab-e
• Running: juicer, wheellog, lifelog, status.py perfoms distance sensor readings every six seconds
• Running: SW_I2C_Stress_with_IMU.py - over 1 million reads with 62 soft errors, no hard failures
• (No USB WiFi dongle - giving 7.7h “playtime” )

19:17:17 1103111
Magnetometer X:  -34.5  Y:  -22.9  Z:  -10.2 Gyroscope X:    0.2  Y:   -0.1  Z:    0.1 Accelerometer X:    0.1  Y:    0.2 Z:    9.5 Euler Heading: 348.1  Roll:   0.4  Pitch:   0.8
19:17:19 1103121
Magnetometer X:  -34.1  Y:  -22.6  Z:  -10.6 Gyroscope X:    0.1  Y:   -0.1  Z:    0.0 Accelerometer X:    0.1  Y:    0.1 Z:    9.4 Euler Heading: 348.1  Roll:   0.4  Pitch:   0.8

 

New Test Started:

Test5: Heavy HW and SW I2C Stress

• Distance Sensor in HW I2C mode
• imulog invocation prevented in crontab-e
• Running: juicer, wheellog, lifelog,
• Running: SW_I2C_Stress_with_IMU.py (10 readings/s), HWDistanceSensor.py (10 readings/s)
• (No USB WiFi dongle - giving ?h “playtime”)

=====
temp=54.8'C
 19:50:42 up 7 days,  5:27,  5 users,  load average: 0.66, 0.78, 0.74
19:49:46 9261:distance from object: 604 mm
19:49:49 9271:distance from object: 615 mm
19:49:52 9281:distance from object: 613 mm
19:49:55 9291:distance from object: 612 mm
19:49:57 9301:distance from object: 611 mm
19:50:00 9311:distance from object: 611 mm
19:50:03 9321:distance from object: 614 mm
19:50:06 9331:distance from object: 602 mm
19:50:09 9341:distance from object: 618 mm
19:50:11 9351:distance from object: 605 mm
======
19:50:26 9341
Magnetometer X:  -33.1  Y:  -23.4  Z:   -6.6 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:28 9351
Magnetometer X:  -34.1  Y:  -24.6  Z:   -6.2 Gyroscope X:    0.1  Y:    0.0  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:31 9361
Magnetometer X:  -34.1  Y:  -25.0  Z:   -1.1 Gyroscope X:   -0.1  Y:    0.0  Z:    0.1 Accelerometer X:    0.1  Y:    0.7 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:33 9371
Magnetometer X:  -33.8  Y:  -26.2  Z:   -1.1 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.7 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:36 9381
Magnetometer X:  -33.8  Y:  -25.8  Z:   -0.8 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
=====
1
IMU (SW I2C) Exceptions 

[jimrh]

So, the conclusions from all this are?

[cyclicalobsessive]

@jimrh

the conclusions from all this are?

HW and SW I2C “banging” continues - very good results so far but no conclusions yet.

The original testing was performed on the exact R4R Oct 17 image.
Current testing is being performed on

• R4R Oct 17 image base
• updated with

sudo apt-get update -y
sudo apt-get full-upgrade -y

One final test remains in plan - to restore the automatic imu logging which uses my “safe” IMU class.

Carl has run very smoothly 24/7 for eight days on this image, at higher processor loads than his usual. He seems quite “happy” so far.

[jimrh]

Thank you!

Though I might be fairly clever, I have not yet reached the pitch of intelligence where I can look at a blob of numbers as raw data and understand what they’re trying to tell me.

Though I have not yet accomplished what you do, your work fascinates me. It’s intriguing. I really enjoy watching the way you carefully and precisely chop up the Dexter/MR designs into tiny pieces and then reassemble them into something that works.

What you’ve discovered is ultimately destined to save me, and others, much time and angst.

Again,

Thank you!

[KeithW]

I’m with @jimrh - thanks for your thorough testing. I’m still very, very new on the platform, so this is quite helpful.
/K

[cleoqc]

We at ModBot also love your reports, @cyclicalobsessive !

[cyclicalobsessive]

Dexter/GoPiGo3/Software/Python/Examples Tests

Control_Panel - Works Fully

easy_Blinkers.py Works Fully

easy_DexEyes.py Works Fully

easy_Distance_Sensor.py Works Fully

easy_Motors.py Works Fully

LED.py Works Fully

Motor_Encoder.py Works Fully

Motor_Position.py Works Fully

Motor.py

• Works but ramps forward fast, then ramps backward slowly not same distance, both are not straight line
   

Motor_Speed.py -

• ISSUE: Works One Time, Second Invocation appears to be working but no motor commands get issued until I move bot with Control Panel, then it goes out of control - I could not stop it with Control Panel.

• Probably would run fine when it is the only instance of a GoPiGo3 class. Carl had several EasyGoPiGo3(use_mutex=True) classes running during this test.
   

Motor_Turn.py Works Fully

Read_Info.py Works Fully

Servo.py Works Fully

 

Did Not Test

easy_Button.py

easy_Buzzer.py

easy_LED.py

easy_Light_Sensor.py

gps_reading.py

Grove_Buzzer.py

Grove_I2C.py

Grove_IR.py

Grove_LED.py

Grove_Light_Sensor.py

Grove.py

Grove_US2.py

Grove_US.py

Line_Sensor.py

Motor_Follow.py

Remote_Control.py

[cyclicalobsessive]

=== 20Nov2020: All Tests Passed - Carl has “Blessed” this beta OS version ==

Disclaimers - YMMV:

1. Carl runs a Pi3B
2. IPv6 disabled
3. File system expanded w/raspi-config
4. Carl is a robot - Robots think different than humans.
5. Remote Desktop does not show Desktop widgets, work around is to open desktop folder, or LogOut and Log back in.

[cyclicalobsessive]

@cleoqc

This is an issue with the VNC server. I’m using a straight VNC viewer and running into the same thing. Usually it fixes itself. It looks like the VNC server has a sleep mode

There is this setting checked:

Today, I also lost the desktop. When I connect through noVNC or through remote desktop, I get to a ModRobotics desktop without the desktop widgets.

Work around is to open desktop folder, or LogOut and Log back in.
(Does not happen if Logout then Disconnect Session, Then Reconnect and Login)

[cyclicalobsessive]

Difficulty configuring RPI-Monitor on R4R PiOS beta

For the last several days, I have been fighting to get RPI-Monitor to properly start at boot on both Carl, and my “DeskPi.”

Carl is running R4R over PiOS, while the DeskPi is running PiOS. Carl was seeing:

Web server not started because of error: Address already in use

I found that starting the RPI-Monitor with Carl’s reserved IP prevents the error to allow this useful tool to work.

And immediately, RPI-Monitor showed how smart Carl actually is!

Recently, I changed Carl’s recharger which lowered his maximum recharge rate from 1A to 0.9A. Carl was getting off his dock before the trickle charge light came on due to “Rule 230c - Probable Trickle Not Detected after 3.5 hour charging”, so I changed the time for that rule to 4.0 hours to provide a little more time.

Today is the first day I have RPI-Monitor running that I can conveniently review the recharge voltage curve, and as I was doing so Carl announced “Trickle Charging - Dismounting”. I quickly looked at the charger to see the status LED showing solid green which means Carl was indeed trickle charging.

Looking through his recharging log showed it still was due to rule 230c - Probable Trickling Not Detected (after 4 hrs), but reviewing the RPI-Monitor shows that the charger had indeed not long before switched to trickle charging - Carl is so smart!

(I would have preferred to see that Carl detected the trickle charge when the average voltage started going down instead of up, but the important thing is to get off the dock when the charger is trickling.)

Who would believe the happiness playing with Carl can give? I’ve been up for two hours and still haven’t prepared my breakfast.

Update: The next morning Carl recognized trickle at 3.8 hours on the dock (rule 230: over 12v and mean voltage has negative slope), before the 4 hour “probable trickle” rule 230c could fire!