Raspbian For Robots Oct 17, 2020 Release Test Log For My GoPiGo3

cyclicalobsessive · November 3, 2020, 3:51pm

The beta GoPiGo OS / DexterOS has TensorFlowLite and OpenCV installed by default.

(I don’t have the R4R Oct 17 image on a card anymore - so can’t look for myself - sorry.)

Q) Does the / Will the next Raspbian For Robots also have TFLite and OpenCV packages already installed?

cleoqc · November 3, 2020, 8:05pm

Probably not. Raspbian for Robots lets you install whatever you want. And having those libraries in makes the image so much bigger

cyclicalobsessive · November 10, 2020, 5:31pm

Trying again without running the IMU this time. Carl will test just hardware I²C.

If Carl survives for a week or two, I’ll add an IMU reliability test.

cyclicalobsessive · November 10, 2020, 10:33pm

Now we’re talkin’ fun!

Installed the TensorFlowLite image classification example and modified it to print to the command-line if an object classification probability is greater than 60%. It successfully classified a clock that I held about 6 inches from Carl’s face…but then it also thought it saw some items not likely to be in my home!

pi@Carl:~/Carl/Examples/TF/examples/lite/examples/image_classification/raspberry_pi $ ./run_it.sh
analog clock 0.71 310.1ms
analog clock 0.60 309.5ms
analog clock 0.62 309.8ms
sliding door 0.60 337.7ms
sliding door 0.61 312.8ms
sliding door 0.66 311.9ms
sliding door 0.69 312.2ms
sliding door 0.62 312.3ms
sliding door 0.74 309.8ms
sliding door 0.67 309.7ms
sliding door 0.70 309.8ms
sliding door 0.66 309.7ms
turnstile 0.69 309.9ms
turnstile 0.79 309.8ms
^C
Exiting

Carl was getting low on juice and the TFLite program ran the battery down pretty quick as he exclaimed “Turning around to Docking Approach Point”.

Have to test more when he gets juiced up.

cleoqc · November 13, 2020, 6:53pm

Liking the new ‘toy’ ?

In GoPiGo OS, I have modified the same script to print to console too. And by experience, you need to handle a confidence level of at least 85% to avoid the non existing sliding door. It did manage to identify my Shih Tzu as an actual Shih Tzu too

cyclicalobsessive · November 13, 2020, 7:07pm

Be sure to look at my version on git, it has all the options plus a save annotated image to file

cyclicalobsessive · November 18, 2020, 12:41am

HW and SW I2C “banging” continues - good news so far:

over 1.1 million SW I2C reads
concurrent with over 23k HW I2C reads

=== Nov 2020 HW and SW I2C Investigation ===

PiOS based Raspbian For Robots 17 Oct 2020 beta release
No Change to DI_Sensors from 17Feb2019 version

Test1: Regular Operation on beta OS release

Distance Sensor in HW I2C mode
IMU in SW I2C mode
Running: juicer, imulog, wheellog, lifelog
imulog (SW I2C) failed after 26 hours
juicer detected fatal I2C failure on HW I2C after 95 hours operation

Test2: Regular Operation w/o using SW I2C/IMU on beta OS

Distance Sensor in HW I2C mode
imulog invocation prevented in crontab-e
Running: juicer, wheellog, lifelog and numerous TensorFlow Lite gpg_classify_picamera.py
(No USB WiFi dongle - giving 8.3h “playtime” vs 6.4h with)
No fatal errors detected in 115 hours

Test3: Add HW I2C stress on Distance Sensor

Distance Sensor in HW I2C mode
imulog invocation prevented in crontab-e
Running: juicer, wheellog, lifelog and numerous TensorFlow Lite gpg_classify_picamera.py
Running: HWDistanceSensor.py
(No USB WiFi dongle - giving 8.3h “playtime”)
No fatal errors detected in 22 hrs, 328k extra HW I2C readings of distance sensor

Test4: Add SW I2C Stress

Distance Sensor in HW I2C mode
imulog invocation prevented in crontab-e
Running: juicer, wheellog, lifelog, status.py perfoms distance sensor readings every six seconds
Running: SW_I2C_Stress_with_IMU.py - over 1 million reads with 62 soft errors, no hard failures
(No USB WiFi dongle - giving 7.7h “playtime” )

19:17:17 1103111
Magnetometer X:  -34.5  Y:  -22.9  Z:  -10.2 Gyroscope X:    0.2  Y:   -0.1  Z:    0.1 Accelerometer X:    0.1  Y:    0.2 Z:    9.5 Euler Heading: 348.1  Roll:   0.4  Pitch:   0.8
19:17:19 1103121
Magnetometer X:  -34.1  Y:  -22.6  Z:  -10.6 Gyroscope X:    0.1  Y:   -0.1  Z:    0.0 Accelerometer X:    0.1  Y:    0.1 Z:    9.4 Euler Heading: 348.1  Roll:   0.4  Pitch:   0.8

New Test Started:

Test5: Heavy HW and SW I2C Stress

Distance Sensor in HW I2C mode
imulog invocation prevented in crontab-e
Running: juicer, wheellog, lifelog,
Running: SW_I2C_Stress_with_IMU.py (10 readings/s), HWDistanceSensor.py (10 readings/s)
(No USB WiFi dongle - giving ?h “playtime”)

=====
temp=54.8'C
 19:50:42 up 7 days,  5:27,  5 users,  load average: 0.66, 0.78, 0.74
19:49:46 9261:distance from object: 604 mm
19:49:49 9271:distance from object: 615 mm
19:49:52 9281:distance from object: 613 mm
19:49:55 9291:distance from object: 612 mm
19:49:57 9301:distance from object: 611 mm
19:50:00 9311:distance from object: 611 mm
19:50:03 9321:distance from object: 614 mm
19:50:06 9331:distance from object: 602 mm
19:50:09 9341:distance from object: 618 mm
19:50:11 9351:distance from object: 605 mm
======
19:50:26 9341
Magnetometer X:  -33.1  Y:  -23.4  Z:   -6.6 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:28 9351
Magnetometer X:  -34.1  Y:  -24.6  Z:   -6.2 Gyroscope X:    0.1  Y:    0.0  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:31 9361
Magnetometer X:  -34.1  Y:  -25.0  Z:   -1.1 Gyroscope X:   -0.1  Y:    0.0  Z:    0.1 Accelerometer X:    0.1  Y:    0.7 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:33 9371
Magnetometer X:  -33.8  Y:  -26.2  Z:   -1.1 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.7 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
19:50:36 9381
Magnetometer X:  -33.8  Y:  -25.8  Z:   -0.8 Gyroscope X:   -0.1  Y:   -0.1  Z:   -0.1 Accelerometer X:    0.1  Y:    0.6 Z:    9.4 Euler Heading: 345.2  Roll:   0.4  Pitch:   3.8
=====
1
IMU (SW I2C) Exceptions

jimrh · November 18, 2020, 4:58pm

So, the conclusions from all this are?

cyclicalobsessive · November 18, 2020, 6:17pm

HW and SW I2C “banging” continues - very good results so far but no conclusions yet.

The original testing was performed on the exact R4R Oct 17 image.
Current testing is being performed on

R4R Oct 17 image base
updated with

sudo apt-get update -y
sudo apt-get full-upgrade -y

One final test remains in plan - to restore the automatic imu logging which uses my “safe” IMU class.

Carl has run very smoothly 24/7 for eight days on this image, at higher processor loads than his usual. He seems quite “happy” so far.

jimrh · November 18, 2020, 6:23pm

Thank you!

Though I might be fairly clever, I have not yet reached the pitch of intelligence where I can look at a blob of numbers as raw data and understand what they’re trying to tell me.

Though I have not yet accomplished what you do, your work fascinates me. It’s intriguing. I really enjoy watching the way you carefully and precisely chop up the Dexter/MR designs into tiny pieces and then reassemble them into something that works.

What you’ve discovered is ultimately destined to save me, and others, much time and angst.

Again,

Thank you!

KeithW · November 18, 2020, 6:56pm

I’m with @jimrh - thanks for your thorough testing. I’m still very, very new on the platform, so this is quite helpful.
/K

cleoqc · November 18, 2020, 11:24pm

We at ModBot also love your reports, @cyclicalobsessive !

cyclicalobsessive · November 20, 2020, 5:21pm

Dexter/GoPiGo3/Software/Python/Examples Tests

Control_Panel - Works Fully

easy_Blinkers.py Works Fully

easy_DexEyes.py Works Fully

easy_Distance_Sensor.py Works Fully

easy_Motors.py Works Fully

LED.py Works Fully

Motor_Encoder.py Works Fully

Motor_Position.py Works Fully

Motor.py

Works but ramps forward fast, then ramps backward slowly not same distance, both are not straight line

Motor_Speed.py -

ISSUE: Works One Time, Second Invocation appears to be working but no motor commands get issued until I move bot with Control Panel, then it goes out of control - I could not stop it with Control Panel.

Probably would run fine when it is the only instance of a GoPiGo3 class. Carl had several EasyGoPiGo3(use_mutex=True) classes running during this test.

Motor_Turn.py Works Fully

Read_Info.py Works Fully

Servo.py Works Fully

Did Not Test

easy_Button.py

easy_Buzzer.py

easy_LED.py

easy_Light_Sensor.py

gps_reading.py

Grove_Buzzer.py

Grove_I2C.py

Grove_IR.py

Grove_LED.py

Grove_Light_Sensor.py

Grove.py

Grove_US2.py

Grove_US.py

Line_Sensor.py

Motor_Follow.py

Remote_Control.py

cyclicalobsessive · November 20, 2020, 6:51pm

=== 20Nov2020: All Tests Passed - Carl has “Blessed” this beta OS version ==

Disclaimers - YMMV:

Carl runs a Pi3B
IPv6 disabled
File system expanded w/raspi-config
Carl is a robot - Robots think different than humans.
Remote Desktop does not show Desktop widgets, work around is to open desktop folder, or LogOut and Log back in.

cyclicalobsessive · November 21, 2020, 3:08am

There is this setting checked:

Screen Shot 2020-11-20 at 8.35.57 PM

Today, I also lost the desktop. When I connect through noVNC or through remote desktop, I get to a ModRobotics desktop without the desktop widgets.

Work around is to open desktop folder, or LogOut and Log back in.
(Does not happen if Logout then Disconnect Session, Then Reconnect and Login)

cyclicalobsessive · November 26, 2020, 2:08pm

Difficulty configuring RPI-Monitor on R4R PiOS beta

For the last several days, I have been fighting to get RPI-Monitor to properly start at boot on both Carl, and my “DeskPi.”

Carl is running R4R over PiOS, while the DeskPi is running PiOS. Carl was seeing:

Web server not started because of error: Address already in use

I found that starting the RPI-Monitor with Carl’s reserved IP prevents the error to allow this useful tool to work.

And immediately, RPI-Monitor showed how smart Carl actually is!

Recently, I changed Carl’s recharger which lowered his maximum recharge rate from 1A to 0.9A. Carl was getting off his dock before the trickle charge light came on due to “Rule 230c - Probable Trickle Not Detected after 3.5 hour charging”, so I changed the time for that rule to 4.0 hours to provide a little more time.

Today is the first day I have RPI-Monitor running that I can conveniently review the recharge voltage curve, and as I was doing so Carl announced “Trickle Charging - Dismounting”. I quickly looked at the charger to see the status LED showing solid green which means Carl was indeed trickle charging.

Looking through his recharging log showed it still was due to rule 230c - Probable Trickling Not Detected (after 4 hrs), but reviewing the RPI-Monitor shows that the charger had indeed not long before switched to trickle charging - Carl is so smart!

(I would have preferred to see that Carl detected the trickle charge when the average voltage started going down instead of up, but the important thing is to get off the dock when the charger is trickling.)

Who would believe the happiness playing with Carl can give? I’ve been up for two hours and still haven’t prepared my breakfast.

Update: The next morning Carl recognized trickle at 3.8 hours on the dock (rule 230: over 12v and mean voltage has negative slope), before the 4 hour “probable trickle” rule 230c could fire!

jimrh · November 26, 2020, 3:17pm

Any idea what causes this in general? What must be done to (generally) avoid this error in the future?

IOW: Given a user who has never used this before, what must be changed in either the code or it’s documentation to prevent this from occurring?

cyclicalobsessive · November 26, 2020, 3:57pm

I suspect that RPI-Monitor is trying to launch a web server on 127.0.0.1 which the R4R apache2 server has claimed:

pi@Carl:~/Carl $ ps -ef | grep apache
root       753     1  0 Nov25 ?        00:00:06 /usr/sbin/apache2 -k start
www-data 17069   753  0 00:00 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 17070   753  0 00:00 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 17072   753  0 00:00 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 17073   753  0 00:00 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 17074   753  0 00:00 ?        00:00:00 /usr/sbin/apache2 -k start
pi       21655  1483  0 10:27 pts/0    00:00:00 grep --color=auto apache
pi@Carl:~/Carl $ sudo systemctl status rpimonitor
● rpimonitor.service - RPi-Monitor daemon
   Loaded: loaded (/etc/systemd/system/rpimonitor.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-25 15:46:13 EST; 18h ago
 Main PID: 20282 (rpimonitord)
    Tasks: 5 (limit: 1939)
   CGroup: /system.slice/rpimonitor.service
           ├─20282 /usr/bin/perl /usr/bin/rpimonitord -a 10.0.0.186
           ├─20291 /usr/bin/perl /usr/bin/rpimonitord -a 10.0.0.186
           ├─20292 /usr/bin/perl /usr/bin/rpimonitord -a 10.0.0.186
           ├─21683 sh -c /home/pi/Carl/plib/getChargingStateStr.py 2>/dev/null
           └─21684 python3 /home/pi/Carl/plib/getChargingStateStr.py

Nov 25 15:46:13 Carl systemd[1]: Started RPi-Monitor daemon.





pi@Carl:~/Carl $ sudo systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-11-25 10:18:27 EST; 1 day 4h ago
     Docs: https://httpd.apache.org/docs/2.4/
  Process: 17057 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
 Main PID: 753 (apache2)
    Tasks: 11 (limit: 1939)
   CGroup: /system.slice/apache2.service
           ├─  753 /usr/sbin/apache2 -k start
           ├─ 1884 /usr/sbin/apache2 -k start
           ├─ 1892 /usr/sbin/apache2 -k start
           ├─ 1893 /usr/sbin/apache2 -k start
           ├─ 1894 /usr/sbin/apache2 -k start
           ├─ 1898 /usr/sbin/apache2 -k start
           ├─17069 /usr/sbin/apache2 -k start
           ├─17070 /usr/sbin/apache2 -k start
           ├─17072 /usr/sbin/apache2 -k start
           ├─17073 /usr/sbin/apache2 -k start
           └─17074 /usr/sbin/apache2 -k start

Nov 25 10:18:22 Carl systemd[1]: Starting The Apache HTTP Server...
Nov 25 10:18:26 Carl apachectl[515]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' dir
Nov 25 10:18:27 Carl systemd[1]: Started The Apache HTTP Server.
Nov 26 00:00:55 Carl systemd[1]: Reloading The Apache HTTP Server.
Nov 26 00:00:55 Carl apachectl[17057]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' d
Nov 26 00:00:55 Carl systemd[1]: Reloaded The Apache HTTP Server.

thus the success of instructing the RPI-Monitor to use the reserved DHCP address I assigned to Carl’s on-board WiFi instead of the 127.0.0.1 localhost commonly used as a default server address.

This is the pertinent installation change that I made to make it work:

RPi-Monitor for GoPiGo3 Carl

* Documentation:

https://xavierberger.github.io/RPi-Monitor-docs/11_installation.html
https://xavierberger.github.io/RPi-Monitor-docs/index.html

* Installation  (first two added for "site refused to connect")
sudo apt-get install libregexp-common-perl
sudo apt-get install liburi-perl
sudo apt-get install dirmngr
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 2C0D3C0F
sudo wget http://goo.gl/vewCLL -O /etc/apt/sources.list.d/rpimonitor.list
sudo apt-get update
sudo apt-get install rpimonitor


==== Nov 2020 PiOS R4R install ====

something weird at boot
Runs just fine when run:
ps -ef | grep rpi
(kill off all rpimonitor - rpimonitord + three perl jobs)
sudo rpimonitord -a 10.0.0.186 -l /var/log/rpimonitord.log &


pkg install of rpimonitor places /etc/init.d/rpimonitor which is the "old" method of starting services.  The "new" method is to create /etc/systemd/system/rpimonitor.service , and enable it.

(see https://github.com/XavierBerger/RPi-Monitor/issues/302)




For some reason I have to set the address to 10.0.0.186 or it gets  

Web server not started because of error: Address already in use



sudo nano /etc/systemd/system/rpimonitor.service

# #####
[Unit]
Description=RPi-Monitor daemon
Before=multi-user.target
After=remote-fs.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
Restart=on-failure
KillMode=mixed
Nice=19
ExecStart=/usr/bin/rpimonitord -a 10.0.0.186
ExecStop=/bin/kill $MAINPID
StandardOutput=append:/var/log/rpimonitor.log
StandardError=append:/var/log/rpimonitor.log

[Install]
WantedBy=multi-user.target
# ######

then 

sudo systemctl daemon-reload
sudo systemctl enable rpimonitor
sudo systemctl start rpimonitor


(reboot to confirm startup at boot)

That gets the RPI-Monitor up showing processor load and temperature, disk and memory statistics.

Getting the RPI-Monitor to track and display Carl’s battery voltage is a complex configuration of somewhat Carl specific configuration that could easily be generalized for any GoPiGo bot.

If you decide to try installing RPI-Monitor, let me know when it is up and running. I’ll create an “any GoPiGo3 config” for you.

jimrh · November 26, 2020, 6:33pm

Thanks!

I don’t think that will be necessary any time soon, there are enough web servers running on Charlie right now, just with the VNC server and the Remote Camera Robot stuff.

However, it is interesting to see you analysis of the problem and your suspicion that it wanted to appropriate localhost. (And why do you have an Apache instance running anyway? )

cyclicalobsessive · November 26, 2020, 7:11pm

Raspbian For Robots (PiOS version) uses apache2 to serve up the R4R landing page:

/var/www/html/index_buster.php 

<title>Dexter Industries Raspbian for Robots</title>
...
<section>
  <p>
    Welcome to Raspbian for Robots, our custom software for your Dexter Industries robots! There are two ways to view and program your robot.
  </p>
  <p>
    The easiest and most user friendly way for beginners is to go in through VNC (virtual network connections), which will show you a little desktop in your browser wi
th icons and folders.
  </p>
  <p>
    If you are more advanced and want to work in the command line, choose Terminal and have fun!
  </p>
</section>

And also interesting the apache error log:

pi@Carl:~/Carl $ sudo more /var/log/apache2/error.log
[Thu Nov 26 00:00:56.003163 2020] [mpm_prefork:notice] [pid 753] AH00163: Apache/2.4.38 (Raspbian) configured -- resuming normal operations
[Thu Nov 26 00:00:56.003303 2020] [core:notice] [pid 753] AH00094: Command line: '/usr/sbin/apache2'
[Thu Nov 26 14:03:13.900447 2020] [php7:notice] [pid 17069] [client 10.0.0.129:50219] PHP Notice:  Undefined offset: 0 in /var/www/html/index.php on line 4
[Thu Nov 26 14:03:13.930312 2020] [php7:notice] [pid 17069] [client 10.0.0.129:50219] PHP Notice:  Undefined variable: ips in /var/www/html/index.php on line 76
[Thu Nov 26 14:03:13.930438 2020] [php7:warn] [pid 17069] [client 10.0.0.129:50219] PHP Warning:  Invalid argument supplied for foreach() in /var/www/html/index.php on
 line 76