GoPi5Go-Dave Achievements, Plans and Travails

[cyclicalobsessive]

GoPi5Go-Dave breathed his first trons on March 17, 2024 with the mating of a Raspberry Pi5 to the GoPiGo3 “Dave” stack.

As of this date (Mar 21 2024) Dave has working:

• IP Announcement at boot
• GoPiGo3 Power Button recoded with Python gpiod for Pi5
• Life Logging
• Pi5 Real-Time-Clock based napping with reboot at end of naps
• C++ GoPiGo3 API
• Basic “Drive Dave with GoPiGo3 C++ API from key presses” in C++

*** GoPi5Go Dave TOTAL LIFE STATISTICS ***
Total Awake:   51.4  hrs
Total Naps:     27.05  hrs
Total Life:    78.45  hrs (since Mar 17, 2024)
Sessions (boot):  17
Average Session:  3.0 hrs
Safety Shutdowns:  0
Total Travel:  0.0 meters 0 feet

So the plan is now

• Continue relearning C++ by implementing (with test and debug)
  • the non-sensor methods from easygopigo3.py
• Characterize GoPiGo3-Dave’s power budget and CPU load limit with GoPiGo3 power from GPIO connector
• Recreate the Docker ROS2 environment on Dave’s Pi5
• Implement a no sensor, no servo C++ gopigo3_node
• Either recode my Python ROS 2 odometer node in C++ or use as is since ROS2 Python will work for this
• Implement Text-To-Speech ROS action server to give Dave’s ROS behaviors voice
  • with speech.log
  • observes “quiet time”
  • offers “whisper”, “normal”, “shout” volume levels
• create Dave’s docking_manager node
  • shouts “HEY YOU HUMAN, I NEED A LITTLE HELP HERE… (Please) put me on the dock”
  • Can detect “charging”, “fully charged”, “not charging”, “charge low”, “charge critically low”
  • Drives off the dock if battery is fully charged and time of day between 8am and 10pm
• Create a C++ Grove Ultrasonic Ranger node to protect Dave (LIDAR doesn’t see black holes)
• Create a C++ “safe wander” action server (or mod the Python Create3-Wali safe wander node)
  • Create3 had a virtual wall detector - might investigate if the Grove IR Remote detector sees the iRobot virtual wall)
• Create a C++ wall follower action server
• [Automate docking?]
• Integrate a ROS 2 node for the MPU9250 IMU
• Return to learning RTABmap/Stereo Depth Camera/with LIDAR for mapping/localization/navigation

Just a year or two worth of “pleasure and pain” ahead.

[KeithW]

@cyclicalobsessive

Just a year or two worth of “pleasure and pain” ahead.

My High School Marine Corps JROTC drill instructor always used to say “No pain, no gain.”
Of course he said that standing by the side of the track smoking a cigarette as we all ran…
/K

[cyclicalobsessive]

@cyclicalobsessive

Characterize GoPiGo3-Dave’s power budget and CPU load limit with GoPiGo3 power from GPIO connector

GoPiGo3 Supports Raspberry Pi5 Running FULL-BORE!

Early testing is encouraging:

• Idle Playtime (no ROS active, Oak-D-Lite and LIDAR powered direct from battery but idle):
  • Raspberry Pi5 4GB with Pi5 cooler
  • Pulling 6.5W at 10v from battery to GoPiGo3 red board
  • 3h 48m to battery protection shutdown
  • Data collected with C++ EasyGoPiGo3 API, plotted with Python matplotlib)

• 3 CPU stress test pulls 10.9W from battery 10v 59C max temp
• 4 CPU stress test pulls 11.5W ave 12.1W max from battery 10v, 62C max temp

[KeithW]

@cyclicalobsessive

3h 48m to battery protection shutdown

And a solid 3h above 10V, which is a pretty good amount of run time, and provides some additional safety margin for the battery.
/K

[cyclicalobsessive]

@KeithW

solid 3h above 10V, which is a pretty good amount of run time, and provides some additional safety margin for the battery.

Humble Dave was programmed to start flashing his LED at 10v and safety shutdown was at 9.75v

I’ll probably use the 9.75 safety shutdown, and I want to add the “if it is 10v and between 8am and 10pm, shout HEY HUMAN, A LITTLE HELP HERE…”

Encoder Travails:

I was feeling good about my progress porting the Python easygopigo3.py to C++, and even figured out how to use VSCode to remote edit and build everything.

Today, the encoder handling is all over the floor. Whoever wrote the C++ GoPiGo3 used int32_t for the encoders, but 0 ticks minus 1 encoder tick returns a big positive integer like unsigned math, not -1 like signed int32.

I thought I was only going to be porting the EasyGoPiGo3 class but it seems like I need to do some mods on the C++ GoPiGo3 class as well. It is fun relearning C++; although C++ is a total PITA made even more crazy when I can’t trust the C++ that I’m attempting to learn from.

Decided to add my own explicitly unsigned integer encoder and raw encoder methods to the GoPiGo3 class to use in my EasyGoPiGo3 class.

(Using signed int methods and parameters for the encoders is just too confusing for me. For a 6-tick GoPiGo3 MOTOR_TICKS_PER_DEGREE is 2.0. Dividing an unsigned integer UINT_MAX by float 2.0 gives UINT_MAX/2 if done correctly, but UINT_MAX as an integer is -1 and dividing integer -1 by 2.0 is ???. Not sure how to use the official C++ encoder methods, so made my own unsigned int cognizant methods.)

[KeithW]

@cyclicalobsessive

although C++ is a total PITA

that pretty much sums up why I stick to Python
/K

[cyclicalobsessive]

@KeithW

why I stick to Python

Finally Broke Down And “Fixed” Python on the Pi5

What better way to understand some C++ code than to look at the same code in Python!
I had previously done all the work to get the GoPiGo3 Python API working on PiOS Bookworm, but the Pi5 introduced a new issue - no pigpiod on the Pi5.

Well guess how simple the fix for Pi5 was…just commented out the import and the five lines that used pigpio - done.

Python with all the sensors, motors, LEDs working like a charm!

Years we’ve been dancing around this pigpiod package and now I find out it isn’t needed and no one knew!

[KeithW]

@cyclicalobsessive

Years we’ve been dancing around this pigpiod package and now I find out it isn’t needed and no one knew!

That’s hilarious. It must have been there for something at some point.
/K

[cyclicalobsessive]

@KeithW

there for something at some point.

I came to the conclusion those lines of code are for setting up the SPI interface, which perhaps are needed if you don’t manually enable SPI in raspi-config (which I always do along with I2C as part of my base OS prep). I only needed to remove the pigpiod dependency for Pi5, but it may be it could have been eliminated long ago. (I cannot test other configs now that Dave has a Pi5 - only one PiOS will work on Pi5.)

This whole process (Bookworm PiOS and now Pi5) has led me to diverge farther and farther from the official ~/Dexter/GoPiGo3/Software. I think the only officially supported OS is the pre-packaged GoPiGo OS image. Probably still the right business decision since new GoPiGo3 users don’t seem to be showing up every Monday and Thursday asking for the latest PiOS, but if they will show up I think they can follow my documentation to get up and running on Pi4 or Pi5.

[cyclicalobsessive]

[Solved] New Scare: Input/output error - (SDCard Extender is Cause)

Have the feeling the sdCard is going bad - getting frequent “Input/output error” that causes terminal to report:

$ ls
-bash: /usr/bin/ls: Input/output error

Every post on the Internet talks about disks going bad, so I’m backing up the suspect card to a new sdCard.

@jimrh how do I force an fsck on the next reboot w/o setting up tune2fs.

@jimrh

how to program a periodic forced fsck, look up “tune2fs” - the “c”, “C”, and “i” options are your friend! I typically set up my Linux filesystems with -c 5 -C 6 -i 5 -e "remount-ro"

[KeithW]

@cyclicalobsessive

I’m backing up the suspect card to a new sdCard.

good luck - hope that fixes the problem
/K

[cyclicalobsessive]

@KeithW

hope that fixes the problem

Had GoPi5Go-Dave do 101 360-degree rotations (Gotta love Python…) and at the end - no cigars:

===== Spin 360.0 WHEEL_DIAMETER:66.05 WHEEL_BASE_WIDTH:105.06 mm at 150 dps ========
Encoder Values: 56644 -56686
Turn Processor Time:3815.9ms Wall Time:3.8s
Encoder Value: 57216 -57258
Delta Value: 572 572
Delta Degrees: 359.6 359.6
Accuracy: 99.9% 99.9%
Spin Rate:94.3 dps Wheel Rate:149.9 dps (includes start/stop effect)
=============
 ^^^^ Turn 100 ^^^^

Spin 360 Wheel Dia:66.05 Base:105.06 mm?  (? for help)
^CExiting...
pi@GoPi5Go:~/GoPi5Go/systests/gpg_config $ ls
-bash: /usr/bin/ls: Input/output error

[cyclicalobsessive]

@cyclicalobsessive

how do I force an fsck on the next reboot

Found @jimrh post on this:

@jimrh

In Charlie’s case, I set a more aggressive schedule:

root@Charlie:/home/pi# tune2fs -e remount-ro -i 17 -c 5 -C 6 /dev/mmcblk0p2
tune2fs 1.44.5 (15-Dec-2018)
Setting maximal mount count to 5  [<== Check Charlies file-system every fifth reboot.]
Setting current mount count to 6  [<== start checking and reset at the next reboot.]
Setting error behavior to 2  [<== remount read-only if the filesystem prangs to limit the damage.]
Setting interval between checks to 1468800 seconds  [<== force a re-check every 17 days if it doesn't happen sooner.]

My thinking is that a system like Charlie - or maybe Carl? - is more likely to have “interesting” things happen to it than the typical sits-and-plays-Solitare desktop system. As a consequence, having a system that is more vigilant about disk integrity checking is a Good Thing.

Are there any really good reasons NOT to set up, (at the very least), some kind of periodic integrity test on our 'bots?

References:
[1] fsck - to run the e2fsck utility which checks a Linux ext2/3/4 filesystem for errors.

Super - created last_fsck.sh:

#!/bin/bash

echo -e "Filesystem: /dev/mmcblk0p2 (/)"
sudo tune2fs -l /dev/mmcblk0p2 | grep "Last checked"

and setup_tune2fs.sh:

#!/bin/bash

echo -e "Last filesystem check:"
sudo tune2fs -l /dev/mmcblk0p2 | grep "Last checked"

echo -e "Configuring to run e2fsk next boot"
echo -e "and every 5 boots"
echo -e "and every 7 days"
echo -e "and remount read-only if errors are found"
echo -e "\n"
sudo tune2fs -c 5 -C 6 -i 7d -e remount-ro /dev/mmcblk0p2

and check_filesystem_for_read_only.sh:

#!/bin/bash

echo -e "Root file system status: (ro: read-only,  rw: read-write)"
more /proc/mounts | grep /dev/mmcblk0p2

which reported after the first fsck happened:

pi@GoPi5Go:~/GoPi5Go $ utils/check_filesys_for_read_only.sh 
Root file system status: (ro: read-only,  rw: read-write)
/dev/mmcblk0p2 / ext4 rw,noatime 0 0

Since it didn’t mount as read-only, it “blessed” the image.

Second card - same “eventual” Input/output error after running GoPiGo3 in both C++ and Python. I hate mysteries.

[jimrh]

@jimrh

Are there any really good reasons NOT to set up, (at the very least), some kind of periodic integrity test on our 'bots?

Short answer:
No.

Longer answer:
Hell no!!

Especially with a system where strange things can, and do, happen an aggressive fsck schedule is extremely important.

The best way to do that is by setting the filesystem’s parameters so that it forces a regular and frequent fsck using tune2fs.

I set Charlie, (and every other Linux system I have) to a five-day, five mount, forced fsck at boot time.

• It fsck’s every 6th mount.
• It fsck’s after five days if it hasn’t been caught by the mount-count based forced fsck.

Additionally, when I set these parameters in tune2fs, I deliberately set the mount-count to “6”, which forces a fsck on the very next reboot.

One last thing:  I set the “on error” behavior to “remount read-only” to prevent further damage.

Viz.:
tune2fs -c 5 -C 6 -i 5 -e "remount-ro" /dev/sda1 (or whatever your device is.)

• -c 5  Set it to check after every 5th mount if it hasn’t been caught by something else¹.
• -C 6  Set the current mount count to “6” so it immediately forces a fsck when rebooted.
• -i 5  Set the interval based checking to every five days if it hasn’t been caught by something else¹.
• -e "remount-ro"  If something is found, immediately forces the filesystem to remount as a read-only filesystem.

Note this only works if you regularly reboot. If your robot stays on-line continuously, it will never “catch”.

You set this for each partition individually if there are more than one ext-formatted partitions.

====================

1. “Catch/caught”:
   Anything that compels an e2fsck resets the interval and mount-count dependent checking as it is “[condition] since last fsck”.

P.S.
If you are going to perform a manual fsck, also pass the “-D” parameter as it allows e2fsck to optimize the directory tree structure to make it more efficient.  Note that it doesn’t change the actual hierarchical structure, (the order of the directories doesn’t change), but optimizes the physical structure within the filesystem to make sure it’s organized efficiently and doesn’t use more i-nodes than necessary.  Sort of like defragging, but different.

[cyclicalobsessive]

@jimrh

/dev/sda1 (or whatever your device is.)

pi@GoPi5Go:~/GoPi5Go $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev             1896432       0   1896432   0% /dev
tmpfs             414240    5856    408384   2% /run
/dev/mmcblk0p2  30085184 6841856  21695016  24% /
tmpfs            2071152      32   2071120   1% /dev/shm
tmpfs               5120      64      5056   2% /run/lock
/dev/mmcblk0p1    522230   76542    445688  15% /boot/firmware
tmpfs             414224     128    414096   1% /run/user/1000

I set it up for /dev/mmcblk0p2 but that misses the /boot/firmware partition, no?

This is what it says when I try for the whole parent device /dev/mmcblk0:

pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -c 5 -C 6 -i 7d -e remount-ro /dev/mmcblk0
tune2fs 1.47.0 (5-Feb-2023)
tune2fs: Bad magic number in super-block while trying to open /dev/mmcblk0
Found a dos partition table in /dev/mmcblk0



pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -l /dev/mmcblk0p1
tune2fs 1.47.0 (5-Feb-2023)
tune2fs: Bad magic number in super-block while trying to open /dev/mmcblk0p1
/dev/mmcblk0p1 contains a vfat file system labelled 'bootfs'
pi@GoPi5Go:~/GoPi5Go $ 

[jimrh]

Rule 1:
Tune2fs is used for a partition¹,  (i.e.  /dev/sda1, /dev/mmcblk0p2, etc.),  not an entire block device,  (i.e.  /dev/sda, /dev/mmcblk0, etc.).

Rule 2:
Tune2fs is only valid with ext partitions.  ext2, ext3 and ext4.

@cyclicalobsessive

I set it up for /dev/mmcblk0p2 but that misses the /boot/firmware partition, no?

That’s correct.  /boot isn’t really a part of root’s ext4 filesystem, being mounted on top of the /boot directory there.

e2fsck only works on the exe2/3/4 partition itself, skipping any overlay mounts as it functions at the partition filesystem level.

tune2fs actually does nothing to the partition per-se as it simply sets partition metadata/flags that control the way the partition operates with respect to its filesystem.  For example, “has-journal” is a filesystem attribute.  Likewise, so is the current mount count, the maximum mount count, the filesystem’s UUID, label, etc.

If you look up the man-page for ‘mkfs’ will be useful as you can set flags/properties at format time.  These properties can also be passed via a manual mount or within the mount string in fstab.

Especially interesting is the “discard” option which enables “discard”/TRIM for solid-state media.  (It is disabled by default.)  Virtually all SSD’s, both internal and external, support it and some of the later, brand-named, SD cards also support it, or ignore it if specified.

====================

1. It is possible to format an entire block device, (there are some early systems that used a partitionless hard drive format, such as early Winchester-type MFM/RLL hard drives), but it’s very unusual and is not normally done in a modern operating system.
   • I actually did that once while trying to create one of my very first RAID systems on a 'nix box, but it was strange and did weird things - I ended up moving the data off of it and reformatting it in a more conventional way.

[jimrh]

P.S.
Credit where credit is due:

Though I might be good at this kind of stuff, you absolutely beat the pants off of me when it comes to down-and-dirty coding!

At risk of mangling a great line from Fiddler on the Roof, you code things that would cross a rabbi’s eyes!

[cyclicalobsessive]

@jimrh

I might be good at this kind of stuff,

And GoPi5Go-Dave really needs a hardware doctor. I do not believe my GoPiGo code can make the sd card suddenly non-readable to an “ls” command, (and prevent further remote connections).

The GoPiGo3 programs continue driving Dave, and reading the GoPiGo3 battery voltage as long as the program operates on RAM, but die if they try to write to the SDCard.

I have a status program running that says no temperature or voltage throttling is or has occurred, (and that low voltage detection is really sensitive and trustworthy, at least it always has been on Pi3 and Pi4).

Since the only way to recover is to reboot, I don’t know how to see a log entry of why the SDCard became inaccessible.

[jimrh]

Stupid questions:

1. Did you try a hard reboot? (i.e. full shutdown, remove power, wait 30 seconds, and reboot.)
2. Did you try removing the SD card and inserting into a different Linux system? You can fsck it there, even if it remounts r/o, because fsck only works on unmounted partitions.

[cyclicalobsessive]

@jimrh

1. Did you try a hard reboot? (i.e. full shutdown, remove power, wait 30 seconds, and reboot.)

No other way to recover, since any command I enter, such as “shutdown -h now” tries to find the shutdown program on the card which is not readable. I have to jerk the power, which luckily is no longer risky because the SDCard is not accessible at that point.

2. Did you try removing the SD card and inserting into a different Linux system? You can fsck it there, even if it remounts r/o, because fsck only works on unmounted partitions.

The tune2fs -c 5 -C 6 caused the check before the partition was mounted at boot time, as I understand it, but no, I have not tried from another Linux system.