GoPi5Go-Dave Achievements, Plans and Travails

cyclicalobsessive · March 26, 2024, 12:36pm

I came to the conclusion those lines of code are for setting up the SPI interface, which perhaps are needed if you don’t manually enable SPI in raspi-config (which I always do along with I2C as part of my base OS prep). I only needed to remove the pigpiod dependency for Pi5, but it may be it could have been eliminated long ago. (I cannot test other configs now that Dave has a Pi5 - only one PiOS will work on Pi5.)

This whole process (Bookworm PiOS and now Pi5) has led me to diverge farther and farther from the official ~/Dexter/GoPiGo3/Software. I think the only officially supported OS is the pre-packaged GoPiGo OS image. Probably still the right business decision since new GoPiGo3 users don’t seem to be showing up every Monday and Thursday asking for the latest PiOS, but if they will show up I think they can follow my documentation to get up and running on Pi4 or Pi5.

cyclicalobsessive · March 26, 2024, 1:45pm

[Solved] New Scare: Input/output error - (SDCard Extender is Cause)

Have the feeling the sdCard is going bad - getting frequent “Input/output error” that causes terminal to report:

$ ls
-bash: /usr/bin/ls: Input/output error

Every post on the Internet talks about disks going bad, so I’m backing up the suspect card to a new sdCard.

@jimrh how do I force an fsck on the next reboot w/o setting up tune2fs.

KeithW · March 27, 2024, 12:06pm

good luck - hope that fixes the problem
/K

cyclicalobsessive · March 27, 2024, 3:20pm

Had GoPi5Go-Dave do 101 360-degree rotations (Gotta love Python…) and at the end - no cigars:

===== Spin 360.0 WHEEL_DIAMETER:66.05 WHEEL_BASE_WIDTH:105.06 mm at 150 dps ========
Encoder Values: 56644 -56686
Turn Processor Time:3815.9ms Wall Time:3.8s
Encoder Value: 57216 -57258
Delta Value: 572 572
Delta Degrees: 359.6 359.6
Accuracy: 99.9% 99.9%
Spin Rate:94.3 dps Wheel Rate:149.9 dps (includes start/stop effect)
=============
 ^^^^ Turn 100 ^^^^

Spin 360 Wheel Dia:66.05 Base:105.06 mm?  (? for help)
^CExiting...
pi@GoPi5Go:~/GoPi5Go/systests/gpg_config $ ls
-bash: /usr/bin/ls: Input/output error

cyclicalobsessive · March 27, 2024, 5:06pm

Found @jimrh post on this:

Use tune2fs to enable periodic mandatory disk-checking on Pi based devices?

In Charlie’s case, I set a more aggressive schedule:
root@Charlie:/home/pi# tune2fs -e remount-ro -i 17 -c 5 -C 6 /dev/mmcblk0p2
tune2fs 1.44.5 (15-Dec-2018)
Setting maximal mount count to 5  [<== Check Charlies file-system every fifth reboot.]
Setting current mount count to 6  [<== start checking and reset at the next reboot.]
Setting error behavior to 2  [<== remount read-only if the filesystem prangs to limit the damage.]
Setting interval between checks to 1468800 seconds  [<== force a re-check every 17 days if it doesn't happen sooner.]
My thinking is that a system like Charlie - or maybe Carl? - is more likely to have “interesting” things happen to it than the typical sits-and-plays-Solitare desktop system. As a consequence, having a system that is more vigilant about disk integrity checking is a Good Thing.

Are there any really good reasons NOT to set up, (at the very least), some kind of periodic integrity test on our 'bots?

References:
[1] fsck - to run the e2fsck utility which checks a Linux ext2/3/4 filesystem for errors.

Super - created last_fsck.sh:

#!/bin/bash

echo -e "Filesystem: /dev/mmcblk0p2 (/)"
sudo tune2fs -l /dev/mmcblk0p2 | grep "Last checked"

and setup_tune2fs.sh:

#!/bin/bash

echo -e "Last filesystem check:"
sudo tune2fs -l /dev/mmcblk0p2 | grep "Last checked"

echo -e "Configuring to run e2fsk next boot"
echo -e "and every 5 boots"
echo -e "and every 7 days"
echo -e "and remount read-only if errors are found"
echo -e "\n"
sudo tune2fs -c 5 -C 6 -i 7d -e remount-ro /dev/mmcblk0p2

and check_filesystem_for_read_only.sh:

#!/bin/bash

echo -e "Root file system status: (ro: read-only,  rw: read-write)"
more /proc/mounts | grep /dev/mmcblk0p2

which reported after the first fsck happened:

pi@GoPi5Go:~/GoPi5Go $ utils/check_filesys_for_read_only.sh 
Root file system status: (ro: read-only,  rw: read-write)
/dev/mmcblk0p2 / ext4 rw,noatime 0 0

Since it didn’t mount as read-only, it “blessed” the image.

Second card - same “eventual” Input/output error after running GoPiGo3 in both C++ and Python. I hate mysteries.

jimrh · March 28, 2024, 8:30am

Short answer:
No.

Longer answer:
Hell no!!

Especially with a system where strange things can, and do, happen an aggressive fsck schedule is extremely important.

The best way to do that is by setting the filesystem’s parameters so that it forces a regular and frequent fsck using tune2fs.

I set Charlie, (and every other Linux system I have) to a five-day, five mount, forced fsck at boot time.

It fsck’s every 6th mount.
It fsck’s after five days if it hasn’t been caught by the mount-count based forced fsck.

Additionally, when I set these parameters in tune2fs, I deliberately set the mount-count to “6”, which forces a fsck on the very next reboot.

One last thing: I set the “on error” behavior to “remount read-only” to prevent further damage.

Viz.:
tune2fs -c 5 -C 6 -i 5 -e "remount-ro" /dev/sda1 (or whatever your device is.)

-c 5 Set it to check after every 5th mount if it hasn’t been caught by something else¹.
-C 6 Set the current mount count to “6” so it immediately forces a fsck when rebooted.
-i 5 Set the interval based checking to every five days if it hasn’t been caught by something else¹.
-e "remount-ro" If something is found, immediately forces the filesystem to remount as a read-only filesystem.

Note this only works if you regularly reboot. If your robot stays on-line continuously, it will never “catch”.

You set this for each partition individually if there are more than one ext-formatted partitions.

====================

“Catch/caught”:
Anything that compels an e2fsck resets the interval and mount-count dependent checking as it is “[condition] since last fsck”.

P.S.
If you are going to perform a manual fsck, also pass the “-D” parameter as it allows e2fsck to optimize the directory tree structure to make it more efficient. Note that it doesn’t change the actual hierarchical structure, (the order of the directories doesn’t change), but optimizes the physical structure within the filesystem to make sure it’s organized efficiently and doesn’t use more i-nodes than necessary. Sort of like defragging, but different.

cyclicalobsessive · March 28, 2024, 11:55am

pi@GoPi5Go:~/GoPi5Go $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev             1896432       0   1896432   0% /dev
tmpfs             414240    5856    408384   2% /run
/dev/mmcblk0p2  30085184 6841856  21695016  24% /
tmpfs            2071152      32   2071120   1% /dev/shm
tmpfs               5120      64      5056   2% /run/lock
/dev/mmcblk0p1    522230   76542    445688  15% /boot/firmware
tmpfs             414224     128    414096   1% /run/user/1000

I set it up for /dev/mmcblk0p2 but that misses the /boot/firmware partition, no?

This is what it says when I try for the whole parent device /dev/mmcblk0:

pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -c 5 -C 6 -i 7d -e remount-ro /dev/mmcblk0
tune2fs 1.47.0 (5-Feb-2023)
tune2fs: Bad magic number in super-block while trying to open /dev/mmcblk0
Found a dos partition table in /dev/mmcblk0



pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -l /dev/mmcblk0p1
tune2fs 1.47.0 (5-Feb-2023)
tune2fs: Bad magic number in super-block while trying to open /dev/mmcblk0p1
/dev/mmcblk0p1 contains a vfat file system labelled 'bootfs'
pi@GoPi5Go:~/GoPi5Go $

jimrh · March 28, 2024, 1:05pm

Rule 1:
Tune2fs is used for a partition¹, (i.e. /dev/sda1, /dev/mmcblk0p2, etc.), not an entire block device, (i.e. /dev/sda, /dev/mmcblk0, etc.).

Rule 2:
Tune2fs is only valid with ext partitions. ext2, ext3 and ext4.

That’s correct. /boot isn’t really a part of root’s ext4 filesystem, being mounted on top of the /boot directory there.

e2fsck only works on the exe2/3/4 partition itself, skipping any overlay mounts as it functions at the partition filesystem level.

tune2fs actually does nothing to the partition per-se as it simply sets partition metadata/flags that control the way the partition operates with respect to its filesystem. For example, “has-journal” is a filesystem attribute. Likewise, so is the current mount count, the maximum mount count, the filesystem’s UUID, label, etc.

If you look up the man-page for ‘mkfs’ will be useful as you can set flags/properties at format time. These properties can also be passed via a manual mount or within the mount string in fstab.

Especially interesting is the “discard” option which enables “discard”/TRIM for solid-state media. (It is disabled by default.) Virtually all SSD’s, both internal and external, support it and some of the later, brand-named, SD cards also support it, or ignore it if specified.

====================

It is possible to format an entire block device, (there are some early systems that used a partitionless hard drive format, such as early Winchester-type MFM/RLL hard drives), but it’s very unusual and is not normally done in a modern operating system.
- I actually did that once while trying to create one of my very first RAID systems on a 'nix box, but it was strange and did weird things - I ended up moving the data off of it and reformatting it in a more conventional way.

jimrh · March 28, 2024, 1:10pm

P.S.
Credit where credit is due:

Though I might be good at this kind of stuff, you absolutely beat the pants off of me when it comes to down-and-dirty coding!

At risk of mangling a great line from Fiddler on the Roof, you code things that would cross a rabbi’s eyes!

cyclicalobsessive · March 28, 2024, 1:26pm

And GoPi5Go-Dave really needs a hardware doctor. I do not believe my GoPiGo code can make the sd card suddenly non-readable to an “ls” command, (and prevent further remote connections).

The GoPiGo3 programs continue driving Dave, and reading the GoPiGo3 battery voltage as long as the program operates on RAM, but die if they try to write to the SDCard.

I have a status program running that says no temperature or voltage throttling is or has occurred, (and that low voltage detection is really sensitive and trustworthy, at least it always has been on Pi3 and Pi4).

Since the only way to recover is to reboot, I don’t know how to see a log entry of why the SDCard became inaccessible.

jimrh · March 28, 2024, 1:30pm

Stupid questions:

Did you try a hard reboot? (i.e. full shutdown, remove power, wait 30 seconds, and reboot.)
Did you try removing the SD card and inserting into a different Linux system? You can fsck it there, even if it remounts r/o, because fsck only works on unmounted partitions.

cyclicalobsessive · March 28, 2024, 1:37pm

No other way to recover, since any command I enter, such as “shutdown -h now” tries to find the shutdown program on the card which is not readable. I have to jerk the power, which luckily is no longer risky because the SDCard is not accessible at that point.

The tune2fs -c 5 -C 6 caused the check before the partition was mounted at boot time, as I understand it, but no, I have not tried from another Linux system.

jimrh · March 28, 2024, 1:38pm

Also run a sudo tune2fs -l /dev/mmcblk0p2 (note that “-l” is a lower-case “L”)

This will list all the current attributes of the partition. Post it here.

jimrh · March 28, 2024, 1:42pm

You can also put a generic file system into Dave temporarily, then insert the card using an SD card USB adapter and fsck it there.

Note that there are two utilities that should be installed on every linux system, including the robots:

gparted.
ddrescue

sudo apt-get install gparted gddrescue will get them for you. This should be a standard part of your initial setup routine.

cyclicalobsessive · March 28, 2024, 1:46pm

pi@GoPi5Go:~/GoPi5Go $ ls
config  Graphics  LICENSE  logs  plib  README.md  saved  status.sh  systests  utils
pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -l /dev/mmcblk0p2
tune2fs 1.47.0 (5-Feb-2023)
Filesystem volume name:   rootfs
Last mounted on:          /
Filesystem UUID:          fc7a1f9e-4967-4f41-a1f5-1b5927e6c5f9
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file dir_nlink extra_isize metadata_csum
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              1886976
Block count:              7659648
Reserved block count:     382982
Free blocks:              5810832
Free inodes:              1709668
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      321
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8064
Inode blocks per group:   504
Flex block group size:    16
Filesystem created:       Fri Mar 15 11:11:08 2024
Last mount time:          Thu Mar 28 07:27:57 2024
Last write time:          Thu Mar 28 09:28:10 2024
Mount count:              4
Maximum mount count:      5
Last checked:             Wed Mar 27 12:49:21 2024
Check interval:           604800 (1 week)
Next check after:         Wed Apr  3 12:49:21 2024
Lifetime writes:          63 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      754db390-ca56-4b3a-b935-14c23d4e6fdb
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0

jimrh · March 28, 2024, 1:55pm

That looks OK, now you need to move it to another system, (or Dave with a copy of GoPiGoO/S), and run a sudo fsck -f -v -D /dev/mmcblk0p2

You might want to run it the first time with a “-n” just to see what it would do. Ultimately you’re going to run it as listed to actually fix it.

cyclicalobsessive · March 28, 2024, 1:58pm

Tested with journalctf -kf running and got a hit after only 6 rotations of Dave:

********* GoPi5Go-Dave STATUS *****
2024-03-28  09:53:36 up  2:25,  3 users,  load average: 0.24, 0.05, 0.02
Current Battery 12.3v    (Reading 11.55v)
5v Supply: 4.98
Processor Temp: 46.6'C
Clock Frequency: 1.50 GHz
throttled=0x0

********* GoPi5Go-Dave STATUS *****
Traceback (most recent call last):
  File "/home/pi/GoPi5Go/plib/status.py", line 131, in <module>
  File "/home/pi/GoPi5Go/plib/status.py", line 119, in main
  File "/home/pi/GoPi5Go/plib/status.py", line 79, in printStatus
  File "/home/pi/GoPi5Go/plib/status.py", line 74, in getUptime
  File "<frozen os>", line 987, in popen
  File "/usr/lib/python3.11/subprocess.py", line 1024, in __init__
  File "/usr/lib/python3.11/subprocess.py", line 1901, in _execute_child
OSError: [Errno 5] Input/output error: '/bin/sh'
./status.sh: error reading input file: Input/output error


pi@GoPi5Go:~/GoPi5Go $ journalctl -kf
Mar 28 07:27:59 GoPi5Go kernel: macb 1f00100000.ethernet: gem-ptp-timer ptp clock registered.
Mar 28 07:27:59 GoPi5Go kernel: Bluetooth: BNEP (Ethernet Emulation) ver 1.3
Mar 28 07:27:59 GoPi5Go kernel: Bluetooth: BNEP filters: protocol multicast
Mar 28 07:27:59 GoPi5Go kernel: Bluetooth: BNEP socket layer initialized
Mar 28 07:27:59 GoPi5Go kernel: Bluetooth: MGMT ver 1.22
Mar 28 07:27:59 GoPi5Go kernel: NET: Registered PF_ALG protocol family
Mar 28 07:27:59 GoPi5Go kernel: brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
Mar 28 07:28:13 GoPi5Go kernel: Bluetooth: RFCOMM TTY layer initialized
Mar 28 07:28:13 GoPi5Go kernel: Bluetooth: RFCOMM socket layer initialized
Mar 28 07:28:13 GoPi5Go kernel: Bluetooth: RFCOMM ver 1.11
Mar 28 09:53:35 GoPi5Go kernel: mmc0: Card stuck being busy! __mmc_poll_for_busy

My rotation program continues to operate just fine because it does not invoke anything that needs to access the sdcard.

jimrh · March 28, 2024, 2:01pm

You need to know the status of the SD card and/or filesystem. There is a distinct possibility that the SD card has begun to go all pear-shaped.

Have you tried removing it and cleaning all the contacts with a clean eraser? Do that and then fsck it.

cyclicalobsessive · March 28, 2024, 2:12pm

Interesting - As part of the recovery from that “mmc0: Card stuck being busy!” (exclamation point theirs…), I plugged the sdcard directly into the Pi5, rather than the sdcard extender.

Also since the mount count was at 4, restarted twice to trigger the fsck:

pi@GoPi5Go:~/GoPi5Go $ utils/last_fsck.sh 
Filesystem: /dev/mmcblk0p2 (/)
Last checked:             Thu Mar 28 10:08:53 2024
pi@GoPi5Go:~/GoPi5Go $ sudo tune2fs -l /dev/mmcblk0p2
tune2fs 1.47.0 (5-Feb-2023)
Filesystem volume name:   rootfs
Last mounted on:          /
Filesystem UUID:          fc7a1f9e-4967-4f41-a1f5-1b5927e6c5f9
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file dir_nlink extra_isize metadata_csum
Filesystem flags:         unsigned_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              1886976
Block count:              7659648
Reserved block count:     382982
Free blocks:              5808805
Free inodes:              1709688
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      321
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8064
Inode blocks per group:   504
Flex block group size:    16
Filesystem created:       Fri Mar 15 11:11:08 2024
Last mount time:          Thu Mar 28 10:08:56 2024
Last write time:          Thu Mar 28 10:08:53 2024
Mount count:              1
Maximum mount count:      5
Last checked:             Thu Mar 28 10:08:53 2024
Check interval:           604800 (1 week)
Next check after:         Thu Apr  4 10:08:53 2024
Lifetime writes:          63 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      754db390-ca56-4b3a-b935-14c23d4e6fdb
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x76313ff1

Going to try test again with SDcard in the Pi5 rather than extender… 20 rotations so far no busy notice… maybe… (30 spin360 completed and Dave is only 2 to 3 degrees off original heading… not too shabby)

cyclicalobsessive · March 28, 2024, 2:48pm

450 rotations no “card busy!” issues - only the GoPiGo3 EasyGoPiGo3.target_reached() issue when the 32-bit encoder values roll over, which happens 2.7 times faster with a 16-tick GoPiGo3!

Probably not the best idea to have the sdCard extender ribbon cable going right between those high current motor wires and high frequency encoder wires:

The good thing about all this testing, I have refined Dave’s WHEEL_BASE by 0.03mm making his turns just a little more accurate, and added “o Spin opposite direction” command to my wheelBaseRotateTest.py program. New WHEEL_BASE=105.09mm and WHEEL_DIAMETER=66.05mm