[Solved? Maybe. . .] Upgrading GoPiGo OS causes a loss of GoPiGo functionality

[jimrh]

Issue:
Doing a complete upgrade, (allowing everything to upgrade), on GoPiGo O/S causes the GoPiGo functionality to stop working.

• It cannot find the GoPiGo hardware
• It cannot detect the robot.
• And as a consequence, nothing else GoPiGo related works.

Note that I am using the term “complete upgrade” deliberately, as running a “full” upgrade may, and can, do additional version related upgrades that are outside the scope of my research.  By “complete” upgrade, I mean running apt-get upgrade and allowing it to install all the upgrades it suggests without modification.

You can restore limited functionality by running sudo pigpiod, but that only restores a small subset of the robot’s functions.

My initial research showed that doing the upgrade, but “holding back” the pigpio, pigpio-tools, and pigpiod updates does not cause the robot to fail.

Also, after a complete upgrade, the logs show errors that pigpiod cannot bind to port 8888

=====================

Subsequent research has been confusing as sometimes the upgrade is completely successful and everything works without exception - but other times the upgrade fails with a loss of GoPiGo functionality as noted above.

Current research shows the following possible suspects:

• The pgpio tools and libraries themselves.
• The updated rpicamera libraries - especially if you accept the “maintainer’s version” of the rpicamera config file.
  • This is the current focus of my investigation at the present time.
  • Note that these libraries do not include the new libcamera libraries from Bullseye.
• IPv6 related issues.
• Power throttling issues.
• Something else, as yet undiscovered.

Additionally, it should be noted that the original “as shipped” image for GoPiGo O/S 3.0.1 is not correctly packaged and the root partition within the image is truncated at the end of the partition.  Since it is not possible to know exactly where individual parts of the data are stored within the original media, there is the possibility that something important was lost.

I cannot prove that this is true, but the possibility of data corruption cannot be ignored either.

I do not consider this a likely cause for the issue being described, especially as a subsequent e2fsck does not disclose any issues immediately after the filesystem has been re-expanded and rebooted.  As a consequence, I am assuming the image is not damaged, however it would be better if the original image were not truncated.

I have discovered that I can create a failing installation by updating a freshly built SD card image, though I have trouble duplicating this when I update an installation on my SSD.

It should be noted that the SD card installation is generated by “flashing” the as-shipped image using Etcher.  However the SSD installation is done by rsync’ing a complete image from a SD card after it has booted and re-expanded the partition(s), and rebooted again.  I do not yet know the significance of this difference, if any.  To help clear this up, I have asked Nicole to re-package the pristine GPGOS 3.0.1 image with partitions that are not truncated.  This way, I can rsync directly from the image file without having to re-create it.

Research is continuing. . . .

Any ideas or contributions would be appreciated.

[cyclicalobsessive]

@jimrh

Any ideas or contributions would be appreciated.

Too far beyond my journeys to even understand, let alone have ideas.

[jimrh]

Hopefully. . . .

Once this research is complete I will have a definite answer as to why this happens.  Once that is known, I can communicate that to Nicole and she can guard against that in the next version.

[cyclicalobsessive]

@jimrh

Any ideas or contributions would be appreciated.

Question - Does running a GoPiGo3 update after running an OS update fix the problems you are seeing?

(Assuming Network Router Connection is configured as default)

1. Run OS update
2. Run GoPiGo3 update:

curl -kL dexterindustries.com/update_gopigo3 | bash
curl -kL dexterindustries.com/update_sensors | bash

I have sometimes seen where I had to run the GoPiGo3 update twice to get a good update for some reason. I have not researched it to be able to repro it, but it has happened enough that I now run the update twice if something doesn’t work after an update.

[jimrh]

@cyclicalobsessive

Does running a GoPiGo3 update after running an OS update fix the problems you are seeing?

An excellent question.

I have not tried that as the primary focus of this research has been on why an update causes the system to fail in the first place.

My opinion is that a normal system update should not cause the robot to fail if it was working before the update.

[cyclicalobsessive]

@jimrh

My opinion is that a normal system update should not cause the robot to fail if it was working before the update.

Except that the GoPiGo3 software is built over the OS, and the OS update does not attempt to protect or understand the myriad of possible configurations of the OS.

The GoPiGo update installs and configures a specific version of wiringpi,
and builds/installs a fresh pigpiod (based on latest libraries or kernel?) so it would appear that DI is aware of some dependency.

update_gopigo3.sh:

check_if_run_with_pi

parse_cmdline_arguments "$@"
install_rfrtools_repo

clone_gopigo3
install_python_pkgs_and_dependencies
install_gopigp3_power_service
install_list_of_serials_with_16_ticks

install.sh:

# Enable GoPiGo3 Power Services in Systemd
# Make sure the gopigo3_power.py program will run at boot

# Update the Service File with the new directory path
SERVICEFILE=$GOPIGO3_DIR/Install/gpg3_power.service # This should be changed to the location of the sh file first.
SCRIPTFILE=$GOPIGO3_DIR/Install/gpg3_power.sh
SERVICECOMMAND=" ExecStart=/usr/bin/env bash "
# Remove line 8 from the service file, the default location of the Service File
sudo sed -i '6d' $SERVICEFILE
# Add new path and file name of gpg3_power.sh to the service file
sudo sed -i "6i $SERVICECOMMAND $SCRIPTFILE" $SERVICEFILE

check_root_user
install_dependencies
install_pigpio
enable_spi
install_wifi_antenna

AND ALL THIS TO CHANGE THE OS PIGPIOD TO NON-FORKING

$ wget https://github.com/joan2937/pigpio/archive/master.zip
$ unzip master.zip
$ cd pigpio-master
$ make
$ sudo make install
$ cd ..
$ rm master.zip

=== pigpiod

wget https://raw.githubusercontent.com/joan2937/pigpio/master/util/pigpiod.service

(Replacing with the non-forking version - don’t know why)

$ sudo nano pigpiod.service

[Unit]
Description=Pigpio daemon
After=network.target syslog.target
StartLimitIntervalSec=60
StartLimitBurst=5
StartLimitAction=reboot

[Service]
Type=simple
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_time=300
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_intvl=60
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_probes=5
# Don't fork pigpiod
ExecStart=/usr/local/bin/pigpiod -g
ExecStop=
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

$ sudo cp pigpiod.service /etc/systemd/system

$ sudo systemctl enable pigpiod.service
$ sudo systemctl start pigpiod.service
$ systemctl status pigpiod.service

[KeithW]

@cyclicalobsessive

Run GoPiGo3 update:

I did not even know that this existed. Fantastic.
/K

[cyclicalobsessive]

@jimrh

The solution is that if any previously installed package, (especially if it is a part of the standard installation), is being modified outside of the package managers control, the modifying process/script should mark that package as “manual” to prevent it from being overwritten during any subsequent update.

so does holding/marking pigpio stuff (and possibly wiringpi and RPI.gpio) keep the update from breaking pigpiod?

but if pigpiod is build/linking to something that gets updated perhaps it is deeper than just pigpiod getting updated

[cyclicalobsessive]

@jimrh

The solution is to mark all the files touched by package “B” as “hold” so they don’t get modified.

I guess I appreciate the review of package management, but all I really care about is the “so what”. Sorry, but I really want concise list of issues, and work-around s for each or “the link to the filed issue”. I can’t find anything in these Dostoevsky-an threads. I get the feeling we need more discipline to hold discussion separate from issues. As the expert on GoPiGo OS 3.0.1 could you create an issue thread that has only one post per issue, and restrict discussion of details, and possible work-arounds, to separate single issue threads.

Maybe I could then not feel lost in your excellent treatises wondering if at the end of all the Ed, there is a conclusion and tested work-around.

[jimrh]

Sorry.

Problem solved.

I will try to be more succinct in the future.

[jimrh]

Problem:
Updating the GoPiGo packages causes problems.

Root cause:
I don’t know.

Possible workaround:
If you want to update do the following:

1. apt-mark hold pigpio pigpio-tools pigpiod
2. When prompted to accept the package maintainer’s version of the raspicam config, say ‘no’.

If you still have problems, let us know.

==========================================

Update:

Re:  Possible filesystem issues with GoPiGo OS 3.n:

Possible file-system issue with GoPiGo OS 3.0.n?

This is the text of a support e-mail sent to @mitch.kremm and @cleoqc: ============================================ Mitch/Nicole, While working with the GoPiGo OS 3.0.1 image and trying to reproduce the “update causes robot functionality to fail” issue, I have been getting inconsistent results. As a result I have been looking more closely at the images themselves. There are certain inconsistencies with the as-shipped GoPiGo OS 3.0.1 image that I would like to bring to your attention. As…

As a consequence of this, I am going to abandon my research on updating the GoPiGo OS until I get an image that behaves consistently when I run a filesystem test on it.

[jimrh]

Update:

Viz:

Possible file-system issue with GoPiGo OS 3.0.n?

I have a clean image and I am uploading it to MediaFire. It will take awhile to upload so I will post a link tomorrow with details on what I did and what happened. ======================== Update: The corrected image is available via the links shown below. What I did to correct it: Flashed image using Etcher and the JMCR SD Card interface built into my laptop. Used rsync -aAxXv to copy the content of the second partition to my Linux partition on my laptop using Mint 19.3 Re-formatted the …

As noted in the above posting, I have created a clean image to test with.  if anyone else wishes to test with it, they’re welcome to download it via the links there.

I am going to continue testing using this image.

[jimrh]

Update:

There is something strange going on here. . .

Previously, it was not possible to update the GoPiGo O/S 3.0.1 without it going all pear-shaped - unless you held back the pigpio utilities.

I am now getting consistent updates, (five?  six?  maybe a few more?), in a row now, including both isolated SD card images and installs that are members of my multi-booting SSD.

Things appear to work correctly, though I will admit to not exhaustively testing the O/S.  In fact my wife had company over yesterday and I “weirded them out” by driving Charlie into the kitchen and having him move his “head” around looking at them before he drove off.

All this was done on a completely updated image, aside from the two or three major raspi-UI update packages that were held back automagically.

As far as I can tell, the pigpio utilities have not changed - at least their version hasn’t changed - so I am not sure what happened.

Perhaps:

• Some other package was updated and that allowed pigpio to work correctly?
• Totally disabling IPv6 on the command line might have helped?
• Or, perhaps, cleaning up the filesystem image did the trick?  (All these new tests were performed on my corrected V2 and V3 images.)

Right now it’s working and - though it really deserves additional research - I hesitate to mess with what appears to be a working system.

Though I realize that everyone is hideously busy, it would be nice of other people could install one of my corrected images and see if it updates successfully.

Thanks!

[cleoqc]

Taking a look at this and I was able to fix my full update by editing this file

/lib/systemd/system/pigpiod.service

so it contains the following, taken from the official pigpiod github

[Unit]
Description=Pigpio daemon

[Service]
Type=forking
PIDFile=pigpio.pid
ExecStart=/usr/bin/pigpiod

[Install]
WantedBy=multi-user.target

[jimrh]

@cleoqc,

Great find!

Supposedly this fixes the “pigpiod” causes a delay on shutdown/reboot issue.

My question is how/why does this fix that issue?  I am curious so that I understand this stuff better.

Thanks!

P.S.
I updated my pigpiod file, saving the old one as a .old file.  We’ll see what happens.

[cyclicalobsessive]

@cleoqc

ExecStart=/usr/bin/pigpiod

OUCH - 2 year old thread … but I’m right exactly on this topic with my building my ROS Noetic Ubuntu20.04-GoPiGo3 image.

in pigpio-master: sudo make install puts the daemon in /usr/local/bin/pigpiod, not /usr/bin/pigpiod where the “comes with pigpio” utils/pigpiod.service expects it.

Interestingly - GoPiGo3 OS has /usr/bin/pigpiod and uses the utils/pigpiod.service exactly as is which specifies Type=forking.

I think the PiOS comes with the pigpiod installed in /usr/bin but I’m downloading from github and sudo make install is putting it in /usr/local/bin.

No biggie, I can make mine work from /usr/local/bin and use the simple forking pigpio.service with changing to /usr/local/bin, (and have an ExecStop for good measure)

[este]

@cyclicalobsessive I think you mean “ROS Noetic-Ubuntu20.04-GoPiGo3 image”, yes?

[cyclicalobsessive]

@este

you mean “ROS Noetic-Ubuntu20.04-GoPiGo3 image”, yes?

Good catch, updated with noetic. I get pushing myself to go, go, go on this project, I start making mistakes.

This pigpio daemon as a service changed from time to time without recording the why and what. I had a more complex service unit file than the GoPiGo OS and I cannot figure out where I got it, nor what all the complexity benefits. For my Noetic OS, I chose to follow what the GoPiGo OS has, with the location the daemon actually is installed, and added an ExecStop function so the daemon can not only started, it can be stopped. Usefull for testing but perhaps not needed in the normal use case.

[este]

Thanks again for what you are doing in this project!