[Solved? Maybe. . .] Upgrading GoPiGo OS causes a loss of GoPiGo functionality

Issue:
Doing a complete upgrade, (allowing everything to upgrade), on GoPiGo O/S causes the GoPiGo functionality to stop working.

  • It cannot find the GoPiGo hardware
  • It cannot detect the robot.
  • And as a consequence, nothing else GoPiGo related works.

Note that I am using the term “complete upgrade” deliberately, as running a “full” upgrade may, and can, do additional version related upgrades that are outside the scope of my research.  By “complete” upgrade, I mean running apt-get upgrade and allowing it to install all the upgrades it suggests without modification.

You can restore limited functionality by running sudo pigpiod, but that only restores a small subset of the robot’s functions.

My initial research showed that doing the upgrade, but “holding back” the pigpio, pigpio-tools, and pigpiod updates does not cause the robot to fail.

Also, after a complete upgrade, the logs show errors that pigpiod cannot bind to port 8888

=====================

Subsequent research has been confusing as sometimes the upgrade is completely successful and everything works without exception - but other times the upgrade fails with a loss of GoPiGo functionality as noted above.

Current research shows the following possible suspects:

  • The pgpio tools and libraries themselves.
  • The updated rpicamera libraries - especially if you accept the “maintainer’s version” of the rpicamera config file.
    • This is the current focus of my investigation at the present time.
    • Note that these libraries do not include the new libcamera libraries from Bullseye.
  • IPv6 related issues.
  • Power throttling issues.
  • Something else, as yet undiscovered.

Additionally, it should be noted that the original “as shipped” image for GoPiGo O/S 3.0.1 is not correctly packaged and the root partition within the image is truncated at the end of the partition.  Since it is not possible to know exactly where individual parts of the data are stored within the original media, there is the possibility that something important was lost.

I cannot prove that this is true, but the possibility of data corruption cannot be ignored either.

I do not consider this a likely cause for the issue being described, especially as a subsequent e2fsck does not disclose any issues immediately after the filesystem has been re-expanded and rebooted.  As a consequence, I am assuming the image is not damaged, however it would be better if the original image were not truncated.

I have discovered that I can create a failing installation by updating a freshly built SD card image, though I have trouble duplicating this when I update an installation on my SSD.

It should be noted that the SD card installation is generated by “flashing” the as-shipped image using Etcher.  However the SSD installation is done by rsync’ing a complete image from a SD card after it has booted and re-expanded the partition(s), and rebooted again.  I do not yet know the significance of this difference, if any.  To help clear this up, I have asked Nicole to re-package the pristine GPGOS 3.0.1 image with partitions that are not truncated.  This way, I can rsync directly from the image file without having to re-create it.

Research is continuing. . . .

Any ideas or contributions would be appreciated.

2 Likes

Too far beyond my journeys to even understand, let alone have ideas.

2 Likes

Hopefully. . . .

Once this research is complete I will have a definite answer as to why this happens.  Once that is known, I can communicate that to Nicole and she can guard against that in the next version.

2 Likes

Question - Does running a GoPiGo3 update after running an OS update fix the problems you are seeing?

(Assuming Network Router Connection is configured as default)

  1. Run OS update
  2. Run GoPiGo3 update:
curl -kL dexterindustries.com/update_gopigo3 | bash
curl -kL dexterindustries.com/update_sensors | bash

I have sometimes seen where I had to run the GoPiGo3 update twice to get a good update for some reason. I have not researched it to be able to repro it, but it has happened enough that I now run the update twice if something doesn’t work after an update.

2 Likes

An excellent question.

I have not tried that as the primary focus of this research has been on why an update causes the system to fail in the first place.

My opinion is that a normal system update should not cause the robot to fail if it was working before the update.

2 Likes

Except that the GoPiGo3 software is built over the OS, and the OS update does not attempt to protect or understand the myriad of possible configurations of the OS.

The GoPiGo update installs and configures a specific version of wiringpi,
and builds/installs a fresh pigpiod (based on latest libraries or kernel?) so it would appear that DI is aware of some dependency.

update_gopigo3.sh:

check_if_run_with_pi

parse_cmdline_arguments "$@"
install_rfrtools_repo

clone_gopigo3
install_python_pkgs_and_dependencies
install_gopigp3_power_service
install_list_of_serials_with_16_ticks

install.sh:

# Enable GoPiGo3 Power Services in Systemd
# Make sure the gopigo3_power.py program will run at boot

# Update the Service File with the new directory path
SERVICEFILE=$GOPIGO3_DIR/Install/gpg3_power.service # This should be changed to the location of the sh file first.
SCRIPTFILE=$GOPIGO3_DIR/Install/gpg3_power.sh
SERVICECOMMAND=" ExecStart=/usr/bin/env bash "
# Remove line 8 from the service file, the default location of the Service File
sudo sed -i '6d' $SERVICEFILE
# Add new path and file name of gpg3_power.sh to the service file
sudo sed -i "6i $SERVICECOMMAND $SCRIPTFILE" $SERVICEFILE

check_root_user
install_dependencies
install_pigpio
enable_spi
install_wifi_antenna

AND ALL THIS TO CHANGE THE OS PIGPIOD TO NON-FORKING

$ wget https://github.com/joan2937/pigpio/archive/master.zip
$ unzip master.zip
$ cd pigpio-master
$ make
$ sudo make install
$ cd ..
$ rm master.zip

=== pigpiod

wget https://raw.githubusercontent.com/joan2937/pigpio/master/util/pigpiod.service

(Replacing with the non-forking version - don’t know why)

$ sudo nano pigpiod.service
[Unit]
Description=Pigpio daemon
After=network.target syslog.target
StartLimitIntervalSec=60
StartLimitBurst=5
StartLimitAction=reboot

[Service]
Type=simple
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_time=300
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_intvl=60
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_probes=5
# Don't fork pigpiod
ExecStart=/usr/local/bin/pigpiod -g
ExecStop=
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
$ sudo cp pigpiod.service /etc/systemd/system

$ sudo systemctl enable pigpiod.service
$ sudo systemctl start pigpiod.service
$ systemctl status pigpiod.service
2 Likes

I did not even know that this existed. Fantastic.
/K

2 Likes

so does holding/marking pigpio stuff (and possibly wiringpi and RPI.gpio) keep the update from breaking pigpiod?

but if pigpiod is build/linking to something that gets updated perhaps it is deeper than just pigpiod getting updated

2 Likes

I guess I appreciate the review of package management, but all I really care about is the “so what”. Sorry, but I really want concise list of issues, and work-around s for each or “the link to the filed issue”. I can’t find anything in these Dostoevsky-an threads. I get the feeling we need more discipline to hold discussion separate from issues. As the expert on GoPiGo OS 3.0.1 could you create an issue thread that has only one post per issue, and restrict discussion of details, and possible work-arounds, to separate single issue threads.

Maybe I could then not feel lost in your excellent treatises wondering if at the end of all the Ed, there is a conclusion and tested work-around.

2 Likes

Sorry.

Problem solved.

I will try to be more succinct in the future.

2 Likes

Problem:
Updating the GoPiGo packages causes problems.

Root cause:
I don’t know.

Possible workaround:
If you want to update do the following:

  1. apt-mark hold pigpio pigpio-tools pigpiod
  2. When prompted to accept the package maintainer’s version of the raspicam config, say ‘no’.

If you still have problems, let us know.

==========================================

Update:

Re:  Possible filesystem issues with GoPiGo OS 3.n:

As a consequence of this, I am going to abandon my research on updating the GoPiGo OS until I get an image that behaves consistently when I run a filesystem test on it.

2 Likes

Update:

Viz:

As noted in the above posting, I have created a clean image to test with.  if anyone else wishes to test with it, they’re welcome to download it via the links there.

I am going to continue testing using this image.

1 Like

Update:

There is something strange going on here. . .

Previously, it was not possible to update the GoPiGo O/S 3.0.1 without it going all pear-shaped - unless you held back the pigpio utilities.

I am now getting consistent updates, (five?  six?  maybe a few more?), in a row now, including both isolated SD card images and installs that are members of my multi-booting SSD.

Things appear to work correctly, though I will admit to not exhaustively testing the O/S.  In fact my wife had company over yesterday and I “weirded them out” by driving Charlie into the kitchen and having him move his “head” around looking at them before he drove off.

All this was done on a completely updated image, aside from the two or three major raspi-UI update packages that were held back automagically.

As far as I can tell, the pigpio utilities have not changed - at least their version hasn’t changed - so I am not sure what happened.

Perhaps:

  • Some other package was updated and that allowed pigpio to work correctly?
  • Totally disabling IPv6 on the command line might have helped?
  • Or, perhaps, cleaning up the filesystem image did the trick?  (All these new tests were performed on my corrected V2 and V3 images.)

Right now it’s working and - though it really deserves additional research - I hesitate to mess with what appears to be a working system.

Though I realize that everyone is hideously busy, it would be nice of other people could install one of my corrected images and see if it updates successfully.

Thanks!

2 Likes

Taking a look at this and I was able to fix my full update by editing this file

/lib/systemd/system/pigpiod.service

so it contains the following, taken from the official pigpiod github

[Unit]
Description=Pigpio daemon

[Service]
Type=forking
PIDFile=pigpio.pid
ExecStart=/usr/bin/pigpiod

[Install]
WantedBy=multi-user.target
3 Likes

@cleoqc,

Great find!

Supposedly this fixes the “pigpiod” causes a delay on shutdown/reboot issue.

My question is how/why does this fix that issue?  I am curious so that I understand this stuff better.

Thanks!

P.S.
I updated my pigpiod file, saving the old one as a .old file.  We’ll see what happens.