Upgrading GoPiGo OS causes a loss of GoPiGo functionality

Issue:
Doing a complete upgrade, (allowing everything to upgrade), on GoPiGo O/S causes the GoPiGo functionality to stop working.

  • It cannot find the GoPiGo hardware
  • It cannot detect the robot.
  • And as a consequence, nothing else GoPiGo related works.

Note that I am using the term “complete upgrade” deliberately, as running a “full” upgrade may, and can, do additional version related upgrades that are outside the scope of my research.  By “complete” upgrade, I mean running apt-get upgrade and allowing it to install all the upgrades it suggests without modification.

You can restore limited functionality by running sudo pigpiod, but that only restores a small subset of the robot’s functions.

My initial research showed that doing the upgrade, but “holding back” the pigpio, pigpio-tools, and pigpiod updates does not cause the robot to fail.

Also, after a complete upgrade, the logs show errors that pigpiod cannot bind to port 8888

=====================

Subsequent research has been confusing as sometimes the upgrade is completely successful and everything works without exception - but other times the upgrade fails with a loss of GoPiGo functionality as noted above.

Current research shows the following possible suspects:

  • The pgpio tools and libraries themselves.
  • The updated rpicamera libraries - especially if you accept the “maintainer’s version” of the rpicamera config file.
    • This is the current focus of my investigation at the present time.
    • Note that these libraries do not include the new libcamera libraries from Bullseye.
  • IPv6 related issues.
  • Power throttling issues.
  • Something else, as yet undiscovered.

Additionally, it should be noted that the original “as shipped” image for GoPiGo O/S 3.0.1 is not correctly packaged and the root partition within the image is truncated at the end of the partition.  Since it is not possible to know exactly where individual parts of the data are stored within the original media, there is the possibility that something important was lost.

I cannot prove that this is true, but the possibility of data corruption cannot be ignored either.

I do not consider this a likely cause for the issue being described, especially as a subsequent e2fsck does not disclose any issues immediately after the filesystem has been re-expanded and rebooted.  As a consequence, I am assuming the image is not damaged, however it would be better if the original image were not truncated.

I have discovered that I can create a failing installation by updating a freshly built SD card image, though I have trouble duplicating this when I update an installation on my SSD.

It should be noted that the SD card installation is generated by “flashing” the as-shipped image using Etcher.  However the SSD installation is done by rsync’ing a complete image from a SD card after it has booted and re-expanded the partition(s), and rebooted again.  I do not yet know the significance of this difference, if any.  To help clear this up, I have asked Nicole to re-package the pristine GPGOS 3.0.1 image with partitions that are not truncated.  This way, I can rsync directly from the image file without having to re-create it.

Research is continuing. . . .

Any ideas or contributions would be appreciated.

2 Likes

Too far beyond my journeys to even understand, let alone have ideas.

2 Likes

Hopefully. . . .

Once this research is complete I will have a definite answer as to why this happens.  Once that is known, I can communicate that to Nicole and she can guard against that in the next version.

2 Likes

Question - Does running a GoPiGo3 update after running an OS update fix the problems you are seeing?

(Assuming Network Router Connection is configured as default)

  1. Run OS update
  2. Run GoPiGo3 update:
curl -kL dexterindustries.com/update_gopigo3 | bash
curl -kL dexterindustries.com/update_sensors | bash

I have sometimes seen where I had to run the GoPiGo3 update twice to get a good update for some reason. I have not researched it to be able to repro it, but it has happened enough that I now run the update twice if something doesn’t work after an update.

2 Likes

An excellent question.

I have not tried that as the primary focus of this research has been on why an update causes the system to fail in the first place.

My opinion is that a normal system update should not cause the robot to fail if it was working before the update.

2 Likes

Except that the GoPiGo3 software is built over the OS, and the OS update does not attempt to protect or understand the myriad of possible configurations of the OS.

The GoPiGo update installs and configures a specific version of wiringpi,
and builds/installs a fresh pigpiod (based on latest libraries or kernel?) so it would appear that DI is aware of some dependency.

update_gopigo3.sh:

check_if_run_with_pi

parse_cmdline_arguments "$@"
install_rfrtools_repo

clone_gopigo3
install_python_pkgs_and_dependencies
install_gopigp3_power_service
install_list_of_serials_with_16_ticks

install.sh:

# Enable GoPiGo3 Power Services in Systemd
# Make sure the gopigo3_power.py program will run at boot

# Update the Service File with the new directory path
SERVICEFILE=$GOPIGO3_DIR/Install/gpg3_power.service # This should be changed to the location of the sh file first.
SCRIPTFILE=$GOPIGO3_DIR/Install/gpg3_power.sh
SERVICECOMMAND=" ExecStart=/usr/bin/env bash "
# Remove line 8 from the service file, the default location of the Service File
sudo sed -i '6d' $SERVICEFILE
# Add new path and file name of gpg3_power.sh to the service file
sudo sed -i "6i $SERVICECOMMAND $SCRIPTFILE" $SERVICEFILE

check_root_user
install_dependencies
install_pigpio
enable_spi
install_wifi_antenna

AND ALL THIS TO CHANGE THE OS PIGPIOD TO NON-FORKING

$ wget https://github.com/joan2937/pigpio/archive/master.zip
$ unzip master.zip
$ cd pigpio-master
$ make
$ sudo make install
$ cd ..
$ rm master.zip

=== pigpiod

wget https://raw.githubusercontent.com/joan2937/pigpio/master/util/pigpiod.service

(Replacing with the non-forking version - don’t know why)

$ sudo nano pigpiod.service
[Unit]
Description=Pigpio daemon
After=network.target syslog.target
StartLimitIntervalSec=60
StartLimitBurst=5
StartLimitAction=reboot

[Service]
Type=simple
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_time=300
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_intvl=60
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_keepalive_probes=5
# Don't fork pigpiod
ExecStart=/usr/local/bin/pigpiod -g
ExecStop=
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
$ sudo cp pigpiod.service /etc/systemd/system

$ sudo systemctl enable pigpiod.service
$ sudo systemctl start pigpiod.service
$ systemctl status pigpiod.service
2 Likes

I did not even know that this existed. Fantastic.
/K

2 Likes

Note:

This posting contains a number of significant errors of fact and should be disregarded, at least in part.

I have attempted to redact and correct the errors I found, but others may still exist.

Please refer to the updated posting located below
https://forum.dexterindustries.com/t/upgrading-gopigo-os-causes-a-loss-of-gopigo-functionality/8625/12

=======================

Viz.:

Actually the package management software does a pretty good job of “understand[ing] the myriad of possible configurations of the OS” using a variety of techniques as noted below.

Once you understand how the package manager keeps track of things, it’s a simple matter to keep things intact.

P.S.
I had a “crash” (:exploding_head:) course in package management this last week.
:wink:

=================

Assuming that this, (overwriting a previously installed package), is the root cause, the fix is simple:

After the custom package modifications are installed, the installation script should “mark” the packages as “manually” installed “hold” (apt-mark?) so that apt/apt-get knows to leave them alone.

If I remember correctly, (I am not in front of my computer), apt can mark a package to be in one of several statuses:

  • Automatic:  (default)
    This represents a package that is a part of the standard installation, (or was installed using the package manager), and is fair game for updates.
     
    Automatic packages are packages that were installed as dependencies of a “manually” installed package.

  • Hold:  (AKA “held back”)
    This represents a package that was automatically manually installed, but that you, (possibly temporarily), don’t want touched - at least for the time being.  This status is also used by apt/apt-get to “automagically” hold back a package that might be disruptive to a plain-vanilla update, or would modify another package that shouldn’t be touched for whatever reason.

  • Manual
    This indicates to the package manager that this package, and it’s dependencies, are not to be touched under any circumstances whatsoever.  If updating a different package will touch something this package depends on, that package, though marked “automatic”, will become “held back” and not updated.
     
    This represents packages that were “side-loaded” so to speak - that is packages that were specifically downloaded, compiled, installed, or whatever - completely outside of the normal package management process and context.

Side-loaded packages should be, (are supposed to be), automatically classified as “manual” by the package manager.

Though “manual” and “held” are essentially identical functionally, “manual” helps you to know that this is something that was installed outside the package manager’s control.

  • Manual
    Manual packages are those that were installed either by the distribution maintainer or by the user, using the package manager.

================

What may be happening here is that an “automatically” [or manually] installed package is being [. . .] overwritten in ways the package manager cannot detect or track, therefore it gets updated.

The solution is that if any previously installed package, (especially if it is a part of the standard installation), is being modified outside of the package managers control, the modifying process/script should mark that package as “manual” “hold”, (held back), to prevent it from being overwritten during any subsequent update.

2 Likes

so does holding/marking pigpio stuff (and possibly wiringpi and RPI.gpio) keep the update from breaking pigpiod?

but if pigpiod is build/linking to something that gets updated perhaps it is deeper than just pigpiod getting updated

2 Likes

That was what I was researching before a file-copy went awry and caused me to loose a huge chunk of data.

The absolutely “correct” answer is “I don’t know.”  However we can make some assumptions that are likely to be true, and generate an answer that has a good chance of being correct.

IF
The sensitive packages are discoverable, and/or they contain libraries that are known dependencies of these packages, (i.e. “known” by apt), then marking the packages as “manual” will solve the problem, as apt won’t touch any dependencies of a known “manual” package, even if the dependencies themselves are not marked “manual”.

IF
There are cross-dependencies, (i.e. a modified package shares a library with an unmodified package that does something else).

THEN
There are two possibilities:

  1. If the library is something that was specifically modified by DI, and the package manager knows that this modified library is a dependency of a package marked “manual”, then the library won’t be updated.  If another package, “X”, also uses that library and wishes to update it, then apt won’t touch it and will not update “X” - it will be “held back”.

  2. If the library is owned by something else, (i.e. it’s a part of a different package, but our package also uses it), then updating the different package will update the library/whatever as apt may not know we are using it too.

    • A corollary of the above are libraries/software “imported” or “included” by other software.  Though there may be a software dependency, there is no package dependency and apt is likely to be oblivious of this.

================

Normally, packages that are “side-loaded”, (i.e. compiled from source or manually copied into the filesystem), aren’t a part of apt’s database and don’t usually exist as package files per se, so apt neither knows nor cares about them.  YOU put it in, so YOU own the maintenance of it.  Custom scripts, hand-copied executables, git-cloned stuff, and such-like are examples of this.

These kind of “packages” are usually entire unto themselves, have their own libraries if needed, and don’t depend on anything other than normally expected system utilities.

On the other hand, consider a package that IS a part of the apt database and that apt assumes is fair game.  (i.e.  I  (apt) installed it, so I  am responsible for maintaining it.)

Now, sneak in - unbeknownst to apt - and make changes to the package’s files.

Apt assumes, (and rightly so, because nobody told it differently), that this package has never been touched since it’s original installation, and apt considers it fair game for updates.

The problem isn’t that apt updates the package - that’s apt’s job.  The problem is that you touched it, and didn’t tell apt.  That’s where marking the packages comes in - that tells apt that you touched it and to stay away.

1 Like

This is brilliant research that I had not known about as I am dealing with pre-existing systems, not hand-built ones and had not gone into the nitty-gritty of the Dexter update process.

It should be noted at this point that one of the huge differences between R4R and GPGOS is that GPGOS does NOT include the ability to do a “Dexter Update” - at least not from the desktop.

If I assume that the difference between a hand-built system, (like an Ubuntu clone), and a Dexter-built system is in the bells-and-whistles and that, under the hood, everything is the same curl’d system, this would make much sense and will save me a considerable amount of time and trouble.
 

(AFAIK), it’s based on a specific library version or set of functionality.  It’s when you update to the latest pigpiod that things go bad.

Knowing that marking the pigpio package set, (pigpio, pigpio-tools, and pigpiod), prevents the problem, (I am still researching this), is only half the story.

The important question becomes “Why?”

Why did Dexter install a specific, (and possibly obsolete?), version of the pigpio stack?  Why does updating pigpio break the Dexter functions?  What within the Dexter functionality needs to change to make it compatible with the as-shipped/as-updated pigpio packages?

@cleoqc,
Care to comment/elaborate/provide guidance here?

Thanks!

1 Like

Update:  (and a needed re-tuning of my mnemonica)

The information as noted above has significant errors.
(I am planning to go back and redact parts of that post, and point people to this posting.)

The error:
“Marking a package as ‘manual’ tells Aptitude to not touch it.”

The truth:
“Marking a package as ‘manual’ indicates that it was installed either by the user of the system or the distribution maintainer when the distribution was created.”  It has no relationship with packages marked “hold” unless they share dependencies.

================

Packages can be in one of four possible states:

  1. Manual
  2. Automatic
  3. Hold  (marked to be “held back”)
  4. Packages?  We don’t need no stinkin’ packages!  :wink:
  • The first, and most important state, is manual.
    • This state indicates packages that were installed using aptitude, (apt, apt-get, or other front-ends for aptitude).&nbsp, This includes packages that are either part of the original distribution as created by the distribution manager, or those specifically installed by the user of the system.  These are, in essence, “top level” packages that represent important and/or visible functionality within the system.
    • Examples of this are packages like the kernel, the package manager, the file manager, editors, and other essential functionality that is visible to the user or is so essential that the system cannot function without it.
       
  • The next important state is automatic.
    • This state indicates packages and/or libraries that were installed to satisfy dependencies of either a manually installed package, or another automatically installed package.
    • Examples of this are libraries or other packages that the manually installed packages need.  For example, kernel header files, (needed by the kernel), graphical libraries or filesystem specific libraries needed by GParted, etc.
       
  • Hold packages are those packages that are either:
    • Marked by the user to prevent them from being modified.
    • Marked by the system because they conflict with other packages that are marked “hold”.
    • Marked by the system (how?) as being potentially disruptive.  (not entirely sure about that.)
       
  • The forth category, :wink:, is for those packages that were manually physically installed, manually downloaded and compiled, “curl’d”, “git-clone’d” or installed in some way that did not involve the package manager.  As a consequence, the package manager has no idea that they even exist.

How this works:

Assume you are going to install a new package, (gparted), for example.  When you run apt-get install gparted mtools (mtools is so gparted can work with 'M’icrosoft file systems, like FAT.), you get some additional information:

The following additional packages will be installed:
libatkmm-1.6-1v5 libcariomm-1.0-1v5 libglibmm-2.4-1v5
libgtkm-2.4-1v5 libpangomm-1.4-1v5 libsigc++=2.0-0v5

These packages represent dependencies required by either gparted or mtools, but are not included in the actual package file for gparted or mtools.  As a consequence, they will be downloaded and installed, but they will be marked “automatic” - that is, dependencies of a “manually” installed package.

Automatic packages only have existence as a part of a manually installed package.  If an automatically installed package no longer has any manually installed packages that depend on it, it is considered “ripe for removal” and generates the following message:

The following package was automatically installed and is no longer required:
   python-colorzero
Use 'sudo apt autoremove' to remove it.

As a general rule, any “automatic” package that is no longer needed by a manually installed package is assumed to be an orphan and can be removed.

Hold packages are a horse of an entirely different hue:
Any package that is marked, either by the user or by the system, as “hold” is specifically avoided by the package manager. Also, any specific dependencies of a “hold” package are also untouchable.

A consequence of this action is that if there are two packages: “X” and"Y" - and package “X” is marked “hold” and a a dependency of “X” is also a dependency of “Y”, package “Y” will also be automatically marked “hold” (held back) because the package manager will not be able to touch that dependency.

Independently installed packages may - inadvertently - collide with installed packages by modifying a file that is part of a package that the package manager controls.

Example:

  • Package “A” was manually installed, or is a part of the original distribution.
    • It has four dependencies:  lib-this, lib-that, lib-those, and lib-these.
  • Package “B” was independently installed, outside the control of the package manager, but it overwrote library “lib-those” with a version that includes a necessary feature that other packages don’t need.

At a later point in time, package “A” gets an update and, because it has no idea that package “B” even exists, it updates lib-those along with the other three.  As a result, package “B” is broken because the necessary feature is now missing.

The solution is to mark all the files touched by package “B” as “hold” so they don’t get modified.

2 Likes