Updating the firmware on a GoPiGo-3, using a Pi-4 board, results in a dead system

GoPiGo
, ,
1

Greetings!

Note that I am inviting @cleoqc, (Nicole), to share and comment on this topic when she has the time.

I have been noticing some interesting things happening with Charlie as I have been working on him and began some investigations that resulted in some extremely startling results.

Configuration:

  • Pi-4 4gig with latest stable firmware.
  • Raspbian for Robots 2019-12-19 with current updates, but NOT the “distribution” update to the latest version of Buster.
  • Power provided to the GoPiGo controller board by an external 12v 3A power adapter, (NOT a wall-cube), in place of a battery pack. Note that “dual power” via both the Raspberry Pi’s power connector and the battery connector was NOT attempted.)
     

While troubleshooting these issues, I attempted both a Dexter software update and a Firmware update to Charlie’s GoPiGo-3 board which is version 3.3.2.

The result was a completely inoperative 'bot and a totally un-responsive GoPiGo controller board.

(The “thunk!” you just heard was me fainting dead away thinking I had killed off Charlie. . . .)

After my wife successfully resuscitated me by applying CPR and an automatic defibrillator she “just happened” to have her pocketbook (:scream:  :astonished:  :crazy_face:)

. . . .I began immediate troubleshooting steps. (I won’t detail them, just the conclusions.) However, I was seriously considering buying a 'scope and logic analyzer to help troubleshoot this issue.

Results of the investigation:

  • Any attempt to update the firmware using a Raspberry Pi-4 results in a completely unresponsive GoPiGo3 controller board and a “detected robot” of “None” on reboot.
    • “Unresponsive” means that no LED’s were active at all, including the power LED, the servos were unresponsive, and the /home/pi/Dexter/detected_robot.txt file shows “None”. (i.e. NO robot controller detected.)
    • This is true even if the Pi-4 board is completely disconnected from everything else except video, mouse/keyboard, and power through the GoPiGo3’s battery connector when the firmware update is attempted.
  • Attempts to update the firmware using a Raspberry Pi-3 always succeeded.
    • Note that this was done with a Pi-3 that was completely disconnected from everything else except the controller board, mouse, keyboard, video, and power as noted above.
    • Also note that prior to upgrading Charlie to a Pi-4, I had attempted firmware updates using a Pi-3 while fully installed and connected without problems.
  • Rebooting after an unsuccessful flash using a Pi-4 requires the power button to be held down until the system completely boots up. Letting it go earlier causes the Pi to loose power as if it had been unplugged.
  • Removing the GoPiGo3 controller board from the Pi-4 and installing it on a stand-alone Pi-3, (using the same SD card), results in the same unresponsive behavior on reboot , including the need to hold down the power button to fully restart the Pi.
  • Re-flashing an unresponsive GoPiGo3 controller board’s firmware from the software updater, while attached to a Pi-3 board, appears to return the board to full functional status.
    • If it works like a 'bot, walks like a 'bot, swims like a 'bot and quacks like a 'bot, I’m assuming that it IS a fully functional 'bot.
  • After re-flashing the firmware using a Pi-3, reattaching it to a Pi-4 works as expected.
  • The only difference visible to me is that the updater uses a "rpi-2.cfg file after identifying the Raspberry Pi board as a Pi-3 and it uses a rpi-4.cfg file when it identifies the board as a Pi-4.
     

I have not done any further troubleshooting or investigation yet.

1 Like
2

Update:

I am digging into what actually goes on behind the scenes when the firmware gets flashed.

I still have a long way to go, but one thing I’ve learned is that they use a tool called Open On-Chip Debugger (http://openocd.org).

I looked at the web site and I think I’m in love! It’s a JTAG/Microcontroller flashing tool/debugger, totally GPL’d and free as a bird!

My hardware/micro-controller self is in heaven, though the command-line syntax appears to be pure hell.

More when I know more.

3

What version was Charlie’s firmware before the update, and what version is it after the update?

My “troubleshooting” reports my board is 3.x.x and firmware is 1.0.0 and the firmware on github has not changed in 3 years so I have been under the impression that all GoPiGo3 boards shipped with the latest firmware. Is that wrong?

1 Like
4

Short answer: No

Longer answer:
3.3.2 is the revision level of the earlier version GoPiGo3 controller board I have.

The firmware version, AFAIK, is and remains 1.0.0. However, in /home/pi/Dexter/Firmware/archives there are a number of sub-1.0 firmware versions available for whatever reason.

Having jinked around with the hardware, (particularly by throwing stuff on the SPI buss, not knowing the potential side effects at the time), caused periodic glitches and potential hardware issues that I solved by “nuking” the base hardware configuration “back to the stone-age” with a firmware reload.

Possibly not necessary, but it’s fast and easy and guarantees that the hardware configuration is totally reset to its out-of-box condition.

It’s also a worthy test in its own right. Some poor sod out there, who doesn’t have the hardware experience, might do this and have a massive coronary, (like I almost had!), thinking he’s totally cooked his semi-expensive 'bot.

BTW, it appears that the only really substantiative difference between the rpi2.cfg and the rpi4.cfg interface configuration files are the programming bit-clock and the location of the firmware base address where the code is loaded on the device during programming.

I am trying to verify this information, just in case.

P.S.
It turns out that the Pi-3 and Pi-4 have no less than six different SPI busses available on the GPIO, assuming you’re not using the pins for something else. They also include up to two chip-enable, (interface hardware select), pins for multi-slave SPI architectures.

If you wanted to get really nutzoid, a one-of-four address select chip would allow as many as three different slaves per buss, assuming that 0x00 is reserved for the “nothing selected, buss idle” condition.

I may experiment with the Velleman touch-screen again if I can get it to work on a different SPI buss than the GoPiGo’s controller.

1 Like
5

I can confirm the issue.
PLEASE DO NOT ATTEMPT TO UPDATE THE FIRMWARE WHILE ON A PI4.

Sorry for yelling, but the problem is there. Something changed in the last OS because that was working last fall. Sigh.

In the meantime, there is no reason to upgrade the firmware at all. It’s been on 1.0.0 for a while and no changes are planned.

Cleo
PS. thanks for reporting this, @jimrh

2 Likes
6

Initial conclusions :
So far I have been able to confirm that the hardware base address is correct as of the latest version of the documentation.

I have to assume that Nicole and the others at MR/DI know what hardware pins to use on the Pi and controller board.

Therefore, the next suspect is the bit-clock.

Kinda’ like the baud rate using a serial interface, if it’s even slightly off, No Enchiladas for you, Senior!

The fundamental problem derives from the fact that this particular firmware flashing algorithm was designed to be used with embedded systems evaluation boards, where things like the clock frequency never change.  Since the system clock of the Pi can change, there is a risk of a blown firmware update even on non-Pi-4 systems, depending on processor temperature, etc.

Note that due to the processor, (SoC), design of the Pi-4, and the tricks being done with the system clock to keep the chip cool and reduce the power consumption, the risk of a blown firmware update on the Pi-4 is far greater than on any previous system.

I have not yet figured out how Nicole et. al. determine the clock constants in the rpi4 config file. (@cleoqc: Any help would be greatly appreciated!)

Next question:
What version of the Pi-4 were you using and what firmware version was installed when you did the initial tests?

Thought:
The bit-clock is derived from the system clock, which cannot be assumed to always be a constant and unchanging value. Therefore, any values that assume a specific value of system clock can, (and have), fail[ed].

Action item:
Once I know what assumptions were used for the Pi-4, I can:

  • Try to determine/verify correct values for the current systems.
  • Try to create a calculated value for these constants based on the actual system clock frequency in use at the time.
    This would insure that the clock constants would be correct for any current or future board or firmware version.
     

What say ye?

7

Thought:
Perhaps this can be migrated out of the Dexter Update panel on production versions of the software and relegated to a maintenance module that can be downloaded and run only if absolutely necessary?

The ability to re-flash firmware may well be important on occasion. However, having a button there for every “button pushin’ cowboy” to hit - especially if they don’t know better - is asking for trouble.

1 Like