Simple Pleasures: Dave's I2C Mutex Keeps Him Healthy

GoPi5Go-Dave is actually a “Re-incarnated Humble-Dave”.

  • Humble Dave was built upon the ROS 2 Humble / Ubuntu 22.04 for GoPiGo3 SDcard image,
  • and was put to sleep when Create3-WaLI joined the household.

GoPi5Go-Dave was reincarnated when Create3-WaLI proved not up to the challenge.

  • (Create3-WaLI had docking / battery management, setting a standard for the re-incarnated Dave)
  • A Raspberry Pi 5 running Raspberry Pi OS Bookworm replaced Humble-Dave’s Pi4 running Ubuntu 22.04
    • (required GoPiGo3 API changes)
  • ROS 2 Humble / Ubuntu 22.04 moved into a Docker container
    • (required learning Docker build/configure)
  • docking contacts were added to allow 24/7/365 “life” -
    • (requires new program to manage battery level)

“Life” for GoPi5Go-Dave has been complicated by these changes to the point that I had to stop all ROS development on his docking logic. Dave’s safetyShutdown program, which resides outside the Docker container, was interfering with Dave’s battery monitor node, which resides inside the Docker container.

After working with @Jimrh on a mutex for his hardware conflicts, a light-bulb moment gave me a clue how to configure the Docker container to respect the I2C mutex that is supposed to marshall accesses to the INA219 voltage and current sensor.

Today, for the first time in months, I launched GoPi5Go-Dave’s ROS 2 battery node without messing up the docking program running outside the container in the PiOS environment.

I can restart development of Dave’s ROS docking management node. (It has been difficult to call GoPi5Go-Dave a ROS robot when his “life” was totally dependant on non-ROS code and I could not run his ROS nodes without causing Dave to have health issues.)

2 Likes

Care to elaborate on how you did it?

It sounds like a piece of work!

I’m (hopefully!), finished with the repairs to Charlie and the updates to Charlene, so now I will begin to work with the SPI mutex code that @cyclicalobsessive was generous enough to help with.

Hopefully I will get the e-Paper displays working well with the rest of the robot’s code!

Thanks @cyclicalobsessive!

2 Likes

One line added to Docker invocation that makes Docker use the same /var/lock/ folder as PiOS uses. (Unless told otherwise programs in a Docker container only see the container file system, so the mutex inside the container was using the container’s /var/lock/ folder while the PiOS was using the host file system’s /var/lock/ folder.)

This is the command that starts my “GoPi5Go API over ROS 2 Humble Desktop Plus” container running:

run_detached_gopi5goROS2.sh

#!/bin/bash

cd ~/GoPi5Go/ros2ws
# --rm    remove container after running

docker run -dt --net=host \
 -v /dev/snd:/dev/snd \
 -v /dev/input:/dev/input \
 -v /home/pi:/home/pi \
 -v /dev/bus/usb:/dev/bus/usb \
 -v /dev/ttyUSB0:/dev/ttyUSB0 \
 -v /var/lock:/var/lock \              <<---- added this mapping to make the I2C mutex work
 -e TZ=America/New_York \
 -w /home/pi/GoPi5Go/ros2ws \
 --privileged \
 --rm \
 --name gopi5goROS2 \
 gopi5gor2hdp

2 Likes

Why do you remove the container after you went to so much trouble to get it finally configured the way you want to make it work??!!!

This makes about as much sense as formatting your hard drive because you’re shutting down for the day. . .
pound_head

1 Like

I like to always start from a known, clean state. If you stop a Docker container without removing it, it remembers a lot of stuff that I don’t care about, and I don’t have to remember (or check) what was running when it was stopped. (it is standard practice to remove a container when not in use, but the option to only stop it exists.)

Stopping and removing the container does not remove the “Docker Image” which was custom built and configured. The container is a running (or stopped in situ) Docker image.

As a “ROS bot”, Dave boots up in PiOS and starts his life logger and safetyShutdown programs, then launches the “GoPi5Go ROS 2 Humble / Ubuntu 22.04” Docker container (boots Ubuntu), then runs the start_robot_dave.sh script which launches the ROS 2 nodes which make Dave a ROS bot:

  • ros_gopigo3_node

  • battery node (maintains state and statistics for last charge cycle and last discharge cycle)

  • state and joint state publisher (updates position of every component of the robot when robot moves)

  • teleop_twist_joy (wireless joystick control)

  • odometer node (logs all travel in ROS)

  • LIDAR node

  • TTS speech server node

  • [optionally] distance_sensor_node, imu_sensor_node, ultrasonic_ranger node, piCam node

  • [optionally] SLAM toolbox (maintains “Where Am I” estimate using LIDAR, IMU, and encoders)

  • [optionally] Oak-D-W 3D Stereo Depth Camera

  • [optionally] RTABmap visual SLAM (Adds Vision and 3D depth to "Where Am I’ estimate)

  • docking_node <<-- Current focus of development effort

During development, I am shutting down and restarting one or more nodes for incremental feature or issue repair tests, but the container (with ROS/Ubuntu, and the GoPi5Go-Dave nodes) remains running in the background.

When I want to perform the “go for broke” test, I stop/remove the Docker container, reboot the Pi5 and let chron start “ROSbot GoPi5Go-Dave” from dead to “living”, walking, talkin’ Dave.

2 Likes

Congrats on the progress!
/K

2 Likes

Good progress always includes a few steps backward.

I decided I would tolerate my docking_node going direct to the GoPiGo3 API drive_cm() function, to work around the massive “drive_distance node” effort. I had a few options:

  1. Have both the docking_node and the gopigo3_node talking to the red board
  2. Add a /drive_distance service to the gopigo3_node
  3. Create a drive_distance_node that offers a /drive_distance service by talking to the gopigo3 node

Both 2) and 3) involve pretty hairy design so I decided to “cheat a little” and have two processes talking to the GoPiGo3 red board.

With that simplification, my really complex, already debugged Create3-Wali wali_node could be refactored to dave_node.py and GoPi5Go-Dave started getting off his dock and returning to his dock all under ROS 2 control.

Of course, when I tried to have the dave_node access the “EasyINA219 Sensor” a weird hurdle appeared in Dave’s path. Dave was reporting using less power running the ROS docking_node than when running the straight Python in PiOS test_docking.py program, and even when I ran a battery.py program. Both were reading the same INA219 at the same time. Both were getting the same average voltage values, but different “average power” values. Turns out I forgot to add the mutex protection on the average power method. The EasyINA219 sensor is a totally separate GitHub repo with a setup install to GoPi5Go-Dave.

That mystery solved, and the dave_node managing docking at 10 volts and undocking when charge current drops below -175 mA, it was time to work on the logging and persisting important variables across reboots and sleeping.

I think that is mostly sorted so GoPiGo3-Dave is attempting his first full playtime under his new ROS 2 management nodes:

  • dave_node
  • docking_node
  • battery_node

Talking to the GoPiGo3 motors and I2C INA219 voltage/current/power sensor.

pi@GoPi5Go:DOCKER:~/GoPi5Go/ros2ws $ ros2 topic echo --once /battery_state
header:
  stamp:
    sec: 1724711985
    nanosec: 624423703
  frame_id: base_link
volts: 10.365546226501465
milliamps: 731.219482421875
watts: 7.568292617797852
watthours: 3.1786320209503174
charging: false
capacity: 24
percent: 12
last_charge: 0.0
last_discharge: 0.0
---


pi@GoPi5Go:DOCKER:~/GoPi5Go/ros2ws $ ros2 topic echo --once /dock_status 
is_docked: false
is_charging: false
---

pi@GoPi5Go:DOCKER:~/GoPi5Go/ros2ws $ tail nohup.out 
2024-08-26 18:40:42 dave_main_cb(): executing
2024-08-26 18:40:42 dave_main_cb(): dave.state = ready_to_dock
2024-08-26 18:40:42 dave_main_cb(): playtime: battery_status 10.4v 662 mA 7.0W
2024-08-26 18:40:43 dock_status_cb(): dock_status.is_docked False 
2024-08-26 18:40:43 battery_state_cb(): battery_state.charging False 


pi@GoPi5Go:~/GoPi5Go $ tail logs/life.log
** ACCELERATED TEST **
2024-08-26 16:15|dave_node.py| ** GoPi5Go-Dave node started - Undock:-700 mA  Dock:11.00 v **
2024-08-26 16:33|dave_node.py| ** Dave Docking xxx: success at battery 12v after 0.4 hrs playtime **
2024-08-26 16:44|dave_node.py| ** Dave Undocking at battery 100%, docked for 0.2 hrs **
...
2024-08-26 18:08|lifelogger.dEmain execution: 368.15
---- FIRST NORMAL OPERATION TEST
2024-08-26 18:09|dave_node.py| ** GoPi5Go-Dave node started - Undock:-175 mA  Dock:10.00 v **  
...
2024-08-26 19:16|dave_node.py| ** GoPi5Go-Dave ROS Docking 928: success at battery 10v after 2.5 h playtime **

Investment:

  • dave_node: 10 hours (not counting the 20 hours for Create3-Wali wali_node)
  • docking_node: 15 hours
  • battery_node: 6 hours
  • ROS 2: too many hours to guess
  • GoPiGo3: 7 years

Dave has his fingers crossed!

Once this is working it will be time to fire up the other redboard accessing node - gopigo3_node to make sure the “Being a ROS 2 GoPiGo3 robot does not interfere with being a ROS 2 GoPi5Go-Dave robot”.

2 Likes

Sounds like you’re getting this thing knocked!

Yes, progress up this several months long climb. Getting “this thing knocked” got me knackered.

1 Like

 

This is me reworking Charlie, (next: Charlene), to add a voltage select switch to the on-board voltmeter so I can look at either the 5v or 12v rail.
 

And this is the modification to the auxiliary power supply’s enable sense line, (the grey wire), to turn the +5v auxiliary power source on when the GoPiGo controller’s +12 is turned on by the power push-button.
 

This is me getting ready to add a power isolation diode to the GoPiGo, to prevent the Raspberry Pi from back-feeding the controller board and causing the +12v rail’s voltage to skyrocket!
 

And THIS is the “adult beverage” I use to help keep the “I want to throw this :face_with_symbols_over_mouth: thing against a wall!” angst meter under control.

2 Likes

A month on, life inside (the Docker container) and “out of the box” is peaceful on the I2C bus. Yesterday I kept track of the voltage versus discharge time and versus charge time, and updated my software model to more accurately report percent remaining during discharge, and percent remaining during charging with ROSbot GoPi5Go-Dave running his basic nodes (LIDAR / Oak-D-W sensors not running, Localization package not running).

Feeling good about the tiny steps progress while docking tests are going on. (Increased the docking maneuver speed to see if Dave can use inertia to overcome the occasional docking failures.)

Here are three voltage checks: 1) GoPiGo3 redboard, 2) ROS accessing the INA219 sensor, 3) PiOS accessing the INA219 sensor:

pi@GoPi5Go:DOCKER:~/GoPi5Go/ros2ws $ ./check_battery.sh 

*** Checking Battery with GoPiGo3 API
GoPiGo3 Battery Voltage: 10.3 volts

*** Checking Battery with ROS2
ros2 topic echo --once /battery_state
header:
  stamp:
    sec: 1725894738
    nanosec: 739532615
  frame_id: base_link
volts: 10.149746894836426
milliamps: 776.280517578125
watts: 7.959755897521973
watthours: 18.424604415893555
charging: false
capacity: 24
percent: 6
last_charge: 21.1393985748291
last_discharge: 19.289514541625977
---

*** Checking Battery from PiOS
Current Battery 10.16v  7.2% Load: 753mA 7.6W

1 Like

Sounds like you’re making a lot of progress!

Pretty soon you’ll be publishing videos of Dave bringing you some iced tea. . .

We gotta talk about Docker containers.

1 Like

You’re anxious to hear how much I hate containers? I am eager for the day I wipe Docker from my disk and memory.

I wish… Today while walking on the treadmill, I watched an absolutely phenomenal introduction to ROS 2 Navigation (Mapping, Localization, Obstacle Avoidance, Path Planning) with the Turtlebot3 Gazebo simulation. The color left my body as I turned green with jealousy of the completeness of the Turtlebot software support.

1 Like