Wanted: A good SPI mutex

One additional question:

Your sample routines contain USER = [xxx] statements and I am assuming that they are simply labels to indicate what’s happening.

I also am assuming that both of the sample programs and the class instance are running as, and are owned by, the “pi” user.

I am also assuming that any file created by the pi user is modifiable by the pi user.

I also assume that the mutex works because “user 2” cannot open the mutex for write access when “user 1” has it.

Question:
Why?

Both processes are owned by the pi user so they should both have write access to the same file, right?

However, it appears that the second process, despite being owned by “pi”, cannot write to an open file it owns.

Possible answer:
open([path/filename], w) returns a file handle, (a numerical pointer to the open file instance), as in FILE_HANDLE = open([path/filename], w) and you access the file using something like write(FILE_HANDLE, bytes).

If a second process, or even a different place in the same process, tries to “open” the same file again via FILE_HANDLE2 = open([path/filename], w), since the file is already open and has an assigned file handle, it can’t be re-opened for write a second time unless the first instance has been formally closed via close(FILE_HANDLE).

Am I correct?

Additional question:

I know that variables can exist within a local function/class scope and multiple instances of the same variable can exist due to scoping rules.

Can there be files with local scope?

Class(Test_Class, [etc.])
    [various statements]

    def(method, [xxx])
         HANDLE =open([path/filename], w)
         (do something with it)
    [more stuff]

(end of class definition)

Process_1

Instance = Test_Class([parameters])
    [do something using the file]
etc

Process_2 (which can run in parallel with process 1)

Instance_2 = Test_Class([parameters])
    [do something using the file]
etc

Are the two file instances local to the class instances or do they reference the same object if [path/file] are identical?

My assumption is that files exist independently of the object creating them, but are “owned” by the creating object.  Therefore the second class instance, if trying to open the same path/filename in an exclusive way (for write) while the first class instance has it open for write will fail and possibly throw an exception.

2 Likes

Translation:
If I try to grab the lock more than once within a particular [process|program|thread|etc.], it will succeed.

However if I have two separate [process|program|thread|etc.]. . .

. . . the calling [whatever] has a unique ID that is not shared by any different [whatever] that may want to use the lock file - and the different [whatever] will fail to get the lock.

Does failing to get the lock throw an exception or simply fail silently w/o a file handle or a non-zero return code?

[. . . .]
throw
    grab_lockfile([path/filename], w)
    [more stuff]

catch
    print("oops! it didn't work!!")
    [more stuff]

or more like

#  if "grab_lockfile" fails to get the lock,
# it returns a non-zero value which evaluates as "true".
if (grab_lockfile([path/filename, w)):
    then:
        print("oopsie!  Someone else has it!")
        [more stuff]
    else:
        print("I got it!  I got it!!")
        [more stuff]
fi
2 Likes

Doesn’t that happen automagically behind the scenes?

My reading of your sample mutex code does not have any reference to “chage the [x] to [y]” and I assumed that exclusive use was an intrinsic attribute of lock files.

2 Likes

(deleted by author that cannot delete for 23 hours)

2 Likes

I’m beginning to feel like Costello in Who’s on first?

  1. A lock file can be successfully opened for write by more than one independent program object.  (i.e. Two entirely independent programs can successfully open the lock file for write.)

  2. However at some point in time, (this is what confuses me), the second process does a “something” that fails, and that’s what tells it that it has to wait.

Let me see if I have this right:

  • Handle = open(path/filename, w) # Always succeeds and returns a|the same file handle.
  • However, fcntl.flock(self.Handle, fcntl.LOCK_EX | fcntl.LOCK_NB) attempts to change some attribute of the file specified by “handle” and only the first one gets it.  Subsequent attempts by different programs always fail.

Because of the asynchronous nature of messing with files/handles, and the way the kernel may or may not queue requests, it’s possible that process2 might execute the “fcntl” statement first, beating process1 to the punch, even if process1 started first, right?

Is it possible for the “fcntl” requests to collide?  (i.e. Happen at the identically same time?)  Or is that intrinsically handled by the kernel, and the kernel forces each operation to be atomic, exclusive unto itself?  (I am assuming "yes the kernel somehow queues the I/O requests and prevents them from colliding so two programs cannot execute the exact same file command at the identically exact same time.)

Then again, that asks the question:  Does the kernel force file operations to be atomic?  (i.e. two requests to open the same file cannot interleave with parts of one request being mixed with parts of another.)  I am also assuming that a low-level kernel function like write_bytes(sector, bytes) is executed more like write_bytes(sector, bytes, atomic).  Therefore, absent a mutex, two different programs can execute the same thing, but not absolutely simultaneously.  (i.e. The data will not be interleaved and blended together, but one can overwrite the other unless prevented somehow.)

Am I understanding this correctly?

1 Like

OK, now that we have the python path issue solved. . .

Given:

  1. The existence of mutex code somewhere in the GoPiGo libraries.

  2. The GoPiGo library creates a lockfile named “LOCKFILE” in /home/pi/.

  3. I have additional code located somewhere else, doing something else, and I incorporate some mutex code into it.

  4. It also wants to create a lockfile called “LOCKFILE” in /home/pi.

  5. These two code objects are not synchronized, and - in fact - have no idea that the other program(s) are even running.

Question:

  1. Will this work?
    Is the mere fact that the lockfile is given the same name and is located in the same directory sufficient?

  2. Is this an appropriate use for a mutex/lockfile?

2 Likes

yes, yes…exactly the purpose and usage

2 Likes

One last wrinkle:

It appears that there is already a system lockfile directory located at /run/lock, and dexter libraries appear to use it for i2c mutexes.

It also appears to be owned by root:root.

It is also annoying to have to make a special lockfile directory/place just for the waveshare code.

Aside from running the code via sudo, is there anything else I can do? (Guess: Stinks being me.)

1 Like

The big problem I want to avoid is creating and managing a mutex in a dozen different places, managing filenames and everything for anything that thinks about using SPI.

My thought is creating a library file that contains a “SPI_Mutex” class that would manage the whole thing for me - from placing the lock-file in the correct location to managing the lock and arbitrating the access.

With this I could include SPI_Mutex as mutex and then use the methods, (for example mutex.get and mutex.release), in each of the various things I want to have a mutex for the SPI buss.

Would you give me a good class template that I can use?

1 Like

OK - I’ve learned a bit through this - guess what - DI already has you covered, we just had to find out how!

The GoPiGo3 API has a class called DI_Mutex that allows passing a name parameter to do everything for us.

The pattern for every user program (located anywhere on the GoPiGo3):

from di_mutex import DI_Mutex

spi_mutex = DI_Mutex("SPI")


try:
    spi_mutex.acquire()
    # do protected SPI access stuff here
finally:
    spi_mutex.release()

I built two examples for testing:

wget https://github.com/slowrunner/GoPi5Go/raw/main/systests/mutex/di_mutex_user.py
chmod +x di_mutex_user.py

wget https://github.com/slowrunner/GoPi5Go/raw/main/systests/mutex/d2_mutex_user.py
chmod +x d2_mutex_user.py


In Terminal 1  $ ./di_mutex_user.py
In Terminal 2  $ ./d2_mutex_user.py

Note acquire times never overlap,
and the mutex file /run/lock/DI_Mutex_SPI will be used for inter-process mutual exclusion

On my system the DI_Mutex is located:

$ unzip -l /usr/local/lib/python3.7/dist-packages/Dexter_AutoDetection_and_I2C_Mutex-0.0.0-py3.7.egg | grep di_mutex
     1700  2020-11-10 13:08   di_mutex.py

PS. I tried to clean up this thread by deleting all my off-track learning posts but got:

1 Like

I really hate doing that unless the posts are 99% angst and another 10% frustration talking, as I believe the process that gets you to the destination is as important as the destination itself.  There’s usually a lot of irreplaceable experimental knowledge along the way!

And You deleted all the example classes!!!  I wanted to copy them!!!  (Growl, growl, hiss, hiss)

Wasn’t it Einstein who said “I’ve learned much more from my failures than I ever did from my successes.”

:wink:

1 Like

Oh, and P.S.

Great find!

I’m sorry that I can’t give that posting five or ten likes as it is absolutely the correct solution, (Qualifier: Within the context of the GoPiGo robot, but that’s the requirement anyway so it’s all good!)

Thanks SO MUCH for all your help and effort you’ve put into to this project.  And the education you’ve given me along the way!

1 Like

You mean the several “doesn’t work” and the one “hard” way to do it on the GoPiGo3? No one should ever look at those. One of the principles of learning is never learn wrong stuff, and another is don’t learn from bad examples.

If you want an example - read the beautiful code the amazing DI folks wrote (on your robot at):
/home/pi/Dexter/lib/Dexter/RFR_Tools/miscellaneous/di_mutex.py

and on Github at: https://github.com/DexterInd/RFR_Tools/blob/master/miscellaneous/di_mutex.py

It is pure Python and Linux, no Dexter dependancies, and will run on any robot using the:

  • put it somewhere in the Python path

or

  • put it somewhere and modify the Python path with:
import os
os.path.insert(1,'/path_to_di_mutex.py/')

1 Like

I found the following thread:

. . . and there was discussion of making the use of the mutex the default.

Question:
Aside from the “pedagogical benefit” of making an inadvertent fool out of the unsuspecting student, is there any real disadvantage to having the mutex on full-time?

1 Like

Yes, it increases the risk that the mutex lock file will get locked by a zombie, and require a ghost-buster visit.

1 Like

Ref:  The following quotes:

. . . and then

Does this mean you do this, but accept the risk that you may end up hung somewhere if the mutex gets deadlocked[1]?

If that’s true, how often do you experience this, if at all?

========== Footnotes ==========

  1. That is, the risk is POSSIBLE though UNLIKELY, as in “going outside” has various risks, (getting hit by lightning[2], an errant airplane landing on your head, getting bitten by a biker-gang of badgers, etc.), though the probability may be quite small.

  2. That actually happened to an uncle of mine many years ago.
    He was out playing golf at a golf course in Norfolk, Va., with a bunch of his buddies on a warm and sunny day, and he was using an iron to get from the fairway to the green.  When he swung the club it got struck by lightning, (from who-knows-where), and it struck him dead on the spot.  Nobody knows for sure, but the suspicion is “heat lightning”.  (i.e.  A “bolt from the blue” as noted below.)

Ref:
https://www.theweathernetwork.com/ca/news/article/lightning-safety-tips

Yes I have to use mutex=True, and even one step further for programs which need read access to the GoPiGo3 but will never issue any effector commands, I created a no_init_easygopigo3.py which allows the initialization option (use_mutex=True, no_init=True) to turn off resetting the GoPiGo3 default speed to 300. Most of my programs set the default to 150 DPI, so if a second program that just wants to read the battery resets the speed to 300, the first program’s commands can be changed mid-motion.

Both Carl and Dave set use_mutex=True and roughly once every 4 to 9 months I have to reboot to clear the I2C error 5 of unknown cause.

Don’t know if you recall, but I had to create an I2C mutex protected BNO055 DI EasyIMU class - for some reason the DI IMU did not get the use_mutex=True option when they added the Software I2C with three channel I2C mutex (global, AD1, and AD2 connected I2C).

Dave no longer has a DI IMU (requiring clock stretching), and I have removed his DI Distance Sensor in light of having a 360 degree LIDAR). I have not seen an I2C error 5 on Dave since.

I do not think I have ever seen the mutex zombie locked, but Carl’s health checker does not test for it.

It is “daily thunderstorm season” here, so much of my days are spent counting the speed of sound seconds before venturing out of the house. We have “armour piercing lightning” roll through that did not read the book on Faraday Shields.

1 Like

prior updated with this after you saw it.

1 Like

When I was “a wee lad” growing up in the southeast corner of Virginia, there was one steadfast, engraved in depleted Uranium, don’t even THINK of disobeying rule:

  • Thunder stops EVERYTHING and everyone immediately goes indoors.
    Even if it’s not raining, if you hear thunder - that’s it.

On the other hand, a summer shower, sans thunder, was a treat because it was God’s sprinkler we kids used to cool off in the 90°+ heat.

1 Like