My "New" Remote Camera Robot Project

GoPiGo
,
1

Continuing the discussion from Questions about the "Remote Camera Robot" project:

Since that thread is rather stale, (and yes, it’s been over a year since I posted anything there), I decided to re-create the conversation here.

If you’re just joining the conversation, you would do well to review the original topic.

First:
What am I even doing?

It has been said, (in other contexts), that unless a robot is completely autonomous, it’s NOT really a robot.

OK, so maybe Charlie is headed toward a life of ignominious failure, achieving as his peak being a fancy R/C car with a pan-and-tilt camera, living his life as a wannabe punter with delusions of grandeur.

Bummer. . . . .

For less than $100-or-so, I can get a FPV drone that flies around and lets me peek wherever I want.  So far, I have (literally) hundreds of dollars invested in Charlie and he can’t even imagine flying around.

  • Lane finding?  Nope.

  • Heat seeking and able to navigate toward or away from a simulated fire?  Nope.

  • Able to return to a charging dock?  Charlie doesn’t even HAVE a “charging dock” - aside from the supplied charger for his battery packs.

  • Mapping?  Fagedaboutit!

  • He can’t recognize colors, shapes, faces, objects, or anything else he sees.

In essence it’s true - he’s a very fancy and very expensive  R/C car that doesn’t even drive that well.

Sigh. . . .

Well, what I HAVE  achieved so far is:

  • Charlie can move, (be driven), around using the mouse and keyboard.

  • Charlie can use the keyboard arrow keys, (and the “home” key), to look around his environment and send pictures back to my laptop while he’s being driven around.

  • I have Charlie configured for remote code development using VS Code and GitHub, with the code being located directly on Charlie.

    • This has the advantage of being able to use the fancy development tools and Python extensions within VS Code, while maintaining the code-base on the 'bot itself.
       

  • I can push/pull from GitHub using my original master branch without having to create multiple projects - and, (hopefully), sync it to my local laptop if I need to.

  • I have set up a key-pair so that VS Code can automatically login via SSH and get things ready to go.

  • I can run code from within VS Code and have it execute directly on the robot.

And all of this without having to use kludges like FileZilla to move code back and forth.

I guess it’s time to get back to my wannabe punter.  Hopefully, eventually, I will be able to create something that really qualifies as a robot.

3 Likes
2

Really something to be proud of.

Now quit resting on your laurels, helping everyone but Charlie.

Help him set a New Year’s Resolution.

2 Likes
3

Over here, they do New Year’s the same way we do Christmas in the States.  They have the tree and on New Year’s Day, they open presents - and the question they ask is “What do you want for New Year’s?”

What do I  want for New Year’s?  An operating system and/or programming environment for Charlie that I can understand and work with, like the GoPiGo O/S.  And I would like one that doesn’t break or go completely pear-shaped every time I sneeze.

This is what I have been working on - getting GoPiGo O/S “ready for prime-time” - so that I have a foundation that I can actually build upon.  I want to be able to work with Charlie, robot or not, and not have to wonder if what just happened is a programming fault of mine, or did the O/S just throw its guts up again?

Charlie deserves no less - and if it helps everyone else, so much the better.

============================

You folks have already been where I am now.  You have generations of experience with Heathkit and Rug Warrior robots, (etc.), and you have already done the exploration that I am doing now.  I am still learning those foundational lessons that you folks learned long ago - about the operating system itself, the libraries, the classes and methods - all about what the various pieces do to make your robot what you want it to be.

As a consequence, Charlie doesn’t have a “goal” right now, he’s still learning how to be what he is and how to be what he is most effectively.

Autonomy?  I need to learn how to make him do what I want him to do before I try to let him do whatever he wants.

Right now, the best he’s achieved is to wander around, bump into something, back up, change direction, and then go bump into something else.  I had battery-powered mechanical toys  in the '60’s that did that.

Eventually, I will want Charlie to actually DO something, not just wander around mindlessly.  Exactly what that “something” is, I don’t know yet.

Even if Charlie never gets to that point, I will have learned much and, hopefully, had fun along the way.

2 Likes
4

Update:

“Getting the (power hungry) monkey off of Charlie’s back.”

Now that GoPiGo O/S has stabilized at V2/V3, and while I wait for a new version from M/R, I have collapsed back into the joystick controlled Remote Camera Robot project.

Since I am not testing operating systems against each other, I really don’t need a whole multi-boot environment; so I decided to take the current version of Charlie’s operating system and project files, and move them to the largest and fastest SD card I have, getting rid of the power hungry SSD.

What I did to accomplish this:

  1. Took the SD Card, (a 128gb A2 rated SD card), and used Gparted to create three partitions:
    • Partition 1:  200 MB, (which is hugely sufficient), for the boot partition.
    • Partition 2:  16 GB for the root partition.
    • Partition 3:  The rest of the device for the Project_Files partition.
    • I named them “boot”, “rootfs”, and “Project” respectively so they would not collide with the names of any other partitions.
       
  2. I mounted the device on my 19.3 Mint system, (which is where I created the partitions), and mounted the original SSD.
    • I used Gparted prior to that to determine that the partitions being used on the SSD for the V2/V3 GoPiGo OS image I am using were partition 5 for the boot partition, partition 8 for the root partition, and partition 11 for the Project Files partition.
       
  3. I used rsync -aAxXv [source_path] [destination_path] to copy the content of partition 5 to the SD card’s boot partiton, partition 8 to the rootfs partition, and selected project file directories, (I don’t need to copy over all my backups while I was testing), to the Project directory.
    • Note that when I copied over the boot and root partitions, I copied [source_path/] - with the following “/” so that it would copy the contents of the directory without the enclosing directory itself.
    • When I copied over the certain, specific subdirectories in the Project_Files partition, I copied [source_path} - without the trailing slash as I wanted both the enclosing directory itself, AND the content to be copied over.
       
  4. Fixups:  (So that the resulting SD card would be bootable)
    • I unmounted and removed the SSD to make things easier.
    • I ran sudo fdisk -l which provides a long listing of all the partitions, both “virtual” and physical.
      Viz.:
       

 

  1. Note the entry Disk identifier: 0xcde45b2d.  This is the “PARTUUID” that is used in the cmdline.txt and fstab to identify the partitions involved.

    • You take this value, (without the 0x), and go to the cmdline.txt file and the fstab file and substitute the correct PARTUUID and partition number.
    • In the /boot/cmdline.txt file I replaced
      root=PARTUUID=ca78f9f0-08 with
      root=PARTUUID=cde45b2d-02.
    • In the /etc/fstab file I replaced
      PARTUUID=ca78f9f0-05 /boot vfat defaults 0 2
      – with –
      PARTUUID=cde45b2d-01 /boot vfat defaults 0 2
       
      – and –
      PARTUUID=ca78f9f0-08 / ext4 defaults,noatime 0 1
      PARTUUID=ca78f9f0-0b /home/pi/Project_Files ext4 defaults,noatime,nofail 0 1
      – with –
      PARTUUID=cde45b2d-02 / ext4 defaults,noatime 0 1
      PARTUUID=cde45b2d-03 /home/pi/Project_Files ext4 defaults,noatime,nofail 0 1
       

  2. Test by booting it up.

    • First boot went OK, except that I forgot to rename the third partition back to “Project_Files”.
       

  3. Try running code remotely in VS code - which failed because I needed to update a path in the program to reflect the new location.

  4. Optional update if using a recent, high performance SD card:
    Enable automatic trim by executing sudo systemctl enable fstrim.timer.

    • You can test if this is possible and/or will work for you by trying sudo fstrim /, and if discard, (trim), is supported on your SD card, it will respond with a certain amount of bytes trimmed.

After that, everything seems to be OK.

Next Steps:
Go back through my code and project notes, and try to figure out what the :face_with_symbols_over_mouth: I was doing when I left the project.

2 Likes
5

Update:

We’re Baaaaaaack!

Viz.:
https://forum.dexterindustries.com/t/how-to-get-a-joystick-a-real-joystick-to-work-in-a-browser/8729/10

Within that thread, I accomplished getting a web browser to read, and display, the motion and button presses of the joystick.  Likewise, I added a bit of button/controller logic to determine when certain motions and activities would take place.

The next steps are to communicate this information to the robot itself so that the robot will respond to the joystick’s commands.

One thought I had was to normalize the joystick values to values that made sense for the action being performed - for example, making the head motion read degrees from 0 to 180 based on joystick position.  I have had second thoughts about that and have decided to return “raw” values to the robot and let the robot’s code normalize them.

Why?

For example, take head motion:
Obviously, using the servos I am using, head motion is constrained to a range of 0 to 180 in each axis, with 90 being centered.  However, it may not be so easy:

  1. The mechanical limits of the servo may not extend over a full 180 degrees, due to the presence of obstructions.  As a consequence, I may have to program “limit stops” to prevent the robot from damaging itself.

  2. The servos may not be exactly centered - that is 90/90 may not be straight ahead and level - so I may have to introduce an offset.  Likewise, this offset may be different for different robots, so hard-coding the data in the joystick routine itself probably isn’t wise.

  3. Returning raw data, (+1 to -1 for the axes and either zero or one for the buttons), allows me to tailor the response to the joystick directly on the robot instead of within the browser.

One additional thing that will have to be done in the actual browser-side code is making the buttons and axes changeable - your buttons and axes may not be the same as MY buttons and axes so there will have to be a way to map the appropriate joystick controls so the software works properly.

Next steps:
Getting the joystick to communicate with the robot.

2 Likes
6

Thanks!

In many ways this particular effort is/was entirely experimental - can I make the controller do THIS particular thing?

Now that I have an idea how to make the controller respond in certain ways to certain input conditions, the next step is to determine the correct balance of “do this in the browser” and “do that on the robot itself”.

My thinking is that an efficient division of responsibility would be to take things that are purely controller related and perform them within the browser-based code - and take the things that involve interpretation of the controller action and put that on the robot.

Of course, some are close calls - the buttons controlling what set of axes, (body or head motion), are active, or the logic for enabling motion at all.

At the present time, my thinking is to try to not dump too much stuff on either set of code, so I do some pre-processing in the browser code:

  • Is “force”, (the robot is not allowed to move unless force > 0), active or not?

  • How is the magnitude of “force” determined?
    (In this case by the absolute value of the y-axis - IF certain buttons are pressed.)

  • Which set of axes, (body or head motion?), are active?

Other things, like direction, (based on the ratio of the wheel speeds, which is based on the position of the joystick), or the absolute position of the pan-and-tilt, (based on head-axis position), will be done on the robot.

The most important requirement is that the robot’s motion and reaction should be reasonable based on a typical user’s expectation of the joystick’s motion.

Any thoughts, ideas, or suggestions would be appreciated!

1 Like
7

Python programming question:

This is the definition of the gamepad’s data structure in JavaScript:

      //  Formal definition of "gopigo3_joystick"
      var gopigo3_joystick = {
        controller_status: 'Disconnected',
        motion_state: 'Waiting for Joystick',
        angle_dir: 'Stopped',
        x_axis: 0.00,
        y_axis: 0.00,
        head_x_axis: 0.00,
        head_y_axis: 0.00,
        force: 0.00,
        trigger_1: 0,   // Partial trigger press (slow speed)
        trigger_2: 0,   // Full trigger press  (faster speed)
        head_enable: 0  // Pinky-switch press  (enable joystick to move head)
      };

. . . and I refer to each element in the structure in dot-notation:  (i.e. gopigo3_joystick.x_axis is how I reference that particular element of that structure). If I specify gopigo3_joystick without a specific element being named, it references all of them, all at once.  (i.e. The function call foo (gopigo3_joystick) passes the entire structure into the function.)

What I want to do is send the entire structure to the robot as a neatly packaged whole.  (i.e. send_data (gopigo3_joystick)), and receive it at the robot end, in Python, as a similar structure that I can identify in a similar way instead of receiving a whole string of stuff.  (i.e. receive_browser_data (gopigo3_joystick)?)

Disclaimer: I don’t know if sending and receiving an entire structure as a package is even possible.

Question:
How do I create a structure like this, probably like a union in C, but I’m not sure what to call it - in Python?

2 Likes
8

Update:

Apparently you don’t.  (Create a structure to receive the responses)

Here’s a snippet of the code that sends some data to the robot:

gopigo3_orientation.state = 'move';
gopigo3_orientation.angle_degrees = data.angle.degree;
if('direction' in data) {
    gopigo3_orientation.angle_dir = data.direction.angle;
}
gopigo3_orientation.force = data.force;
query_string = '?' + $.param(gopigo3_orientation);

// use the following function instead the usual $.post
// so that we can throttle the rate of sending data
// good practice for not "choking" the network
send_throttled_data(server_address, query_string, gopigo3_orientation);

Here’s a snippet of the code that receives the data from the browser:

@app.route("/robot", methods = ["POST"])
def robot_commands():
    # get the query
    args = request.args

    state = args['state']

    angle_degrees = int(float(args['angle_degrees']))

    angle_dir = args['angle_dir']

    force = float(args['force'])

If I’m reading that correctly, (and my money’s on “I’m not” :wink:), the entire structure is sent as name:value pairs where “force” references/contains the value for the force sent - and you can reference an argument more than once to re-read it’s value.

I am going to have to experiment with this.

2 Likes
9

that’s the way most of the time because it is the easiest

2 Likes
10

It turns out that this is correct.

As an experiment, I decided to try to print out the “args” variable that contains the returned arguments with the following result:

INFO:werkzeug:172.31.100.209 - - [31/Dec/2021 16:54:43] "POST /robot?state=stop&angle_degrees=0&angle_dir=none&force=0 HTTP/1.1" 200 -
ImmutableMultiDict([('state', 'Home'), ('angle_degrees', '0'), ('angle_dir', 'none'), ('force', '0')])

. . . with the “ImmutableMultiDict” being the structure with all the name-value pairs.

That makes things SOOOOO much easier as far as collecting the data is concerned. . .

. . . except that the difference between a browser talking to itself and talking to a server isn’t trivial. . .
:thinking:  :roll_eyes:

1 Like
11

From the earlier examples (e.g. state = args['state']) it looked like the results were being stored as a dictionary. But then I saw this. I wasn’t familiar with MultiDict in Python - apparently it lets you have multiple values per key - the values get returned as a list. Still, in the example code it looks like everything works like a regular dictionary (i.e. the assumption is that each key returns a single value).

/K

2 Likes
12

I presume that this allows for the possibility that a particular “key” might have multiple values returned, such as:
“axes” returning 0.00, 0.00, 0.00, 0.00 for the 0th, 1st, 2nd and 3rd controller axes - and it’s up to you to figure it out in code.

Me, I’m returning one value per key to make things easier.

2 Likes
13

Update:

Browsers are getting too smart for their own good - or maybe it’s the malware writers are getting too smart for OUR own good.

One manifestation of this is that browsers are now severely restricting “cross site resource sharing” (CORS) requests - even if it’s on your own filesystem - because of some clever malware writers.

The result is that in order to make something like my New Remote Camera Robot software work, I have to specifically allow cross-site resource sharing.

You can read about it here if you want.

Allow “CORS” (Cross Origin Resource Sharing) in pages served by the GoPiGo

The next problem to solve is that browsers are not allowing gamepad support unless it’s a secure site. . . and how are we supposed to generate a secure site for any robot being used by anyone?

Sigh. . .

2 Likes
14

Here’s a theory kind of question:

Given:

  1. Motion and servo movement requests can be (relatively) time intensive.
  2. Browser “blocking” - where the browser appears to stall - is a Bad Thing.

The Usual Case:
The “usual case” for browser/server interaction is that the browser does something the server must respond to, the server does it, and then sends a “Yo! I’m done and everything’s a Thing of Beauty!”

My Specific Case:
In my case, the completion of the task, or failure to complete the task, isn’t really the browser’s business.  What’s important is “Did the server understand the request - or was it effed-up beyond all recognition?”

So, this becomes a theory question, (that I will experiment with, but would like your considered input on).

Should I respond as early as I can?  Or, should I wait for the expensive requests to finish?

My thought for the time being - while I get this thing working - is to only return when something is done so I don’t risk overloading the server with requests.

Once things are working more like I want to - then I will try different things.

However your opinions are still welcome.

1 Like
15

What are the “expensive requests” you are referring to?

Have you tried commenting out the werkzeug logging unless in debug mode?

But looking at the number of werkzeug logging strings, maybe the blocking is the “too many posts limiter”?

Forget waiting for anything to complete, in my opinion, but program everything in a fire and forget manner. If you truly have something that needs babysitting to stop it, launch the babysitter in a thread.

2 Likes
16

Update:

Partial success.

I can now “drive” the robot using the joystick.  Well, at least while it’s on blocks, the wheels move in a reasonable manner.  :wink:

I have not tried it on the ground yet, nor have I implemented joystick controlled head movement.

It still doesn’t work with a remote browser because it claims one of the variables in my JavaScript browser routine is undefined.  It works fine on the GoPiGo, but not in either Firefox or Chrome on my Windows box.

Still scratching my head. . .

Next steps:
Head motion control.

2 Likes
17

Update:

Asking @mitch.kremm and @cleoqc to join the topic

Well, I lied.  Sort of. . .

It turns out that it works fine in Chromium, (on the 'bot), and in Chrome, (on my Windows box), but does NOT work with Firefox anywhere.

The only indication I get is that the variable returned when I instantiate the “gamepad controller” (AKA “Joystick”), it says that the instantiated variable is “undefined”

I don’t know this for certain, but the only possibility I can see right now is that Firefox will only allow gamepads from secure sites  (i.e. sites that begin with "https://).

Aside from just not using Firefox, (as I expect Chrome to follow suit), the only solution I know if is to (somehow) install a self-signed certificate on the 'bot.

Unfortunately, you can’t just do a certificate that is valid for everything, and if Modular Robotics figures out - somehow or other - how to create a generic certificate for the GoPiGo O/S - as soon as you rename the robot, it’s no longer valid.

I’m going to research this, and ask at the Mozilla developer site, but I’m not expecting much.

====================================

Progress report:

Charlie got his first on-the-floor run and though it worked, it wasn’t pretty as control sensitivity and linearity was all over the place and controlling the 'bot was almost impossible.  I didn’t crash into anything, but it was a challenge!

I was going to video the run for everyone’s benefit, but I’m glad I didn’t - it would have been HUGELY embarrassing!

2 Likes
18

Update:
@cleoqc
@mitch.kremm

This is a verbatim copy of my posting over at the Mozilla Developer Network, located at:
https://discourse.mozilla.org/t/reprise-the-gamepad-api-works-in-chrome-but-not-in-firefox/91077

. . . . requesting help with the gamepad API’s and browser policies.

Comments and suggestions are always welcome.

Update:

I was able to reproduce this in Chrome if the joystick was physically disconnected.

Interesting!

1 Like
19

Update:

It’s been a while since I provided an update, and it’s been a wild and woolly ride!

  1. Creating a generic joystick event:

    • Research indicates that there is a thing called a “web worker” which is a subsidiary script that can be bound to one or more “main” scripts within a web page.  JavaScript is not truly multi-threading, (this is why browsers “stall” on some pages), but using web workers you can insulate potentially expensive processes, (like polling loops), from the main logic of the page.
       
    • Unfortunately, web workers cannot interact with all the logic on the page - and the further up the food-chain the logic and methods are, the less able a web worker is able to access it.
       
    • Likewise, not every method available to a top-level script is available to a web worker.
       
    • Therefore, I will have to research if I can move the polling loop for the joystick to a worker and have it send events back to the main script when “something happens”.
       

  2. Who moved my cheese?!!

    • One of the continuing challenges of this project from day one is keeping a stable working environment.  This is complicated by both the browser companies and Microsoft redefining how things work on what almost seems like a daily basis.  This ranges the entire gamut from suddenly needing special CORS headers in the server responses to suddenly noticing that using a joystick requires a secure, (https), context.
       
      This last is is especially controversial and there is a huge wailing and gnashing of teeth over on the W3C forums about it - as that will, at a stroke, make many applications for joysticks in browsers either impossible or expose tremendous security risks.  It seems like they’ve really dug in their heels about this, but hopefully saner minds will prevail.
       
      It is not uncommon for me to go to bed one day with a working, (at least partially), project and wake up the next morning to a quivering lump of GAGH!  Microsoft is especially notorious in redefining essential parts of the operating system, removing features and introducing new ones, all under the guise of being a “security update” - which is hogwash.  Maybe 20% of the update was security related but the other 80% dinks around with my carefully crafted settings and the basic operation of the O/S.  And they don’t tell you until something breaks and you start digging.
       
    • Another HUGE issue is browser caching.  For whatever reason, browsers are REALLY AGGRESSIVE and possessive of their cache to the point where it virtually impossible to turn it off - especially regarding storing scripts.  I can’t tell you how many times I have “dump cache and do a hard reload” in both Firefox and Chrome, only to find that the running script is one from three commits ago - and has absolutely no resemblance to the current version of the script on the server.
       
    • I do not know this for an absolute fact, and I have not seen this documented anywhere, but I suspect that there is a hidden “background cache” somewhere that continues to serve stale data even after the main cache is dumped.  Sort of like Window’s “Virtual Store” - a separate program cache that must be dumped when updates are done, otherwise you never see the effect.
       

  3. Visual Studio Code

    • Visual Studio Code is a very powerful development environment and it has a lot of nifty features, but as Descartes said:  “With great power comes great responsibility”.  There are times when I am trying to work with Visual Studio Code that I feel like an eight year old trying to handle grandpa’s old 12-gague double-barrel shotgun.  It’s just too big a weapon for such small hands.
       
    • The other side of that is there are a lot of people willing to help the nugget/neo navigate through the maze that a complex, professional-grade IDE is and I am learning a lot.  At times it reminds me of you folks working on ROS where it seems sometimes like you’ve bitten off more than you can chew, but you do the web searching and ask, (I am primarily on Stack Overflow), and there is usually a guiding hand to help you along.
       

  4. The actual coding:
    This is a large part of the learning curve within this project.

    • Take something as simple as “rounding a number”.
       
    • I bet you didn’t know that there are many different ways to round a number - they’re all different and they’re all implementation specific.   (i.e. Different browsers do the same rounding function in different ways.)  The most obvious rounding method, “rounding away from zero” where +1.5 becomes +2.0 and -1.5 becomes -2.0, seems to be the most difficult to find.
       
      I discovered that little gem while trying to drive at slow-speeds backwards, or with the joystick to the left, when one of the terms rounds toward zero and i end up with division-by-zero exceptions all over the place.  As a first try, I’ve added extra digits of precision to keep the number non-zero for as long as I can.  If I can’t figure this out, I may have to implement my own rounding routine based on the absolute value of the number.

The bottom line is that this is turning out to be a more involved project than I originally imagined - and will become even more involved as things progress - but I plan to keep my teeth buried in it until I have this knocked cold.

2 Likes
20

But it sounds like you’ve made net progress. So Congrats!!
/K

2 Likes