My "New" Remote Camera Robot Project

Update:

Apparently you don’t.  (Create a structure to receive the responses)

Here’s a snippet of the code that sends some data to the robot:

gopigo3_orientation.state = 'move';
gopigo3_orientation.angle_degrees = data.angle.degree;
if('direction' in data) {
    gopigo3_orientation.angle_dir = data.direction.angle;
}
gopigo3_orientation.force = data.force;
query_string = '?' + $.param(gopigo3_orientation);

// use the following function instead the usual $.post
// so that we can throttle the rate of sending data
// good practice for not "choking" the network
send_throttled_data(server_address, query_string, gopigo3_orientation);

Here’s a snippet of the code that receives the data from the browser:

@app.route("/robot", methods = ["POST"])
def robot_commands():
    # get the query
    args = request.args

    state = args['state']

    angle_degrees = int(float(args['angle_degrees']))

    angle_dir = args['angle_dir']

    force = float(args['force'])

If I’m reading that correctly, (and my money’s on “I’m not” :wink:), the entire structure is sent as name:value pairs where “force” references/contains the value for the force sent - and you can reference an argument more than once to re-read it’s value.

I am going to have to experiment with this.

2 Likes

that’s the way most of the time because it is the easiest

2 Likes

It turns out that this is correct.

As an experiment, I decided to try to print out the “args” variable that contains the returned arguments with the following result:

INFO:werkzeug:172.31.100.209 - - [31/Dec/2021 16:54:43] "POST /robot?state=stop&angle_degrees=0&angle_dir=none&force=0 HTTP/1.1" 200 -
ImmutableMultiDict([('state', 'Home'), ('angle_degrees', '0'), ('angle_dir', 'none'), ('force', '0')])

. . . with the “ImmutableMultiDict” being the structure with all the name-value pairs.

That makes things SOOOOO much easier as far as collecting the data is concerned. . .

. . . except that the difference between a browser talking to itself and talking to a server isn’t trivial. . .
:thinking:  :roll_eyes:

1 Like

From the earlier examples (e.g. state = args['state']) it looked like the results were being stored as a dictionary. But then I saw this. I wasn’t familiar with MultiDict in Python - apparently it lets you have multiple values per key - the values get returned as a list. Still, in the example code it looks like everything works like a regular dictionary (i.e. the assumption is that each key returns a single value).

/K

2 Likes

I presume that this allows for the possibility that a particular “key” might have multiple values returned, such as:
“axes” returning 0.00, 0.00, 0.00, 0.00 for the 0th, 1st, 2nd and 3rd controller axes - and it’s up to you to figure it out in code.

Me, I’m returning one value per key to make things easier.

2 Likes

Update:

Browsers are getting too smart for their own good - or maybe it’s the malware writers are getting too smart for OUR own good.

One manifestation of this is that browsers are now severely restricting “cross site resource sharing” (CORS) requests - even if it’s on your own filesystem - because of some clever malware writers.

The result is that in order to make something like my New Remote Camera Robot software work, I have to specifically allow cross-site resource sharing.

You can read about it here if you want.

Allow “CORS” (Cross Origin Resource Sharing) in pages served by the GoPiGo

The next problem to solve is that browsers are not allowing gamepad support unless it’s a secure site. . . and how are we supposed to generate a secure site for any robot being used by anyone?

Sigh. . .

2 Likes

Here’s a theory kind of question:

Given:

  1. Motion and servo movement requests can be (relatively) time intensive.
  2. Browser “blocking” - where the browser appears to stall - is a Bad Thing.

The Usual Case:
The “usual case” for browser/server interaction is that the browser does something the server must respond to, the server does it, and then sends a “Yo! I’m done and everything’s a Thing of Beauty!”

My Specific Case:
In my case, the completion of the task, or failure to complete the task, isn’t really the browser’s business.  What’s important is “Did the server understand the request - or was it effed-up beyond all recognition?”

So, this becomes a theory question, (that I will experiment with, but would like your considered input on).

Should I respond as early as I can?  Or, should I wait for the expensive requests to finish?

My thought for the time being - while I get this thing working - is to only return when something is done so I don’t risk overloading the server with requests.

Once things are working more like I want to - then I will try different things.

However your opinions are still welcome.

1 Like

What are the “expensive requests” you are referring to?

Have you tried commenting out the werkzeug logging unless in debug mode?

But looking at the number of werkzeug logging strings, maybe the blocking is the “too many posts limiter”?

Forget waiting for anything to complete, in my opinion, but program everything in a fire and forget manner. If you truly have something that needs babysitting to stop it, launch the babysitter in a thread.

2 Likes

Update:

Partial success.

I can now “drive” the robot using the joystick.  Well, at least while it’s on blocks, the wheels move in a reasonable manner.  :wink:

I have not tried it on the ground yet, nor have I implemented joystick controlled head movement.

It still doesn’t work with a remote browser because it claims one of the variables in my JavaScript browser routine is undefined.  It works fine on the GoPiGo, but not in either Firefox or Chrome on my Windows box.

Still scratching my head. . .

Next steps:
Head motion control.

2 Likes

Update:

Asking @mitch.kremm and @cleoqc to join the topic

Well, I lied.  Sort of. . .

It turns out that it works fine in Chromium, (on the 'bot), and in Chrome, (on my Windows box), but does NOT work with Firefox anywhere.

The only indication I get is that the variable returned when I instantiate the “gamepad controller” (AKA “Joystick”), it says that the instantiated variable is “undefined”

I don’t know this for certain, but the only possibility I can see right now is that Firefox will only allow gamepads from secure sites  (i.e. sites that begin with "https://).

Aside from just not using Firefox, (as I expect Chrome to follow suit), the only solution I know if is to (somehow) install a self-signed certificate on the 'bot.

Unfortunately, you can’t just do a certificate that is valid for everything, and if Modular Robotics figures out - somehow or other - how to create a generic certificate for the GoPiGo O/S - as soon as you rename the robot, it’s no longer valid.

I’m going to research this, and ask at the Mozilla developer site, but I’m not expecting much.

====================================

Progress report:

Charlie got his first on-the-floor run and though it worked, it wasn’t pretty as control sensitivity and linearity was all over the place and controlling the 'bot was almost impossible.  I didn’t crash into anything, but it was a challenge!

I was going to video the run for everyone’s benefit, but I’m glad I didn’t - it would have been HUGELY embarrassing!

2 Likes

Update:
@cleoqc
@mitch.kremm

This is a verbatim copy of my posting over at the Mozilla Developer Network, located at:
https://discourse.mozilla.org/t/reprise-the-gamepad-api-works-in-chrome-but-not-in-firefox/91077

. . . . requesting help with the gamepad API’s and browser policies.

Comments and suggestions are always welcome.

Update:

I was able to reproduce this in Chrome if the joystick was physically disconnected.

Interesting!

1 Like

Update:

It’s been a while since I provided an update, and it’s been a wild and woolly ride!

  1. Creating a generic joystick event:

    • Research indicates that there is a thing called a “web worker” which is a subsidiary script that can be bound to one or more “main” scripts within a web page.  JavaScript is not truly multi-threading, (this is why browsers “stall” on some pages), but using web workers you can insulate potentially expensive processes, (like polling loops), from the main logic of the page.
       
    • Unfortunately, web workers cannot interact with all the logic on the page - and the further up the food-chain the logic and methods are, the less able a web worker is able to access it.
       
    • Likewise, not every method available to a top-level script is available to a web worker.
       
    • Therefore, I will have to research if I can move the polling loop for the joystick to a worker and have it send events back to the main script when “something happens”.
       
  2. Who moved my cheese?!!

    • One of the continuing challenges of this project from day one is keeping a stable working environment.  This is complicated by both the browser companies and Microsoft redefining how things work on what almost seems like a daily basis.  This ranges the entire gamut from suddenly needing special CORS headers in the server responses to suddenly noticing that using a joystick requires a secure, (https), context.
       
      This last is is especially controversial and there is a huge wailing and gnashing of teeth over on the W3C forums about it - as that will, at a stroke, make many applications for joysticks in browsers either impossible or expose tremendous security risks.  It seems like they’ve really dug in their heels about this, but hopefully saner minds will prevail.
       
      It is not uncommon for me to go to bed one day with a working, (at least partially), project and wake up the next morning to a quivering lump of GAGH!  Microsoft is especially notorious in redefining essential parts of the operating system, removing features and introducing new ones, all under the guise of being a “security update” - which is hogwash.  Maybe 20% of the update was security related but the other 80% dinks around with my carefully crafted settings and the basic operation of the O/S.  And they don’t tell you until something breaks and you start digging.
       
    • Another HUGE issue is browser caching.  For whatever reason, browsers are REALLY AGGRESSIVE and possessive of their cache to the point where it virtually impossible to turn it off - especially regarding storing scripts.  I can’t tell you how many times I have “dump cache and do a hard reload” in both Firefox and Chrome, only to find that the running script is one from three commits ago - and has absolutely no resemblance to the current version of the script on the server.
       
    • I do not know this for an absolute fact, and I have not seen this documented anywhere, but I suspect that there is a hidden “background cache” somewhere that continues to serve stale data even after the main cache is dumped.  Sort of like Window’s “Virtual Store” - a separate program cache that must be dumped when updates are done, otherwise you never see the effect.
       
  3. Visual Studio Code

    • Visual Studio Code is a very powerful development environment and it has a lot of nifty features, but as Descartes said:  “With great power comes great responsibility”.  There are times when I am trying to work with Visual Studio Code that I feel like an eight year old trying to handle grandpa’s old 12-gague double-barrel shotgun.  It’s just too big a weapon for such small hands.
       
    • The other side of that is there are a lot of people willing to help the nugget/neo navigate through the maze that a complex, professional-grade IDE is and I am learning a lot.  At times it reminds me of you folks working on ROS where it seems sometimes like you’ve bitten off more than you can chew, but you do the web searching and ask, (I am primarily on Stack Overflow), and there is usually a guiding hand to help you along.
       
  4. The actual coding:
    This is a large part of the learning curve within this project.

    • Take something as simple as “rounding a number”.
       
    • I bet you didn’t know that there are many different ways to round a number - they’re all different and they’re all implementation specific.   (i.e. Different browsers do the same rounding function in different ways.)  The most obvious rounding method, “rounding away from zero” where +1.5 becomes +2.0 and -1.5 becomes -2.0, seems to be the most difficult to find.
       
      I discovered that little gem while trying to drive at slow-speeds backwards, or with the joystick to the left, when one of the terms rounds toward zero and i end up with division-by-zero exceptions all over the place.  As a first try, I’ve added extra digits of precision to keep the number non-zero for as long as I can.  If I can’t figure this out, I may have to implement my own rounding routine based on the absolute value of the number.

The bottom line is that this is turning out to be a more involved project than I originally imagined - and will become even more involved as things progress - but I plan to keep my teeth buried in it until I have this knocked cold.

2 Likes

But it sounds like you’ve made net progress. So Congrats!!
/K

2 Likes

Update:

@superbam
@cyclicalobsessive
@KeithW

I am curious how you calculated your turns.

I am working on how to calculate the ratio of the wheel speeds to get the degree of turn I desire based on how far forward the joystick is pressed and how far to the side it is pushed.

(Right now, I am ignoring sign which is forward/back and left/right since once I get a smooth turn based on the two axes, I can always flip the sign for the other direction.)

My current calculations are based on the following:

  • Assume the joystick is being pushed forward, so the y-axis value is increasing.
    • It’s actually increasing in the negative direction, but I am ignoring sign for the time being)
  • Assume a x-axis deflection between zero and one.  (centered to all the way to one side.
  • Y-axis further forward, (closer to 1), means faster travel up to the limit set by max_speed. (actual_speed = |max_speed * y_axis|)
  • As the x-axis is moved further away from center, the ratio of the speeds of the outside wheel to the inside wheel should become greater.  (i.e. The inside wheel should become slower as the x-axis is deflected more.

I came up with one formula where the inside wheel was a fraction of max_speed, which didn’t work.

I have modified the formula as follows:
(Note: I haven’t tried this on Charlie yet.)

actual_speed = max_speed * abs(y_axis)

percent_speed = int(((y_axis / x_axis) / 2 ) * 100)

Where

  • actual_speed is the forward speed if x_axis = 0 and is the speed of the outside wheel during a turn.
  • percent_speed is the percentage of actual_speed that the inside wheel gets.
  • “int” is a function that I will probably have to write so that both positive and negative numbers round away from zero to the nearest integer.
    • +1.5 => +2
    • -1.5 => -2
      As it is, the rounding methods available in Python round negative numbers toward zero.
    • +1.5 => +2
    • -1.5 => -1
    • -0.5 => ZERO!

Sigh. . . .

I actually don’t. I just use the ROS functionality.

Sorry.
/K

1 Like

Update:  (Failure and an idea)

Issue:
The “normal” way the original remote camera robot project sent data to the robot was a combination of an event driven interface and a short timer to keep from flooding the internet connection.  This was possible because the virtual joystick library, (nipple.js), used mouse and keyboard activity events to regulate the volume of data sent.  (i.e. Data was sent ONLY  if there was something useful to send.)

Unfortunately that cannot be done with a real joystick because the current implementation of the gamepad interface does NOT  generate generalized activity events - the software has to continually poll the joystick.  This results in either the server being flooded with data, (even if limited by a brief timer), or if a activity polling loop is used, the browser stalls.

I attempted to mitigate this by using a web-worker thread of execution to poll the joystick and only send useful events to the main thread.  Unfortunately, that does not work because web-worker threads cannot access the “window” attribute of the page and therefore cannot interact with the joystick directly.

I have an idea that might work:

  1. Abandon the web-worker attack on the problem.  (At least for the time being.  I may return to that if true activity events are implemented.)

  2. Allow the current code to work the way it does - flooding the server with POST data messages.  However, instead of allowing them every time the animation loop is run, I will try wrapping the “send_data” routine in a test for the joystick’s time_stamp attribute - has it changed or not?

    • If the time_stamp attribute HAS changed, that means something happened and I will allow send_data to send the current data-frame to the robot.
    • If not, there’s nothing interesting to send and I’ll just skip the send_data routine.
    • Keyboard events, (for moving the robot’s head), don’t cause changes in the joystick’s time_stamp.  Therefore, if there is a keypress to transmit to the robot, I’ll just dump some bogus value into the time_stamp so that the send_data process triggers.

Let’s see if that helps.  Hopefully I will be able to generate the functional equivalent of gamepad activity events without bogging down the browser or making joystick response sluggish.

2 Likes

Update:

Problem solved!

The browser now only sends data when there is something interesting to send - and is much more responsive since it doesn’t stall in a wait-loop.

The robot motion logic itself still needs considerable help.

  • The motion isn’t as smooth and controllable as I’d like, and it sometimes sticks “running”, but these should not be insurmountable problems.

The big problem is that, right now, it only works in Chrome because Firefox won’t fire joystick events unless you’re within a secure context - and I don’t know how to negotiate a secure context even with a self-signed certificate.

It is very likely that Chrome will follow Firefox’s lead very soon and I won’t be able to continue development.  :face_with_symbols_over_mouth:

2 Likes

I have posted issues on both Mozilla’s site and the W3C group’s GitHub page about requiring a secure context to use a gamepad/joystick within a browser.

Unfortunately, the move to restrict everything possible to a secure context has a lot of momentum and the people who develop the relevant specifications seem to believe that a secure context is a panacea that will solve all the Internet’s problems - and that’s just not true.

Requiring a secure context to use a hardware device to prevent fingerprinting is not the correct solution because those sites that wish to fingerprint you, the Google’s, Meta’s, (facebook), and the rest of the skum-sites that want to follow you around already have secure sites!  Therefore, requiring a secure site does absolutely nothing to protect you from fingerprinting.

I have submitted objections to this on both Mozilla Developpment and the W3C GitHub page.

Viz.:

Anyone who wishes to comment on either of those sites objecting to this pointless solution that actually makes things worse, is welcome to do so.

What say ye?

2 Likes

Update:

Re: The requirement for a “secure context”.  (See previous message.)

  • I discovered that the domain “gopigo3.com” was available, so I “bought” it.
    • Along with the domain, I purchased a simple e-mail plan so I could receive e-mail at the xxx@gopigo3.com address.
       
  • I have also purchased a simple site certificate from Comodo.
    • Unfortunately, I cannot provision the certificate until the e-mail becomes active which might take a couple of days.  I am in contact with the support people at Dirctnic and if it’s not working by tomorrow, I will open a support ticket.
       
  • I have sent an e-mail to the support/sales people at Comodo asking how to solve the problem of requiring a secure context to use just about any hardware device, (aside from a mouse or keyboard), within a browser.
    • The “obvious” solution of using a self-signed certificate for the robot is not as simple as it appears.  (I tried that.)
    • Using a self-signed certificate - at least with Firefox - is a horrible user experience.
    • Within Firefox the user has to traverse no less than three warning screens, each scarier than the one before, predicting doom, destruction, the end of the world, along with incurable dandruff and hideously bad breath.  “It should be intuitively obvious” that this is a less than stellar user experience, virtually guaranteed to scare away most everyone.

Re:  Joysticks and gamepads.

  • I purchased two different gamepads.
    • One from “Ritmix”, ($8 US), which is hideous.
      • The controls are not linear - the axes traverse the entire range in each direction with less that a third of their travel.
      • It has a flimsy feel, the buttons stick, and it generally gives the impression of being a cheap piece of :poop:.
      • The default game mode is some strange controller mode where everything, (including the joystick axes), maps as a button.  There is a secondary mode where the gamepad acts as a gamepad, but it is not the default and has to be selected each time the gamepad is plugged in.
      • Attempting to reach the ritmixrussia.ru web, (to see if there is calibration software), site throws several “bogus site” warnings claiming that the site is fraudulent and infested with Trojans.  Malwarebytes absolutely refused to allow access to the site without a specific exception added.  I decided not to go there. . . .
         
    • The second one was from Logitech, (about $25 US), which has much better control, the joystick axes behave as one would expect, and it’s default game mode is as a “standard” gamepad.
      • The gamepad “feel” is much more solid, the button action is firm and positive, it defaults to a standard gamepad format, and the general impression is that of a solidly built device.
         
    • “Da’ bitch part”, (as they said in Blazing Saddles), is that the gamepad button mapping is totally different than the button mapping for my joystick.  (i.e. The left-hand trigger on a gamepad is button 7 instead of button 0 and the right-hand trigger is button 8.)
      • I suspect that as the project nears completion, I will have to add the capability to define the mapping between the actual joystick buttons and the logical buttons within the software either via a JSON or XML configuration file.
      • I’d really like to use XML as that will allow me to define one parameter per line, as opposed to the “everything on one line” JSON format.
1 Like

is inure to such warnings and will simply see this as just another existential threat to tip-toe away from.

Same effect, but to think a warning, one, two or three, will induce fear is to give too much credit to humanity.

Quite ingenious approach you have discovered. I applaud your tenacity.

One thing I don’t understand is how you getting a real cert will be useful to the next person wanting to use a remote machine connected joystick?

Additionally, what advantages does your remote machine browser connected joystick have over a “Wireless USB Gamepad” connected to the GoPiGo3?

2 Likes