New Remote Camera Robot: Server data, overlays, animations, and such like

[jimrh]

Continuing, (actually forking), the discussion from My "New" Remote Camera Robot Project - I have created a sub-topic to discuss visual/functional enhancements to the browser display being shown to the user.

The goals of this topic are:

1. Display useful server-side data as an overlay on the browser display of the robots FOV. (i.e. Battery state, distance to an obstacle, bumper impacts, etc.)

2. Force the browser to display the video in the correct aspect ratio, regardless of the browser’s dimensions.  I believe I can do this by defining a viewpoint within the overall browser frame with a fixed aspect ratio - and directing the video stream there.

3. Allow the server to transmit “configuration” data to the browser:

   • The video-stream’s aspect ratio.
   • The contents of a joystick config file so that the browser can send the correct control inputs back to the server.
     • I could “include” a configuration file in the same way that I am already including the script and CSS data.  Then I could somehow read/unpack the data and apply it as required.
     • Another way would be to forget about any data-manipulation on the browser itself and always send the entire game controller data-frame to the server, and let the server itself pick-and-choose based on the content of the controller configuration file.  I have avoided doing this up to this point to allow the browser to do some of the computations and data-filtering and prevent the server from being overloaded with processing that can be done elsewhere.
        
   • Real-time, (or near-real-time), data from the server to the client - like battery voltage, etc.
     • There are two major ways to transmit asynchronous data back to the server:  Web Sockets and HTML5 Server Side Events.
     • Web Sockets would allow me to transmit the entire data-stream - both control and data - over a single persistent “socket” connection.  However, that would require a major re-design and re-write of everything.  Maybe later?  Then again, maybe not?
     • Server Side Events are a way for the browser to create an event handler that listens to the server for the presence of data packets.  Since I am already doing something similar for the joystick and keyboard inputs, this should be relatively easy to implement on the browser side.  Server side complexity is, as yet, unknown.
        

4. Provide browser animations, (like moving “pippers” on the axes), based on control inputs.

   • Viz.: https://dai.ly/x2no8hr

Ideas are always welcome.

[cyclicalobsessive]

@jimrh

Provide browser animations, (like moving “pippers” on the axes), based on control inputs.

Oh that compass (IMU) heading thing would be so cool and useful as well for finding doorways and navigation when a room has multiple exits or points of interest

[jimrh]

@cyclicalobsessive

Oh that compass (IMU) heading thing would be so cool and useful

I’m a long way from including IMU data. 

And, (if I remember rightly), aren’t YOU the one who called down every ancient and modern curse on the IMU because of its i²c fussiness and because using it was a pain in the tush? 

I suspect that in order to get a centered, non-repeating overlay, I will have to create a viewpoint window centered within the main window, just for the display items like the graticule and “pippers”.

[cyclicalobsessive]

@jimrh

aren’t YOU the one who called down every ancient and modern curse on the IMU because of its i2c fussiness and because using it was a pain in the tush?

But I can see you are into S-M, so it is only natural you would want to finally use that IMU for something.

[jimrh]

@cyclicalobsessive

But I can see you are into S-M

I am in to S/M?  You say that I am into S/M?

You, Mr. ROS, who likes nothing better than to wage The Battle Of The Mutexes and is waiting for a good opportunity to re-write the i²c routines for the GoPiGo?

You say I am into S/M??!!!

And you’d be right, but I think it’s a common disorder among robot hobbyists. 
 

Oh - I think I found your IMU bug. . .

[jimrh]

By the way, getting the browser to absolutely, totally, and completely refresh and drop cache is a non-trivial exercise, so I did something sneaky:

 

I created several versions that are different colors.  Every time I make a change, I reference a different colored overlay image.  This way I know when the browser is REALLY running the most recent code.

[jimrh]

Update:

More progress to report.

I now have a secondary overlay on the right hand side for status information from the robot.

I am also researching Server Sent Events which, though it has its own issues, (and what doesn’t these days!), it seems perfect for the kind of periodic message updates I am looking to implement.

The current status box on the right-hand side is just a static display right now.  Once I get Server Sent Events working, I’ll be able to update it.

P.S.
I don’t know if I am going to keep the graticule or not.  I’m still looking into a way to animate a “pipper” on the axes.

[KeithW]

Wow - that’s looking really good.
/K

[jimrh]

“Da’ bitch part” is that for SSE’s to work, they have to be in a separate thread.

Threading = classes.
Classes = complexity.
Complexity = “How do I troubleshoot this thing when, (not if, when), it decides to go all pear-shaped and doesn’t do what I want?”

I’m having enough trouble getting css and html to play nice together in the sandbox.

And to be honest, threading scares the *&^%$#@!!! outta me as I have no clue how to chase problems in a piece of code that has gone off somewhere else to do whatever it wants.

Part of the problem is the lack of basic understanding of what a fair amount of this does - like classes, the streaming methods, Flask itself as a server environment, etc. etc. etc. - so I just keep cutting-and-pasting the stuff together into a Frankenstein’s Monster of code snippets glued together with bits of stretchy-string, duct tape, bubble-gum and good 'ole fashioned spit.

I, myself, will admit that this code is in desperate need of being seriously refactored - bits and pieces are scattered around everywhere - and keeping track of what what is, (and who’s on first), is becoming a non-trivial burden.

I am actually seriously considering abandoning research temporarily and taking some time to sort this mess out.

Ideally, what I would like to see is the programmatic equivalent of “cubelets” - small pieces of code that can be put together to do what is needed with a minimum of interface complexity.  And, (AFAIK), that means classes again - as a way to make “cubelets” out of my processes and procedures.

That being done, these little blocks can be shared with others - like the gentlemen trying to get smooth motor control out of their robots - in a way that makes their life easier.

And, to be perfectly honest, though the EasyGoPiGo3 classes and methods are a huge advancement over the “Arduino-like” low level programming that a lot of robotic projects have had to use, they’re still, (IMHO), needlessly complex.  Ditto the ROS solutions where everyone listens and publishes more than The New York Times.

I want to see a “class” of motion actuator routines that can take simple x-y values, (like those that come from a standard gamepad/joystick), and translate that into smooth, repeatable and controllable motion.  I also want to see something similar for pan-and-tilt head motion.  You say where and I get you there.  Smoothly and accurately.

Along with that, I want to see an expansion of the calibration capabilities.

For example, virtually nobody’s pan-and-tilt will be perfectly centered when set to “90-90”, so there needs to be a way to create and set calibration constants for the servos.  Ditto tweaking the wheel ratios so that when commanded to travel straight, the robot travels straight.  Ditto ditto the calibration of the joystick/gamepad layout.  And so forth.

And I want to do this in such a way that the user doesn’t have to have a dual major in quantum mechanics and rocket science to understand and use it.

One thing that has been an epiphany, is dictionaries.  This is a clever way to create structures that allow the user to manipulate relatively complex things by setting a simple attribute, or collect all the information about what the robot is doing by reading the entire dictionary.

Like this:

var gopigo3_joystick = {
    controller_status: 'Disconnected',
    motion_state: 'Waiting for Joystick',
    angle_dir: 'None',
    time_stamp: 0,  // a large integer that, (sometimes), becomes a float (shrug shoulders)
    x_axis: 0.00,  //  x-axis < 0, joystick pushed left - x-axis > 0, joystick pushed right
    y_axis: 0.00,  // y-axis < 0, joystick pushed forward - y-axis > 0 , joystick pullled back
    head_x_axis: 0.00,  //  head x and y axes mirror the joystick x and y axes
    head_y_axis: 0.00,  //  if the pinky-switch, (head motion enable), is pressed
    force: 0.00,  //  force is the absolute value of the y-axis deflection
    trigger_1: 0,   // Partial primary trigger press (motion enabled)
    trigger_2: 0,   // Full primary trigger press  (faster speed - not yet implemented)
    head_enable: 0  // Pinky-switch press  (enable joystick to move head)
};

It’s relatively simple to create a data-structure that contains key-value pairs for everything you need.

Obviously, there’s a lot to be considered and there will have to be compromises, but as Edward Teller said about inventing the H-Bomb:

It had to be done. [referring to Stalin and the US/Soviet arms race.]

Since it had to be done, and I could help, I did, and I was never sorry for having done it.

I’m going to have to bite, (byte), the bullet and figure it out, but I’m not expecting it to be easy.

[jimrh]

Server Sent Events:  Ready for Prime Time?

Here’s an interesting article I stumbled upon reading about SSE’s:

Server Sent Events are still not production ready after a decade. A lesson...

In this article the author talks about a client-server scenario that uses SSE’s as a way to broker messages back and forth between a server and client system - and in this case he’s using it to sync and establish login sessions!

Read the article and then see if you agree with my assessment:

1. The guy’s basic premise is a total ***-up, since. . .
2. You don’t use SSE’s for time-sensitive data, and. . .
3. You damn-sure don’t use them in a situation, like a login, where a delay - even a short one - can make things go all pear-shaped, especially if you’re using a captcha that’s time sensitive.
   — and —
4. This guy needs a better lab for testing scalability and network robustness.
   • I have a story I can tell about how I totally trashed a distributed network system that hadn’t been tested outside of the single network segment it was dev’d on.

My understanding is that SSE’s are a good choice for:

1. Data that is not guaranteed to be regular and/or data that IS guaranteed to be “whatever, whenever”.
2. Data that is not time sensitive.  (i.e.  The “push” notification that someone replied to your posting on this forum - someone won’t die if it takes a bit longer than normal.)
3. Data that’s not “mission critical”.  (i.e.  I wouldn’t use it to control an aircraft in flight, a reactor, or a real-time process.)
4. Or, data that is being sent between known endpoints, over a known, well defined network, like the connection between my GoPiGo and a laptop on the same network.

In other words, SSE’s are like a UDP connection - you’re pushing messages “up the wire” and hoping that there’s someone at the other end to get them.  If it’s an “absolutely, positively has to get there overnight” kind of time-sensitive message, you should be using something else.

What say ye?

[jimrh]

Accidentally posted a message in the wrong place.

I moved it to: https://forum.dexterindustries.com/t/my-new-remote-camera-robot-project/8728/51

[KeithW]

@jimrh

UDP connection

I know a great joke about UDP.

I could tell you, but you might not get it…

/K

[jimrh]

@KeithW

I know a great joke about UDP.

I could tell you, but you might not get it…

You’re gonna get a SMACK!

[jimrh]

Update:
Ref: https://forum.dexterindustries.com/t/my-new-remote-camera-robot-project/8728/51

Now that I have accomplished that major goal, the next goal is real-time status reporting from the 'bot to the client.

What does that mean?

• Right now, the status window on the client’s browser is populated by the client and represents the status that the client is sending/has sent to the server on the robot.  It does NOT necessarily represent the server’s status - only what the client thinks the server status should be.

• What I want to achieve is a status window that shows the actual robot status as of that point in time.  (i.e. Shutting down, moving, direction, battery voltage, processor temp, etc., depending on what I feel is important - and what will fit.)

Server Sent Events looks like the way to do this with a minimum of overhead.  However, because of the way SSE’s are transmitted to the client, they’re not absolutely guaranteed to arrive - though there are workarounds by using things like a message number.  (and a database/queue to store previously sent messages - which I probably won’t implement)

The big question is how to do that.

I have several interesting articles I am looking at - and trying to understand - and I hope to have something working soon.