The SpinDown Problem

In a solution where the number of users is such that multiple servers are required to meet demand, it is very easy to spin up more servers as demand increases.  The only real difficulty in spinning up additional servers is making sure that requests are sent to the same server where it is required.  This is the concept of session-bound request processing as opposed to session-less request processing.  Typically in a Genero context, GUI Genero applications and  Sticky web services use session-bound request processing, non sticky web services use session-less request processing.

Session-bound request processing is required to send consecutive requests from the same client to the same fglrun process on the same server.  In the case of a GUI Genero application, this is so that the request is processed by the same fglrun process which is typically sitting in a dialog statement (DIALOG, INPUT, CONSTRUCT, MENU, DISPLAY ARRAY, MENU).  In the case of a sticky web service, this is typically so that the request is processed by the same fglrun process which has a FETCH NEXT iterating through an open database cursor.

In the case of non-sticky web services with their session-less request processing, each individual request is stateless and can be processed by any one of the available fglrun processes on any of the available servers waiting ready to process the next request that comes in.  With one server, the Services Pool concept allows you to have multiple fglrun processes ready to process Web Service requests as they come in.  With multiple servers, you then have the Load Balancer sitting in front of that distributing requests to one of the many servers.

When configuring your Web Server and/or Load Balancers to handle session-bound requests, you are typically looking for words like session-bound, session-persistence, stateful.  For example, with nginx you need Nginx-plus and to configure for session-persistence.

As the saying goes, what goes up must come down.  If you have spun up multiple servers to meet demand, at some point you must spin down some of these servers as demand drops.  When is it safe to shut down a server?

Using the gasadmin tool there are a number of commands that are at the disposal of the system administrator or can form part of an automated system.

• There is the enable and disable verbs for applications and services that allow you to disable a service or application, thus preventing any new requests from being accepted by that applicaton or web service on that server.
• There is the list-session, idle-time,   close-session, close-all-session verbs that allow you to iterate through all sessions and see how long since a session had activity, and if so close it.
• You also have the ability using the send-message verb to send messages to users advising them that a server will soon shut down.

What hurts in these scenarios are long running processes.  These long running processes maybe genuine such as month-end, year-end batch processes, or they maybe unintentional such as user going out to lunch and leaving their Genero application running in the middle of a screen.  However they become present, it can be frustrating that when you go to spin down a server you no longer need, that you will be stymied by one or two fglrun processes remaining.

There are things you can do with your Genero application and system configuration to minimise long running processes interrupting a server shutdown.

With genuine long running processes, you can run them in the background.  RUN "export FGLGUI=0; fglrun program-name" WITHOUT WAITING rather than running with a GUI screen that says “Processing row N of M”.  With these long running processes in the background, you can also schedule them to be run on the last server that will be shut down.  If you do run it in the background, you do lose the ability for the user to cancel a long running process as discussed here so take care with any input parameters, you may need to add extra validation or an extra “Are You Sure?” dialog.

For GUI applications, you can use ON IDLE, or AUTO_LOGOUT to ensure that if there is prolonged period of inactivity, the program can be made to  stop.

For Sticky Web Services, the KEEP_ALIVE element in the .xcf can be configured to shut down a process once requests stop coming into it.

At the moment, there is nothing in Genero that allows you to pick up a fglrun process, shift its memory/stack to another server, and direct future requests to the fglrun process on new server.  Whilst that would technically be possible to implement in certain conditions, the difficulty will be things like database connections and open database cursors and database transaction, file handles, open base.Channel connections, LOCATE etc..  One exercise that might be good to go through is identify scenarios where your application is idle and make sure that resources have been freed.

This last scenario is one I recall from my days as a junior programmer.  We had a POS (Point of Sale) application that could be used by any store employees so once a user had finished a sale, it went back to a login/password screen that the employee would key in to start processing a new sale. You may have seen this in your local bar/tavern/restaurant where an employee has to login or tap a swipe card before they start processing your purchase.  We were running out of resources and what we identified was that when our application was sitting at this login/password screen it still had database cursors open, memory being consumed etc.  What we did is that at the end of the sale, add code to close cursors etc.  In a multiple server scenario, when an application is at this stage, the process is in a good condition to be shutdown and hence the server can be spun down.

Another scenario you can investigate is that for enquiry only screens, using the XSLT transformer classes to turn web services output into a web page.  I should do an Ask-Reuben on this but it means these screens are stateless and clicking on a link on one of these screens can be served by any server.  Thus you can spindown a server quicker without worrying about long running processes.

I have waffled a but in this article, I want to finish with a few succinct points

What is the SpinDown problem? – The SpinDown problem occurs in a multiple server load balanced scenario where demand drops and you have to choose a server to shut down.  You have to choose a server and decide what to do with the processes running on that server.

Is it a Genero problem? – It is not a problem unique to Genero. Any framework that is stateful will encounter this issue.

What tools do I have to help me? – The gasadmin command has a number of options that allow you to measure inactivity, disable applications and services to prevent new requests from starting, send messages to users asking them to shut down, and forcibly close a session.  These would be used in combination with the tools your Load Balancer has to control what server a process is allocated to.

What other things can I do to help? – Avoid long running processes, that can mean running long running processes in the background without UI and assigning to a server that would be the last one shut down, or it can mean shutting down programs after periods of inactivity, or it can mean writing a program in such a way it is not running for longer than is necessarily.

It isn’t an impossible scenario, but you may need to make sure that expectations are lowered.  You can shutdown a server but you may have to make sure that no one will lose any work, so it may mean you have to give the users time to finish what they are doing.