Ask Reuben

Operating Systems 101

Questions where the answer is because …

  • LD_LIBRARY_PATH (or its equivalent) is not set correctly
  • PATH (or its equivalent) is not set correctly
  • The process does not have permissions to read/write that file 
  • The process is not running in that directory
  • The process is not running on that server
  • The process can’t communicate with another process

This weeks Ask Reuben on Operating Systems is one I have been attempting to write for a few weeks and is a little different.  I have been wanting to write it as I am conscious that I have concentrated on Genero code issues and I want to branch out into our other products such as Genero Application Server, Genero Studio, Genero Report Writer, Genero Mobile and this touches across all our products.

Genero does not do anything special with regards Operating Systems.  We have to respect and follow the rules operating systems put in place.  There are a number of questions I see in the support portal where there is a lack of understanding of

  • how to configure or use Genero products, and
  • what is wrong with the result of the configuration and why you need to change the configuration.

Frequently I see users concentrating on the how but what they need to do is step back and look at the what and why.  For example I don’t think I have ever been asked how to set LD_LIBRARY_PATH in Genero Application Server or Genero Studio configuration, but we have had cases raised where the answer is that LD_LIBRARY_PATH is not set to the value needed, but the person asking the question has not identified that LD_LIBRARY_PATH not being correct is the problem.

When a process is running on a server, it has certain properties …

  • the name of the command
  • the directory it is running from
  • the owner of the process
  • the arguments passed to the command
  • the environment of the process i.e the value of all the environment variables.

When typing and executing a command from the command line, we have a good idea of what those properties are.  (Examples below with Linux, other operating systems have equivalent commands)

  • fglcomp, fglform or fglrun is the name of the command
  • pwd gives us the name of the current working directory
  • ps or who gives us the name of the process owner (and you can use ls -l to look at permissions of directories and files)
  • fglrun -d program arg1 arg2 we can see that -d is an argument to fglrun, (as indeed are program, arg1, arg2), and arg1, arg2 are arguments to the Genero application that can be found by using base.Application methods.
  • we can use env to get the value of environment variables
  • which fglrun can tell us what fglrun is actually being executed based on PATH
  • ldd fglrun can tell us what shared libraries are loaded based on LD_LIBRARY_PATH
  • ps command and others can give more details about the process as it runs.
  • lsof command to identify files open by a process

Acts such as compiling or running a program from Genero Studio, building and deploying a Genero Mobile package in Genero Studio, running a Genero Web Application by typing a URL into a browser, making a Genero Web Service call, if you look closely on the operating system you will see there is a command being executed and this command will have a name, have the concept of a current working directory, have an owner, have arguments passed to it, have an environment.

The Genero Application Server configuration allows you to see this in one place.  If you look in the EXECUTION node, you will see …


<EXECUTION [ Using=component-id ] 
   [ AllowUrlParameters={"TRUE"|"FALSE"} ] [ AllowUnsafeSession={"TRUE"|"FALSE"} ] >
        
   [<ENVIRONMENT_VARIABLE>...</ENVIRONMENT_VARIABLE>][...]
   [<PATH>...</PATH>]
   [<DVM>...</DVM>]
   [<MODULE>...</MODULE>]
   [<PARAMETERS>...</PARAMETERS>]
   [<ACCESS_CONTROL>...</ACCESS_CONTROL>]
   [<DELEGATE>...</DELEGATE>]
   [<WEB_COMPONENT_DIRECTORY>...</WEB_COMPONENT_DIRECTORY>]
</EXECUTION>

… and a good way to think of this is that this is the equivalent of typing at the command line …

export ENVIRONMENT
cd PATH
DVM MODULE PARAMETERS

… if you do a ps e.g. (ps -ef | grep ‘fglrun’) you should find the running process.  You should also be able to find the proxy and dispatcher processes as per the Genero Application Server architecture.  If you look in these proxy and dispatcher logs, you will find that the value of the environment can be output in the logs when the process starts, see this tip here about making sure CATEGORIES_FILTER includes CONFIGURATION.

There are two common misconceptions I see with programs launched via GAS.

  • Who is the process owner? Those coming from command line are used to the process owner being who they are logged into the server as.  When using GDC with Direct Connection short-cuts this is also logging into the server as the passed in user. (in both cases typically the name of the person between the keyboard and chair) When using Genero Application Server either with GDC or via GBC, it is not logging into the server, the Web Server launches the dispatcher which in turn launches the proxy which in turn launches the fglrun process, all as the same system user as opposed to the name of the user between keyboard and chair.  This in turn has a flow on effect in terms of permissions, database connections (value of USER in SQL operations), and values of environment variables USER, HOME and other variables typically set in login shells etc.  There is a note to this effect that you need to consider when moving from GDC – Direct Connection to GDC via HTTP and GBC.  Typically you need to introduce the concept of an application user that a lot of the logic that was tied to the process owner should be tied too.  If someone has trouble with this point, I will either point them at the database processes and ask what system user are they owned by? or I might say if you had terminal access to a Facebook server, could you logon to that server with your Facebook login and password?
  • Launching programs via GAS or GST also requires a different way to think about setting of environment variables.  At your terminal, and with GDC Direct Connection, environment variables are typically set via login shells.  With Genero Application Server, environment variables are set by the ENVIRONMENT settings in the .xcf.  However you also need to consider the inheritance of environment variables from their parent.  This actually comes handy into passing environment variables from the Web Server as documented here with the FGL_WEBSERVER family of environment variables and how these values are then available in your 4gl program.  But when testing with httpdispatch, you also need to consider how httpdispatch is launched and what was in its environment when it was launched.  A common trap I have seen is environment variables such as LD_LIBRARY_PATH have been set via login shells for the developer who runs httpdispatch, these are then inherited in the fglrun program if in the .xcf they have been set with entries that are equivalent of LD_LIBRARY_PATH=some-path:$LD_LIBRARY_PATH.  That is the value is inherited from, or includes the value from  the parent process of httpdispatch.  When the .xcf is run from a Web Server, this value is not in the parent process and hence LD_LIBRARY_PATH is not set as expected or as it was when testing using httpdispatch.  The rule of thumb here is to set the complete environment you need in the .xcf.

You can also find the environment dumped into the log files as mentioned above and in Genero Studio you can right-click on the application node to get a dump of the environment.

A tip I also give is to add logic similar to  …

IF some-condition THEN
   RUN "env | sort > /tmp/yourname.env"
END IF

… as close as possible after MAIN to look at the actual environment the process has when it runs.  A number of support calls have been solved doing that and the developer realising that when their program executed it did not have the environment variable value they were expecting.

I mentioned that the process operating is set by the EXECUTION node in the .xcf when using GAS.  With Genero Studio a similar process occurs so that when you build, execute, package, deploy etc, a command is built-up and executed with a particular environment in a particular directory.  The places where it is set are more varied.   The Genero Configuration Management dialog has details of the runtime, environment sets and what user interface client to use.  There is a little more to it than just what appears on that dialog but if you right-click on an Application node and investigate the entries Display Environment, Edit Build Rules, Edit Package Rules, Advanced Properties you can either see where the command was derived or check its value. The Build Rules being a good one to see the steps being executed when a compile is taking place.  The key point to takeaway is that Studio like GAS is building up a command and executing it.

This concept of environment variable inheritance also applies to the RUN command.  The environment will be the same as per the fglrun command, if you need a different environment variable value then you need to set it as part of the RUN command.  It is not a bug in Genero if the value of LD_LIBRARY_PATH needed for fglrun and its communication with a database breaks a command launched in the RUN.  As part of the RUN command you need to set your environment to what is required.

One question I get after asking developers to try and have an understanding about what processes are running, is what is gsproxy?  You won’t find a mention of it in the Genero Studio documentation.  Where you will see it is in the .xcf files created by Genero Studio when you run a program using the Genero Application Server.  If you find them in FGLASDIR/appdata/app you will see two .xcf files are created with your appname_yourname_studio.xcf and appname_yourname.xcf.  If you study these two .xcf files you will see different entries in the EXECUTION node …

#! appname_yourname.xcf
    <PATH>/Applications/fourjs/fgl/3.20.10/demo</PATH>
    <DVM>"/Applications/fourjs/fgl/3.20.10/bin/fglrun"</DVM>
    <MODULE>"demo"</MODULE>
    <PARAMETERS/>

#! appname_yourname_studio.xcf
    <DVM>"/Applications/Genero Studio 3.20.08-167272.app/Contents/MacOS/gsproxy"</DVM>
    <MODULE></MODULE>
    <PARAMETERS>
      <PARAMETER>--host</PARAMETER>
      <PARAMETER>127.0.0.1</PARAMETER>
      <PARAMETER>--port</PARAMETER>
      <PARAMETER>50457</PARAMETER>
      <PARAMETER>--console</PARAMETER>
      <PARAMETER>/Applications/fourjs/fgl/3.20.10/bin/fglrun</PARAMETER>
      <PARAMETER>demo</PARAMETER>
    </PARAMETERS>

… the file that ends  _studio.xcf has gsproxy as the process launched and some arguments that include the program name.   This .xcf ending _studio.xcf is the one you will see in the browser URL field.  gsproxy acts as an intermediary between Genero Studio and the running instance of fglrun -d, and is used to pass the debugger instructions from when you click next/step in Studio to the fglrun -d instance, and in turn brings the variable values back from the debugger to Genero Studio.

If after the program has terminated you copy and paste the URL ending _studio into another browser tab, this will fail.  The URL you need in this instance is the one that does not end _studio.xcf.  So the two tips here are

  • after running program from Genero Studio in browser, if you want to run it again outside of Studio, copy and paste the URL and remove the _studio
  • this .xcf that does not end _studio.xcf is a close model of what a production .xcf should look like, I say close, a potential improvement is instead of repeating environment variables in many individual .xcf files, consider using an abstract application so they are only defined once.

Final point of operating systems is consider what servers processes are running on and how are they communicating.  With GDC Direct Connection, we know that fglrun is running on one server and GDC is running on your local workstation.  That is typically FGLDIR/bin/fglrun is running on the back-end Linux server and gdc.exe is running on your front-end workstation.  FGLSERVER controls the port that fglrun communicates on, -p controls the port that GDC listens in.  But do you know the present working directory of the gdc.exe executable? Why can’t FGL_GETFILE find a file, have you specified file absolutely or relatively, if relative what is it relative to …. This isn’t unseen Genero magic, this is following operating systems, if I am specifying a file relatively, what is it relative to?, and does the owner of the gdc.exe process have permissions to read that file etc.

I encourage you if necessary to draw a diagram of the various pieces.  This diagram comes in handy with Genero Report Writer, in the architecture there are 3 pieces and so issues with fonts, images, files not found, normally come down to the fact not considering these 3 pieces (the designer, the engine, the viewer).  Is the font or image used by the designer on their PC, is it available on the engine, and on the viewer PC etc

Some tidbits to end.

  • Behind the scenes, FGLSERVER is used for fglrun to communicate to gdc.  When running via Genero Application Server, you don’t ever have to explicitly set it but FGLSERVER is also set for fglrun to talk to the proxy process.   If you are wondering why FGLSERVER is set when using GAS and inside a Genero Mobile app, that is why
  • Within a Genero Mobile application,  a mobile app consists of an fglrun thread and a UI thread. Again fglrun process talking to UI so FGLSERVER will be set, and if you think about client-server (runOnServer) or development mode, that is effectively just using the UI thread, the fglrun is on the connected server.
  • Look at GREDIR/src/bin/gviewdoc.4gl to see what is happening when you preview a report, see the completed report being moved or copied and then viewed.
  • Code completion, real time syntax checking, code beautifier and even code quality,  deep down that is simply an execution of fglcomp with certain arguments as you might have seen in this vim setup guide.

I appreciate I may have raised more answers than answered with this article as it is widespread.  Don’t be afraid to dig deep down to the O/S level and try and understand what is happening as that may help you analyse and resolve issues, and improve your usage of our products.