Ask Reuben

Composability and Genero

How can I increase the composability of my Genero system?

How can I break up my monolithic Genero system into smaller parts?

Composability is a design principle that deals with how well your system interacts with other systems. Genero systems have historically been one standalone monolithic system. Now there are demands to take that system and have it interface to another system, or to replace parts of another system with a Genero system, or the other way round replace parts of your Genero system with a 3rd party system. These require the ability to break your Genero system into smaller pieces.

Similarly in cloud environments there are requirements to distribute your monolithic Genero system across multiple servers. By breaking your system into several smaller modules, you can allocate resources more efficiently. Rather than trying to squeeze the same monolithic system onto many servers, place one module in one environment, place another module in another environment, have these different environments scale independently and still communicate with one another.

In this article, I am going to use the following words

Application/Program – a single Genero program (.42r or .42m with MAIN)
System – all of your Genero programs that are packaged together as …
- in the case of an ISV- implement the system sold to your customer(s),
- in the case of an Enterprise customer – implement your organisations system
Module – a subset of your System that contains many programs.

The use of module like this fits in with how it is typically used from a marketing perspective to represent a subset of a system. This is different from the Genero documentation where the term module refers to a single .4gl file.

If you are reading this article, you maybe approaching this article with the following goals in mind …

one or more modules in my system are really good and I want to investigate selling just the module(s) to someone else without having to include my whole system
have my system or modules of it interface with a third party application or system.
to make better use of cloud architectures, by having modules on seperate servers

To do this requires you to break up your monolithic system into clearly defined pieces that have clearly defined interfaces. if you do this well it means you have a composable system.

Coding Standards

In order to do any work on your code base, you should have the ability to set and enforce coding standards upon your developers.

You do not want to do the work and then 12 months later, find a developer has done something that undoes your investment in your code.

You need to be in a position whereby you can reject a developer’s work and tell them to go back and rework their code to fit within the standards of your system.

The success of your organisation and the ability for your system or parts of your system to interface with other systems will depend upon the reliability of your systems functionality within that defined interface. If your part of the solution does not do what it is supposed to do, then you may find it being removed and replaced.

A good example of code standards in recent years has been with the advent of light and dark themes. To do this successfully requires a coding standard that says no hard coded colours in your source code. A development environment that allows hard coded colours in their sources e.g. ERROR error_text ATTRIBUTES(RED) will find it difficult to adapt to light and dark themes compared to one that defines color once in a presentation style e.g. ERROR error_text ATTRIBUTES(STYLE="error").

Coding standards are more than just coding conventions. Code can comply with all your coding conventions but still not do what it is supposed to do. Code standards enforce the ongoing success of your development and also reduce ongoing maintenance costs by capturing coding errors early.

Code Reviews

Code Reviews, both automatic and manual are an important way of making sure that developers code adheres to your coding standards.

An automated code review is where a program analyses code and identifies any infractions of your coding standards. Before I joined Four Js I implemented scripts that ran overnight on the checked in code base and sent me an e-mail of any coding standard infractions. This allowed me to catch any infractions the day after the developer checked them in and then re-educate the developer when it was fresh in their mind. Today you might add this testing to the compile stage or to the check-in stage to catch the infractions even earlier. In Genero Studio we recently introduced some Code Quality tools which you should investigate if you have not already.

A manual code review is where another developer reviews your changes and makes sure that they still adhere to your coding standards. A manual code review is subject to human error in detecting conformance but is able to detect infractions where it is difficult to automate a rule.

Ideally your development environment has both automatic and manual code reviews.

When making a system more composable, particular attention will be made to the defined interfaces within the system. Your coding standards and code reviews will check that the interface remains the same and that the individual program stays within the bounds of what its interface says it will do.

You may also have rules as to what syntax is allowed in certain files. Code reviews will check the ongoing conformance of your code base to these rules.

Well written coding standards help in the adherence too, and the detection of any infractions in your coding standards by code reviews. They need to be simple and understandable in order to be documentable, adhered to, and enforceable.

Modern Code

In general, if you want your application to fit in with other modern applications then you should be prepared to embrace some modern coding techniques.

Other languages and other development environments will have similar concepts.

If you continue to do things the way Informix-4gl did it, you will eventually paint yourself into a corner. If the rest of the world uses RESTful Web Services, JSON to interface then you should be prepared to use RESTful Web Services, JSON etc to interface. That is one of the great things of Genero is that we keep adding modern syntax and concepts to the language so that you can incorporate them in your Genero programs. Yes it means sometimes you have to tinker with your code but that tinkering and minor surgery is much less than the cost of a full rewrite every time something new comes along.

IMPORT FGL and PACKAGE

Informix-4gl programs that make up systems are not by themselves structured. With the linking concept, developers have had the ability to add any .4gl as and when needed. Unless there are some clearly defined code standards, a Genero system potentially can turn into a barely decipherable mess with extra .42m being added to the fgllink instruction as and when required. Developers often find that when they added one .42m then they need to another and then another. Then someone would make a change on in one .4gl and then they needed to recompile other .4gl and recompile other Genero programs.

By using IMPORT FGL, and PACKAGE, you can structure your sources which is good for maintenance, reliability, reusability, and deployment.

you can see at a glance what files are required by a file by the IMPORT FGL at the top.
the compiler can check the number of parameters and returning values in functions calls.
autocompletion and real time syntax checking in source code editors is improved as it can suggest and test all imported symbols.
build rules can ensure the right files are rebuilt.

If you have not already, you should review what it would take to remove the use of fgllink from your build process and make your Genero programs “no-link” by use of IMPORT FGL and if necessary PACKAGE.

Genero Studio has Build Rules that are link and no-link. Have a look at the different build, and link and execution rules in Genero Studio.

Also note as part of any reorganisation the source code documentation generator and how it assumes a structured code base.

Circular Dependencies

When structuring your sources to use IMPORT FGL, you want to avoid circular dependencies. Yes they are allowed but if you want a clear structure to your source files you should try and avoid them. If you end up in a situation where A->B->C->A then that means everywhere you use A you also have to use B and C.

The last place you normally find circular dependencies is in your exception handling. Even here you should avoid it by making sure your exception handling code is self-contained and does not have any library references.

No More Globals

Whilst GLOBALs are legal syntax, the advice for many years has been to remove them and replace them with IMPORT FGL and PUBLIC variables. See the note in the Informix-4gl to Genero migration guide.

Dependency Diagram

You can utilise the Dependency Diagram inside Genero Studio to get a pictorial relationship of the relationship between your source files. The first time you attempt this on all your source files you are going to get a barely decipherable mass of dotted lines and white rectangles. The key I believe to using this diagram effectively as per this article is to filter out certain library calls using the Filter View and in conjunction with the the Expand and Collapse functionality.

A good starting point for any code structure exercise is to use Genero Studio to render a dependency diagram, and use the Poster Printing facility (File -> Poster Printing Setup) to physically print it using multiple pages and stick it on a wall. Be in a position where you can compare your before and after.

Efforts to split code into modules, functionality should result in the possibility of producing a simpler dependency diagram that is interpretable by a human being, even though there maybe hundreds of source files.

If you have successfully removed all circular dependencies, all the arrows will go up, each arrows base will be in the top half of a file, each arrows points will be in the bottom half of a file.

Naming Convention

When it comes to splitting a system into smaller parts, one of the “free” tools you have at your disposal is naming conventions. In particular assigning a concept of “module”, or “purpose” to source code files and database tables by the inclusion of a small code in a filename or database table. If the source code filename or database table can tell you its “module”, or “purpose” by part of its name, it is then very easy to identify if all of its code or references are valid for its “module” or “purpose”.

Your code standard then becomes making sure that there is no code or references that does not fit the “module” or “purpose” of that code.

Chances are your source code files names already have some form of naming convention, but do you have any coding standards that restrict the code that can be used in an individual source code file based on the filename?

Splitting By Business Functionality

One of the first ways a system is broken up into smaller pieces is to group programs into modules based on business functionality. A module is normally given a short two or three code to help identify it. For an ERP system this might involve the following …

lib – library – common code used throughout the system
sys – system – configuration parameters for the system to differentiate individual use of that system by a customer.
gl – General Ledger
dl – Debtors Ledger
cl – Creditors Ledger
al – Asset Ledger
cl – Cash Ledger
st – Stock / Inventory
so – Sales Orders
po – Purchase Orders
wo – Works Orders
hr – Human Resources
py – Payroll

A common idea is to use these short two or three letter codes is to identify the database tables and source code that belongs to each individual module. So for example…

all your library files would begin lib_ e.g. lib_date.4gl would contain all your date related library functions, lib_string would contain all your string related library functions.
all your general ledger database tables would begin gl e.g. gl_account, gl_transaction, gl_balance
all your general ledger source code files would begin gl e.g. gl_account_maintenance.4gl, gl_account_maintenance.per

Your coding standards then enforce this breaking up of your application into modules. You have coding standards that say database tables that begin “gl” can only by referenced by .4gl code that begin “gl”. Forms that begin “gl” can only be opened from within .4gl files that begin “gl”. .4gl files that begin “gl” can only be imported by other modules that begin “gl”. Such coding rules are easy to enforce, both manually, and automatically. For example, “IMPORT FGL dl…” from within a .4gl whose name began “gl…” would be an easy red flag for both a machine and human to identify.

There may also be a concept of hierarchy within your modules. You would have a rule that says library can be called from any module. If commercially you said that if you buy the Debtors Ledger module then you must also buy your General Ledger module then it would be permissible to allow “gl” tables and code to be referenced by “dr” code, however if your goal is to allow your Debtors Ledger module to reference any other General Ledger module then you would not allow such code.

If your goal is for various modules to be truly independent then you have to review how your modules interact with each other. For instance a sale in the “so” Sales Order module may need to check a debtors balance from your “dl” Debtors Ledger module.

You would definitely not allow direct reference of a “dl” database table from your “so” code like this…

#! so_sales_entry.4gl
...
   SELECT dl_current_balance
   FROM dl_balance                   # BAD - should not reference dl table from so code.
   WHERE dl_id = ?

You would similarly avoid referencing a dl module from your “so” code” …

#! so_sales_entry.4gl
IMPORT FGL dl_balance             # BAD - should not import dl code from so code.
...
   IF dl_balance.get_current_balance(?) < 0 THEN

Instead the better way is for your “dl” module to expose the needed information via a Web Service call. Your 4gl code might then look like

#! so_sales_entry.4gl
IMPORT FGL so_dl_interface           # GOOD - imports so code from so code
...
    IF so_dl_interface.get_current_balance(?) < 0

The key difference is that you can replace so_dl_interface.42m with different interfaces to different debtors ledger systems. Inside so_dl_interface.4gl it can reference your debtors ledger module or it can reference another third party vendors debtors ledger module.

This works both ways. By exposing your in this case Debtors Ledger functionality via web service and making it standalone, then other systems can in turn utilise your debtors ledger module.

I will get into the details of exposing a Genero function via a High-level RESTful Web Service later but in short to interface between different modules, you would use …

4gl calls if you define a module as being dependent on a particular module your provide e.g. library module
web service calls if the modules are independent.

By splitting by business functionality, you can break your single system into several modules that can interface with modules written by third parties. You can identify and enforce this break by coding standards that ensure that your modules are truly independent of one another by not allowing references to one module in another.

Earlier I mentioned using the Dependency Diagram. If done well, you should be able to use Collapse to collapse the code for a module down to a single white rectangle. This then allows you to check relationships between modules.

Once you have split your code base into modules based on business functionality, you can then …

install different modules onto different machines. Configure machines appropriate for the module load.
sell a single module without having to provide your whole system
replace a single module with a third party module
combine your system with one or more third party system to produce a larger system

Splitting By Code Functionality

There is another way to split code that is also beneficial. In particular it helps in exposing code as web services, and it can also help with database and report tool independence. That is to split and identify code by code functionality, again a little two or three letter code as part of the naming convention helps here …

ui – User Interface
br – Business Rules
ct – Controller
ws – Web Service
db – Database
rp – Report

ui for User Interface tends to be a very common shortening. The terms for business rules and controller can vary more so you may see other words and codes used.

The controller runs the program, showing the user interface, and then having the business rules. So a…

ct program can import a br and ui file
br program cannot import a ct or ui file
ui file cannot import a ct file
the ui file can be replaced to indicate a different user interface
the ui file can be replaced with a ws to indicate that a web service is setting/retrieivng values rather than a screen.

ct ----> ui ----> br
ct -------------> br

With these splits in place, you then enforce coding standards that control what type of syntax is allowed in particular .4gl code files.

A key rule to illustrate this point is to only have ui code in certain .4gl files and more importantly not to have ui code in certain .4gl files.

User interface syntax such as OPEN WINDOW, INPUT, ERROR, DISPLAY etc can only appear in 4gl files that have _ui_ as part of their file name.

There is a very good reason for this rule. When it comes to exposing business rules as a web service, it is important that your web service does not have any ui.

This is best illustrated by code similar to …

AFTER FIELD number_field
   IF number_field < 0 THEN
      ERROR "Number must not be negative"
      NEXT FIELD number_field
   END IF

The goal is that this business rules should also be able to be called from a web service. That is define this business logic ONCE rather than having it be coded independently in two places.

With the following, note the common mistake …

#! ?_ui_?.4gl
IMPORT FGL ?_br_? AS br
...
AFTER FIELD number_field
   CALL ?_br_?.validate_number(number) RETURNING ok
   IF NOT ok THEN
      NEXT FIELD number_field
   END IF

#! ?_br_?.4gl
FUNCTION validate_number(number) RETURNS (ok)
   IF number < 0 THEN
      ERROR "Number must not be negative"   -- BAD, ui code in br file
      RETURN  FALSE
   END IF
   RETURN TRUE
END FUNCTION

… Note the ERROR inside the business rules code. This is bad. If the intent is to also expose this business rule (br) function via a web service, you cannot have this ui code (the ERROR) executed by a Web Service.

The better code is to have something like …

#! ?_ui_?.4gl
IMPORT FGL ?_br_? AS br
...
AFTER FIELD number_field
   CALL br.validate_number(number) RETURNING ok, error_text
   IF NOT ok THEN
      ERROR error_text  -- GOOD, ui code in ui file
      NEXT FIELD number_field
   END IF

#! ?_br_?.4gl
FUNCTION validate_number(number) RETURNS (ok, error_text)
   IF number < 0 THEN
      RETURN "Number must not be negative", FALSE  -- Good, no UI code in br file
   END IF
   RETURN TRUE
END FUNCTION

… the ui code contains no business logic, the business logic code contains no user interface. The business logic is defined once. The business logic can be safely called from multiple places, from a GUI program, and from a Web Service.

For example, the web service code could reuse the same validation like …

#! ?_ws_?.4gl
IMPORT FGL ?_br_? AS br
...
CALL br.validate_number(number) RETURNING ok, error_text  
IF NOT ok THEN    
    # handle error
END IF
# continue success

So by splitting code by functionality, naming conventions and coding standards can be used to ensure that only certain syntax is utilised in certain .4gl files. This aids the composability and code reviews ensure these levels of composability are maintained.

Database and Report Independence

In the naming conventions, I floated the idea of separating database code (db) and reporting code (rp) into seperate source files. Whilst not something you would do in a first pass, consider the case where …

instead of a database connection, you interact with databases via another method such as a web service or JDBC.
you use a non-ODBC database such as a NoSQL database.
replacing your ASCII report tool with another tool such as Genero Report Writer

… both tasks would be more easily accommodated if you are able to seperate the code that communicates with the database, with the reporting tool etc.

Interfacing via Web Services

I don’t want to educate on basics of Web Services. If the topic of Web Service is new to you with Genero, have a look at this series of articles in one, two, three parts or start with the documentation.

I would encourage you to research the high-level RESTful Web Services framework. Yes you can do SOAP or you can do things at a low-level but I think the best success will be with high-level RESTful Web Services.

There are two key exercises that I think will help as building blocks.

The first is to start with a simple function, let’s call it foo(). Create a genero program containing a.4gl and b.4gl where a.4gl contains foo() and b.4gl calls foo() …

#a.4gl
FUNCTION foo()
...
END FUNCTION

#b.4gl
IMPORT FGL a
...
CALL a.foo()

Now do the following …

create a Genero program using a.4gl that exposes foo() as a Web Service. Consume that function using curl, Postman or similar to show that you can consume that web service from a 3rd party
create a program using another language that does the equivalent of foo() (or perhaps use something like Postman). Create a Genero program using b.4gl that consumes this web service via CALL foo(). Use fglrestful to create part of this program.
create two programs. The first program is as in part 1. The second program is like the program in part 2 but instead it consumes the web service provided by the Genero program created in part 1.

They say a picture says a thousand words so to illustrate with a picture the original program is the green arrow, the 3 parts to the exercise each represent a red arrow …

The above illustrates how your Genero application can call and expose a function, both to itself, to 3rd parties, and to other Genero applications. I should perhaps flesh that out as an example in GitHub.

The second skillset is being able to identify and rationalise the interfaces. With your dependency diagram, a starting point is the arrows that exist between modules.

If we take a module such as General Ledger (gl) module, we will find interfaces …

admin – create, modify, delete a GL account
transactional – post a series of GL transactions
enquiry – get details of a single account, get details of a single transaction, get details of a list of transactions, get a balance of an GL account.

These are your interfaces that you will need an API for. You will find commonality between your modules as to the interfaces. For instance in your Debtors Ledger (dl) module, you will find ….

admin – create, modify, delete a Debtors Ledger account
transactional – post a series of Debtors Ledger transactions
enquiry – get details of a single account, get details of a single transaction, get details of a list of transactions, get a balance of a Debtors Ledger account.

What you are looking to do is identify patterns in your interfaces. These can be repeated between modules and lead to positives that consistency brings. You are also looking to identify unnecessary interfaces and/or consolidate interfaces, do you need multiple interfaces that return a single field or can they be combined in one interface that gets all fields. Remember that any interface will need to be documented, if you can consolidate you can simplify the documentation.

Error Handling with Web Services

What does your program do if the web service it calls is not running or is slow to respond? You now have to add in some extra handling that was not there before. You don’t want your calling program to crash because a web service it relied on was down …

Before your code might simply have been …

WHENEVER ANY ERROR CALL serious_error
...
LET standard_price = get_price(product_code)

… but now consider adding code to handle an error occurring in the web service call

IMPORT FGL price

    CALL price.get(product_code) RETURNING wsstatus, price_response.* -- call webservice
    IF wsstatus = 0 THEN
        LET standard_price = price_response.price -- the webservice returned a value
    ELSE 
       ... -- code to handle the case that something went wrong with the webservice
   END IF

There will be a little more complex code around the web service call to make sure that you handle the case of something going wrong inside the web service call.

Speed of a Web Service call

A single function call will outperform a single web service call. A web service call needs to do the following …

the calling parameters are packed up into a request by the consumer
the request is sent across the network/internet to the provider
the request is unpacked by the provider
the provider determines the values to return
the return values are packed into a response by the provider
the response is sent back across the network/internet to the consumer
the response is unpacked to provide the return value by the consumer

If you have a function that is called repeatedly from inside a loop (FOR, FOREACH, WHILE), this performance difference will be magnified. Whilst you may not a difference between a function call taking 0.001 seconds and a web service call taking 0.1 seconds, you will notice that difference if that call is inside a loop that is executed 1000 times. Something that took 1 second is now taking 100 seconds.

The technique is to bundle repeated calls into a single web service. So rather than a web service to return debtor balance taking an input parameter of a single debtor code and returning a single balance amount, the exposed web service may take an input parameter of an array of debtor codes and return an array of balances. This exposed web service is then only called once. Using the above maths, this web service would take approximately 1.1 seconds ( 1 second for the 1000 calculations, 0.1 for the single packed and unpacking).

So when it comes to deciding to expose some business functionality as a web service, consider the case where instead of processing a single request, you have the ability to combine multiple requests into a single request. Similarly when consuming a web service, examine if it is possible to combine request.

User Interface Composability

There is another topic that creeps into any discussion on composability of applications. When you are selling or using a system that contains two or more applications from different vendors, you typically want them to have similar User Interfaces. To the untrained eye, you want the end user to think they are using one application. Even if the end user is aware they are two seperate applications, you want them to have a similar user experience so that when the user instinctively does something in a Genero application, it is similar to what they do in the other application(s).

This is where Genero Browser Customization comes into play. At its simplest this might involve changing the primary color, changing the browser tab icon etc to more complex cases such as changing the calendar used by DateEdit widget.

The important point to note is that you can go a long way to making a Genero application look like another third party application. The challenge then becomes identifying the Genero program and identifying the non-Genero program (apart from the ua/r in the URL :-)) . You are not constrained to a white background with a blue chromebar.

Summary

I hope I have not scared you.

Some Genero developers have had coding standards that split by business functionality, split by code functionality as described here since the mid to late 90’s.

You might not know it but if you look closely you might have some elements in your development environment already. All your library code might be in code beginning lib… Your code might be structured into modules already by the prefix of a filename. What you might not have is the clearly defined interfaces.

If you have not already, try examine your code using the Dependency Diagram. The Dependency Diagram is one of those things that when it is right, it looks right. Does your diagram look right?