Example prometheus

An example program that records how it is used, with a Counter, a Gauge, and a Histogram metric.

The example records how an orders program is used. It counts how often the program starts, how each session ends, how many sessions are open at any moment, and how long each session lasts.


IMPORT util
IMPORT prometheus

--
-- Records how the orders program is used, through the Prometheus API.
--

DEFINE runs      prometheus.Counter
DEFINE exits     prometheus.Counter
DEFINE active    prometheus.Gauge
DEFINE duration  prometheus.Histogram
DEFINE startedAt FLOAT

MAIN
    -- A user who closes the browser never reaches the end of MAIN,
    -- so the abandoned outcome is recorded from the close handler.
    OPTIONS ON CLOSE APPLICATION CALL onClose

    -- A program stopped with SIGTERM does not reach the end of MAIN either.
    -- This option is supported on UNIX-like systems only.
    OPTIONS ON TERMINATE SIGNAL CALL onTerminate

    CALL initMetrics()

    LET startedAt = util.Datetime.toSecondsSinceEpoch(CURRENT YEAR TO FRACTION(3))
    CALL runs.inc([])
    CALL active.inc([])

    MENU "Orders"
        COMMAND "Enter"
            # ... program code
        COMMAND "Exit"
            EXIT MENU
    END MENU

    -- The end of MAIN was reached, so this session ended normally.
    CALL recordExit("normal")
END MAIN

#
# initMetrics: creates the metrics, once, before any user interaction.
# The name given to create() is the name you query in Prometheus.
#
FUNCTION initMetrics()

    LET runs = prometheus.Counter.create("orders_runs_total",
        "Times the orders program was started", [])

    LET exits = prometheus.Counter.create("orders_exits_total",
        "Times the orders program ended, by outcome", ["outcome"])

    LET active = prometheus.Gauge.create("orders_sessions_active",
        "Orders sessions open now", [])

    LET duration = prometheus.Histogram.create("orders_session_seconds",
        "How long an orders session lasted",
        [5.0, 30.0, 120.0, 600.0, 1800.0], [])

    -- A counter series does not exist until its first inc(). Creating each
    -- series at zero means the metric appears in /metrics from the start.
    CALL runs.add(0, [])
    CALL exits.add(0, ["normal"])
    CALL exits.add(0, ["abandoned"])
    CALL exits.add(0, ["killed"])

END FUNCTION

#
# recordExit: records the end of the session.
# outcome -> value of the "outcome" label: "normal", "abandoned" or "killed"
# Closes the session, counts the exit, and observes how long the session lasted.
#
FUNCTION recordExit(outcome STRING)
    DEFINE elapsed FLOAT

    LET elapsed = util.Datetime.toSecondsSinceEpoch(CURRENT YEAR TO FRACTION(3)) - startedAt

    CALL active.dec([])
    CALL exits.inc([outcome])
    CALL duration.observe(elapsed, [])
END FUNCTION

#
# onClose: shutdown handler triggered from the client.
# Fires when the user closes the browser or the application tab.
#
FUNCTION onClose()
    CALL recordExit("abandoned")
END FUNCTION

#
# onTerminate: shutdown handler triggered by the SIGTERM signal.
# The program continues after this callback, so it stops itself here.
# Without EXIT PROGRAM the session would later record a second, normal exit.
#
FUNCTION onTerminate()
    CALL recordExit("killed")
    EXIT PROGRAM 1
END FUNCTION
      
Table 1. Metrics created by the example
Metric Type Question it answers
orders_runs_total Counter How many times was the program started?
orders_exits_total Counter How did sessions end, normal, abandoned, or killed?
orders_sessions_active Gauge How many sessions are open now?
orders_session_seconds Histogram How long did a session last? Distribution, mean, and percentiles.

A counter series does not exist until the first call to inc(). The example therefore calls add(0, ...) on orders_runs_total and on each outcome value of orders_exits_total, creating each series with an initial value of zero.

The program records only the endings that reach recordExit():

  • An ending that does not reach it updates none of the three metrics: orders_exits_total does not count the exit, orders_session_seconds observes no duration, and orders_sessions_active is not decremented.

  • A session killed on Windows is one of those endings, because ON TERMINATE SIGNAL is supported on UNIX-like systems only.

  • onClose() does not run the moment the browser closes. Under the Genero Application Server (GAS) it can wait until the user agent timeout expires, so abandoned sessions can appear on a dashboard later than expected.

For the conditions under which onTerminate() and onClose() run, go to Application termination and Front-end termination.

Important: An ending that does not reach recordExit() leaves orders_sessions_active one too high, and the count stays too high until the GAS restarts. An exit or duration that was never recorded is absent from the data, but the gauge still reports a number that looks correct. Read that gauge as an indicator of how busy the program is rather than as an exact count of open sessions.

Querying the metrics

You can check the metrics before you query them. Open the GAS metrics endpoint, http://localhost:6394/metrics on a standalone dispatcher, and search the page for orders_. All four metrics appear as soon as the program starts:

# HELP orders_runs_total Times the orders program was started
# TYPE orders_runs_total counter
orders_runs_total 1
# HELP orders_exits_total Times the orders program ended, by outcome
# TYPE orders_exits_total counter
orders_exits_total{outcome="killed"} 0
orders_exits_total{outcome="normal"} 1
orders_exits_total{outcome="abandoned"} 0
# HELP orders_sessions_active Orders sessions open now
# TYPE orders_sessions_active gauge
orders_sessions_active 0
# HELP orders_session_seconds How long an orders session lasted
# TYPE orders_session_seconds histogram
orders_session_seconds_bucket{le="5.0"} 1
orders_session_seconds_bucket{le="30.0"} 1
orders_session_seconds_bucket{le="+Inf"} 1
orders_session_seconds_sum 0.576
orders_session_seconds_count 1

The help text is the description passed to create(), and the bucket boundaries are the ones the example gives Histogram.create().

The metrics endpoint shows the values at the moment you read the page. To see how they change over time, you query them with PromQL, the Prometheus query language, and draw the result as a graph, on the Prometheus server itself or in a monitoring tool such as Grafana. Graphed over time, the same four metrics show how often the program is run, how the outcomes divide, how many sessions are open across the day, and how session length is distributed.

Where Prometheus is already scraping the GAS, you can run the following queries yourself on the Prometheus query page, http://localhost:9090/query on a default installation. The histogram includes a sum and a count, so two of its series give the mean session length in seconds:

orders_session_seconds_sum / orders_session_seconds_count

The buckets answer a question the metrics endpoint cannot. This query reports the session length that 95% of sessions completed within:

histogram_quantile(0.95, sum(rate(orders_session_seconds_bucket[5m])) by (le))

For more PromQL, and for building dashboards on the metrics, refer to the Monitoring the GAS with Prometheus section of the Genero Application Server User Guide and its Using Grafana with Prometheus metrics topic.