Example prometheus
An example program that records how it is used, with a Counter, a Gauge, and a Histogram metric.
The example records how an orders program is used. It counts how often the program starts, how each session ends, how many sessions are open at any moment, and how long each session lasts.
IMPORT util
IMPORT prometheus
--
-- Records how the orders program is used, through the Prometheus API.
--
DEFINE runs prometheus.Counter
DEFINE exits prometheus.Counter
DEFINE active prometheus.Gauge
DEFINE duration prometheus.Histogram
DEFINE startedAt FLOAT
MAIN
-- A user who closes the browser never reaches the end of MAIN,
-- so the abandoned outcome is recorded from the close handler.
OPTIONS ON CLOSE APPLICATION CALL onClose
-- A program stopped with SIGTERM does not reach the end of MAIN either.
-- This option is supported on UNIX-like systems only.
OPTIONS ON TERMINATE SIGNAL CALL onTerminate
CALL initMetrics()
LET startedAt = util.Datetime.toSecondsSinceEpoch(CURRENT YEAR TO FRACTION(3))
CALL runs.inc([])
CALL active.inc([])
MENU "Orders"
COMMAND "Enter"
# ... program code
COMMAND "Exit"
EXIT MENU
END MENU
-- The end of MAIN was reached, so this session ended normally.
CALL recordExit("normal")
END MAIN
#
# initMetrics: creates the metrics, once, before any user interaction.
# The name given to create() is the name you query in Prometheus.
#
FUNCTION initMetrics()
LET runs = prometheus.Counter.create("orders_runs_total",
"Times the orders program was started", [])
LET exits = prometheus.Counter.create("orders_exits_total",
"Times the orders program ended, by outcome", ["outcome"])
LET active = prometheus.Gauge.create("orders_sessions_active",
"Orders sessions open now", [])
LET duration = prometheus.Histogram.create("orders_session_seconds",
"How long an orders session lasted",
[5.0, 30.0, 120.0, 600.0, 1800.0], [])
-- A counter series does not exist until its first inc(). Creating each
-- series at zero means the metric appears in /metrics from the start.
CALL runs.add(0, [])
CALL exits.add(0, ["normal"])
CALL exits.add(0, ["abandoned"])
CALL exits.add(0, ["killed"])
END FUNCTION
#
# recordExit: records the end of the session.
# outcome -> value of the "outcome" label: "normal", "abandoned" or "killed"
# Closes the session, counts the exit, and observes how long the session lasted.
#
FUNCTION recordExit(outcome STRING)
DEFINE elapsed FLOAT
LET elapsed = util.Datetime.toSecondsSinceEpoch(CURRENT YEAR TO FRACTION(3)) - startedAt
CALL active.dec([])
CALL exits.inc([outcome])
CALL duration.observe(elapsed, [])
END FUNCTION
#
# onClose: shutdown handler triggered from the client.
# Fires when the user closes the browser or the application tab.
#
FUNCTION onClose()
CALL recordExit("abandoned")
END FUNCTION
#
# onTerminate: shutdown handler triggered by the SIGTERM signal.
# The program continues after this callback, so it stops itself here.
# Without EXIT PROGRAM the session would later record a second, normal exit.
#
FUNCTION onTerminate()
CALL recordExit("killed")
EXIT PROGRAM 1
END FUNCTION
| Metric | Type | Question it answers |
|---|---|---|
orders_runs_total |
Counter |
How many times was the program started? |
orders_exits_total |
Counter |
How did sessions end, normal,
abandoned, or killed? |
orders_sessions_active |
Gauge |
How many sessions are open now? |
orders_session_seconds |
Histogram |
How long did a session last? Distribution, mean, and percentiles. |
A counter series does not exist until the first call to inc(). The example
therefore calls add(0, ...) on orders_runs_total and on each
outcome value of orders_exits_total, creating each series with an initial value of
zero.
The program records only the endings that reach recordExit():
-
An ending that does not reach it updates none of the three metrics:
orders_exits_totaldoes not count the exit,orders_session_secondsobserves no duration, andorders_sessions_activeis not decremented. -
A session killed on Windows is one of those endings, because
ON TERMINATE SIGNALis supported on UNIX-like systems only. -
onClose()does not run the moment the browser closes. Under the Genero Application Server (GAS) it can wait until the user agent timeout expires, so abandoned sessions can appear on a dashboard later than expected.
For the conditions under which onTerminate() and
onClose() run, go to Application termination and Front-end termination.
recordExit() leaves
orders_sessions_active one too high, and the count stays too high until the GAS
restarts. An exit or duration that was never recorded is absent from the data, but the gauge still
reports a number that looks correct. Read that gauge as an indicator of how busy the program is
rather than as an exact count of open sessions.Querying the metrics
You can check the metrics before you query them. Open the GAS metrics endpoint,
http://localhost:6394/metrics on a standalone dispatcher, and search the page
for orders_. All four metrics appear as soon as the program starts:
# HELP orders_runs_total Times the orders program was started
# TYPE orders_runs_total counter
orders_runs_total 1
# HELP orders_exits_total Times the orders program ended, by outcome
# TYPE orders_exits_total counter
orders_exits_total{outcome="killed"} 0
orders_exits_total{outcome="normal"} 1
orders_exits_total{outcome="abandoned"} 0
# HELP orders_sessions_active Orders sessions open now
# TYPE orders_sessions_active gauge
orders_sessions_active 0
# HELP orders_session_seconds How long an orders session lasted
# TYPE orders_session_seconds histogram
orders_session_seconds_bucket{le="5.0"} 1
orders_session_seconds_bucket{le="30.0"} 1
orders_session_seconds_bucket{le="+Inf"} 1
orders_session_seconds_sum 0.576
orders_session_seconds_count 1
The help text is the description passed to create(), and the bucket
boundaries are the ones the example gives Histogram.create().
The metrics endpoint shows the values at the moment you read the page. To see how they change over time, you query them with PromQL, the Prometheus query language, and draw the result as a graph, on the Prometheus server itself or in a monitoring tool such as Grafana. Graphed over time, the same four metrics show how often the program is run, how the outcomes divide, how many sessions are open across the day, and how session length is distributed.
Where Prometheus is already scraping the GAS, you can run the following queries yourself on the Prometheus query page, http://localhost:9090/query on a default installation. The histogram includes a sum and a count, so two of its series give the mean session length in seconds:
orders_session_seconds_sum / orders_session_seconds_count
The buckets answer a question the metrics endpoint cannot. This query reports the session length that 95% of sessions completed within:
histogram_quantile(0.95, sum(rate(orders_session_seconds_bucket[5m])) by (le))
For more PromQL, and for building dashboards on the metrics, refer to the Monitoring the GAS with Prometheus section of the Genero Application Server User Guide and its Using Grafana with Prometheus metrics topic.