Many organisations have chosen IBM Datacap as their preferred solution for intelligent data capture and workflow management. However, as with any enterprise system, monitoring its performance and health is crucial for ensuring uptime and identifying potential bottlenecks. Following our recent blog on using Prometheus and Grafana to monitor Document Capture systems, many readers asked how to apply this in a Datacap environment. In this article, we’ll walk through what’s involved in real-time monitoring of IBM Datacap with Grafana, with a visualisation example.
Why Prometheus and Grafana?
Prometheus: An open-source systems monitoring and alerting toolkit designed for reliability and scalability. It’s optimised for collecting time-series data, storing it efficiently, and offering powerful query capabilities.
Grafana: A widely-used visualisation tool that integrates seamlessly with Prometheus to display the collected metrics in a visually appealing and actionable way.
A Datacap Monitoring Dashboard in Grafana:
Consider the Grafana dashboard above. It provides extensive information about the health of a Datacap system, enabling real-time monitoring of IBM Datacap with Grafana. Organisations can customise dashboards to meet their specific needs. This dashboard provides a list of Datacap Applications on the left. The user selects one of these applications and the statistics displayed pertain to the selected application. Any items in red indicate that there is a problem that needs to be investigated. In our screenshot, the wTM Service Status is red, indicating that there is a problem. Likewise, if the Number of Batches Waiting exceeded a pre-defined threshold, this would also turn red. You can configure red items in the dashboard to trigger alerts to support staff, either via email or any of the available integrations to platforms such as Google Chat, Discord, Slack or Microsoft Teams.
Step 1: Pushing Datacap Metrics to Prometheus
While Prometheus typically “pulls” or “scrapes” metrics like CPU load or memory usage, from the monitored system, to monitor Datacap performance, we can push metrics from Datacap to Prometheus using the Pushgateway, because Datacap’s metrics cannot be directly scraped by Prometheus.
a). Expose Datacap Metrics: As already mentioned, Datacap doesn’t natively expose metrics in the Prometheus format. Therefore, the first step is to collect relevant performance data from Datacap. This typically involves:
– Accessing Datacap’s comprehensive REST API (specifically the wTM endpoints) to pull statistics such as task duration, throughput, and job status. This screenshot shows some of the endpoints that are exposed by Datacap’s REST API (official documentation here):
– Using the Datacap Engine Database to surface data such as the number of batches waiting for background processing tasks (with a view to sending an alert to a group or individual if this breaches a certain threshold).
– Writing a custom exporter or script that converts Datacap data into a Prometheus-compatible format then using the Pushgateway to push the data to Prometheus.
For the purpose of this blog, we wrote a simple PowerShell script that leverages the Datacap wTM Rest API to get some of the data. Other data was retrieved from the Engine Database of a Datacap Application. You can schedule these scripts to run regularly via Windows Task Scheduler.
This particular Grafana visualisation shows the individual tasks that form part of a Datacap Job and how many batches are currently awaiting processing by each task:
We make a call to the API using the /Session/Logon endpoint. This logs in to Taskmaster and returns a session ID as a cookie. Here’s some PowerShell which achieves this:
# === CONFIGURATION ===
$baseUrl = "http://localhost:83/service"
$logonEndpoint = "$baseUrl/Session/Logon"
$logoffEndpoint = "$baseUrl/Session/Logoff"
# === Log on to get session token ===
$logonBody = @{
user = "admin"
password = "admin"
station = "1"} | ConvertTo-Json
$response = Invoke-WebRequest -Uri $logonEndpoint -Method Post -Body $logonBody -ContentType "application/json" -SessionVariable webSession
# === Get the wTMID ===
$wTMID = $webSession.Cookies.GetCookies($logonEndpoint)["wTMID"].Value
Write-Host "Captured wTMID: $wTMID"
ForegroundColor Cyan
# === Log Off ===
$response3 = Invoke-WebRequest -Uri $logoffEndpoint -Method Post -SessionVariable webSession
# Check if the response is not empty
if ($response3.StatusCode -eq 200) { $
data = $response3.Content | ConvertFrom-Json
Write-Host "Logoff Successful" -ForegroundColor Cyan}
We then make a call to the appropriate Statistics endpoint using the session ID obtained earlier, in this case the /Statistics/BatchByTask endpoint.
We send the data contained in the response to the Prometheus Pushgateway. We then use the /Session/Logoff endpoint to log out of Datacap.
Step 2: Visualising Metrics with Grafana
After Prometheus starts collecting Datacap metrics, the next step is to visualise them using Grafana.
- Create a Dashboard in Grafana:
- Go to Create > Dashboard.
- Click Add new panel and choose the Prometheus data source.
- Select the Visualisation type (In this example “Stat”, which displays statistics)
Write Prometheus queries to display the relevant statistics. For example, to visualise the number of batches waiting for VScan you could use the query:
datacap_task_count{step="VScan"}- Customise the visualisation settings (e.g., colours, display units etc.) to fit your needs.
- Set Alerts (Optional):
- You can also use Grafana to set up alerts based on certain thresholds.
- For instance, if the number of batches waiting for Profiler or Export exceeds a certain threshold, you can create an alert to notify administrators.
This diagram sums up the process:
Conclusion
Hopefully, you have seen that real-time monitoring of IBM Datacap with Grafana is a real possibility – organisations can gain visibility into system performance, identify potential issues early, and maintain a high level of operational efficiency. While the setup process involves creating custom exporters and writing Prometheus queries, the flexibility of these tools enables a highly tailored and scalable monitoring solution.
The integration forms a vital component of any enterprise monitoring stack, providing a powerful, visual means to track key metrics and ensure that the IBM Datacap system runs smoothly.
We’d love to hear from you! Drop your questions or thoughts in the comments section below — whether you’re exploring monitoring options, facing challenges with your current setup, or just want to dive deeper into the technical implementation details. Let’s start a conversation!
Don’t forget – Contact Us to find out how Open Connections can help you to unlock these hidden insights today!
0 Comments