---
title: "ELK Stack & Logstash Integration | DreamFactory Docs"
source: "https://docs.dreamfactory.com/system-settings/logstash"
canonical_url: "https://docs.dreamfactory.com/system-settings/logstash"
converted_at: "2026-07-31T23:49:23.078Z"
format: "markdown"
converted_by: "html-to-md-ai"
---
This guide provides step-by-step instructions for installing and configuring Elasticsearch, Logstash, and Grafana to work with DreamFactory's Logstash connector for API activity monitoring and analytics.

## Table of Contents[​](#table-of-contents)

1. [Prerequisites](#prerequisites)

2. [Installation](#installation)

- [Elasticsearch](#elasticsearch)

- [Logstash](#logstash)

- [Grafana](#grafana)

3. [Configuration](#configuration)

- [Elasticsearch Configuration](#elasticsearch-configuration)

- [Logstash Configuration](#logstash-configuration)

- [Grafana Configuration](#grafana-configuration)

4. [DreamFactory Integration](#dreamfactory-integration)

5. [Dashboard Setup](#dashboard-setup)

6. [Monitoring Data Consumption by App and API Key](#monitoring-data-consumption-by-app-and-api-key)

7. [Troubleshooting](#troubleshooting)

---

## Prerequisites[​](#prerequisites)

- Ubuntu Server (20.04 or later recommended)

- Root or sudo access

- Minimum 2GB RAM (4GB+ recommended)

- Java 11 or later

- DreamFactory instance with Logstash connector enabled

---

## Installation[​](#installation)

### Elasticsearch[​](#elasticsearch)

### Step 1: Install Java[​](#step-1-install-java)

```
sudo apt updatesudo apt install -y openjdk-11-jdk
```

### Step 2: Add Elasticsearch Repository[​](#step-2-add-elasticsearch-repository)

```
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.listsudo apt update
```

### Step 3: Install Elasticsearch[​](#step-3-install-elasticsearch)

```
sudo apt install -y elasticsearch
```

### Step 4: Configure Elasticsearch Memory[​](#step-4-configure-elasticsearch-memory)

For systems with limited RAM, reduce heap size:

```
sudo mkdir -p /etc/elasticsearch/jvm.options.dsudo tee /etc/elasticsearch/jvm.options.d/heap.options << EOF-Xms256m-Xmx256mEOF
```

### Step 5: Start and Enable Elasticsearch[​](#step-5-start-and-enable-elasticsearch)

```
sudo systemctl daemon-reloadsudo systemctl enable elasticsearchsudo systemctl start elasticsearch
```

### Step 6: Verify Installation[​](#step-6-verify-installation)

Wait 30 seconds, then check status:

```
sudo systemctl status elasticsearch
```

Get the default password:

```
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
```

Test connection:

```
curl -k -u elastic:YOUR_PASSWORD https://localhost:9200
```

---

### Logstash[​](#logstash)

### Step 1: Install Logstash[​](#step-1-install-logstash)

```
sudo apt install -y logstash
```

### Step 2: Configure Logstash Memory[​](#step-2-configure-logstash-memory)

```
sudo tee /etc/logstash/jvm.options << EOF-Xms256m-Xmx256mEOF
```

### Step 3: Create Logstash Configuration[​](#step-3-create-logstash-configuration)

Create the DreamFactory configuration file:

```
sudo tee /etc/logstash/conf.d/dreamfactory.conf << 'EOF'input {  gelf {    port => 12201    type => "dreamfactory"  }}filter {  if [type] == "dreamfactory" {    # Rename host field to avoid conflict with ECS (Elastic Common Schema) host object    # Note: ECS is a vendor-neutral standard that works across all cloud providers and on-premise    if [host] {      mutate {        rename => { "host" => "source_hostname" }      }    }        # Add host information as object for ECS compatibility    # This follows the ECS standard and works on AWS, Azure, GCP, on-premise, etc.    mutate {      add_field => {         "log_source" => "dreamfactory"        "[host][name]" => "%{source_hostname}"      }    }        # Parse log level if present    if [level] {      mutate {        uppercase => [ "level" ]      }    }        # Extract user_id from JWT session token    ruby {      code => '        begin          # Access nested field directly          event_data = event.get("event")          if event_data && event_data.is_a?(Hash)            request_data = event_data["request"]            if request_data && request_data.is_a?(Hash)              headers = request_data["headers"]              if headers && headers.is_a?(Hash)                token = headers["x-dreamfactory-session-token"]                if token && token.is_a?(String) && token.include?(".")                  parts = token.split(".")                  if parts.length >= 2                    payload_b64 = parts[1]                    # Add padding for base64 URL-safe decoding                    padding = (4 - payload_b64.length % 4) % 4                    payload_b64 += "=" * padding                    # Decode and parse                    require "base64"                    require "json"                    payload_json = Base64.urlsafe_decode64(payload_b64)                    payload = JSON.parse(payload_json)                    if payload["user_id"]                      event.set("[user][id]", payload["user_id"])                    end                  end                end              end            end          end        rescue => e          # Silently fail        end      '    }  }}output {  elasticsearch {    hosts => ["https://localhost:9200"]    user => "elastic"    password => "YOUR_ELASTICSEARCH_PASSWORD"    ssl_enabled => true    ssl_verification_mode => "none"    index => "logstash-dreamfactory-%{+YYYY.MM.dd}"  }}EOF
```

**Important:** Replace `YOUR_ELASTICSEARCH_PASSWORD` with the password you obtained in Step 6 of Elasticsearch installation.

### Step 4: Set File Permissions[​](#step-4-set-file-permissions)

```
sudo chown logstash:logstash /etc/logstash/conf.d/dreamfactory.confsudo chmod 644 /etc/logstash/conf.d/dreamfactory.conf
```

### Step 5: Test Configuration[​](#step-5-test-configuration)

```
sudo /usr/share/logstash/bin/logstash --path.settings=/etc/logstash --config.test_and_exit
```

### Step 6: Start and Enable Logstash[​](#step-6-start-and-enable-logstash)

```
sudo systemctl daemon-reloadsudo systemctl enable logstashsudo systemctl start logstash
```

### Step 7: Verify Installation[​](#step-7-verify-installation)

```
sudo systemctl status logstashsudo tail -f /var/log/logstash/logstash-plain.log
```

---

### Grafana[​](#grafana)

### Step 1: Install Dependencies[​](#step-1-install-dependencies)

```
sudo apt install -y software-properties-commonsudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
```

### Step 2: Add Grafana GPG Key[​](#step-2-add-grafana-gpg-key)

```
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -sudo apt update
```

### Step 3: Install Grafana[​](#step-3-install-grafana)

```
sudo apt install -y grafana
```

### Step 4: Start and Enable Grafana[​](#step-4-start-and-enable-grafana)

```
sudo systemctl daemon-reloadsudo systemctl enable grafana-serversudo systemctl start grafana-server
```

### Step 5: Access Grafana Web UI[​](#step-5-access-grafana-web-ui)

Open your browser and navigate to:

```
http://YOUR_SERVER_IP:3000
```

Default credentials:

- Username: `admin`

- Password: `admin` (you'll be prompted to change it)

---

## Configuration[​](#configuration)

### Elasticsearch Configuration[​](#elasticsearch-configuration)

### Basic Security Settings[​](#basic-security-settings)

Elasticsearch 8.x has security enabled by default. The default user is `elastic` with a randomly generated password.

### Index Management[​](#index-management)

Logs are automatically indexed with the pattern: `logstash-dreamfactory-YYYY.MM.dd`

View indices:

```
curl -k -u elastic:YOUR_PASSWORD https://localhost:9200/_cat/indices?v
```

---

## Elasticsearch 8 Compatibility: Security Enabled by Default[​](#elasticsearch-8-compatibility-security-enabled-by-default)

Elasticsearch 8.x introduces a significant breaking change from ES7: **TLS encryption and authentication are enabled by default**. If you are migrating from Elasticsearch 7 or configuring a fresh ES8 installation, connections that worked without credentials in ES7 will fail with `connection refused` or `SSL handshake failed` errors in ES8.

When you install Elasticsearch 8 for the first time, the installer outputs a randomly generated password for the `elastic` superuser and enrolls Kibana automatically with a token. You must save these credentials — the password is shown only once. To reset it:

```
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
```

You have two options for configuring your Logstash-to-Elasticsearch connection with ES8:

### Option A: Configure Logstash to Use HTTPS with ES8 Credentials (Recommended for Production)[​](#option-a-configure-logstash-to-use-https-with-es8-credentials-recommended-for-production)

Update the `output` block in your `/etc/logstash/conf.d/dreamfactory.conf` to include TLS settings and credentials. The CA certificate for self-signed ES8 installs is located at `/etc/elasticsearch/certs/http_ca.crt`:

```
output {  elasticsearch {    hosts => ["https://localhost:9200"]    user => "elastic"    password => "YOUR_ELASTICSEARCH_PASSWORD"    ssl_enabled => true    # Option 1: Provide the CA cert for full certificate verification (most secure)    ssl_certificate_authorities => ["/etc/elasticsearch/certs/http_ca.crt"]    # Option 2: Skip verification (acceptable for same-host setups, not for remote ES)    # ssl_verification_mode => "none"    index => "dreamfactory-logs-%{+YYYY.MM.dd}"  }}
```

Copy the CA certificate so Logstash can read it:

```
sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/logstash/sudo chown logstash:logstash /etc/logstash/http_ca.crtsudo chmod 640 /etc/logstash/http_ca.crt
```

Then update the `ssl_certificate_authorities` path in your Logstash config to `/etc/logstash/http_ca.crt`.

Verify the connection works from Logstash's perspective:

```
curl --cacert /etc/elasticsearch/certs/http_ca.crt \     -u elastic:YOUR_PASSWORD \     https://localhost:9200
```

You should see the ES cluster info JSON — if you do, the certificate and credentials are correct.

### Option B: Disable ES8 Security (Non-Production Only)[​](#option-b-disable-es8-security-non-production-only)

warningDisabling Elasticsearch security removes authentication and TLS protection. **Do not use this in production** or on any network-accessible host. This is only appropriate for local development or isolated test environments.

To disable ES8 security, edit `/etc/elasticsearch/elasticsearch.yml` and add:

```
xpack.security.enabled: falsexpack.security.http.ssl.enabled: falsexpack.security.transport.ssl.enabled: false
```

Then restart Elasticsearch:

```
sudo systemctl restart elasticsearch
```

With security disabled, your Logstash output block reverts to the simpler ES7-style config:

```
output {  elasticsearch {    hosts => ["http://localhost:9200"]    index => "dreamfactory-logs-%{+YYYY.MM.dd}"  }}
```

---

## Logstash Configuration Reference[​](#logstash-configuration-reference)

The following is a complete, annotated `logstash.conf` for DreamFactory's GELF log output. Save this as `/etc/logstash/conf.d/dreamfactory.conf`:

```
# =============================================================# DreamFactory → Logstash → Elasticsearch Pipeline# =============================================================input {  # DreamFactory sends logs via GELF (Graylog Extended Log Format) over UDP.  # Default port is 12201. Configure the same port in DreamFactory Admin >  # Services > Logstash connector settings.  gelf {    port => 12201    type => "dreamfactory"  }}filter {  if [type] == "dreamfactory" {    # DreamFactory log fields (top-level):    #   host        - hostname of the DreamFactory server    #   level       - log severity: DEBUG, INFO, WARNING, ERROR    #   short_message - brief description of the event    #   full_message  - full event payload (may include request/response)    #   timestamp   - event time (Unix epoch)    #    # DreamFactory event fields (nested under [event]):    #   [event][request][method]   - HTTP method: GET, POST, PUT, DELETE, PATCH    #   [event][request][uri]      - full request URI, e.g. /api/v2/mydb/_table/orders    #   [event][request][service]  - service name, e.g. "mydb", "system", "files"    #   [event][request][resource] - resource path within the service    #   [event][request][headers]  - request headers map (includes API key and session token)    #   [event][response][status]  - HTTP response status code    #   [event][response][content] - response body (if log level includes response)    # Rename [host] string to avoid conflict with ECS [host] object    if [host] {      mutate {        rename => { "host" => "source_hostname" }        add_field => { "[host][name]" => "%{source_hostname}" }      }    }    mutate {      add_field => { "log_source" => "dreamfactory" }      uppercase => [ "level" ]    }    # Extract user_id from the DreamFactory JWT session token so dashboards    # can show per-user activity without requiring a separate lookup.    ruby {      code => '        begin          event_data = event.get("event")          if event_data&.is_a?(Hash)            token = event_data.dig("request", "headers", "x-dreamfactory-session-token")            if token.is_a?(String) && token.include?(".")              parts = token.split(".")              if parts.length >= 2                require "base64"                require "json"                padding = (4 - parts[1].length % 4) % 4                payload = JSON.parse(Base64.urlsafe_decode64(parts[1] + "=" * padding))                event.set("[user][id]", payload["user_id"]) if payload["user_id"]              end            end          end        rescue => e          # silently skip malformed tokens        end      '    }  }}output {  elasticsearch {    # Use https:// for Elasticsearch 8.x (security enabled by default).    # Switch to http:// only if you have explicitly disabled xpack.security.    hosts => ["https://localhost:9200"]    user => "elastic"    password => "YOUR_ELASTICSEARCH_PASSWORD"    ssl_enabled => true    ssl_certificate_authorities => ["/etc/logstash/http_ca.crt"]    # Index pattern: creates one index per day for easier lifecycle management.    # Grafana datasource index pattern should be set to "dreamfactory-logs-*"    index => "dreamfactory-logs-%{+YYYY.MM.dd}"  }}
```

After saving the file, test the configuration before starting:

```
sudo /usr/share/logstash/bin/logstash \  --path.settings=/etc/logstash \  --config.test_and_exit
```

A successful test outputs `Configuration OK`. Then restart Logstash:

```
sudo systemctl restart logstashsudo journalctl -u logstash -f
```

---

## Grafana Dashboard Setup[​](#grafana-dashboard-setup)

Once Logstash is indexing DreamFactory events into Elasticsearch, connect Grafana to visualize the data.

### Connecting Grafana to the `dreamfactory-logs-*` Index[​](#connecting-grafana-to-the-dreamfactory-logs--index)

In Grafana, navigate to **Connections → Data Sources → Add data source → Elasticsearch**. Use these settings:

FieldValueURL`https://localhost:9200`AccessServer (default)Basic AuthEnabledUser`elastic`PasswordYour Elasticsearch passwordTLS/SSL modeSkip TLS verification (or provide CA cert)Index name`dreamfactory-logs-*`Time field`@timestamp`Version`8.0+`

Click **Save & Test** — you should see "Data source connected and index found."

### Dashboard Panels to Create[​](#dashboard-panels-to-create)

Build panels that answer the operational questions your team cares about most:

**1. Requests per Minute**

- Visualization: Time series

- Query: Count all documents with `type:dreamfactory`

- Bucket aggregation: Date histogram on `@timestamp`, interval `1m`

- This shows throughput trends and helps identify traffic spikes.

**2. Error Rate by HTTP Status Code**

- Visualization: Bar chart or time series

- Query: `event.response.status:>=400`

- Bucket aggregations: Date histogram → Terms on `event.response.status`

- Use a threshold alert at >5% error rate to catch service degradation early.

**3. Top API Consumers by API Key / User**

- Visualization: Table

- Query: `event.request.method:*`

- Bucket aggregation: Terms on `user.id` (size 20, order by count desc)

- Join with DreamFactory's `/api/v2/system/user` endpoint (via Grafana Infinity datasource) to resolve user IDs to email addresses.

**4. Slowest Endpoints**

- Visualization: Table sorted by average response time

- Requires DreamFactory to log response duration. If the `full_message` field includes timing data, parse it with a Logstash `grok` filter and store it as `response_ms`.

- Query: `event.request.uri:*`

- Aggregation: Terms on `event.request.uri.keyword` → Avg on `response_ms`

### Importing a Pre-Built Dashboard[​](#importing-a-pre-built-dashboard)

If DreamFactory provides a dashboard JSON export:

1. In Grafana, click **Dashboards → Import**

2. Upload the JSON file or paste its contents

3. Select your `DreamFactory Logs` Elasticsearch datasource when prompted

4. Click **Import**

The dashboard will appear under your configured folder. Set the default time range to **Last 24 hours** and auto-refresh to **1 minute**.

---

## Monitoring Data Consumption by App and API Key[​](#monitoring-data-consumption-by-app-and-api-key)

Beyond operational health, the Logstash connector is DreamFactory's data-consumption feed: every API event it ships answers "which identity called which service and endpoint, when, and with what response." This makes the ELK stack a consumption-metering layer for all DreamFactory-governed APIs, not just an error log.

### What Each Event Contains[​](#what-each-event-contains)

For each event key you enable on the Logstash service, you choose which context to ship:

- **`_event`** — the serialized API request and response: method, URI, service name, resource path, headers, and response status/content (depending on log context). Credentials, session tokens, and API keys are redacted by DreamFactory before the event leaves the platform.

- **`_platform`** — the session and user context for the call: which user and session were active when the request ran.

Because DreamFactory requires every request to present an API key bound to an application and role, the shipped events carry the identity attribution needed for per-consumer reporting — the same identity fabric described in [API Keys](/api-generation-and-connections/api-keys) and [Role-Based Access Control](/Security/role-based-access).

### Consumption Panels for Grafana[​](#consumption-panels-for-grafana)

Extending the four dashboard panels above, two more answer the consumption question directly:

**5. Top Consumers by App / API Key**

- Visualization: Table or bar chart

- Query: `event.request.method:*`

- Bucket aggregation: Terms on the app/API-key field available in your shipped event context (order by count desc)

- This is the per-application counterpart to the per-user panel: one row per consuming application, ranked by request volume.

**6. Request Volume per Service**

- Visualization: Time series (stacked)

- Query: `event.request.service:*`

- Bucket aggregations: Date histogram on `@timestamp` → Terms on `event.request.service.keyword`

- Shows which backend APIs (databases, files, custom services) each slice of traffic is consuming over time.

Together with the existing `user.id` panels, these give you consumption broken down by user, application, service, and endpoint — the raw material for showback reports on data access.

### Correlating with AI Usage and Cost[​](#correlating-with-ai-usage-and-cost)

GELF events tell you *who called what*; token counts and estimated AI cost live in DreamFactory's `ai_usage_log`, not in the GELF stream. To bring AI spend into the same ELK stack, pull the AI audit stream — `GET /_internal/ai/audit-stream` returns NDJSON events designed for Logstash's `http_poller` input (the example configuration lives in [AI Gateway Analytics](/AI/ai-gateway)).

For per-app, per-user, per-role, and per-model token and cost attribution — and for chargeback reporting built on it — see [AI Usage Monitoring & Cost Allocation](/AI/ai-usage-monitoring-and-cost-allocation).

---

## Troubleshooting ELK Integration[​](#troubleshooting-elk-integration)

### Connection Refused to Elasticsearch[​](#connection-refused-to-elasticsearch)

**Symptom**: Logstash logs show `Connection refused - connect(2) for "localhost" port 9200` or Grafana shows "No data."

**Causes and fixes**:

- Elasticsearch is not running: `sudo systemctl status elasticsearch`. Start it if stopped.

- Firewall is blocking port 9200: `sudo ufw allow 9200/tcp` (or adjust your iptables rules).

- ES8 is running on HTTPS but Logstash is configured with `http://`: change `hosts` to `["https://localhost:9200"]`.

### Index Not Created in Elasticsearch[​](#index-not-created-in-elasticsearch)

**Symptom**: Logstash starts without errors but no `dreamfactory-logs-*` index appears.

**Causes and fixes**:

- Logstash is not receiving GELF events from DreamFactory yet — generate some API calls and check: `sudo tcpdump -i lo -n udp port 12201`

- Logstash output is failing silently — check the Logstash log for ES errors: `sudo tail -100 /var/log/logstash/logstash-plain.log | grep -i error`

- Authentication failure: verify your ES password is correct with a direct curl: `curl -k -u elastic:YOUR_PASSWORD https://localhost:9200`

- SSL certificate error: ensure `ssl_certificate_authorities` points to the correct CA cert file and that the `logstash` user can read it.

### Grafana "No Data" on Panels[​](#grafana-no-data-on-panels)

**Symptom**: Grafana datasource test passes, but panels show "No data."

**Causes and fixes**:

- Index pattern mismatch: the Grafana datasource index name must be `dreamfactory-logs-*` (with the wildcard), matching the Logstash output `index` value.

- Time range issue: Grafana's default time range may not overlap with when logs were indexed — try setting the time picker to "Last 7 days."

- `@timestamp` field not mapped: Elasticsearch needs `@timestamp` as a date field. Verify: `curl -k -u elastic:YOUR_PASSWORD "https://localhost:9200/dreamfactory-logs-*/_mapping/field/@timestamp"`

- If the mapping is missing, delete the index and let Logstash recreate it with proper field mappings.

---

### Logstash Configuration[​](#logstash-configuration)

### Key Configuration Points[​](#key-configuration-points)

1. **GELF Input**: Listens on UDP port 12201 for Graylog Extended Log Format (GELF) messages from DreamFactory

2. **JWT Extraction**: Extracts `user_id` from DreamFactory session tokens and adds it as `user.id` field

3. **Field Mapping**: Renames `host` to `source_hostname` to avoid Elasticsearch mapping conflicts

4. **ECS Compatibility**: Uses Elastic Common Schema (ECS) standard for `host.name` field - this is vendor-neutral and works across all cloud providers (AWS, Azure, GCP, etc.) and on-premise installations

5. **Index Pattern**: Creates daily indices for better performance and management

### Verify Logstash is Receiving Logs[​](#verify-logstash-is-receiving-logs)

```
sudo netstat -tlnp | grep 12201
```

### Check Logstash Logs[​](#check-logstash-logs)

```
sudo tail -f /var/log/logstash/logstash-plain.log
```

---

### Grafana Configuration[​](#grafana-configuration)

### Step 1: Add Elasticsearch Data Source[​](#step-1-add-elasticsearch-data-source)

1. Go to **Configuration** → **Data Sources**

2. Click **Add data source**

3. Select **Elasticsearch**

4. Configure:

- **Name**: `DreamFactory Logs`

- **URL**: `https://localhost:9200`

- **Access**: Server (default)

- **Basic Auth**: Enabled

- **User**: `elastic`

- **Password**: Your Elasticsearch password

- **TLS/SSL Mode**: Skip TLS Verification

- **Index name**: `logstash-dreamfactory-*`

- **Time field**: `@timestamp`

- **Version**: `8.0+`

5. Click **Save & Test**

### Step 2: Add Infinity Data Source (for User Enrichment)[​](#step-2-add-infinity-data-source-for-user-enrichment)

**Prerequisites: Create a Least-Privilege API Key**

Before configuring the Infinity data source, create a DreamFactory API key with minimal permissions:

1. **Log into DreamFactory Admin Console**

2. **Navigate to Roles** → Create a new role (e.g., "Grafana Read-Only")

3. **Set Role Permissions** (see [Role-Based Access Control](/Security/role-based-access) for detailed information):

- **Service**: `system`

- **Component**: `user`

- **Access**: `GET` only (read-only)

- **Fields**: Allow access to: `id`, `first_name`, `last_name`, `email` (user/* in the Component section will give permission to view all user fields if needed)

4. **Create API Key**:

- Go to **Users** → Select or create a user

- Assign the "Grafana Read-Only" role to the user

- Generate an API key for this user

- **Note**: This API key will only have read access to user data, not full admin access

**Alternative: Use Existing Role with Limited Permissions**

- If you have a read-only role, you can use an existing API key with that role

- Ensure the role has GET access to `system/user` endpoint

**Configure Infinity Data Source**:

1. Install Infinity plugin (if not already installed):

```
sudo grafana-cli plugins install yesoreyeram-infinity-datasourcesudo systemctl restart grafana-server
```

2. Go to **Configuration** → **Data Sources**

3. Click **Add data source**

4. Select **Infinity**

5. Configure:

- **Name**: `DreamFactory API`

- **URL**: `http://localhost` (or your DreamFactory URL)

- **Authentication**: API Key

- Add header: `X-DreamFactory-API-Key` with your least-privilege API key (created in the Prerequisites section above)

- **Security Note**: Use the read-only API key, not an admin key

6. Click **Save & Test**

---

## DreamFactory Integration[​](#dreamfactory-integration)

### Step 0: Create Least-Privilege API Key for Grafana[​](#step-0-create-least-privilege-api-key-for-grafana)

The Infinity data source only needs read access to user data. Create a minimal-permission API key:

#### Option 1: Create a Custom Read-Only Role (Recommended)[​](#option-1-create-a-custom-read-only-role-recommended)

1. **Log into DreamFactory Admin Console**

2. **Navigate to**: **Roles** → **Create Role**

3. **Role Configuration**:

- **Name**: `grafana-readonly` (or similar)

- **Description**: `Read-only access for Grafana user enrichment`

4. **Set Service Access**:

- **Service**: `system`

- **Component**: `user`

- **Access**: `GET` only

- **Fields**: `id`, `first_name`, `last_name`, `email` (or leave blank for all fields)

5. **Save the role**

#### Option 2: Use Existing Read-Only Role[​](#option-2-use-existing-read-only-role)

If you already have a read-only role with `system/user` GET access, you can use that.

#### Create API Key with the Role[​](#create-api-key-with-the-role)

1. **Navigate to**: **Users** → Select a user (or create a dedicated user like `grafana-service`)

2. **Assign Role**: Add the `grafana-readonly` role to the user

3. **Generate API Key**:

- Go to the user's profile

- Generate or copy the API key

- **Store securely** - this key will be used in Grafana configuration

---

### Step 1: Enable Logstash Connector in DreamFactory[​](#step-1-enable-logstash-connector-in-dreamfactory)

1. Log into DreamFactory Admin Console

2. Navigate to **Services** → **System** → **Logstash**

3. Enable the Logstash connector

4. Configure:

- **Host**: `localhost` (or your Logstash server IP)

- **Port**: `12201`

- **Protocol**: `GELF (UDP)`

- **Log Context**: `Request All`
Service Event: (These are the Events necessary to integrate Logstash with our template Grafana Dashboard)

- Event: user.* | Log level: INFO | Message: System Activity

- Event: db.* | Log Level: INFO | Default Database Activity (db is a local SQLite database typically included with all DreamFactory installations)

- Event: files.* | Log Level: INFO| File Activity

- To add your own Database or other service connection start typing the name of your service and select the service followed by * or your specfic criteria

![Creating a Logstash service in the DreamFactory admin console](/assets/images/logstash-service-create-bb5cadb8f7b9d54436be47b076310df0.png)

1. Save configuration

### Step 2: Verify Logs are Being Sent[​](#step-2-verify-logs-are-being-sent)

Check if DreamFactory is sending logs: (Generate some API requests or logins to see them appear in the logs)

```
sudo tcpdump -i any -n udp port 12201
```

Or check Logstash logs:

```
sudo tail -f /var/log/logstash/logstash-plain.log | grep dreamfactory
```

### Step 3: Verify Logs in Elasticsearch[​](#step-3-verify-logs-in-elasticsearch)

```
curl -k -u elastic:YOUR_PASSWORD "https://localhost:9200/logstash-dreamfactory-*/_search?size=1&pretty"
```

---

## Dashboard Setup[​](#dashboard-setup)

### Creating the User Activity Dashboard[​](#creating-the-user-activity-dashboard)

### Panel 1: Total API Calls by User[​](#panel-1-total-api-calls-by-user)

**Purpose**: Shows the total number of API calls made by each user with email enrichment.

**Query Configuration**:

- **Data Source**: DreamFactory Logs (Elasticsearch)

- **Query Type**: Count

- **Query**: `user.id:* event.request.method:*`

- **Bucket Aggregations**:

- **Terms**: `user.id` (Size: 50, Order: Desc by _count)

- **Time Field**: `@timestamp`

**Second Query** (for email enrichment):

- **Data Source**: DreamFactory API (Infinity)

- **Type**: JSON

- **URL**: `/api/v2/system/user?fields=id,first_name,last_name,email&limit=100`

- **Parser**: Backend

- **Root Selector**: `resource`

**Transformations**:

1. **Join by field**: Join query A and B on `user.id` (Mode: Outer)

2. **Organize fields**:

- Exclude: `first_name B`, `last_name B`, `user.id`, `user.id 1`, `user.id 2`

- Rename: `Count` → `Total API Calls`, `email B` → `User Email`

3. **Filter data by values**:

- Field: `Total API Calls`

- Condition: `IS NOT NULL`

**Field Mappings**:

- Map empty/null email values to "Admin User"

The Dashboard should end up looking similar to this:

[![Grafana Dashboard Template](/img/Logstash/GrafanaDashboardTemplate.png)](#lightbox-grafana-dashboard)
[×![Grafana Dashboard Template](/img/Logstash/GrafanaDashboardTemplate.png)](#)
When there are no calls in the selected time frame for a user the panels should all be empty as the filters we applied should remove any row with no calls.

---

### Panel 2: APIs Accessed by Each User[​](#panel-2-apis-accessed-by-each-user)

**Purpose**: Shows which specific API endpoints each user accessed with HTTP method details.

**Query Configuration**:

- **Data Source**: DreamFactory Logs (Elasticsearch)

- **Query Type**: Count

- **Query**: `user.id:* event.request.method:* event.request.uri:*`

- **Bucket Aggregations**:

1. **Terms**: `user.id` (Size: 20, Order: Desc by _count)

2. **Terms**: `event.request.method.keyword` (Size: 10, Order: Desc by _count)

3. **Terms**: `event.request.uri.keyword` (Size: 25, Order: Desc by _count)

- **Time Field**: `@timestamp`

**Second Query** (for email enrichment):

- Same as Panel 1

**Transformations**:

1. **Join by field**: Join on `user.id` (Mode: Outer Tabular)

2. **Organize fields**:

- Exclude: `first_name B`, `last_name B`, `user.id`, `user.id 1`, `user.id 2`

- Rename:

- `email B` → `User Email`

- `event.request.method.keyword` → `HTTP Method`

- `event.request.uri.keyword` → `API Endpoint`

3. **Filter data by values**:

- Field: `Count` (or your count column)

- Condition: `IS NOT NULL`

---

### Panel 3: User Access to Specific Services[​](#panel-3-user-access-to-specific-services)

**Purpose**: Shows which DreamFactory services each user accessed.

**Query Configuration**:

- **Data Source**: DreamFactory Logs (Elasticsearch)

- **Query Type**: Count

- **Query**: `user.id:* event.request.method:* event.request.service:*`

- **Bucket Aggregations**:

1. **Terms**: `user.id` (Size: 20, Order: Desc by _count)

2. **Terms**: `event.request.service.keyword` (Size: 15, Order: Desc by _count)

3. **Terms**: `event.request.method.keyword` (Size: 10, Order: Desc by _count)

- **Time Field**: `@timestamp`

**Second Query** (for email enrichment):

- Same as Panel 1

**Transformations**:

1. **Join by field**: Join on `user.id` (Mode: Outer Tabular)

2. **Organize fields**:

- Exclude: `first_name B`, `last_name B`, `user.id`, `user.id 1`, `user.id 2`

- Rename:

- `email B` → `User Email`

- `event.request.method.keyword` → `HTTP Method`

- `event.request.service.keyword` → `Service`

3. **Filter data by values**:

- Field: `Count`

- Condition: `IS NOT NULL`

---

### Dashboard Settings[​](#dashboard-settings)

**Time Range**: Set to "Last 24 hours" (adjustable)
**Refresh Interval**: 1 minute (recommended)
**Tags**: `dreamfactory`, `users`, `api`, `activity`

---

## Dashboard Function and Use[​](#dashboard-function-and-use)

### Overview[​](#overview)

The dashboard provides comprehensive visibility into DreamFactory API usage by individual users, helping administrators:

1. **Monitor User Activity**: See which users are actively using the API

2. **Identify API Usage Patterns**: Understand which endpoints and services are most accessed

3. **Troubleshoot Issues**: Track user-specific API calls for debugging

4. **Security Auditing**: Monitor API access patterns for security analysis

### Key Features[​](#key-features)

### User Identification[​](#user-identification)

- **User ID Extraction**: Automatically extracts user IDs from JWT session tokens

- **Email Enrichment**: Joins with DreamFactory user database to show email addresses

- **Admin User Handling**: Maps null/empty emails to "Admin User" for system-level operations

### Data Filtering[​](#data-filtering)

- **Method-Based Filtering**: Only shows users with actual HTTP method calls (GET, POST, etc.)

- **Complete Data Only**: Filters out rows with missing data using transformation filters

- **Real-Time Updates**: Refreshes every minute to show current activity

### Aggregation Levels[​](#aggregation-levels)

1. **User-Level**: Total calls per user

2. **Endpoint-Level**: Specific API endpoints accessed

3. **Service-Level**: DreamFactory services accessed (db, system, files, etc.)

### Use Cases[​](#use-cases)

1. **Activity Monitoring**

- Identify most active users

- Track API usage trends

- Monitor system load by user

2. **Troubleshooting**

- Find which user made a specific API call

- Track user-specific errors

- Debug authentication issues

3. **Security Auditing**

- Monitor unusual access patterns

- Track service access by user

- Identify potential security issues

4. **Capacity Planning**

- Understand API usage patterns

- Plan for user growth

- Optimize service performance

### Interpreting the Data[​](#interpreting-the-data)

- **Total API Calls**: Sum of all HTTP requests made by a user

- **HTTP Method**: Type of request (GET, POST, PUT, DELETE, etc.)

- **API Endpoint**: Specific DreamFactory API path accessed

- **Service**: DreamFactory service category (db, system, files, etc.)

- **User Email**: Email address from DreamFactory user database (or "Admin User" for system operations)

---

## Maintenance[​](#maintenance)

### Regular Tasks[​](#regular-tasks)

1. **Monitor Disk Space**: Elasticsearch indices can grow large

```
curl -k -u elastic:YOUR_PASSWORD "https://localhost:9200/_cat/indices?v"
```

2. **Index Management**: Consider setting up index lifecycle management

```
# Delete old indices (example: older than 30 days)curl -k -u elastic:YOUR_PASSWORD -X DELETE "https://localhost:9200/logstash-dreamfactory-2024.12.*"
```

3. **Log Rotation**: Monitor Logstash logs

```
sudo logrotate -d /etc/logrotate.d/logstash
```

4. **Performance Monitoring**: Check Elasticsearch cluster health

```
curl -k -u elastic:YOUR_PASSWORD "https://localhost:9200/_cluster/health?pretty"
```

---

If you have any issues or need help with a specifc Grafana dashboard panel please reach out to DreamFactory support!