

APIM 3.x Based Analytics

WSO2 API Microgateway provides the capability to publish events to a configured analytics server, in order to generate analytics. The Microgateway is also capable of generating a usage report to gain insights of the analytics generated. This page describes the feature and explains how the feature could be used to generate useful analytics in order to gain important insights into the APIs deployed on the Microgateway.

Overview

WSO2 API Microgateway supports publishing events to an analytics server in one of the two following methods.

1. Periodically publishing events (file-based)
2. Real-time publishing events (gRPC-based)

The Following topics introduce and explain the difference between the two methods, as well as provide instructions on how to use them in the Microgateway to generate analytics.

Periodically Publishing Events (file-based)

API Microgateway is capable of publishing events periodically to an analytics server.  The period in which the data is published can be configured.  A file-based approach is used in this method.

Periodical data publishing has the following advantages:

1. A persistent connection to the analytics server is not a mandatory requirement (lossless publishing)
2. No backpressure to the Gateway if the analytics server is loaded, allowing the gateway to operate smoothly.

Architecture

The current architecture depicting the connection between the API Microgateway and the Analytics server is shown below.

How periodical publishing works

The Microgateway receives the requests from the API. To process the requests, there are two main tasks that run in the Microgateway.

When an API is invoked through the Microgateway, the subsequent events related to the invocations are populated and written in a file. To avoid the files attaining a large size,  the following tasks are executed at particular time intervals.

File rotating task

• This task creates large files by rotating them at particular time intervals. The size of the files depends on the TPS (Transactions Per Second) of the environment in which the Microgateway is running. Once the file is rotated, it is compressed into a zipped file. The rotatingPeriod can be configured as described below) .

File uploading task

• This task uploads the zipped file created by the previous task, to a microservice running on the Analytics Server. This zip file is then persisted in the WSO2AM_STATS_DB database in the Analytics server node. A task running in the Analytics server processes the persisted data and sends it to the event stream. The time interval in which the zipped files are uploaded can be configured by setting the uploadingTimeSpanInMillis parameter.

After successfully completing the upload and persistence is achieved, the corresponding files are deleted from the Microgateway filesystem.

Real-Time Publishing Events (gRPC based)

The Micorgateway is capable of publishing events in real-time to an analytics server using gRPC.

Real-Time data publishing has the following advantages:

• Transferring analytics data over a secure HTTPS connection.
• Efficient, fast and reliable end to end communication.
• Lower communication payload for analytics data transferring.
• Server-fail detection and failure recovery mechanisms.
• Real-Time analytics data viewing capability.

How real-time publishing works

A streaming service is established between the Analytics server and Microgateway when Microgateway is initiated. When an API is invoked through Microgateway, the particular events related to the request are streamed to the gRPC server stub in the Analytics server without persisting, unlike in periodical publishing.

Note

Since there is no persistent data, if the gRPC connection is broken between Microgateway and Analytics server the event data will be lost. Microgateway will attempt to re-establish connection with the timeout that is provided in the configuration under reconnectTimeInMillies.

Configuring Analytics for the Microgateway

The following sections describe how to configure the WSO2 API-M Analytics Server for Microgateway.

Using periodical data publishing (file-based)

Step 1 - Configuring the WSO2 API-M Analytics Server

Note

Before you begin... Configure API Manager Analytics using the Quick setup or Standard Setup . For instructions, see Configuring APIM Analytics .

Create the AM_USAGE_UPLOADED_FILES table in the APIM_ANALYTICS_DB. A sample MySQL script to create the table is given below (This step is only required if you followed the Standard setup when configuring API Manager Analytics).

CREATE TABLE IF NOT EXISTS AM_USAGE_UPLOADED_FILES (
    FILE_NAME varchar(255) NOT NULL,
    FILE_TIMESTAMP TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FILE_PROCESSED tinyint(1) DEFAULT 0,
    FILE_CONTENT MEDIUMBLOB DEFAULT NULL,
    PRIMARY KEY (FILE_NAME, FILE_TIMESTAMP)
);

Warning

Make sure that you correctly configured the APIM_ANALYTICS_DB between WSO2 API Manager and WSO2 API-M Analytics.

Info

Java system properties The following system properties can be passed to the server if the default values need to be changed.

Property	Default value	Description
-Dusage.publishing.frequency=20000	300000 (5 minutes)	The frequency in which the database is read by the event source. This parameter is defined in milliseconds.
-Dusage.cleanup.frequency=36000	1800000 (30 minutes)	The frequency in which the cleanup task is run, to remove the processed files. This parameter is defined in milliseconds.
-Dfile.retention.days=5	N/A	The allowed duration for files to remain in the database. This parameter is defined in days
-Dusage.publishing.thread.count=3	3	The number of worker threads used to read data from the database. Each thread will read from a single file.

Example

    -Dusage.cleanup.frequency=48000  -Dusage.publishing.frequency=20000 -Dfile.retention.days=3

Step 2 - Configuring the Microgateway for Analytics

To do the configurations for Microgateway analytics, open the <MICROGW_HOME>/conf/micro-gw.conf file. The sample below shows the analytics-related configurations.

[analytics.fileUpload]
    enable=true
    uploadingTimeSpanInMillis=600000
    uploadingEndpoint="https://localhost:9444/analytics/v1.0/usage/upload-file"
    rotatingPeriod=60000
    taskUploadFiles=true
    username="admin"
    password="admin"

The configurations are described in the table below.

Property	Description
enable	Set this to true to enable Microgateway analytics. When enabled, the Microgateway starts persisting files with the analytics data in the <MICROGW_HOME>/api-usage-data folder.
uploadingTimeSpanInMillis	The time interval in which the uploading task runs.
uploadingEndpoint	The endpoint URL of the web application, to which the file has to be uploaded. This web app is deployed in the Analytics server, to retrieve files containing analytics data.
rotatingPeriod	The time interval, after which the file is rotated and compressed. This depends on the TPS (Transactions Per Second) capacity of the environment. Tip Best Practice To avoid creating large files, we recommend setting a low rotatingPeriod if your environment has a higher TPS.
taskUploadFiles	Set this to true to enable the file upload task. Note If this property is disabled, the analytics files are not uploaded to the analytics server, although the files are persisted in the Microgateway system. Tip Best Practice In a distributed setup, the analytics data is uploaded to a shared location from multiple Microgateway nodes. To avoid multiple nodes competing to upload the same file, WSO2 recommends enabling the taskUploadFiles property in only one node. Make sure that you disable this in the rest of the Microgateway nodes. You can also opt to have your own design as a workaround.
username	Username used in the Analytics server.
password	Password for the corresponding username.

Note

When adding the analytics configurations to <MICROGW_HOME>/conf/micro-gw.conf file for file-based analytics publishing, make sure to add the version of the API-M Analytics you are using from the following config.

[analytics]
  # The configured API Manager analytics stream version
  streamVersion = "3.2.0"

Using real-time data publishing (gRPC-based)

Step 1 - Configuring the WSO2 API-M Analytics Server

1. Open the <APIM-ANALYTICS-HOME>/conf/worker/deployment.yaml file

2. Locate the siddhi → refs → ref → name → grpcSource parameter section. Change the IP and port of the receiver.url to point to the Microgateway.

   siddhi:
     refs:
       - ref:
           name: 'grpcSource'
           type: 'grpc'
           properties:
             receiver.url : grpc://localhost:9806/org.wso2.analytics.mgw.grpc.service.AnalyticsSendService/sendAnalytics

3. The SSL configurations for the connection can be defined under siddhi → extensions → extension → grpc as follows.

   -
     extension:
       name: 'grpc'
       namespace: 'source'
       properties:
         keyStoreFile : ${sys:carbon.home}/resources/security/wso2carbon.jks
         keyStorePassword : wso2carbon
         keyStoreAlgorithm : SunX509
         trustStoreFile : ${sys:carbon.home}/resources/security/client-truststore.jks
         trustStorePassword : wso2carbon
         trustStoreAlgorithm : SunX509

Step 2 - Configuring the Microgateway for Analytics

1. Open the <MICRO-GW_HOME>/conf/default-micro-gw.conf.template. Locate analytics.gRPCAnalytics located under the analytics section.
2. Copy the analytics.gRPCAnalytics and paste it under the analytics section in the <MICRO-GW_HOME>/conf/micro-gw.conf file.

3. Configure the following parameters in the copied section.

     [analytics.gRPCAnalytics]
       enable = true
       # APIM Analytics endpoint configured to accept gRPC analytics
       endpointURL = "https://localhost:9806"
       # Time interval in milliseconds for gRPC connection recovery task
       reconnectTimeInMillies = 6000

Parameter	Description	Mandatory	Acceptable Values
enable	Enables or disables gRPC based real-time data publishing in the API Microgateway		True - Enables gRPC based real-time analytics False - Disables gRPC based real-time analytics
endpointURL	APIM Analytics endpoint configured to accept gRPC analytics		A valid URL
reconnectTimeInMillies	Defines the time interval for the gRPC reconnect task. It will try to connect to the gRPC supported analytics server according to the time interval defined in milliseconds.		Integers between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807

Please note that when configuring Microgateway for analytics you need to use only one of the configurations explained in using periodical data publishing (file-based) and using real-time data publishing (gRPC-based).

Generating a Microgateway usage report

A report containing the number of requests served by the Microgateway can be generated using the following steps.

Note

Before you begin... Configure API Manager Analytics using the Quick setup or Standard Setup . For instructions, see Configuring APIM Analytics .

1. Start the analytics dashboard in the analytics setup.

2. Login to the analytics dashboard (https://localhost:9643/analytics-dashboard/) and navigate to the Reports dashboard.

   

3. Select a year and a month to generate the report for the respective month.

   

4. Select 'Download' button to get the generated pdf with the usage information.

   

Top