AWS CloudWatch Logs provides a comprehensive solution for monitoring, storing, and accessing log files generated by various sources such as Amazon EC2 instances, AWS CloudTrail, and Route 53.

It offers a centralized and scalable service that allows you to consolidate logs from multiple systems, applications, and AWS services into a single location.

With CloudWatch Logs, you can easily view, search, and filter logs based on specific error codes or patterns, ensuring efficient log management and analysis.

One of the key advantages of CloudWatch Logs is its ability to present logs from diverse sources as a unified and chronological stream of events. Regardless of the original source, you can access all your logs in a consistent manner, ordered by time. This unified view simplifies log analysis and troubleshooting across your entire infrastructure.

That being said, CloudWatch Logs offers advanced features to enhance log querying and analysis. It provides a powerful query language that enables you to extract valuable insights from your logs.

Additionally, CloudWatch Logs supports auditing and data masking capabilities to ensure the security and privacy of sensitive information contained within logs. Furthermore, you can generate metrics from logs using filters or an embedded log format, enabling you to derive meaningful operational insights from your log data.

Exporting AWS CloudWatch logs to LOGIQ

There are two methods to forward CloudWatch logs to LOGIQ:

1. Logiq CloudWatch Exporter Lambda function

2. Running Logstash on a VM (or Docker)

In this guide, we will provide a step-by-step overview of how to export AWS CloudWatch logs to LOGIQ using these two methods.

Method 1: Logiq CloudWatch Exporter Lambda Function

Step 1: Creating a Lambda Function

LOGIQ.AI provides CloudFormation templates to create the LOGIQ CloudWatch exporter Lambda function.

You can Choose the appropriate CloudFormation template based on the type of logs you want to export:

Exporting Lambda Function logs

Use the following CloudFormation template to export AWS Lambda function logs to LOGIQ:


Exporting CloudTrail Logs

Use the following CloudFormation template to export CloudTrail logs to LOGIQ:


Exporting AWS VPC Flowlogs

Use the following CloudFormation template to export Flowlogs logs to LOGIQ:


Exporting Cloudwatch logs from other services

Use the following CloudFormation template to export cloudwatch logs:


This CloudFormation stack creates a Lambda function and its necessary permissions. However, you’ll need to configure the following attributes before using this template:

Application name – a readable name for LOGIQ to partition logs.
Cluster ID – a readable name for LOGIQ to partition logs.
Namespace – a readable name for LOGIQ to partition logs.
IP address or hostname of the LOGIQ server. (Without http or https)
JWT token to securely ingest logs. Refer here to generate ingest token.

Step 2: Configuring the CloudWatch Trigger

  1. After the CloudFormation stack is created, go to the AWS Lambda function (logiq-cloudwatch-exporter) in the AWS Management Console.
  2. Add a trigger to the Lambda function by clicking on the “Add trigger” button. AWS CloudWatch Logs
  3. Select “CloudWatch” as the trigger type.
  4. Choose the desired CloudWatch Logs Log Group to forward to LOGIQ.
  5. Save the trigger configuration.
Thus, with the CloudWatch trigger configured, any new logs coming to the specified CloudWatch Log Group will be streamed to the LOGIQ cluster.

Method 2: Running Logstash on a VM (or Docker)

Step 1: Create the Logstash VM (or Docker)

Install Logstash on an Ubuntu virtual machine by following these commands:
					wget -qO - | sudo apt-key add -
echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update
sudo apt-get install logstash


Step 2: Install Logstash Input Plugin for CloudWatch

Change the directory to Logstash’s installation directory:
					cd /usr/share/logstash
Next, install the Logstash Input Plugin for CloudWatch:
					sudo -u root sudo -u logstash bin/logstash-plugin install logstash-input-cloudwatch

Step 3: Configure Logstash

Logstash comes with no default configuration. Create a new file named /etc/logstash/conf.d/logstash.conf and enter the following configuration, modifying the values as needed:
					input {
  cloudwatch_logs {
    access_key_id => "<Acess-key>"
    secret_access_key => "<secret-access-key>"
    region => "<region>"
    "log_group" => ["<Cloud-watch-log-group"]
    "log_group_prefix" => true
    codec => plain
    start_position => end
    interval => 30
filter {
        ruby {
           path => "/home/<custom-path>/flattenJSON.rb"
           script_params => { "field" => "cloudwatch_logs" }
	mutate {
           gsub => ["cloudwatch_logs.log_group","\/","-"]
           gsub => ["cloudwatch_logs.log_group","^-",""]
	   add_field => { "namespace" => "<custom-namespace>" }
	   add_field => { "cluster_id" => "<custom-cluster-id>" }
	   add_field => { "app_name" => "%{[cloudwatch_logs.log_group]}" }
	   add_field => { "proc_id" => "%{[cloudwatch_logs.log_stream]}" }

output {
 http {
       url => "http://<logiq-endpoint>/v1/json_batch"
       headers => { "Authorization" => "Bearer <SECURE_INGEST_TOKEN>" }
       http_method => "post"
       format => "json_batch"
       content_type => "json_batch"
       pool_max => 2000
       pool_max_per_route => 100
       socket_timeout => 300


You can obtain an ingest token from the LOGIQ UI as described here.

Moreover, you can customize the namespace and cluster_id in the Logstash configuration based on your needs.


In conclusion, forwarding CloudWatch logs to LOGIQ can be accomplished through two methods: using the Logiq CloudWatch Exporter Lambda function or running Logstash on a VM or Docker.

The first method involves creating an AWS Lambda function and configuring a CloudWatch trigger to stream logs to LOGIQ.

The second method requires installing Logstash on a VM or Docker, configuring it with the necessary settings, and leveraging the Logstash Input Plugin for CloudWatch. Both methods enable seamless forwarding of AWS CloudWatch logs to LOGIQ, allowing you to efficiently manage and explore your logs within the LOGIQ interface.

Choose the method that best suits your needs and follow the step-by-step instructions provided to get started with forwarding your CloudWatch logs to LOGIQ.