Logiq is now patented! This solidifies our position as a leader in the field of modern data engineering.   Know More

Masking sensitive information in logs with LogFlow

Masking sensitive information in logs with LogFlow

As a business, you’re bound to have large numbers of interactions with customers, prospects, employees, consultants, partners, and third-party vendors. These interactions generate huge amounts of data and often contain information that are sensitive in nature and possibly personally identifiable.

Sensitive data that could make their way into your logs include user location, names, addresses, payment information, social security numbers, health information, etc. It then becomes your business’ duty to protect such data from theft, misuse, and simply falling into the wrong hands. Moreover, data privacy and security regulations such as GDPR, HIPAA, and PCI-DSS require you to anonymise, store, manage access, and dispose of sensitive information securely.

The machine data your interconnected systems generate can also contain sensitive and potentially-exploitable data such as API keys, network traffic, and IP addresses. By controlling your observability and machine data pipelines with LogFlow, you can prevent sensitive information from being redirected to storage or any target system; right from within your log stream. LogFlow lets you create custom rules to identify patterns or strings of sensitive information and either rewrite, mask, or drop that piece of information entirely from your log stream.

In this article, we’ll focus on the rewrite rule and take you through an example of how you can identify and rewrite strings that contain sensitive information from your logs.

Using rewrite rules to mask sensitive data

In this scenario, LogFlow is ingesting logs from a Tomcat instance. These logs contain email addresses that should normally not be exposed or redirected to any target system. We’ll create a rewrite rule that matches email address within logs and rewrites them with the string we specify.

We begin by navigating to the Logs page on the LogFlow UI and selecting the Tomcat application that contains sensitive data. A quick search of the logs for the email address user@logiq.ai highlights all the log lines that contain this email address. To create a rewrite rule, we do the following.

  1. Next to a log line, select Options > Create rule
The logs containing email addresses
  1. On the Create Rule page, select Rewrite.
  2. Next, Name your rule and select a severity Level.
  3. Either create a new rule Group or select an existing rule group to assign this rule to.
  4. Enter a Description.
  5. In the Match field, enter a regular expression that matches email address patterns.
  6. In the Rewrite field, enter the string you’d like to replace the email addresses in the logs with.
  7. Finally, click Create Rule.
Configuring the Rewrite rule

LogFlow creates this new Rewrite rule in the application and namespace you created it in. This rule is applied the moment it gets created and starts replacing the email addresses in your incoming logs with the string you applied.

The following video demonstrates how you can create a rewrite rule to mask sensitive data in your log streams.

Ajit Chelat

Ajit Chelat

The LOGIQ blog

Let’s keep this a friendly and inclusive space: A few ground rules: be respectful, stay on topic, and no spam, please.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

More insights. More affordable. Less hassle.

Follow Us on LinkedIn
and Twitter!

Before you go, make sure you don’t miss out on our latest updates and insights. Follow us on LinkedIn to stay up-to-date on industry trends, company news, and valuable insights.

Click the “Follow” button below to join our community and stay ahead of the curve. Thank you for visiting our site, and we hope to connect with you soon!