As a business, you’re bound to have large numbers of interactions with customers, prospects, employees, consultants, partners, and third-party vendors. These interactions generate huge amounts of data and often contain information that are sensitive in nature and possibly personally identifiable.

Sensitive data that could make their way into your logs include user location, names, addresses, payment information, social security numbers, health information, etc. It then becomes your business’ duty to protect such data from theft, misuse, and simply falling into the wrong hands. Moreover, data privacy and security regulations such as GDPR, HIPAA, and PCI-DSS require you to anonymise, store, manage access, and dispose of sensitive information securely.

The machine data your interconnected systems generate can also contain sensitive and potentially-exploitable data such as API keys, network traffic, and IP addresses. By controlling your observability and machine data pipelines with LogFlow, you can prevent sensitive information from being redirected to storage or any target system; right from within your log stream. LogFlow lets you create custom rules to identify patterns or strings of sensitive information and either rewrite, mask, or drop that piece of information entirely from your log stream.

In this article, we’ll focus on the rewrite rule and take you through an example of how you can identify and rewrite strings that contain sensitive information from your logs.

Using rewrite rules to mask sensitive data

In this scenario, LogFlow is ingesting logs from a Tomcat instance. These logs contain email addresses that should normally not be exposed or redirected to any target system. We’ll create a rewrite rule that matches email address within logs and rewrites them with the string we specify.

We begin by navigating to the Logs page on the LogFlow UI and selecting the Tomcat application that contains sensitive data. A quick search of the logs for the email address user@logiq.ai highlights all the log lines that contain this email address. To create a rewrite rule, we do the following.

1. Next to a log line, select Options > Create rule
   

1. On the Create Rule page, select Rewrite.
2. Next, Name your rule and select a severity Level.
3. Either create a new rule Group or select an existing rule group to assign this rule to.
4. Enter a Description.
5. In the Match field, enter a regular expression that matches email address patterns.
6. In the Rewrite field, enter the string you’d like to replace the email addresses in the logs with.
7. Finally, click Create Rule.

LogFlow creates this new Rewrite rule in the application and namespace you created it in. This rule is applied the moment it gets created and starts replacing the email addresses in your incoming logs with the string you applied.

The following video demonstrates how you can create a rewrite rule to mask sensitive data in your log streams.