Establishing a data retention policy in the Engine

Contents

Establishing a data retention policy in the Engine

Overview

The Engine stores the real-time data that it receives from the the Collectors in the form of events. Events are very numerous and they usually take most of the memory dedicated to the Engine. The types of events that occupy most of the space in memory are executions, connections, and web requests. When two or more of these events are very similar to each other and they occur in sequence, the Engine may consider that they are actually the same event. In that case, the Engine combines the data of the events and stores only one event in its database. We say then that the Engine aggregates the information of several events into one; thus saving memory space and resulting in a larger history for the Engine.

When you have the web monitoring feature fully enabled, you usually collect a huge number of web domains. In the same spirit of event aggregation, when two or more domain names share their highest level domains, the Engine may group them into one generalized domain by obeying specific rules. This process is known as domain compaction or domain compression and it replaces one or more of the lower level domains in the domain name by the wildcard character *. For instance, the Engine might compact the domains one.example.com and two.example.com into *.example.com. Note however that those domains declared as internal domains are never compacted, and those included in the definition of a web-based service are compacted only up to the point that they match the pattern specified in the definition of the service, as these domains are considered of special interest to you.

Learn here how to set the maximum number of events and establish the policies for both the aggregation of events and the compaction of domains in the Engine.

Setting the maximum number of events

To set the maximum number of events that the Engine can store:

  1. Log in to the Web Console as admin.
  2. Click the Engine tab at the top of the window.
  3. Select the General section from the left-hand side menu.
    EngineParameters.png
  4. Under Parameters, choose a number from the Max stored events drop-down list.
  5. Click Save. Note that the Engine is restarted after saving the changes.

Setting the aggregation policy for events

Choose among four strategies of aggregation for an optimal trade-off between detailed event information and history length. The more aggressive the policy, the fewer individual (non aggregated) events are visible from the Finder.

  1. Log in to the Web Console as admin.
  2. Click the Engine tab at the top of the window.
  3. Select the General section from the left-hand side menu.
  4. Under Parameters, choose one of the following aggregation policies from the list labeled Aggregation policy:
    • very low - normal history, for the traditional minimal aggregation.
    • low - up to 10% more history, for increasing the history 10% approx. while keeping most of the individual events.
    • medium - up to 80% more history (recommended), for a more aggressive aggregation policy to increase history in the Engine up to 80%. This is the recommended setting.
    • high - up to 100% more history, for the most aggressive aggregation policy to practically double the history traditionally available in the Engine.
  5. Click Save. Note that the Engine is restarted after saving the changes.

Setting the compaction level for domains

  1. Log in to the Web Console as admin.
  2. Click the Engine tab at the top of the window.
  3. Select the General section from the left-hand side menu.
  4. Under Parameters, choose one of the following domain compression policies from the list labeled Domain compression:
    • medium (recommended), the default compression policy for domains with more than five levels or with repetitive (or randomly generated) subdomains. This is the recommended setting.
    • high, to apply a compression method to all the stored domain names according to a public list of domain suffixes.
  5. Click Save. Note that the Engine is restarted after saving the changes.

For a detailed explanation of compaction policies, see the section about compaction in the definition of domain.

Related concepts