Web protection size estimates

The table below provides some general rules to help estimate the size of your web protection Log Database, based on whether you are recording visits or hits. and URL hostnames or full URLs. By default, visits and URL hostnames are recorded.

  URL hostnames Full URL logging
Visits 3 MB per user per month 4.5 MB per user per month
Hits 8 MB per user per month 12 MB per use per month

The size increase that accompanies full URL logging can vary widely based on the type of URL being logged. For example, the following types of traffic tend to generate very long URLs:

  • Research database search and results URLs
  • Search strings
  • Social networking sites
  • Online apps and games

Consider the example of a 7500-user organization that wants to store data for three months, using monthly database partition rollover:

  • Visits are enabled
  • Full URL logging is enabled
  • The Internet access policy is permissive
  • A high percentage of web traffic includes longer than average URLs (social networking sites and online games)

Based on this information, estimate 6 MB of data stored per user per month.

Because rollover is monthly, it is necessary to allocate enough space to hold up to 4 months of data. (This ensures that it is always possible to report on 3 months of data, even during the first days or weeks of the most recent month.)

To start, therefore, 180 GB would be needed for the database logging partitions (not including the catalog database and AMT partition).

6 MB * 7500 users * 4 months = 180 GB

In addition, calculate space for the following:

  • Database transaction logs, whose size depends on the recovery model you choose.
    • For full recovery (not recommended), allow for the same size as the data itself (180 GB in our example).
    • For simple recovery, the log volume depends on the amount of database activity. Allow roughly 30 percent of the size of an active database partition.

    In our example, the total size of 4 full partitions is 180 GB, so each partition is 45 GB in size. So for simple recovery:

    0.3 * 45 GB = 13.5 GB

    Note that this space is used only while data is being logged to the database. Inactive partitions, for example, have minimal transaction log activity. Likewise, during time periods in which little or no Internet activity is being recorded, the transaction log is very small.

  • Temporary (tempdb) storage is used heavily during report generation and when the maintenance job is reindexing the database. The amount of tempdb space required depends heavily on the types of reports being generated.

    In most environments, it is sufficient to allow twice the size of a partition:

    45 * 2 = 90 GB

  • Temporary database (tempdb) transaction logs take up about 20 percent of the tempdb storage size. In our example:

    0.2 * 90 GB = 18 GB

The total space required for the web protection Log Database in our example, then, is:

180 GB + 13.5 GB + 90 GB + 18 GB = 301.5 GB

Tip: As a best practice, record visits rather than hits to reduce significantly the amount of storage space required.

See Factors that affect reporting database size, for an explanation of the Forcepoint Web Security visits algorithm, other data reduction options, and the full URL logging setting.