Upgrading Content Gateway to v8.5.x

Applies to:
Forcepoint Web Security, v8.5.x

This section provides upgrade instructions for software-based Content Gateway installations.

Note: When upgrading Content Gateway on an appliance, see the Forcepoint appliance documentation appropriate for your configuration.

Perform an upgrade by running the Content Gateway installer on a machine with a previous version of Content Gateway installed. The installer detects the presence of Content Gateway and upgrades it to the current version.

Versions supported for direct upgrade to v8.5.x

Direct upgrade is supported from v8.1.x, v8.2.x, v8.3.x, and v8.4.x to Content Gateway v8.5, from v8.2, v8.3, v8.4 and v8.5 to Content Gateway v8.5.3, or from v8.4, v8.5, and v8.5.3 to Content Gateway v8.5.4. Upgrades from earlier versions require intermediate upgrades:

v7.0/7.1 > v7.5 > v7.6 > v7.7 > v7.8.4 > v8.4.x>v8.5.x

Follow the upgrade procedures for each intermediate version. Read the Content Gateway Installation Guide and its upgrade supplement for each version.

To perform an intermediate upgrade, download the installer package for that version from the Downloads site at forcepoint.com.

System requirements

Before upgrading Content Gateway, make sure the host machine meets the system requirement outlined in Content Gateway section in System requirements for this version, including hardware specifications, operating system, and browser.

Upgrading distributed components

Content Gateway is the web proxy component of Forcepoint Web Security. Several Forcepoint Web Security components must be upgraded prior to upgrading Content Gateway. Distributed components must be upgraded in a particular order. See Upgrading Web Protection Solutions.

Preparing to upgrade

Before upgrading Content Gateway, be aware of the following.

Most SSL configuration settings are saved and applied to the upgraded Content Gateway, except for dynamic certificates. Note that:
- The Incident list is retained. Before upgrading, consider performing maintenance on the Incident list; remove unwanted entries.
- SSLv2 is not enabled by default. If it is enabled prior to upgrade, the setting is retained.
For user authentication, there is one credential cache for both explicit and transparent proxy mode, and one Global Authentication Options page for setting the caching method and Time-To-Live.
During upgrade, the Cache TTL value is retained from the Transparent Proxy Authentication tab unless the value on the Global Authentication Options tab is not the default. In this case, the customized value is used.
If you use Integrated Windows Authentication (IWA), be aware that IWA domain joins should be preserved through the upgrade process. However, in case the joins are dropped, make a record of the settings before starting the upgrade. Log on to the Content Gateway manager and record the IWA settings, including the names of domains to which IWA is joined. Keep this record where it is easily retrieved after the upgrade.

If you have software instances of Content Gateway, make sure the host system meets the following hardware requirements before upgrading:

CPU	Quad-core running at 2.8 GHz or faster
Memory	6 GB minimum 8 GB recommended
Disk Space	2 disks: 100 GB for the operating system, Content Gateway, and temporary data. Max 147 GB for caching If caching will not be used, this disk is not required. The caching disk: Should be at least 2 GB and no more than 147 GB Must be a raw disk, not a mounted file system Must be dedicated Must not be part of a software RAID Should be, for best performance, a 10K RPM SAS disk on a controller that has at least 64 MB of write-through cache
Network Interfaces	2

In addition, to support transparent proxy deployments:

Router

Must support WCCP v2.

A Cisco router must run IOS 12.2 or later. The latest version is recommended.

To support IPv6, WCCP v2.01 and Cisco router version 15.4(1)T or later are required.

Client machines, the destination Web server, and Content Gateway must reside on different subnets.

-or-

Layer 4 switch

You may use a Layer 4 switch rather than a router.

To support WCCP, a Cisco switch requires the EMI or IP services image of the 12.2SE IOS release (or later).

Content Gateway must be Layer 2 adjacent to the switch.

The switch must be able to rewrite the destination MAC address of frames traversing the switch.

The switch must be able to match traffic based on the layer 4 protocol port (i.e., TCP port 80).

Upgrading Content Gateway

Content Gateway runs on web protection full policy source, user directory and filtering, and filtering only appliances (all of which should already have been upgraded at this point).

Content Gateway is supported on Red Hat Enterprise Linux machines. See the Certified Product Matrix for a list of supported operating systems.

IContent Gateway upgrade instructions

This section describes upgrading Content Gateway on your Red Hat Enterprise Linux host.

Important:

At the beginning of the upgrade procedure, the installer checks to see if the partition that hosts /opt has enough space to hold a copy of the existing Content Gateway log files (copied to /opt/WCG_tmp/logs). If there’s not enough space, the installer prints an error message and quits.

In this situation, if you want to retain the log files you must copy the contents of /opt/WCG/logs to a location that has enough space, and then delete the log files in /opt/WCG/ logs.

When the upgrade is complete, move the files from the temporary location back to /opt/WCG/logs and delete the files in the temporary location.

Note: If you have multiple Content Gateway instances deployed in a cluster, you do not have to disable clustering or VIP (if used). As each member of the cluster is upgraded it will rejoin the cluster.

If your existing web protection solution is deployed with Forcepoint Web Security DLP Module or a data protection product:
1. Log on to the Content Gateway manager.
2. Navigate to the Configure > My Proxy > Basic page.
3. Disable Web DLP.
  When the upgrade is complete:
4. Return to the Configure > My Proxy > Basic page.
5. Enable the new Web DLP option.
6. Restart Content Gateway.
7. Navigate to the Configure > Security > Web DLP page and confirm that automatic registration was successful. If it was not, confirm that the Data module of management console is running as expected.
Log on to the Content Gateway Linux host and acquire root permissions:
su root
Disable any currently running firewall on this machine for the duration of the upgrade. Bring the firewall back up after the upgrade is complete, opening ports used by Content Gateway.
For example, if you are running IPTables:
1. At a command prompt, enter service iptables status to determine if the firewall is running.
2. If the firewall is running, enter service iptables stop.
3. After upgrade, restart the firewall. In the firewall, be sure to open the ports used by Content Gateway on this machine. See Default ports for on-premises Forcepoint security solutions for more information.
  Important: Forcepoint Web Security customers using Red Hat Enterprise Linux or CentOS 7.x must disable firewalld prior to installing Content Gateway.
  
  On the machine where Content Gateway will be installed, execute the following:
  
  systemctl stop firewalld
  
  systemctl disable firewalld
Use the Downloads tab of the My Account page at forcepoint.com to download the Content Gateway version 8.5.x installer, and save it to a temporary directory. For example, place it in:
/tmp/cg_v85
Unpack the Content Gateway installer tar archive:
cd /tmp/cg_v85

tar -xvzf <installer tar archive>

Important: If SELinux is enabled, set it to permissive, or disable it before installing Content Gateway. Do not install or run Content Gateway with SELinux enabled.
If you intend to upgrade Red Hat Enterprise Linux 6.x to a more recent version, perform the upgrade now. See your Red Hat Enterprise Linux documentation.
In the directory where you unpacked the tar archive (for example, /tmp/wcg_8x), start the installation/upgrade script.
./wcg_install.sh

Respond to the prompts.

Content Gateway is installed and runs as root.

Note: Up to the point that you are prompted to confirm your intent to upgrade, you can quit the installer by pressing CTRL+C. If you change your mind after you choose to continue, do not use CTRL+C to stop the process. Instead, allow the installation to complete and then uninstall.
If your server does not meet the minimum hardware requirements or is missing required operating system packages, you will receive error or warning messages. For example:
Error: Content Gateway v8.5.x on x86_64 requires several packages that are not present on your system.

Please install the following packages: <list of packages>

If you are connected to a yum repository you can install these packages with the following command:

yum install <list of packages>

See the Technical Library (support.forcepooint.com/ Documentation) for information about the software requirements for x86_64 installation.

To make it easier to install the needed packages, the Content Gateway distribution includes a Linux “rpm” containing the needed packages. To install its contents, ensure that the operating system has access to the Red Hat Linux distribution library (for example the DVD), and enter:

yum install wcg_deps-1-0.noarch.rpm

Upon successful completion, a list of updated packages displays and then the word “Complete!”.

Here is an example of a system resource warning:
```
Warning: Content Gateway requires at least 6 gigabytes of RAM.
Do you wish to continue [y/n]?
```
Enter n to end the installation and return to the system prompt.

Enter y to continue the upgrade. You should not install or upgrade on a system that does not meet the minimum requirements. If you choose to run Content Gateway after receiving a system resource warning, performance and stability may be affected.
Read the subscription agreement. At the prompt, enter y to accept the agreement and continue the upgrade, or n to cancel.
Do you accept the above agreement [y/n]? y
The installer checks for the presence of an existing Content Gateway installation. When asked, choose to replace the existing version with version 8.4.x.
```
WCG version 8.1.n-nnnn was found.
Do you want to replace it with version 8.5.x-nnnn [y/n]? y
```

Existing settings and logs are copied to backup files and stored. For example:

Stopping Content Gateway processes...done
Copying settings from /opt/WCG to /root/WCG/OldVersions/ 8.1.0-1418-PreUpgrade/...done
Zipping configuration archive...done
Moving log files from /opt/WCG/logs to /opt/WCG_tmp/logs/...done

You can either re-use the installation selections you entered during the last install, or provide new answers to all installation prompts, such as admin password, admin email address, Policy Server IP address, etc.:
```
Previous installation selections </root/WCG/Current/ WCGinstall.cfg> found.
Use previous installation selections [y/n]?
```
Enter y to use previous installation selections.

Enter n to revert to default values, and receive all installation questions and answer them again.
If you answered y at Step 11, then you can also leave proxy settings at their current values or revert to default values (which perform a fresh install!).
Restore settings after install [y/n]?

Enter y to keep the proxy settings as they are.

Enter n to restore default settings for the proxy.

CAUTION:
If you answer n (no), the current installation of Content Gateway is removed, and a fresh install of 8.2.x begins. See Installation Instructions: Forcepoint Web Security for a detailed description of the installation procedure. This is not an upgrade, but rather a fresh install.

The previously installed version of Content Gateway is removed, and the settings and selections you chose to retain are re-used. Details of the upgrade process are output to the screen. Please wait.

*COMPLETED* Content Gateway 8.5.x-nnnn installation.
A log file of this installation process has been written to
/root/WCG/Current/WCGinstall.log
For full operating information, see the Content Gateway Help system.
Follow these steps to start the Content Gateway management interface (Content Gateway Manager):
------------------------------------------------------------
1. Start a browser.
2. Enter the IP address of the Content Gateway server, followed by a colon and the management interface port (8081 for this installation). For example: https://11.222.33.44:8081.
3. Log on using username admin and the password you chose earlier.
A copy of the CA public key used by the Manager is located in /root/WCG/.

The automated portion of the upgrade is now complete, and the proxy software is running.
If you chose to revert to default proxy settings, be sure to configure any custom options.
Check Content Gateway status with:
/opt/WCG/WCGAdmin status
All services should be running. These include:
- Content Cop
- Content Gateway
- Content Gateway Manager
- Analytics Server
Important: If Content Gateway fails to complete startup after upgrade, check for the presence of the no_cop file. Look for: /opt/WCG/config/internal/no_cop. If the file exists, remove it and start Content Gateway using /opt/WCG/WCGAdmin start
To finish the upgrade, be sure to perform the post-upgrade instructions at the end of this document.

Post-upgrade activities

After you have finished upgrading components, refer to the following to ensure that your Content Gateway upgrade is complete.

If at the start of the upgrade process you manually moved your existing log files to a temporary location, move them back to /opt/WCG/logs and delete the files in the temporary location.
Register Content Gateway nodes in Forcepoint Security Manager on the Web > Settings > Content Gateway Access page.
Registered nodes add a link to the Content Gateway manager logon portal and provide a visual system health indicator: a green check mark or a red X.
Configure Content Gateway system alerts on the Settings > Alerts > System page in the Security Manager.
This subset of Content Gateway system alerts can be configured to be sent to administrators, in addition to being displayed in the Content Gateway manager.
If you use SSL support:
1. If your clients don’t yet use a SHA-256 internal Root CA, create and import a SHA-256 Root CA into all affected clients. See Internal Root CA in Content Gateway Help.
2. Using the notes you compiled prior to upgrade, rebuild your Static Incident list.
If you use proxy user authentication, review the settings on the Global Authentication Options page (Configure > Security > Access Control > Global Configuration Options).
If you use IWA user authentication, confirm that the AD domain is still joined. Go to Monitor > Security > Integrated Windows Authentication. If it is not joined, rejoin the domain. Go to Configure > Security > Access Control > Integrated Windows Authentication.
If you use Rule-Based Authentication, review your configuration. Go to Configure > Security > Access Control.
1. Check the Domains page.
  - IWA domains that were joined before upgrade should still be joined.
  - LDAP and Legacy NTLM domains should be listed.
2. Check each rule.
  - Go to the Authentication Rules page and enter the editor.
  - Select each rule and check the configuration.
  - For Multiple Realm Authentication rules that used Cookie Mode Caching, check the cookie list on the Global Authentication Option page.
  - Check that the expected domain is in the Auth Sequence list.
Important: The Rule-Based Authentication feature is very rich and can satisfy many user authentication requirements. To make best use of it, please refer to Rule-Based Authentication.
If a web protection and data protection solution were deployed together, confirm that Content Gateway has automatically re-registered with the Data module of the Forcepoint Security Manager. If it has not, manually re-register.
1. Ensure that the Content Gateway and the Security Manager server system clocks are synchronized to within a few minutes.
2. In the Content Gateway manager:
  - Go to Configure > My Proxy > Basic, ensure that Web DLP: Integrated on-box is enabled, and click Apply.
  - Next to Integrated on-box, click the Not registered link. This opens the Configure > Security > Web DLP registration screen.
  - Enter the IP address of the Security Manager server.
  - Enter a user name and password for logging onto Security Manager. The user must be a Forcepoint DLP administrator with Deploy Settings privileges.
  - Click Register. If registration is successful, a message confirms the result and prompts you to restart Content Gateway. If registration fails, an error message indicates the cause of failure. Correct the problem and perform the registration process again.
If web and data protection products were deployed together and upgraded, you may need to remove stale entries of Content Gateway instances registered in Forcepoint DLP system modules:
1. Log onto Security Manager.
2. Select the Data tab and navigate to the Settings > Deployment > Modules page.
3. Listed are 2 instances of each Content Gateway module registered with the system. Delete the older instances. You can identify these by looking at the version number.
4. Click Deploy.
If web and data protection products were deployed together and configured to use the on-box policy engine, and then reconfigured during upgrade or later to use the ICAP interface, the Content Gateway instance may need to be deleted from the list of Forcepoint DLP system modules or the deployment will fail. Go to the Data > Settings > Deployment > System Modules page, click on the affected Content Gateway instance to open its Details page, click Delete and then Deploy.
If your explicit proxy deployment was customized to support an external load balancer with IWA user authentication, the configuration is preserved during upgrade. You do not need to re-apply the custom configuration. You should, however, test your deployment to verify that the load balancer is performing as expected.
With v8.2.x, the basic functionality for 2 features was changed slightly:
- Send authentication to parent proxy, configured on the Configure > My Proxy > Basic > General page
- X-Forwarded-For, enabled on the Configure > Protocols > HTTP > Privacy
In both cases, header values are forwarded only to a configured parent proxy.

If you are upgrading from v8.1 to v8.5, enabled either of these settings in your previous version, and are expecting header values to be forwarded for all outbound requests, add the appropriate variable to your records.config file (in the /opt/WCG/config directory, by default).
- To add the user name to outbound requests, add:
  CONFIG proxy.config.http.insert_xua_to_external INT
- To send X-Forwarded-For header values directly to the Internet, add:
  CONFIG proxy.config.http.insert_xff_to_external INT 1
If you were using v8.1 with custom cipherlist settings using these variables in records.config:
```
proxy.config.ssl.server.cipherlist
proxy.config.ssl.client.cipherlist
```
You need to reconfigure the custom settings because these variables were replaced in v8.2.
- proxy.config.ssl.server.cipherlist_suffix replaces proxy.config.ssl.server.cipherlist
- proxy.config.ssl.client.cipherlist_suffix replaces proxy.config.ssl.client.cipherlist
The non-default cipherlist being used prior to the upgrade are saved as a comment in records.config, where it can be used for reference. Default values for the new variables are put into place during the upgrade and can be reconfigured after the upgrade is complete.

See Content Gateway Manager Help for more information on how these new variables now work with proxy.config.ssl.server.cipherlist_option and proxy.config.ssl.client.cipherlist_option to create cipher lists.
The Tunnel Skype option on the Configure > Protocols > HTTPS page of Content Gateway Manager was removed in v8.3. Variables stored in the records.config file that apply to Skype are removed during upgrades from v8.1 and v8.2.
The settings on the Configure > Networking > Connection Management > Low Memory Mode page of Content Gateway manager was removed in v8.3. Corresponding variables stored in the records.config file are removed by upgrades from v8.1 and v8.2.
If LOW encryption cipher suites was previously selected on the Configure > SSL > Decryption/Encryption > Inbound or Outbound pages of Content Gateway manager, upgrades from v8.1 or v8.2 will change the setting to MEDIUM. LOW is no longer a valid option on those pages.
The corresponding records.config variables are also updated by the upgrade.
During upgrades from v8.1 or v8.2, the Enable the certificate verification engine on the Configure > SSL > Validation > General page of Content Gateway manager will be changed to ON for any customer who does not already have the feature enabled.
In v8.3 and continued in v8.4 and v8.5, improvements were made to the Adaptive Redirection Module (ARM). The ARM component now utilizes iptables, policy routing, and transparent sockets which are configured during product installation or upgrade.
The Content Gateway Manager was changed to reflect these improvements.
- The Network Address Translation (NAT) section of the Configure > Networking > ARM > General page has been renamed to Redirection Rules to better reflect the contents of the table.
- Text on that page has also been updated.
To facilitate interception and redirection of traffic:
- IPTables rules are configured during upgrade.
  - Forcepoint IPTables chains are inserted.
  - Forcepoint IPTables rules are also inserted into existing chains.
  - Forcepoint chains and rules use “NC_” as a prefix for identification purposes.
- IPTables rules configured outside of Content Gateway Manager must
  - Be inserted after Forecepoint rules.
  - Never be added to Forcepoint chains.
- Forcepoint chains and rules should never be edited.
- If customized chains or rules impact the Forcepoint configuration, navigate to /opt/wcg/bin and execute the following to re-establish the Forcepoint IPTables chains and rules:
  netcontrol.sh -r
For some customers, the GRE Packet Return Method (GRE return) may not be as expected. In all cases, GRE return, as documented by Cisco (see this site), is fully functional. However, tunneling back through a router (enhanced GRE tunnel return) now requires a specific kernel module. Contact Forcepoint Technical Support to enable this functionality.

To provide more appropriate statistical data for the new ARM, the Bypass Statistics now provide information for:
- Total Packets Bypassed
- Packets Dynamically Bypassed
- DNS Packets Bypassed
- Packets Shed
A change was made in v8.4 to resolve customer issues with SSL retry logic. The default values for the following records.config variables are reset to 1 during an upgrade from v8.1, v8.2, or v8.3.
```
proxy.config.http.connect_attempts_max_retries
proxy.config.http.connect_attempts_max_retries_dead_server
```
Automatic updates to the Certificate Authority tree were added to v8.4.
After upgrading from v8.1, v8.2, or v8.3, when the initial CA tree update occurs, CAs in the customer deployment but not in the 8.4 CA db, any CA that is no longer a root CA, and CAs that are no longer trusted are converted to a private CA. This process also removes expired CAs.

After the initial update, review the CA tree on the Configure > SSL > Certificates page of Content Gateway manager and remove any certificates that are no longer trusted or may be revoked.
With v8.5, default IPTables include a rule that will drop traffic that is neither HTTP, HTTPS, nor FTP and not forward it through the proxy.
On upgrade, this feature is disabled by default. To add the rule and not forward traffic that is neither HTTP, HTPTS, nor FTP, add the following to records.config ((located in /opt/WCG/config, by default):

CONFIG proxy.config.arm.forward_unwanted_traffic INT 0

After this entry is added and Content Gateway is restarted, an IPTables rule is added and traffic that is neither HTTP, HTTPS, nor FTP will not be forwarded.
For customers who have purchased the v8.5 Protected Cloud Apps feature, the setting for Parent Proxy on the Configure > Content Routing > Hierarchies page of Content Gateway Manager will be enabled. If you previously enabled and configured Parent Proxy and later disabled the option, the configured settings will be used and should be updated as necessary.
With v8.5, the option of TLSv1 on the Configure > SSL > Decryption/ Encryption page (Inbound and Outbound tabs) and on the Configure > Security > FIPS page of Content Gateway Manager is no longer a default selection. Options for TLSv1.1 and TLSv1.2 are added and enabled by default.
During upgrade, if HTTPS (SSL) was enabled on the Configure > My Proxy > Basic > General page of Content Gateway Manager prior to upgrade, the SSL settings are not changed.

IF HTTPS (SSL) is enabled after the upgrade, the settings will be handled like a fresh installation of the product and TLSv1.1 and TSLv1.2 will be enabled by default. TLSv1 will not be enabled.
Beginning with v8.5.3, Content Gateway will no longer accept nor download SHA-1 intermediate certificates. SHA-1 certificates that were added by Content Gateway will be removed during an upgrade to v8.5.3. Note that SHA-1 certificates that were manually added will not be deleted.
A new variable has been added in v8.5.3 that will disable the automatic adding of new certificates to the certificate database. Upgrades to v8.5.3 will add this new parameter to records.config, set to use the default functionality.

To disable the default functionality edit the following in records.config (located in /opt/WCG/config, by default)

CONFIG proxy.config.ssl.cert.verify.add_cert_to_database INT 0

Reset the value to 1 to restore the default functionality.
Version 8.5.3 adds the ability to manually add a dynamic certificate key. Each key requires a passphrase. Both the key and passphrase are stored in the certificates database.
With v8.5.4, a setting has been added to Content Gateway manager that enables authentication of HTTPS requests over HTTPS, using port 4443.
Open Content Gateway manager and navigate to Configure > Security > Access Control and select Global Authentication Options. A new Redirect Options section contains the Redirect Hostname entry field as well as the option to Redirect for HTTPS Authentication.

Disabled by default, click Enabled to direct all HTTPS requests to authenticate over HTTPS.

Changing the manager options also resets a new records.config variable.

proxy.config.auth.ssl_auth_url
Custom certificates added for use with Captive Portal are not retained when upgrading to v8.5.x. These certificates must be re-added after the upgrade is completed.
A new Socks Server Rule has been added to the "Do not route through SOCKS server" rule type to ensure that traffic that does not need to be directed through a SOCKS server is not sent there.This avoids SOCKS server issues that may result from excessive load.
This rule is also added when upgrading to v8.5.4.

Note: SOCKS traffic from the ip range included in the rule will be routed through a SOCKS server.
To fix a vulnerability, the default value for the following records.config variables has been changed in v8.5.4 and will be updated to the new defaults when upgrading.
proxy.config.ssl.server.cipherlist_suffix

proxy.config.ssl.client.cipherlist_suffix

See Content Gateway Manager Help for more information on how these variables work.
The Session Cache section, previously available on Configure > SSL > Decryption / Encryption > Outbound have been removed in v8.5.4 to avoid Content Gateway restarts. Upgrades to v8.5.4 will automatically disable these options if they had been previously enabled.