Checkpoint IPSO Clustering Issue

       Nokia Clustering Overview

Overview:

IPSO lets you create firewall/VPN clusters that provide fault tolerance and dynamic load balancing. A cluster is made up of multiple appliances (nodes) that share common IP addresses, and it appears as a single system to the networks connected to it.

 

Cluster Terminology:

1. Cluster ID
2. Cluster IP address
3. Cluster MAC Address
4. Cluster Master
5. Cluster Member
6. Cluster Protocol networks

1. Primary cluster protocol
2. Secondary Cluster protocol

7.  Clustering Modes

1. Forwarding
2. Multicast
3. Unicast

Creating and configuring a cluster

Basic steps to create the entire cluster:

1. Create a cluster configuration on the first node.

2. Change the cluster state to up on the first node.

3. Create a cluster configuration on another node.

4. Add the second node to the cluster.

You can use the joining method (recommended) or manual method to add the second node to the cluster

5. Repeat step 3 and step 4 for the remaining nodes.

Once there are at least two nodes in the cluster, you can manage both nodes simultaneously by using Cluster Voyager.

Nokia Clustering configuration via voyager

Cluster Configuration page.

Performing a Detailed Configuration

1. Click Configuration > High Availability > Clustering > Clustering Configuration in the Voyager navigation tree.

2. Enter a cluster ID (0-65535).

3. Choose the cluster topology.

Note - You must use the same password on each node that you add to the cluster. This is also the password that you use to log into Cluster Voyager. Note - If you want to examine or change any of the default selections that IPSO made for you, Click Configuration > High Availability > Clustering > Clustering Configuration

4. Enter a password for the user cadmin.

5. Enter the password for cadmin again (for verification).

6. Click Apply.

7. Click Manually Configure IPSO Cluster.

1. Selecting the Cluster Topology

1. Load balancing
2. N+1:N
3. Active/Hot standby

ii.   Selecting Cluster mode

1. Forwarding
2. Multicast
3. Unicast

iii. Configuring the work assignment method  

1. Dynamic
2. Static

iv.  Failure Interval and Performance ratting

v. Configuring Interfaces

1. Primary cluster interface
2. Secondary cluster interface (optional)
3. Select the Interface and enter the cluster IP address , Repeat same for all the interfaces.

10. Enable or disable firewall monitoring, as appropriate:

•If R70 is running on the node, enable R70 monitoring before you make the cluster active.

•If R70 is not running on the node, disable R70 monitoring before you make the cluster active (so that the cluster can be initialized). After you activate the firewall, enable the monitoring so that the cluster monitors the firewall and leaves the cluster if the firewall fails on the node.

11. Change the cluster state to up.

12. Click Apply.

13. Click Save.

Joining a System to a Cluster

1. Display the Interface Configuration page.

2. Configure interfaces with IP addresses in each of the networks used by the cluster and activate the interfaces.

3. Under High Availability, click Clustering to display the Clustering Configuration page.

4. Enter the ID of the existing cluster.

5. Enter the password for the user cadmin in both password fields.

6. Click Apply.

7. In the Cluster node address field, enter an IP address that meets the following criteria:

•You should use an address of an interface on the cluster node that you configured first.

•The interface must be one of the cluster interfaces. To join a system to a cluster, perform this simple procedure:

•You should use the “real” address of the interface—not its cluster IP address. (If the cluster is in forwarding mode and you supply a cluster IP address for joining purposes, the joining system will copy configuration settings from the master node, which might not be the one you want to copy the settings from.)

8. Click Join.

•If the node successfully joins the cluster, Voyager displays a number of new fields.

•If the node does not successfully join the cluster, you see a message indicating why. Correct the problem and attempt the join again.

Managing Cluster

1. Voyager : Cluster monitoring in monitor Menu
2. Command Line

Clustering Issues:

Problem Reported: Customer having Nokia IP390 devices in load balancing clustering, from outside internet http traffic getting dropped and slow.

Device Hardware and Software Detail:

Device Model: Nokia IP390: 2No.

IPSO Ver.: IPSO 6.2 MR2 GA039

Checkpoint Packages: R70 HFA 30

HA: Clustering (load balancing)

Cluster configuration parameter in both the devices:

Cluster ID: 100

Cluster Topology: Load Balancing

Number on Node: 2

Number of interfaces: 4

Cluster Mode: Forwarding

Work Assignment: dynamic

Performance rating: 1100

Failure interval: 500

Troubleshooting steps:

1.      Cluster monitoring state via Nokia voyager monitoring tab : its ok load balancing equally

2.      Check Smart View tracker logs traffic : Showing traffic passing traffic through both the gateways

3.      Then Start  TCPdump capturing on Incoming interface and outgoing interface for outside

Public IP:  a) - IN interface traffic coming logs showing traffic pass through interface

     b) - OUT interface traffic not coming logs showing blank in capture file

With this output we can’t identify why traffic dropping. 

4.      Also taken CST file backup: Output of this file there is no drop.

Note: If you’re facing same issue in IPSO 6.2 below version MR2 GA039 then first upgrade with new version IPSO MR2 GA039 the check refer SK43954, then check if problem still then go for the below steps.

5.      Taken Putty of both the gateway and started monitoring the session:

 Then started Kernel debug for Packet Drop, execute:

% fw ctl zdebug + drop

And started accessing the sites from outside. (Stopped after 10min)

6.      Opened this capture file and found that the sites not opened from outside is dropped with following message:-

In the above case, zdebug drop shows the following drops

For affected services --fw_log_drop: Packet proto=6 10.207.144.106:80 -> 172.16.193.32:17799 dropped by fwchain_fwd Reason: can't translate interface

7.      This is known issue in IPSO clustering refer SK54500,

Symptoms

·         Intermittent slowness of some services / traffic through the firewall.

·         Stateful failover does not happen for some active connections through the cluster

·         In all the above cases, zdebug drop shows the following drops for affected services --fw_log_drop: Packet proto=6 10.207.144.106:80 -> 172.16.193.32:17799 dropped by fwchain_fwd Reason: can't translate interface;

Solution:

In the objects_5_0.C file on the SmartCenter Server, in properties of the cluster object, the interfaces associated with the missing Virtual IP's, the multicast_address attribute has a multicast MAC address populated in the the value field.

e.g.

: multicast_address ("01:00:5e:0a:0a:02")

: multicast_address ("01:00:5e:10:5f:06")

a). On IP Appliances that run IPSO this field should be blank, in our case there is MAC entry in multicast_address.

b). Ensure that all SmartConsole  connections to the SmartCenter Server are closed.

       c). Then Using the SmartConsole GUIDbedit connect to the SmartCenter Server.

       d).  Search for 'multicast_address', Then I  find the attributes for the cluster object firewall    properties in question and see a multicast address in the values field, then delete it.

       e). Save the object database before exiting GUIDbedit.

       f). Then Installed policy and checked the output of 'cphaprob -a if' on both the cluster members.

      Testing: All the website and services from the internet and internal N/W.

      Result:  Websites opening very fast and no drops in packets.

 Important Note:- After the activity, I executed one more command to save the kernel configuration changes permanently into gateways, otherwise after reboot of any gateway it introduce same problem again.

Command:  fw ctl set int fwha_perfom_chain_forward 0

To make the changes permanent refer sk26202 / sk40988