Troubleshooting Port Forwarding with KVM, libvirt, and iptables

This document outlines the steps taken to troubleshoot and resolve an issue where a service running inside a KVM guest was inaccessible from an external network.

I. Problem Description

Unable to access a service running inside a KVM guest from an external network.

• Specifics:

   *   External IP address: 10.48.29.227
   *   Internal IP address (KVM guest): 192.168.122.10
   *   Port: 8443 (and later, 80)
   *   Service: (Name of the service running inside the KVM guest)

II. Initial Configuration

• Network setup:

   *   Host machine interfaces: enp1s0 (external), virbr0 (internal bridge)
   *   KVM guest interface: enp1s0 (or similar)

• iptables rules (as they were before troubleshooting):

iptables Rules (Before Troubleshooting)

Table	Chain	Rule
nat	PREROUTING	(Original PREROUTING rules)
filter	FORWARD	(Original FORWARD rules)
filter	LIBVIRT_FWI	(Original LIBVIRT_FWI rules)
filter	LIBVIRT_FWO	(Original LIBVIRT_FWO rules)

• • Note:** Replace the placeholders "(Original...rules)" with the actual iptables rules.

III. Troubleshooting Steps

• Step 1: Verify Basic Connectivity

   *   Ping the KVM guest from the host machine.
   *   Ping the host machine from the KVM guest.

• Step 2: Check Application Listening

   *   Command used: netstat -tulnp | grep 8443 (or ss -tulnp | grep 8443)
   *   Expected output: The application should be listening on 0.0.0.0:8443 or 192.168.122.10:8443.
   *   Solution if not listening: Configure the application to listen on the correct IP address and port.

• Step 3: Verify DNAT Rule

   *   Command used: sudo iptables -t nat -L -v -n
   *   Expected output: The DNAT rule should be correctly forwarding traffic to the KVM guest.
   *   Solution if incorrect: Correct the DNAT rule using iptables -t nat -A PREROUTING ...

• Step 4: Verify FORWARD Rules

   *   Command used: sudo iptables -L FORWARD -v -n
   *   Expected output: The FORWARD rules should allow traffic between enp1s0 and virbr0 on port 8443.
   *   Solution if incorrect: Add the necessary FORWARD rules using iptables -A FORWARD ...

• Step 5: Investigate LIBVIRT_FWI and LIBVIRT_FWO Chains

   *   Commands used:
       *   sudo iptables -L LIBVIRT_FWI -v -n
       *   sudo iptables -L LIBVIRT_FWO -v -n
   *   Problem: The default libvirt firewall rules were blocking incoming traffic.
   *   Solution: Add ACCEPT rules to the LIBVIRT_FWI and LIBVIRT_FWO chains to allow traffic on port 8443.
       *   sudo iptables -I LIBVIRT_FWI 1 -o virbr0 -p tcp --dport 8443 -j ACCEPT
       *   sudo iptables -I LIBVIRT_FWO 1 -o virbr0 -p tcp --dport 8443 -j ACCEPT
   *   Troubleshooting deleting incorrect rules: Document the various attempts to delete the incorrect rules and why they didn't work.
   *   Final solution: Flush the LIBVIRT_FWO chain and add back the necessary rules.
       *   sudo iptables -F LIBVIRT_FWO
       *   sudo iptables -A LIBVIRT_FWO -o virbr0 -p tcp --dport 8443 -j ACCEPT
       *   sudo iptables -A LIBVIRT_FWO -i virbr0 -s 192.168.122.0/24 -j ACCEPT
       *   sudo iptables -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable

• Step 6: Disable ufw (if applicable)

   *   Command used: sudo ufw disable
   *   Reason: ufw can interfere with iptables rules.

• Step 7: Verify IP Forwarding

   *   Command used: cat /proc/sys/net/ipv4/ip_forward
   *   Solution if disabled: Enable IP forwarding using sudo sysctl -w net.ipv4.ip_forward=1 and make the change permanent in /etc/sysctl.conf.

• Step 8: Check SELinux/Firewalld

   *   Commands used:
       *   sudo setenforce 0 (SELinux)
       *   sudo systemctl stop firewalld (Firewalld)
   *   Reason: SELinux or Firewalld might be blocking the traffic.

• Step 9: Tcpdump Analysis

   *   Commands used:
       *   sudo tcpdump -i enp1s0 port 8443 (host)
       *   sudo tcpdump -i virbr0 port 8443 (host)
   *   Purpose: To capture traffic and identify where the connection is failing.

IV. Final Configuration

• iptables rules (as they are after troubleshooting):

iptables Rules (After Troubleshooting)

Table	Chain	Rule
nat	PREROUTING	(Final PREROUTING rules)
filter	FORWARD	(Final FORWARD rules)
filter	LIBVIRT_FWI	(Final LIBVIRT_FWI rules)
filter	LIBVIRT_FWO	(Final LIBVIRT_FWO rules)

• • Note:** Replace the placeholders "(Final...rules)" with the actual iptables rules.

V. Lessons Learned

• The importance of verifying that the application is listening on the correct IP address and port.
• The potential for libvirt firewall rules to interfere with port forwarding.
• The need to systematically troubleshoot each component of the network configuration.
• The importance of documenting the troubleshooting steps.

VI. Additional Notes

• Include any other relevant information, such as the specific versions of KVM, libvirt, and iptables that you are using.
• Add screenshots or diagrams to illustrate the network setup and iptables rules.