FortiGate IPsec VPN Configuration & Troubleshooting
Description
Explore a complete FortiGate IPsec VPN guide covering configuration, authentication, advanced features, and troubleshooting. Perfect for beginners and professionals.
Scope
This article is a complete resource guide for IPsec VPN on FortiGate.
It covers:
- VPN concepts and use cases
- All configuration types
- Advanced deployment scenarios
- Real-world troubleshooting cases
It is suitable for:
- Beginners learning VPN
- Network engineers managing enterprise environments
Solution
1. IPsec VPN Overview
IPsec (Internet Protocol Security) secures communication at the IP layer.
Common Use Cases:
- Site-to-Site VPN → Connect offices
- Remote Access VPN (Dial-up) → Connect users
2. Basic VPN Configurations
- Site-to-Site VPN using wizard
- Site-to-Site with Pre-Shared Key (PSK)
- Site-to-Site with Digital Certificates
- Policy-based IPsec VPN (GUI enable required)
3. Dial-up (Remote Access) VPN
Types:
- Full Tunnel (all traffic via VPN)
- Split Tunnel (only internal traffic)
Advanced Dial-up Features:
- External DHCP IP assignment
- DHCP reservation for VPN users
- Multiple user groups (XAUTH/EAP)
- Multiple connections from same IP
- Peer ID selection for multiple tunnels
- Hard timeout enforcement
- DNS configuration (multiple DNS servers)
4. Authentication Methods (Complete)
- Pre-Shared Key (PSK)
- Certificates
- SAML (SSO login)
- Microsoft Entra ID
- OKTA
- FortiAuthenticator (IdP)
- LDAP authentication
- RADIUS authentication
- EAP authentication (multi-group matching)
- Multi-Factor Authentication (FortiToken)
- Certificate + LDAP/RADIUS combined authentication
5. Advanced VPN Features
- ADVPN (Auto Discovery VPN)
- ADVPN 1.0 and ADVPN 2.0 differences
- Loopback-based scalable design
- Shortcut tunnels
- OCVPN (Overlay Controller VPN)
- Full mesh deployment
- Simple deployment options
- IPsec Aggregate & Redundant VPNs
- HA (High Availability) VPN setup
- NAT Traversal (NAT-T) behavior
- IKEv1 vs IKEv2 (Aggressive vs Main Mode)
- IKE SA rekey behavior (Phase 1 & Phase 2)
- Custom IKE port configuration
6. Routing & Network Integration
- Static routing
- Default route via IPsec
- BGP over IPsec tunnel
- OSPF over IPsec (static & dynamic)
- Dynamic dial-up VPN with OSPF
7. SD-WAN Integration
- Add IPsec tunnels as SD-WAN members
- Primary & backup VPN tunnels
- DDNS-based tunnels
- OCVPN with SD-WAN
- Traffic steering using SD-WAN rules
7. SD-WAN Integration
- Add IPsec tunnels as SD-WAN members
- Primary & backup VPN tunnels
- DDNS-based tunnels
- OCVPN with SD-WAN
- Traffic steering using SD-WAN rules
8. Special & Advanced Use Cases (Full List)
- L2TP over IPsec
- VXLAN over IPsec (multi-VLAN support)
- Overlapping subnet VPN
- Hub-to-spoke with overlapping networks
- Multiple tunnels on same interface
- Forward dial-up traffic to site-to-site VPN
- Multi-hop VPN (Site A → B → C communication)
- Access remote resources via existing VPN
- IPv6-only client VPN with IPv4 access
- MTU override and packet fragmentation handling
- Forward Error Correction (FEC)
- Restrict IPs allowed to connect
- IPsec over TCP (ESP over TCP encapsulation)
9. Migration & Upgrade Scenarios
- SSL VPN → IPsec VPN migration
- SAML SSL VPN → IPsec migration
- ADVPN 1.0 → ADVPN 2.0 upgrade
- Version-specific limitations (e.g., FortiOS 7.6 GUI issues)
10. Troubleshooting
Basic Troubleshooting
- Tunnel status check
- Connectivity (ping, routing)
- Firewall policy verification
Logs & Debug Tools
- VPN logs analysis
- Diagnose/debug CLI commands
- Packet capture (sniffer, Wireshark)
- Convert IKE debug to PCAP
Common Errors
- “No proposal chosen / No SA proposal chosen”
- “TS_UNACCEPTABLE”
- “Invalid ESP packet (HMAC failed)”
- “IPv4 pool not configured”
- “Peer SA proposal mismatch”
- “No matching IPsec selector”
- “Ignoring IKE request (interface down)”
- “GW validation failed”
Advanced Troubleshooting Cases
- Tunnel up but traffic not passing
- One-way traffic issues
- Large packet / MTU errors
- Fragmentation issues
- NAT issues between peers
- Traffic selector mismatch
- IKE stuck at AUTH_RESPONSE
- Phase1/Phase2 negotiation failures
- IPv4 exhaustion in mode-cfg
Authentication Troubleshooting
- SAML redirect failures
- FortiToken MFA failures
- LDAP / RADIUS issues
- EAP authentication mismatch
Hardware / System-Level Issues
- NP6 (NPU) packet drops
- Layer 2 padding issues
- ESP traffic drops due to hardware bugs
Client & Version-Specific Issues
- FortiClient disconnect due to DPD
- NAT-T disabled in specific versions
- VPN failure after firmware upgrade
- AES256-SHA512 traffic issues
ADVPN & SD-WAN Troubleshooting
- Shortcut tunnel not forming
- “No match for shortcut-reply”
- ADVPN with SD-WAN debugging
Final Key Insight
- IPsec VPN on FortiGate is powerful but complex
- Success depends on:
- Matching configuration on both sides
- Correct authentication
- Proper routing
Note
- This article is a complete resource collection, not a step-by-step guide
- Always:
- Match Phase1 & Phase2 settings
- Verify authentication methods
- Check routing and policies
- Test in a lab environment before production
- Keep firmware updated to avoid bugs and compatibility issues
FAQ
What is IPsec VPN used for?
To securely connect networks or remote users over the internet.
What is the difference between IKEv1 and IKEv2?
IKEv2 is newer, faster, and more secure than IKEv1.
Why does “No proposal chosen” error occur?
Because encryption or authentication settings do not match on both VPN peers.
What is the most common VPN issue?
Tunnel is up, but traffic is not passing due to routing or policy issues.
Can IPsec VPN work with SD-WAN?
Yes, FortiGate allows full integration with SD-WAN for better traffic control.