OpenVPN-Monitoring-Simple/DOCS/Operations/REBOOT-TEST.md

Reboot test: ENTRY (Timeweb) and EXIT (Hel)

Date: 2026-09-30. Goal: confirm the Hysteria 2 chain recovers by itself after a reboot of each node, with no manual steps.

Nodes: ENTRY 213.226.125.13 (Alpine, OpenRC), EXIT 85.234.86.140 (Debian 13, systemd).

Methodology

Order: ENTRY first, then EXIT. Each node is rebooted separately, so the other side is always up and its reconnect behavior is observable.

1. Pre-flight (before each reboot)

Check	ENTRY	EXIT
Services enabled	rc-update show default (hysteria, hysteria-route, hysteria-edge)	systemctl is-enabled hysteria-server
Kernel module for TUN persists	lsmod | grep tun, grep -x tun /etc/modules	not needed
sysctl persisted	ls /etc/sysctl.d/ (99-hytun.conf, 99-quic.conf)	ls /etc/sysctl.d/99-quic.conf
Temporary test settings removed	n/a	grep -c speedTest /etc/hysteria/config.yaml = 0

Any gap found is fixed before rebooting.

2. Reboot

sync, then a delayed reboot / systemctl reboot (detached, so the SSH command returns). Wait 45 s, then poll SSH every 6 s (up to 20 tries) until uptime answers.

3. Post-boot verification

ENTRY:

1. rc-service hysteria|hysteria-route|hysteria-edge status → started.
2. ls -l /dev/net/tun, ip -br a show hytun.
3. ip rule | grep -E '^10[01]' and ip route show table 100 → both rules, default via hytun, blackhole fallback.
4. sysctl net.ipv4.conf.all.rp_filter net.core.rmem_max → 2 and 7500000.
5. ss -lunp | grep :443 → hysteria-edge listening.
6. Tunnel TCP: su -s /bin/sh hyedge -c 'curl -4 -s https://ifconfig.me' → must equal the EXIT IP.
7. Tunnel UDP: su -s /bin/sh hyedge -c 'nslookup example.com 1.1.1.1' → address returned.
8. Logs: /var/log/hysteria.log, /var/log/hysteria-edge.log; a real client (mobile) reconnecting is a bonus signal.

EXIT:

1. systemctl is-active hysteria-server, ss -lunp | grep :443.
2. sudo /usr/sbin/sysctl net.core.rmem_max net.core.wmem_max (sysctl is not in a non-root user's PATH).
3. journalctl -u hysteria-server -b → "server up and running" and "client connected" from ENTRY.
4. From ENTRY: client log shows a new connected to server (count incremented) and the hyedge curl test (step 6 above) returns the EXIT IP.

Results

ENTRY reboot (Timeweb)

Check	Result
Pre-flight	Gap found: module tun loaded by hand but absent from /etc/modules → after reboot /dev/net/tun could be missing and the tunnel would not start. Fixed (echo tun >> /etc/modules) before rebooting.
SSH back	Yes, about 45-60 s after the reboot command
Services	hysteria, hysteria-route, hysteria-edge all started
TUN / routing	/dev/net/tun present, hytun up (100.100.100.101/30), rules 100 and 101 present, table 100: default dev hytun + blackhole
sysctl	rp_filter=2, rmem_max=7500000 applied
Listener	hysteria-edge on UDP 443
Tunnel TCP	85.234.86.140 (EXIT IP)
Tunnel UDP	DNS answer received
Client	Hiddify (mobile) reconnected on its own right after boot
Noise in logs	A few dial timeouts to individual remote hosts (unreachable destinations, not a tunnel fault)

Verdict: PASS (after the /etc/modules fix).

EXIT reboot (Hel)

Check	Result
Pre-flight	hysteria-server enabled, 99-quic.conf present, speedTest absent from config (0)
SSH back	Yes, about 45-50 s after the reboot command
Service	hysteria-server active, listening on UDP 443
sysctl	rmem_max and wmem_max = 7500000
Journal	"server up and running" at boot, then "client connected" from 213.226.125.13 about 30 s later
ENTRY side	Client reconnected automatically (connected to server, count: 2); brief TUN UDP error ... EOF while EXIT was down; hyedge curl returned 85.234.86.140

Verdict: PASS.

Conclusions

• Both nodes recover unattended; the ENTRY hysteria client (supervise-daemon, lazy: true) reconnects without restart when EXIT comes back.
• During an EXIT outage the anti-leak blackhole route in table 100 keeps hyedge traffic from leaving via ENTRY's own uplink (leak test done separately: removing the tun default route makes the test curl fail instead of exposing the ENTRY IP).
• Not covered: simultaneous reboot of both nodes, power-loss/unclean shutdown, and IKE/strongSwan (separate task, charon is not enabled at boot).

ENTRY reboot after the OpenVPN Monitor deployment (2026-09-30)

Goal: confirm the whole ENTRY stack (Hysteria chain + OpenVPN Monitor, OpenVPN, egress rules, hardening) recovers by itself. Only ENTRY was rebooted; EXIT stayed up.

Pre-flight

• rc-update show default: hysteria, hysteria-route, hysteria-edge, ovpn-hytun, openvpn, ovpmon-api, ovpmon-gatherer, ovpmon-profiler, nginx, fail2ban, sshd, chronyd enabled.
• tun in /etc/modules; sysctl files present (ip_forward=1, rp_filter=2); sshd -t and doas -C /etc/doas.d/ovpmon.conf valid.
• Baseline snapshot /root/pre-reboot-snapshot.txt: service states, ip rule 100-102, table 100, nat/FORWARD rules, listeners, process owners.

Reboot and result

sync, detached reboot; SSH answered again after ~45 s. Post-boot snapshot /root/post-reboot-snapshot.txt is identical to the baseline.

Check	Result
All services in default	started by themselves
API processes	gunicorn/uvicorn/gatherer run as ovpmon
hytun, tun0, rp_filter=2, ip_forward=1	present
ip rule 100/101/102, table 100 (hytun + blackhole), MASQUERADE to hytun, FORWARD DROP guard	restored by hysteria-route / ovpn-hytun
Tunnel TCP as hyedge	85.234.86.140 (EXIT)
Hysteria client	reconnected to EXIT
Panel	https://…:8088/ 200, profiler without token 401, HTTP→HTTPS 301, login tstark works
Helper/doas	service status and process stats OK; gatherer reads openvpn-status.log (root:ovpmon 640 persisted)
/etc/openvpn/crl.pem, /var/lib/ovpmon ownership	intact
fail2ban	3 jails active
SSH	passwordauthentication no, permitrootlogin prohibit-password
Errors in ovpmon-* logs since boot	none (older entries predate the boot)
Real VPN client	reconnected by itself ~3.5 min after boot (UDP client waits for its keepalive 10 120 timeout); ~5 MB downloaded through the tunnel, MASQUERADE counters rose, leak guard FORWARD DROP counter stayed 0

Notes

• Expected client gap after a server reboot is up to ~2-4 minutes for UDP OpenVPN clients (ping-restart), not a fault of the server.
• openvpn-status.log keeps root:ovpmon 640 because the file persists on disk; if it is ever deleted, OpenVPN recreates it as 600 and the gatherer cannot read it until the next ovpmon-helper service start|restart (which fixes the mode). Not triggered by a reboot.