OpenVPN-Monitoring-Simple/DOCS/Operations/Hysteria_Chain_Manifest.md

Hysteria 2 entry→exit chain (manifest)

Client ─hy2/UDP443→ ENTRY (Alpine, OpenRC) ─hy2/UDP443→ EXIT (Debian, systemd) → Internet (IPv4)

Secrets live only in: client-uris.txt, singbox-*.json, /etc/hysteria/* on hosts (chmod 600). Never commit/print them.

Inventory (this deployment)

Role	Host	OS	Access
ENTRY (Timeweb)	213.226.125.13	Alpine 3.23	ssh -i private.key root@ (key must be chmod 600)
EXIT (Gcore Hel)	85.234.86.140	Debian 13	ssh -i hpriv.key debian@ (sudo NOPASSWD)

Prerequisites (verify first, they were the real blockers)

1. UDP 443 open both ways (Security Group on EXIT; ENTRY provider). Test: tcpdump -ni <if> udp port 443 on the receiver while echo x | nc -u -w1 <ip> 443 from the sender. ICMP/TCP passing proves nothing about UDP.
2. Use official binary apernet/hysteria (HyNetworks/hysteria is a fork). Verify sha256sum against hashes.txt of the release: https://github.com/apernet/hysteria/releases/latest/download/{hysteria-linux-amd64,hashes.txt}
3. Hysteria server socks5/http outbounds are TCP-only → for UDP use TUN + policy routing (below), not a SOCKS5 chain.

EXIT (Debian) — server

• /usr/local/bin/hysteria, /etc/hysteria/{server.crt,server.key(600),config.yaml}
• Self-signed cert: openssl req -x509 -nodes -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -days 3650 -subj /CN=<name> -keyout server.key -out server.crt
• config.yaml: listen: :443, tls{cert,key}, auth{type: password, password: <rand>}, masquerade{type: proxy, proxy{url: https://news.ycombinator.com/, rewriteHost: true}}
• systemd unit hysteria-server (ExecStart=hysteria server -c ..., Restart=always, AmbientCapabilities=CAP_NET_BIND_SERVICE)
• /etc/sysctl.d/99-quic.conf: net.core.rmem_max=7500000, net.core.wmem_max=7500000
• No IPv6 uplink here → IPv4 only.

ENTRY (Alpine) — tunnel client + client-facing server

Packages: iproute2 libcap-utils openssl; user hyedge (adduser -S -D -H -s /sbin/nologin); /dev/net/tun must exist and module tun must be in /etc/modules (loaded module, not builtin; without it the tunnel does not come up after reboot).

1. /etc/hysteria/client.yaml (600): server: <EXIT>:443, auth: <pw>, tls{sni, insecure: true, pinSHA256: <EXIT cert fp>}, lazy: true, bandwidth{up: 180 mbps, down: 180 mbps} (Brutal CC; measured path ≈191-195 Mbps via hysteria speedtest, single-flow without it was ~44 Mbps), socks5{127.0.0.1:1080}, http{127.0.0.1:8080}, tun{name: hytun, mtu: 1400, timeout: 5m, address{ipv4: 100.100.100.101/30}} (no route: → no auto-routes).
2. /etc/hysteria/edge.yaml (600, owner hyedge): listen: :443, own self-signed edge.crt/edge.key, auth{type: userpass, userpass{<dev>: <pw>}}, resolver{type: udp, udp{addr: 1.1.1.1:53}}, outbounds[direct{mode: "4"}], trafficStats{listen: 127.0.0.1:9999, secret}, acl.inline: reject 10/8,172.16/12,192.168/16,127/8,169.254/16,100.64/10 then direct(all).
   • Do not add reject(::/0) — it matches any domain having AAAA and rejects dual-stack sites.
   • DoH resolver timed out via tun; UDP resolver works.
3. Binary copy for low port: cp hysteria hysteria-edge; setcap cap_net_bind_service=+ep /usr/local/bin/hysteria-edge.
4. OpenRC (supervisor=supervise-daemon, respawn_delay=3, respawn_max=0):
   • hysteria (client, root) ← need net
   • hysteria-route ← need hysteria; waits for hytun, then:

     ip route replace blackhole default metric 1000 table 100   # anti-leak fallback
     ip route replace default dev hytun metric 10 table 100
     ip rule add priority 100 ipproto udp sport 443 lookup main # edge replies bypass tun
     ip rule add priority 101 uidrange <uid hyedge> lookup 100
     

   • hysteria-edge (command_user=hyedge) ← need net hysteria-route
   • rc-update add hysteria/hysteria-route/hysteria-edge default
5. sysctl (critical): net.ipv4.conf.all.rp_filter=2 and default=2 (/etc/sysctl.d/99-hytun.conf). With strict 1 the TUN SYN-ACKs are dropped → TCP via tun silently hangs.
6. Only uid hyedge traffic and, since 2026-09-30, the OpenVPN client subnet (172.20.1.0/24, rule 102, see below) enter the tunnel; SSH/root traffic untouched.

OpenVPN Monitor & Profiler on ENTRY (added 2026-09-30)

ENTRY also runs the web suite for OpenVPN (native OpenRC services, no containers). OpenVPN clients exit through the same hytun tunnel to EXIT.

Item	Value
Panel	https://213.226.125.13:8088/ (self-signed TLS, login tstark, 2FA; password is set by the owner, not stored here)
OpenVPN	udp/1194, clients 172.20.1.0/24, full tunnel
Services	ovpmon-api, ovpmon-gatherer, ovpmon-profiler (user ovpmon), nginx, openvpn, ovpn-hytun, fail2ban
Egress	ip rule 102 from 172.20.1.0/24 → table 100 (hytun, blackhole fallback) + MASQUERADE on hytun + FORWARD DROP to any other interface (no leak)
Privileges	APIs run unprivileged; a root helper (ovpmon-helper via doas, 6 fixed commands) validates/installs server.conf, publishes the CRL and controls openvpn
Hardening	SSH key-only (00-hardening.conf), fail2ban (sshd, sshd-ddos, ovpmon-login), input validation, CORS restricted
Docs	SUMMARY-ovpn-monitor.md (state, results, security assessment, rollback), IMPLEMENTATION-01/02-*.md, PLAN-validation-and-privilege-drop.md; application repo DOCS/

Check: rc-status | grep -E "ovpmon|openvpn|ovpn-hytun|nginx|fail2ban", ip rule | grep 102, a VPN client on https://ifconfig.me shows the EXIT IP (85.234.86.140).

Clients

• URI: hy2://<user>:<pw>@<ENTRY>:443/?sni=<edge sni>&insecure=1&pinSHA256=<hex, no colons>#name (strip #name when scripting pin extraction).
• Hiddify/NekoBox/Streisand/Shadowrocket: paste URI. sing-box (SFA/SFI, ≥1.12): JSON profile with hysteria2 outbound, password: "<user>:<pw>", embedded tls.certificate PEM (edge.crt), TUN inbound, DoH via outbound, hijack-dns, private + ENTRY IP → direct.

Recommended client apps

Protocol: Hysteria 2 (QUIC/UDP 443). Server cert is self-signed, so the app must support pinSHA256/custom certificate — all apps below do via the URI/QR or sing-box JSON.

Platform	App	Import	Notes
Android	Hiddify (primary)	hy2:// URI or QR	sing-box core, pin support, simplest
Android	NekoBox for Android	hy2:// URI or QR	alternative, more knobs
Android	SFA (sing-box)	JSON profile only	official sing-box client; use profiles/singbox/*.json
iOS	Streisand (primary)	hy2:// URI or QR	free
iOS	Shadowrocket	hy2:// URI or QR	paid, stable
iOS	SFI (sing-box) / Hiddify	JSON profile / URI	sing-box >= 1.12 required for the JSON
Desktop	Hiddify, sing-box CLI, hysteria client	URI / JSON / YAML	sing-box needs root/admin for TUN

Required client settings (otherwise many sites fail or hang):

• IPv6 off (Hiddify: IPv6 mode = Disable / IPv4 only). EXIT has no IPv6; apps sending IPv6 literals get no IPv4 address available.
• Remote DNS via the tunnel (DoH https://1.1.1.1/dns-query); do not leak DNS outside.
• QUIC blocking is not needed (UDP is carried end to end).
• Verify: ifconfig.me shows the EXIT IP (85.234.86.140); test-ipv6.com shows no IPv6.

App availability/names vary by store and region and change over time; this list comes from the project's client testing (Hiddify confirmed working on a phone) and general knowledge, not a current store check. One profile per person.

Profiles (20 client users)

profiles/ (dir 700, files 600; contains secrets): hy2/clientNN.txt (hy2:// URI), singbox/clientNN.json (sing-box >=1.12), qr/<user>.png (QR of the hy2:// URI, 22 files incl. laptop/mobile; made with qrencode -t PNG -s 8 -m 2 -l M; decoding verified), ALL-hy2.txt, INDEX.md (no passwords). Mobile apps: scan QR -> imports hy2:// profile. Users client01..client20 live in edge.yaml auth.userpass (backup edge.yaml.bak4; laptop, mobile kept). Wrong password -> client error authentication error, HTTP status code: 404. Add/revoke a user: edit auth.userpass on ENTRY, rc-service hysteria-edge restart (all sessions reconnect within seconds), then regenerate/delete the profile files. Verified login for client01/10/20 via temp client (egress = EXIT IP).

Verification checklist

su -s /bin/sh hyedge -c 'curl -4 -s https://ifconfig.me'            # = EXIT ip (TCP via tun)
su -s /bin/sh hyedge -c 'nslookup example.com 1.1.1.1'              # UDP via tun
ip route del default dev hytun table 100; <same curl must FAIL>; ip route replace default dev hytun metric 10 table 100
hysteria client (temp, socks 127.0.0.1:1081) -> curl -x socks5h://... https://ifconfig.me  # = EXIT ip; http://10.0.0.1 rejected

Logs: ENTRY /var/log/hysteria.log, /var/log/hysteria-edge.log; EXIT journalctl -u hysteria-server.

Tuning / diagnostics learned

• Path MTU ENTRY↔EXIT = 1500 (DF ping + tracepath); hytun mtu 1400 is safe; MTU is not the cause of site failures here.
• Capacity test: on EXIT temporarily add speedTest: true to config.yaml, restart, run hysteria speedtest -c /etc/hysteria/client.yaml on ENTRY, then remove it.
• su hyedge -c curl <domain> is ~4s slower (system resolver = provider DNS, unreachable via EXIT). Not a client issue: hysteria-edge uses its own resolver (1.1.1.1). Use --resolve or -x socks5h when benchmarking.
• Clients that connect to IPv6 literals get no IPv4 address available (EXIT has no IPv6). Fix on the client: IPv6 off / IPv4-only DNS strategy (Hiddify: IPv6 mode = disable). Server cannot fix it without EXIT IPv6.
• UdpRcvbufErrors rises at ~190 Mbps (quic-go requests 7 MB socket buffers; rmem_max already 7.5 MB) — cosmetic.

Hysteria diagnostics (manual)

Version: v2.12.3. Replace IPs/paths if the layout differs. On ENTRY run as root.

Service state

# ENTRY (OpenRC)
for s in hysteria hysteria-route hysteria-edge; do rc-service $s status; done
ss -lunp | grep -E ':443\b'                       # edge listener (hysteria-edge)
ss -unp | grep hysteria                            # client UDP socket to EXIT
tail -f /var/log/hysteria.log /var/log/hysteria-edge.log
# EXIT (systemd)
systemctl status hysteria-server --no-pager; ss -lunp | grep :443
journalctl -u hysteria-server -f                   # "client connected" / "TCP error"

Tunnel (ENTRY -> EXIT)

hysteria ping -c /etc/hysteria/client.yaml 1.1.1.1:443   # RTT of a connect via the tunnel
hysteria speedtest -c /etc/hysteria/client.yaml          # needs `speedTest: true` on EXIT (remove afterwards)
ip -br a show hytun; ip rule | grep -E '^10[01]'; ip route show table 100
su -s /bin/sh hyedge -c 'curl -4 -s https://ifconfig.me'   # must print the EXIT IP
tcpdump -ni hytun -c 20                                   # traffic entering the tun
tcpdump -ni eth0 'udp port 443 and host 85.234.86.140'    # QUIC to EXIT (on EXIT: enp3s0, host 213.226.125.13)

• Client log connected to server ... count: N – N grows on each reconnect. TUN UDP error ... EOF right after an EXIT restart is normal.
• rc-service hysteria status = crashed and connect error: timeout -> UDP 443 path/Security Group problem (see prerequisites).

Clients (Traffic Stats API on edge, enabled, 127.0.0.1 only)

Config block in edge.yaml: trafficStats: {listen: 127.0.0.1:9999, secret: <in /etc/hysteria/.stats-secret, 600>}.

S=$(cat /etc/hysteria/.stats-secret); A="Authorization: $S"; U=http://127.0.0.1:9999
curl -s -H "$A" $U/online                       # {"mobile":1}  devices per user
curl -s -H "$A" $U/traffic                      # {"mobile":{"tx":..,"rx":..}}  add ?clear=1 to reset counters
curl -s -H "$A" -H 'Accept: text/plain' $U/dump/streams   # live streams: user, bytes, lifetime, req-addr
curl -s -H "$A" -X POST -d '["mobile"]' $U/kick           # drop a user's sessions (they reconnect if creds still valid)

Without the header the API returns 401. To block a user permanently remove it from auth.userpass and restart hysteria-edge.

Useful subcommands

hysteria share -c /etc/hysteria/client.yaml     # hy2:// URI for a client config (prints the password!)
hysteria check-update                           # compare with the latest release
hysteria version

Error -> meaning

Log text	Meaning
no IPv4 address available (edge)	client asked for an IPv6 literal; EXIT has no IPv6 -> disable IPv6 on the client
rejected (edge)	ACL hit (private ranges)
resolve error ... deadline exceeded (edge)	edge resolver unreachable through the tunnel
no certificate matches the pinned hash	wrong pinSHA256 (strip #name from the URI fragment)
authentication failed / 404 on auth	wrong user:password
curl via tun hangs, tcpdump -i hytun shows SYN-ACK from dst on hytun	strict rp_filter -> set =2

Known issues / notes

• Backups on ENTRY: /etc/hysteria/*.bak*. Remote shell: zsh does not word-split variables → use functions for ssh wrappers.
• ENTRY reboot tested (2026-09-30): all three services, hytun, rules, sysctl and tunnel came up by themselves; mobile client reconnected. EXIT reboot tested too: hysteria-server and sysctl came up, ENTRY client reconnected automatically (count 2).
• strongSwan/IKE to Gcore (62.112.222.168) is a separate, unresolved task (needs UDP 500/4500 reachability); charon started manually, not enabled at boot.
• SSH on ENTRY is key-only since 2026-09-30 (a cloud-init drop-in used to re-enable passwords; hardening lives in /etc/ssh/sshd_config.d/00-hardening.conf). A full ENTRY reboot test after the OpenVPN Monitor changes passed (2026-09-30, see REBOOT-TEST.md).