OpenVPN-Monitoring-Simple/DOCS/Operations/Hysteria_Chain_Manifest.md

# Hysteria 2 entry→exit chain (manifest)

Client ─hy2/UDP443→ **ENTRY** (Alpine, OpenRC) ─hy2/UDP443→ **EXIT** (Debian, systemd) → Internet (IPv4)

Secrets live only in: `client-uris.txt`, `singbox-*.json`, `/etc/hysteria/*` on hosts (chmod 600). Never commit/print them.

## Inventory (this deployment)
| Role | Host | OS | Access |
|---|---|---|---|
| ENTRY (Timeweb) | 213.226.125.13 | Alpine 3.23 | `ssh -i private.key root@` (key must be chmod 600) |
| EXIT (Gcore Hel) | 85.234.86.140 | Debian 13 | `ssh -i hpriv.key debian@` (sudo NOPASSWD) |

## Prerequisites (verify first, they were the real blockers)
1. **UDP 443 open both ways** (Security Group on EXIT; ENTRY provider). Test: `tcpdump -ni <if> udp port 443` on the receiver while `echo x | nc -u -w1 <ip> 443` from the sender. ICMP/TCP passing proves nothing about UDP.
2. Use **official** binary `apernet/hysteria` (HyNetworks/hysteria is a fork). Verify `sha256sum` against `hashes.txt` of the release:
   `https://github.com/apernet/hysteria/releases/latest/download/{hysteria-linux-amd64,hashes.txt}`
3. Hysteria server `socks5`/`http` outbounds are **TCP-only** → for UDP use TUN + policy routing (below), not a SOCKS5 chain.

## EXIT (Debian) — server
- `/usr/local/bin/hysteria`, `/etc/hysteria/{server.crt,server.key(600),config.yaml}`
- Self-signed cert: `openssl req -x509 -nodes -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -days 3650 -subj /CN=<name> -keyout server.key -out server.crt`
- `config.yaml`: `listen: :443`, `tls{cert,key}`, `auth{type: password, password: <rand>}`, `masquerade{type: proxy, proxy{url: https://news.ycombinator.com/, rewriteHost: true}}`
- systemd unit `hysteria-server` (`ExecStart=hysteria server -c ...`, `Restart=always`, `AmbientCapabilities=CAP_NET_BIND_SERVICE`)
- `/etc/sysctl.d/99-quic.conf`: `net.core.rmem_max=7500000`, `net.core.wmem_max=7500000`
- No IPv6 uplink here → IPv4 only.

## ENTRY (Alpine) — tunnel client + client-facing server
Packages: `iproute2 libcap-utils openssl`; user `hyedge` (`adduser -S -D -H -s /sbin/nologin`); `/dev/net/tun` must exist and module `tun` must be in `/etc/modules` (loaded module, not builtin; without it the tunnel does not come up after reboot).

1. **`/etc/hysteria/client.yaml`** (600): `server: <EXIT>:443`, `auth: <pw>`, `tls{sni, insecure: true, pinSHA256: <EXIT cert fp>}`, `lazy: true`, `bandwidth{up: 180 mbps, down: 180 mbps}` (Brutal CC; measured path ≈191-195 Mbps via `hysteria speedtest`, single-flow without it was ~44 Mbps), `socks5{127.0.0.1:1080}`, `http{127.0.0.1:8080}`, `tun{name: hytun, mtu: 1400, timeout: 5m, address{ipv4: 100.100.100.101/30}}` (no `route:` → no auto-routes).
2. **`/etc/hysteria/edge.yaml`** (600, owner hyedge): `listen: :443`, own self-signed `edge.crt/edge.key`, `auth{type: userpass, userpass{<dev>: <pw>}}`, `resolver{type: udp, udp{addr: 1.1.1.1:53}}`, `outbounds[direct{mode: "4"}]`, `trafficStats{listen: 127.0.0.1:9999, secret}`, `acl.inline`: `reject` 10/8,172.16/12,192.168/16,127/8,169.254/16,100.64/10 then `direct(all)`.
   - Do **not** add `reject(::/0)` — it matches any domain having AAAA and rejects dual-stack sites.
   - DoH resolver timed out via tun; UDP resolver works.
3. Binary copy for low port: `cp hysteria hysteria-edge; setcap cap_net_bind_service=+ep /usr/local/bin/hysteria-edge`.
4. **OpenRC** (`supervisor=supervise-daemon`, `respawn_delay=3`, `respawn_max=0`):
   - `hysteria` (client, root) ← `need net`
   - `hysteria-route` ← `need hysteria`; waits for `hytun`, then:
     ```
     ip route replace blackhole default metric 1000 table 100   # anti-leak fallback
     ip route replace default dev hytun metric 10 table 100
     ip rule add priority 100 ipproto udp sport 443 lookup main # edge replies bypass tun
     ip rule add priority 101 uidrange <uid hyedge> lookup 100
     ```
   - `hysteria-edge` (`command_user=hyedge`) ← `need net hysteria-route`
   - `rc-update add hysteria/hysteria-route/hysteria-edge default`
5. **sysctl (critical):** `net.ipv4.conf.all.rp_filter=2` and `default=2` (`/etc/sysctl.d/99-hytun.conf`). With strict `1` the TUN SYN-ACKs are dropped → TCP via tun silently hangs.
6. Only uid `hyedge` traffic and, since 2026-09-30, the OpenVPN client subnet (`172.20.1.0/24`, rule 102, see below) enter the tunnel; SSH/root traffic untouched.

## OpenVPN Monitor & Profiler on ENTRY (added 2026-09-30)
ENTRY also runs the web suite for OpenVPN (native OpenRC services, no containers). OpenVPN clients exit through the same `hytun` tunnel to EXIT.

| Item | Value |
|---|---|
| Panel | `https://213.226.125.13:8088/` (self-signed TLS, login `tstark`, 2FA; password is set by the owner, not stored here) |
| OpenVPN | `udp/1194`, clients `172.20.1.0/24`, full tunnel |
| Services | `ovpmon-api`, `ovpmon-gatherer`, `ovpmon-profiler` (user `ovpmon`), `nginx`, `openvpn`, `ovpn-hytun`, `fail2ban` |
| Egress | `ip rule 102 from 172.20.1.0/24 → table 100` (hytun, blackhole fallback) + MASQUERADE on `hytun` + FORWARD DROP to any other interface (no leak) |
| Privileges | APIs run unprivileged; a root helper (`ovpmon-helper` via `doas`, 6 fixed commands) validates/installs `server.conf`, publishes the CRL and controls `openvpn` |
| Hardening | SSH key-only (`00-hardening.conf`), fail2ban (`sshd`, `sshd-ddos`, `ovpmon-login`), input validation, CORS restricted |
| Docs | [`SUMMARY-ovpn-monitor.md`](SUMMARY-ovpn-monitor.md) (state, results, security assessment, rollback), `IMPLEMENTATION-01/02-*.md`, `PLAN-validation-and-privilege-drop.md`; application repo `DOCS/` |

Check: `rc-status | grep -E "ovpmon|openvpn|ovpn-hytun|nginx|fail2ban"`, `ip rule | grep 102`, a VPN client on `https://ifconfig.me` shows the EXIT IP (85.234.86.140).

## Clients
- URI: `hy2://<user>:<pw>@<ENTRY>:443/?sni=<edge sni>&insecure=1&pinSHA256=<hex, no colons>#name` (strip `#name` when scripting pin extraction).
- Hiddify/NekoBox/Streisand/Shadowrocket: paste URI. sing-box (SFA/SFI, ≥1.12): JSON profile with `hysteria2` outbound, `password: "<user>:<pw>"`, embedded `tls.certificate` PEM (edge.crt), TUN inbound, DoH via outbound, `hijack-dns`, private + ENTRY IP → direct.

### Recommended client apps
Protocol: Hysteria 2 (QUIC/UDP 443). Server cert is self-signed, so the app must support `pinSHA256`/custom certificate — all apps below do via the URI/QR or sing-box JSON.

| Platform | App | Import | Notes |
|---|---|---|---|
| Android | **Hiddify** (primary) | hy2:// URI or QR | sing-box core, pin support, simplest |
| Android | NekoBox for Android | hy2:// URI or QR | alternative, more knobs |
| Android | SFA (sing-box) | JSON profile only | official sing-box client; use `profiles/singbox/*.json` |
| iOS | **Streisand** (primary) | hy2:// URI or QR | free |
| iOS | Shadowrocket | hy2:// URI or QR | paid, stable |
| iOS | SFI (sing-box) / Hiddify | JSON profile / URI | sing-box >= 1.12 required for the JSON |
| Desktop | Hiddify, sing-box CLI, `hysteria client` | URI / JSON / YAML | sing-box needs root/admin for TUN |

Required client settings (otherwise many sites fail or hang):
- **IPv6 off** (Hiddify: IPv6 mode = Disable / IPv4 only). EXIT has no IPv6; apps sending IPv6 literals get `no IPv4 address available`.
- Remote DNS via the tunnel (DoH `https://1.1.1.1/dns-query`); do not leak DNS outside.
- QUIC blocking is **not** needed (UDP is carried end to end).
- Verify: `ifconfig.me` shows the EXIT IP (85.234.86.140); `test-ipv6.com` shows no IPv6.

App availability/names vary by store and region and change over time; this list comes from the project's client testing (Hiddify confirmed working on a phone) and general knowledge, not a current store check. One profile per person.

## Profiles (20 client users)
`profiles/` (dir 700, files 600; contains secrets): `hy2/clientNN.txt` (hy2:// URI), `singbox/clientNN.json` (sing-box >=1.12), `qr/<user>.png` (QR of the hy2:// URI, 22 files incl. laptop/mobile; made with `qrencode -t PNG -s 8 -m 2 -l M`; decoding verified), `ALL-hy2.txt`, `INDEX.md` (no passwords). Mobile apps: scan QR -> imports hy2:// profile. Users `client01..client20` live in `edge.yaml` `auth.userpass` (backup `edge.yaml.bak4`; `laptop`, `mobile` kept). Wrong password -> client error `authentication error, HTTP status code: 404`.
Add/revoke a user: edit `auth.userpass` on ENTRY, `rc-service hysteria-edge restart` (all sessions reconnect within seconds), then regenerate/delete the profile files. Verified login for client01/10/20 via temp client (egress = EXIT IP).

## Verification checklist
```
su -s /bin/sh hyedge -c 'curl -4 -s https://ifconfig.me'            # = EXIT ip (TCP via tun)
su -s /bin/sh hyedge -c 'nslookup example.com 1.1.1.1'              # UDP via tun
ip route del default dev hytun table 100; <same curl must FAIL>; ip route replace default dev hytun metric 10 table 100
hysteria client (temp, socks 127.0.0.1:1081) -> curl -x socks5h://... https://ifconfig.me  # = EXIT ip; http://10.0.0.1 rejected
```
Logs: ENTRY `/var/log/hysteria.log`, `/var/log/hysteria-edge.log`; EXIT `journalctl -u hysteria-server`.

## Tuning / diagnostics learned
- Path MTU ENTRY↔EXIT = 1500 (DF ping + tracepath); hytun mtu 1400 is safe; MTU is **not** the cause of site failures here.
- Capacity test: on EXIT temporarily add `speedTest: true` to config.yaml, restart, run `hysteria speedtest -c /etc/hysteria/client.yaml` on ENTRY, then remove it.
- `su hyedge -c curl <domain>` is ~4s slower (system resolver = provider DNS, unreachable via EXIT). Not a client issue: hysteria-edge uses its own `resolver` (1.1.1.1). Use `--resolve` or `-x socks5h` when benchmarking.
- Clients that connect to **IPv6 literals** get `no IPv4 address available` (EXIT has no IPv6). Fix on the client: IPv6 off / IPv4-only DNS strategy (Hiddify: IPv6 mode = disable). Server cannot fix it without EXIT IPv6.
- `UdpRcvbufErrors` rises at ~190 Mbps (quic-go requests 7 MB socket buffers; rmem_max already 7.5 MB) — cosmetic.

## Hysteria diagnostics (manual)
Version: v2.12.3. Replace IPs/paths if the layout differs. On ENTRY run as root.

### Service state
```
# ENTRY (OpenRC)
for s in hysteria hysteria-route hysteria-edge; do rc-service $s status; done
ss -lunp | grep -E ':443\b'                       # edge listener (hysteria-edge)
ss -unp | grep hysteria                            # client UDP socket to EXIT
tail -f /var/log/hysteria.log /var/log/hysteria-edge.log
# EXIT (systemd)
systemctl status hysteria-server --no-pager; ss -lunp | grep :443
journalctl -u hysteria-server -f                   # "client connected" / "TCP error"
```

### Tunnel (ENTRY -> EXIT)
```
hysteria ping -c /etc/hysteria/client.yaml 1.1.1.1:443   # RTT of a connect via the tunnel
hysteria speedtest -c /etc/hysteria/client.yaml          # needs `speedTest: true` on EXIT (remove afterwards)
ip -br a show hytun; ip rule | grep -E '^10[01]'; ip route show table 100
su -s /bin/sh hyedge -c 'curl -4 -s https://ifconfig.me'   # must print the EXIT IP
tcpdump -ni hytun -c 20                                   # traffic entering the tun
tcpdump -ni eth0 'udp port 443 and host 85.234.86.140'    # QUIC to EXIT (on EXIT: enp3s0, host 213.226.125.13)
```
- Client log `connected to server ... count: N` – N grows on each reconnect. `TUN UDP error ... EOF` right after an EXIT restart is normal.
- `rc-service hysteria status` = `crashed` and `connect error: timeout` -> UDP 443 path/Security Group problem (see prerequisites).

### Clients (Traffic Stats API on edge, enabled, 127.0.0.1 only)
Config block in `edge.yaml`: `trafficStats: {listen: 127.0.0.1:9999, secret: <in /etc/hysteria/.stats-secret, 600>}`.
```
S=$(cat /etc/hysteria/.stats-secret); A="Authorization: $S"; U=http://127.0.0.1:9999
curl -s -H "$A" $U/online                       # {"mobile":1}  devices per user
curl -s -H "$A" $U/traffic                      # {"mobile":{"tx":..,"rx":..}}  add ?clear=1 to reset counters
curl -s -H "$A" -H 'Accept: text/plain' $U/dump/streams   # live streams: user, bytes, lifetime, req-addr
curl -s -H "$A" -X POST -d '["mobile"]' $U/kick           # drop a user's sessions (they reconnect if creds still valid)
```
Without the header the API returns 401. To block a user permanently remove it from `auth.userpass` and restart `hysteria-edge`.

### Useful subcommands
```
hysteria share -c /etc/hysteria/client.yaml     # hy2:// URI for a client config (prints the password!)
hysteria check-update                           # compare with the latest release
hysteria version
```

### Error -> meaning
| Log text | Meaning |
|---|---|
| `no IPv4 address available` (edge) | client asked for an IPv6 literal; EXIT has no IPv6 -> disable IPv6 on the client |
| `rejected` (edge) | ACL hit (private ranges) |
| `resolve error ... deadline exceeded` (edge) | edge `resolver` unreachable through the tunnel |
| `no certificate matches the pinned hash` | wrong `pinSHA256` (strip `#name` from the URI fragment) |
| `authentication failed` / 404 on auth | wrong `user:password` |
| curl via tun hangs, `tcpdump -i hytun` shows SYN-ACK from dst on hytun | strict `rp_filter` -> set `=2` |

## Known issues / notes
- Backups on ENTRY: `/etc/hysteria/*.bak*`. Remote shell: zsh does not word-split variables → use functions for ssh wrappers.
- ENTRY reboot tested (2026-09-30): all three services, hytun, rules, sysctl and tunnel came up by themselves; mobile client reconnected. EXIT reboot tested too: hysteria-server and sysctl came up, ENTRY client reconnected automatically (count 2).
- strongSwan/IKE to Gcore (62.112.222.168) is a separate, unresolved task (needs UDP 500/4500 reachability); `charon` started manually, not enabled at boot.
- SSH on ENTRY is key-only since 2026-09-30 (a cloud-init drop-in used to re-enable passwords; hardening lives in `/etc/ssh/sshd_config.d/00-hardening.conf`). A full ENTRY reboot test after the OpenVPN Monitor changes passed (2026-09-30, see `REBOOT-TEST.md`).