repo init

docs/LOCAL_E2E.md

@@ -0,0 +1,92 @@
+# Local end-to-end smoke test
+
+`scripts/run-local-e2e.sh` runs the full system as local processes with no
+real OpenStack cloud and no real internet access:
+
+- **control-api** with `openstack.mode: mock` — the in-memory
+  `openstack.MockClient` stands in for Neutron, pre-seeded with one
+  synthetic floating IP per configured address (see
+  `cmd/control-api/main.go`'s `newOpenStackClient`).
+- **1 validator-agent** (`validator_01`), started with
+  `-stub-ports 12022,18081,18443,18888` — trivial accept-and-close TCP
+  listeners standing in for the base-minimum services (22/80/443/8080) a
+  real validator would run. ICMP needs no stub: the kernel answers echo
+  requests to any local address (127.0.0.0/8) on its own.
+- **3 probers** (`site-1`/`site-2`/`site-3`), all probing `127.0.0.1`.
+- **3 `httpstub` instances** (`scripts/httpstub`) standing in for the real
+  outbound targets (hub.docker.com / github.com / packages.ubuntu.com),
+  always returning `200 OK`.
+
+The generated config uses a short `lease_ttl_seconds: 8` and
+`checking_window_seconds: 15` so the whole run finishes in well under a
+minute instead of using the (much longer) production defaults.
+
+**Why only one IP address (`127.0.0.1`), and why it must be exactly that
+one:** the self-check mechanism (`GET /whatsmyip`, see
+`internal/httpapi/server.go`'s `remoteIP`) compares the TCP source address
+the validator-agent's own outbound connection to control-api arrives with
+against the address it was just assigned. In a real deployment, Neutron
+actually SNATs the validator's egress traffic through whichever floating
+IP is attached, so any configured address self-checks correctly. This
+offline harness has no real network-level SNAT — the agent's traffic to
+control-api always really originates from `127.0.0.1` — so only that
+literal loopback address can ever pass self-check here. This is a
+limitation of the harness's fidelity, not of the self-check mechanism
+itself.
+
+## Running it
+
+```
+scripts/run-local-e2e.sh
+```
+
+It will:
+
+1. Build all three binaries plus `httpstub` into a temp workdir.
+2. Start the 3 stub HTTP targets, control-api, the validator-agent, and the
+   3 probers.
+3. Wait for `GET /healthz` to come up.
+4. A few seconds in, **kill the validator-agent mid-run** and wait past the
+   8s lease TTL, to demonstrate that control-api's lease sweep reclaims the
+   in-flight IP (moves it back to `queued`, bumps `retry_count`) without
+   any special crash-recovery code — it's the same sweep that runs every
+   tick. It then restarts the validator-agent so the queue can finish.
+5. Poll `GET /api/v1/admin/status` until every configured IP has reached a
+   terminal state (`done` or `failed`).
+6. Print the final `/api/v1/admin/status` and `/api/v1/admin/ips` output.
+
+Expect to see `127.0.0.1` end with `"state":"done"` and
+`"overall_result":"pass"` (all egress checks against the stub targets
+succeed, and all 3 probers can reach the stub TCP listeners and get ICMP
+replies from loopback).
+
+## Inspecting a run
+
+The workdir (printed at the end, `/tmp/cloud-ip-validator-e2e.XXXXXX`) is
+**not** deleted automatically, so you can inspect:
+
+- `logs/control-api.log`, `logs/validator-agent.log`, `logs/prober-site-*.log`
+- `control-api.db` — open with `sqlite3` to inspect the `checks` and
+  `events` tables directly, e.g.:
+  ```
+  sqlite3 /tmp/cloud-ip-validator-e2e.XXXXXX/control-api.db \
+    "select ip_address, source, check_type, success from checks order by id"
+  ```
+
+## What this does *not* cover
+
+This harness proves the orchestration, HTTP protocol, and check-running
+logic all work together correctly. It does **not** exercise the real
+`internal/openstack/client.go` (gophercloud) path — that only runs against
+`openstack.mode: real` with actual OpenStack credentials. That path has its
+own read-only smoke test, `internal/openstack/client_live_test.go`, skipped
+by default and gated behind `OPENSTACK_LIVE_TEST=1`:
+
+```
+OPENSTACK_LIVE_TEST=1 \
+OS_AUTH_URL=https://keystone.example:5000/v3 \
+OS_TOKEN=... \
+OS_PROJECT_ID=... \
+OS_TEST_FLOATING_IP=203.0.113.10 \
+go test ./internal/openstack/... -run TestClientLive -v
+```