93 lines
4.0 KiB
Markdown
93 lines
4.0 KiB
Markdown
# Local end-to-end smoke test
|
|
|
|
`scripts/run-local-e2e.sh` runs the full system as local processes with no
|
|
real OpenStack cloud and no real internet access:
|
|
|
|
- **control-api** with `openstack.mode: mock` — the in-memory
|
|
`openstack.MockClient` stands in for Neutron, pre-seeded with one
|
|
synthetic floating IP per configured address (see
|
|
`cmd/control-api/main.go`'s `newOpenStackClient`).
|
|
- **1 validator-agent** (`validator_01`), started with
|
|
`-stub-ports 12022,18081,18443,18888` — trivial accept-and-close TCP
|
|
listeners standing in for the base-minimum services (22/80/443/8080) a
|
|
real validator would run. ICMP needs no stub: the kernel answers echo
|
|
requests to any local address (127.0.0.0/8) on its own.
|
|
- **3 probers** (`site-1`/`site-2`/`site-3`), all probing `127.0.0.1`.
|
|
- **3 `httpstub` instances** (`scripts/httpstub`) standing in for the real
|
|
outbound targets (hub.docker.com / github.com / packages.ubuntu.com),
|
|
always returning `200 OK`.
|
|
|
|
The generated config uses a short `lease_ttl_seconds: 8` and
|
|
`checking_window_seconds: 15` so the whole run finishes in well under a
|
|
minute instead of using the (much longer) production defaults.
|
|
|
|
**Why only one IP address (`127.0.0.1`), and why it must be exactly that
|
|
one:** the self-check mechanism (`GET /whatsmyip`, see
|
|
`internal/httpapi/server.go`'s `remoteIP`) compares the TCP source address
|
|
the validator-agent's own outbound connection to control-api arrives with
|
|
against the address it was just assigned. In a real deployment, Neutron
|
|
actually SNATs the validator's egress traffic through whichever floating
|
|
IP is attached, so any configured address self-checks correctly. This
|
|
offline harness has no real network-level SNAT — the agent's traffic to
|
|
control-api always really originates from `127.0.0.1` — so only that
|
|
literal loopback address can ever pass self-check here. This is a
|
|
limitation of the harness's fidelity, not of the self-check mechanism
|
|
itself.
|
|
|
|
## Running it
|
|
|
|
```
|
|
scripts/run-local-e2e.sh
|
|
```
|
|
|
|
It will:
|
|
|
|
1. Build all three binaries plus `httpstub` into a temp workdir.
|
|
2. Start the 3 stub HTTP targets, control-api, the validator-agent, and the
|
|
3 probers.
|
|
3. Wait for `GET /healthz` to come up.
|
|
4. A few seconds in, **kill the validator-agent mid-run** and wait past the
|
|
8s lease TTL, to demonstrate that control-api's lease sweep reclaims the
|
|
in-flight IP (moves it back to `queued`, bumps `retry_count`) without
|
|
any special crash-recovery code — it's the same sweep that runs every
|
|
tick. It then restarts the validator-agent so the queue can finish.
|
|
5. Poll `GET /api/v1/admin/status` until every configured IP has reached a
|
|
terminal state (`done` or `failed`).
|
|
6. Print the final `/api/v1/admin/status` and `/api/v1/admin/ips` output.
|
|
|
|
Expect to see `127.0.0.1` end with `"state":"done"` and
|
|
`"overall_result":"pass"` (all egress checks against the stub targets
|
|
succeed, and all 3 probers can reach the stub TCP listeners and get ICMP
|
|
replies from loopback).
|
|
|
|
## Inspecting a run
|
|
|
|
The workdir (printed at the end, `/tmp/cloud-ip-validator-e2e.XXXXXX`) is
|
|
**not** deleted automatically, so you can inspect:
|
|
|
|
- `logs/control-api.log`, `logs/validator-agent.log`, `logs/prober-site-*.log`
|
|
- `control-api.db` — open with `sqlite3` to inspect the `checks` and
|
|
`events` tables directly, e.g.:
|
|
```
|
|
sqlite3 /tmp/cloud-ip-validator-e2e.XXXXXX/control-api.db \
|
|
"select ip_address, source, check_type, success from checks order by id"
|
|
```
|
|
|
|
## What this does *not* cover
|
|
|
|
This harness proves the orchestration, HTTP protocol, and check-running
|
|
logic all work together correctly. It does **not** exercise the real
|
|
`internal/openstack/client.go` (gophercloud) path — that only runs against
|
|
`openstack.mode: real` with actual OpenStack credentials. That path has its
|
|
own read-only smoke test, `internal/openstack/client_live_test.go`, skipped
|
|
by default and gated behind `OPENSTACK_LIVE_TEST=1`:
|
|
|
|
```
|
|
OPENSTACK_LIVE_TEST=1 \
|
|
OS_AUTH_URL=https://keystone.example:5000/v3 \
|
|
OS_TOKEN=... \
|
|
OS_PROJECT_ID=... \
|
|
OS_TEST_FLOATING_IP=203.0.113.10 \
|
|
go test ./internal/openstack/... -run TestClientLive -v
|
|
```
|