# Local end-to-end smoke test

`scripts/run-local-e2e.sh` runs the full system as local processes with no
real OpenStack cloud and no real internet access:

- **control-api** with `openstack.mode: mock` — the in-memory
  `openstack.MockClient` stands in for Neutron, pre-seeded with one
  synthetic floating IP per configured address (see
  `cmd/control-api/main.go`'s `newOpenStackClient`).
- **1 validator-agent** (`validator_01`), started with
  `-stub-ports 22022,28081,28443,28888` — trivial accept-and-close TCP
  listeners standing in for the base-minimum services (22/80/443/8080) a
  real validator would run. ICMP needs no stub: the kernel answers echo
  requests to any local address (127.0.0.0/8) on its own.
- **3 probers** (`site-1`/`site-2`/`site-3`), all probing `127.0.0.1`.
- **3 `httpstub` instances** (`scripts/httpstub`) — `/` always returns
  `200 OK`, standing in for the real outbound egress targets (hub.docker.com
  / github.com / packages.ubuntu.com); `/ip` echoes the caller's remote
  address as plain text, standing in for the external IP-echo service the
  validator-agent's self-check normally queries in production (see
  `internal/agentcore.detectPublicIP` and `config.SelfCheckCfg.IPEchoURLs`).

The generated config uses a short `lease_ttl_seconds: 8` and
`checking_window_seconds: 15` so the whole run finishes in well under a
minute instead of using the (much longer) production defaults.

**Why only one IP address (`127.0.0.1`), and why it must be exactly that
one:** self-check compares the source address the validator-agent's own
request to the IP-echo target arrives with against the address it was
just assigned. In a real deployment that target is a real external
IP-echo service, and Neutron actually SNATs the validator's egress
traffic through whichever floating IP is attached, so any configured
address self-checks correctly. This offline harness points
`self_check.ip_echo_urls` at the local `httpstub`'s `/ip` route instead
(see `validator-agent.yaml` generated by the script) — there's no real
network-level SNAT here, the agent's traffic to that stub always really
originates from `127.0.0.1`, so only that literal loopback address can
ever pass self-check in this harness. This is a limitation of the
harness's fidelity, not of the self-check mechanism itself.

## Running it

```
scripts/run-local-e2e.sh
```

It will:

1. Build all three binaries plus `httpstub` into a temp workdir.
2. Start the 3 stub HTTP targets, control-api, the validator-agent, and the
   3 probers.
3. Wait for `GET /healthz` to come up.
4. A few seconds in, **kill the validator-agent mid-run** and wait past the
   8s lease TTL, to demonstrate that control-api's lease sweep reclaims the
   in-flight IP (moves it back to `queued`, bumps `retry_count`) without
   any special crash-recovery code — it's the same sweep that runs every
   tick. It then restarts the validator-agent so the queue can finish.
5. Poll `GET /api/v1/admin/status` until every configured IP has reached a
   terminal state (`done` or `failed`).
6. Print the final `/api/v1/admin/status` and `/api/v1/admin/ips` output.
7. Force a re-check of the finished address via `POST /api/v1/admin/ips`
   and wait for it to drain (`attempt_number` advances).
8. Exercise the **automatic cycle** (`/api/v1/admin/auto-cycle`): set the
   smallest allowed interval (60s) and a 120s run limit, `start` it, and
   wait for the first cycle to finish. The script then asserts that the
   outcome is `completed`, the phase is `waiting` with `runs_total=1`, and
   that the registry's `total_cycles` for `127.0.0.1` grew (the cycle
   cleared the queue, re-scanned the mock floating IP and re-checked it).
   Finally it `stop`s the cycle and asserts it is `idle`. The second cycle
   (the interval wait) is covered by unit tests, so the script does not
   sit through the 60s pause. The script exits non-zero if any assertion
   fails.

Expect to see `127.0.0.1` end with `"state":"done"` and
`"overall_result":"pass"` (all egress checks against the stub targets
succeed, and all 3 probers can reach the stub TCP listeners and get ICMP
replies from loopback).

## Inspecting a run

The workdir (printed at the end, `/tmp/cloud-ip-validator-e2e.XXXXXX`) is
**not** deleted automatically, so you can inspect:

- `logs/control-api.log`, `logs/validator-agent.log`, `logs/prober-site-*.log`
- `control-api.db` — open with `sqlite3` to inspect the `checks` and
  `events` tables directly, e.g.:
  ```
  sqlite3 /tmp/cloud-ip-validator-e2e.XXXXXX/control-api.db \
    "select ip_address, source, check_type, success from checks order by id"
  ```

## What this does *not* cover

This harness proves the orchestration, HTTP protocol, and check-running
logic all work together correctly. It does **not** exercise the real
`internal/openstack/client.go` (gophercloud) path — that only runs against
`openstack.mode: real` with actual OpenStack credentials. That path has its
own read-only smoke test, `internal/openstack/client_live_test.go`, skipped
by default and gated behind `OPENSTACK_LIVE_TEST=1`:

```
OPENSTACK_LIVE_TEST=1 \
OS_AUTH_URL=https://keystone.example:5000/v3 \
OS_TOKEN=... \
OS_PROJECT_ID=... \
OS_TEST_FLOATING_IP=203.0.113.10 \
go test ./internal/openstack/... -run TestClientLive -v
```