docs/ADMIN_CLEANUP.md: what is in the control-api database and what must
not be touched, preparation (stop, backup, checks), ready SQL for a full
reset before a new run, the event log, the check registry, single
addresses and compaction, verification after the cleanup, restore from a
backup, and what to do through the API instead. Every SQL block was run
on a copy of the production backup. Linked from the README.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A mass check on 2026-10-02 stalled 7 of 20 validators and sent 42
addresses to fail without a single check. A validator busy with slow
checks went silent, was marked unreachable, and its next heartbeat put it
back to idle while it still held the address; it was handed a second one,
whose association never ran (the in-flight guard was keyed by validator),
and both waited for their leases to expire.
- Heartbeat/re-register return an unreachable validator to assigned when
it still holds an address, else idle.
- A validator is released only from the address it currently holds
(ReleaseFIP, RequeueOrFail, MarkFIPOccupied, FreeValidator); an
unreachable validator stays unreachable until its next heartbeat, so a
dead validator is no longer handed a new address every lease period.
- ClaimNextQueued refuses a validator that still has an address; a
ReconcileValidators pass on every tick repairs rows that disagree with
the queue.
- Association guard is keyed by address, not validator.
- The agent sends heartbeats from their own goroutine.
- Clear queue / delete: detach only floating IPs of unfinished rows (done,
failed and occupied rows kept their fip_id and made a clear issue >1000
sequential cloud calls: 256 s), at most 8 in parallel; the operation no
longer dies with the client connection (10 minute limit).
Includes the incident analysis and the plan under analysis/ and
docs/changes/, and rebuilt bin/control-api and bin/validator-agent.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Run from the jump host: on each validator it updates the git clone in
/opt/cloud-ip-validator, builds the image there, stops and removes the
current container and starts a new one from the new image. Run
parameters live in an env file (deploy/ansible/env/validator-agent.env,
git-ignored, template committed).
The image is built before the running container is touched, so a failed
build leaves the old container running. Hosts are updated in waves
(1, 4, rest) and any failure stops the run. validator_id comes from the
inventory and is checked against the running container before it is
replaced. Only ansible.builtin modules are used, so the validators need
no extra packages.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
control-api is hosted outside the cloud and validators reach it directly,
so it sees the floating IP as the connection's source address. New open
route GET /api/v1/agents/{id}/observed-ip returns that address (taken only
from the TCP peer; forwarding headers are ignored so a validator cannot
forge it).
The agent gets self_check.methods, a priority-ordered list of ip_echo
(unchanged) and control_api; the default stays [ip_echo]. The self-check
passes when any method confirms the address; the next method is tried on
no answer and on a mismatch. Each method has its own timeout so a hung
first method cannot starve the fallback, and control_api uses a new TCP
connection per call (a connection opened before the floating IP was
attached would keep reporting the old address).
Also: docker agent template/env, example config, docs, plan in
docs/changes, e2e script switch E2E_SELF_CHECK_METHODS, rebuilt
bin/control-api and bin/validator-agent.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The "Scan Floating IP" button failed with a client timeout: the project now
holds ~6.4k floating IPs and the scan listed them all in one unpaginated,
timeout-less Neutron request on the HTTP request context.
openstack: ListFreeFloatingIPs reads marker-based pages (fields= keeps them
small) with per-page retry/backoff on transport errors, 5xx and 429, and every
request now has a timeout (also ends hangs inside the orchestrator tick).
orchestrator: the scan is a single-flight background job on the process
context with progress (clearing/listing/enqueuing/done/error), dry_run, full
discovery before anything is enqueued, then SubmitIPs in chunks of 500 in
ascending IP order; a failed read leaves the queue untouched. The auto-cycle
gets a "scanning" phase that polls the job, so the control loop and
autoCycleMu are never held across OpenStack/DB work; it recovers after a
restart and waits for (instead of adopting) a scan started by someone else.
db: migration 0009 (indexes), paged ListIPsPage/ListRegistryPage, GROUP BY
counters, EXISTS completion check, set-based ClearAllIPs.
API: POST /admin/ips/scan -> 202 (dry_run, wait), GET /admin/ips/scan, paging
and filters on /admin/ips and /admin/registry (bare arrays without limit),
results_by_overall in /admin/status.
dashboard: scan progress panel and dry-run button, paginated /ips and
/registry with server-side filters, Overview on counters and capped lists
with progress/ETA, "select all N by filter", hx-params fix for per-row
buttons, real counts in confirmations.
Also: docs (API, USAGE, DASHBOARD, README), plan and review under
docs/changes/, bin/ rebuilt with new SHA256SUMS.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
control-api: every route now carries a mandatory access level (admin / agent /
open) in a route table. All /api/v1/admin/* require the admin token; the
write calls of validator-agent and prober (self-check, events, results,
complete) require a separate static agent token; register, heartbeat and
fetching the assignment stay open. Tokens come from env vars, are compared in
constant time and never logged. An empty token leaves that level open with a
startup warning (backward compatible).
validator-agent / prober: apiclient sends the agent token only to control-api.
admin-dashboard: login/password (from env) with a stateless HMAC session
cookie, Origin-based CSRF check, per-IP brute-force throttle, HX-Redirect for
htmx polls, logout in the sidebar; the dashboard calls control-api with the
admin token. Login page layout fixed after review.
Also: env plumbing in docker-compose/rxprod-compose/systemd/config examples,
e2e script with token assertions, tests, docs (API, SETUP, USAGE, DASHBOARD,
README), plan and review under docs/changes/, bin/ rebuilt with new
SHA256SUMS.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Follow the section layout used in the ipam_control project: quick start,
configuration tables, architecture with a directory tree, data-model rules
(queue states, results, registry, auto-cycle), API table, security notes,
operations, dashboard pages, tests, docs index and change history. Details
stay in docs/ and are linked rather than duplicated.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An admin-controlled scenario that repeats what the operator does by hand:
clear the IP queue, scan and enqueue all free Floating IPs, wait until every
queued address reaches a terminal state (so results are in the Registry),
then wait a configurable interval and start over.
- control-api: new auto_cycle singleton table (migration 0008) holding
enabled/interval/max-run settings and persisted phase state, so the cycle
survives restarts; engine in orchestrator/autocycle.go driven from the
existing loop tick with an injectable "now" for deterministic tests.
- Interval (default 1h, min 60s) and max wait (default unlimited, timeout
outcome) are runtime settings, never hardcoded.
- The periodic fip_scan_interval_seconds scan is skipped while the cycle is
enabled. An emptied queue mid-cycle counts as finished; stopping during
the pause keeps the last cycle's outcome.
- API: GET/PUT /api/v1/admin/auto-cycle, POST .../start, POST .../stop.
- admin-dashboard: "Автоматический цикл" panel on /settings and an
"Автоцикл активен" indicator on /overview.
- Tests for db, orchestrator, httpapi and dashboard; run-local-e2e.sh now
exercises a full auto cycle; docs updated; bin/ rebuilt with refreshed
SHA256SUMS.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replaced the duplicated Docker walkthrough (now fully covered in
docs/SETUP.md) with a single link, and turned the prose description of
each component into an explicit role/state/host table so the architecture
reads at a glance. Cuts the file roughly in half.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
deploy/docker/RUN.txt only documented docker build/run per component with
no explanation of docker-compose, multi-host profiles, or how the images
relate to bin/*. Adds a full "Развёртывание в Docker" section to SETUP.md
covering requirements, single-host and multi-host docker-compose flows,
per-component docker build/run (absorbing RUN.txt's content with more
context), rebuilding images after code changes, and troubleshooting.
README.md now points here as the primary source, with RUN.txt kept as a
quick command reference.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standalone HTML page (docs/CONTROL_DATA_PLANE.html) showing the
docker-compose deployment topology (hosts, COMPOSE_PROFILES) and the
egress/inbound check traffic, refined through several presentation
review passes. Linked from README.md's docs table and docs/DIAGRAMS.md
alongside the existing Mermaid diagrams.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeVbMVEiE7XQAkBd7HQgj6
control-api was the only component deployable via a bare Dockerfile alone
(no entrypoint/template, no default mountable config), and there was no
compose file wiring the four services' network, healthcheck ordering, or
DB volume together. Adds a base docker-compose.yml plus dev (override,
auto-loaded) and prod overlays, split by Compose profiles matching the
real deployment topology (control-plane/dashboard/prober/validator), a
ready-to-copy mock config for control-api so `docker compose up` works
out of the box, and the repo's first .gitignore for the local env/config
copies developers create from the committed examples.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeVbMVEiE7XQAkBd7HQgj6
ayurishchevandClaude Sonnet 5.5