Stage B of review fixes: daemon state lock, image build check, RIPEstat retries

Job state is changed under one RLock, the Dockerfile copies all root
modules and imports them at build time, RIPEstat requests go through a
retrying session with the sourceapp parameter (ripestat_sourceapp) and a
capped Retry-After. Adds the summary and marks review findings 5-10 fixed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
ayurishchevandClaude Sonnet 5 committed 2026-09-21 10:12:48 +03:00
1 parent d6b69d0842
commit aaf41adc79
8 files changed
+164 -28

No files matched your search

+3 -1
View File
@@ -45,6 +45,7 @@ The system consists of **two independent processes**: the collector daemon (`col
}
```
`ttl_days` - how long an address is kept after it was last seen (default `90`, `0` = keep forever).
Optional `ripestat_sourceapp` (default `ripe-cidr-collector`) - the application name sent to RIPEstat as `sourceapp`; RIPEstat asks clients to identify themselves, you can add a contact (`my-collector admin@example.org`).
Optional `allow_non_global_ips` (default `false`) - by default only public addresses resolved from FQDNs are stored; loopback, private, link-local and unspecified addresses (`127.0.0.1`, `10.x`, `0.0.0.0`, ...) are ignored and logged. Set `true` for internal names.
Optional `backup_keep` - how many database backups to keep (default `7`); the backup cron is `schedule.backup` (default `30 4 * * *`), see section 9.
Optional `changes_retention_days` - how long the change journal for `/addresses/diff` is kept (default `30`, `0` = forever).
@@ -423,7 +424,7 @@ python3 cidr_collector.py run --mode fqdn
When the collector runs (whether manually or via schedule):
1. **Instantiation**: Creates a new instance of `CIDRCollector` or `FQDNCollector`. This forces a fresh read of `config.json`, ensuring any added ASNs/FQDNs are immediately processed.
2. **Fetching**:
* **ASN**: Queries RIPE NCC API (`stat.ripe.net`).
* **ASN**: Queries RIPE NCC API (`stat.ripe.net`) with the `sourceapp` parameter. Connection failures and codes 429/500/502/503/504 are retried up to 3 times with growing pauses (0, 2, 4 s; a `Retry-After` pause is capped at 30 s), 10 s per attempt - at worst about 50 s per ASN. If all attempts fail the ASN is skipped for this run and nothing is deleted.
* **FQDN**: Uses Python's `socket.getaddrinfo` to resolve A and AAAA records. Non-public addresses are dropped (see `allow_non_global_ips`); if nothing is left the run is treated like a DNS failure (nothing is deleted).
3. **Merge in one transaction**: the fetched addresses are merged into the SQLite table `addresses` (`db.py`) in a single write transaction, so readers (the API) never see a half-updated state.
4. **Accumulation with TTL**: Each address has `first_seen`/`last_seen`.
@@ -483,6 +484,7 @@ docker compose down # stop; data stays in the volume
### Notes and risks
- **Port 8000 is published without TLS**, so `X-API-Key` travels in clear text. Restrict access with a firewall or put a TLS reverse proxy in front (bind the port to `127.0.0.1` by changing `ports` in `docker-compose.yml`).
- The image copies all `*.py` modules from the project root and imports the main ones during the build, so a missing or broken module fails the build instead of the start.
- Dependencies are pinned (see "Dependency versions" below), so rebuilds give the same libraries. The base image `python:3.11-slim` is not pinned by digest: Python patch releases arrive on rebuild.
- Files written by the services in the volume (`config.json`, `status.json`) have mode `600`; both services run as the same user.
- `docker compose` uses the image name `ripe-cidr-collector`; `Dockerfile.test` is used only for running the tests (section 1, step 5).