Harden input handling and filter non-global DNS addresses
X-API-Key compared as bytes (401 instead of 500 on non-ASCII), strict parsing of the diff "since" parameter (400 instead of 500), FQDN resolution keeps only global addresses (allow_non_global_ips to opt out). Adds the code review report and the plan/summary for this change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
1 parent
02f7b49e12
commit
46d56654d8
8 files changed
+168
-9
No files matched your search
@@ -45,6 +45,7 @@ The system consists of **two independent processes**: the collector daemon (`col
|
||||
}
|
||||
```
|
||||
`ttl_days` - how long an address is kept after it was last seen (default `90`, `0` = keep forever).
|
||||
Optional `allow_non_global_ips` (default `false`) - by default only public addresses resolved from FQDNs are stored; loopback, private, link-local and unspecified addresses (`127.0.0.1`, `10.x`, `0.0.0.0`, ...) are ignored and logged. Set `true` for internal names.
|
||||
Optional `backup_keep` - how many database backups to keep (default `7`); the backup cron is `schedule.backup` (default `30 4 * * *`), see section 9.
|
||||
Optional `changes_retention_days` - how long the change journal for `/addresses/diff` is kept (default `30`, `0` = forever).
|
||||
|
||||
@@ -337,7 +338,7 @@ curl "http://localhost:8000/addresses/diff?since=0"
|
||||
# {"since":"0","now":"2026-09-21T04:09:44Z","cursor":42,"added":["3.0.0.0/24"],"removed":["2.0.0.0/24"]}
|
||||
curl "http://localhost:8000/addresses/diff?since=42" # next sync: use the returned cursor
|
||||
```
|
||||
- `since` is a **cursor** from the previous answer (recommended: exact, independent of clocks) or an ISO 8601 time (no time zone = UTC; write `+` in a URL as `%2B`). Time is compared with millisecond precision, borders are inclusive, so an entry may be reported twice - repeating it is harmless.
|
||||
- `since` is a **cursor** from the previous answer (recommended: exact, independent of clocks) or an ISO 8601 time (no time zone = UTC; write `+` in a URL as `%2B`). Time is compared with millisecond precision, borders are inclusive, so an entry may be reported twice - repeating it is harmless. A cursor is at most 30 ASCII digits; anything else that is not an ISO 8601 time (including out-of-range dates) is `400`.
|
||||
- The result is the net effect: an address added and removed (or the other way) within the interval is not reported; an address that another source still holds is not reported as removed. Both CIDRs and IPs of FQDNs count. Output is JSON only, no aggregation.
|
||||
- The first sync: take the full `/addresses` list and the cursor from its response header `X-Changes-Cursor` (read before the data), then call `/addresses/diff?since=<that cursor>` regularly.
|
||||
- `400` - invalid `since`; `410 Gone` - `since` is older than the journal (see `changes_retention_days`) or the cursor does not belong to this database: fetch the full `/addresses` list and continue from the new `cursor`.
|
||||
@@ -423,7 +424,7 @@ When the collector runs (whether manually or via schedule):
|
||||
1. **Instantiation**: Creates a new instance of `CIDRCollector` or `FQDNCollector`. This forces a fresh read of `config.json`, ensuring any added ASNs/FQDNs are immediately processed.
|
||||
2. **Fetching**:
|
||||
* **ASN**: Queries RIPE NCC API (`stat.ripe.net`).
|
||||
* **FQDN**: Uses Python's `socket.getaddrinfo` to resolve A and AAAA records.
|
||||
* **FQDN**: Uses Python's `socket.getaddrinfo` to resolve A and AAAA records. Non-public addresses are dropped (see `allow_non_global_ips`); if nothing is left the run is treated like a DNS failure (nothing is deleted).
|
||||
3. **Merge in one transaction**: the fetched addresses are merged into the SQLite table `addresses` (`db.py`) in a single write transaction, so readers (the API) never see a half-updated state.
|
||||
4. **Accumulation with TTL**: Each address has `first_seen`/`last_seen`.
|
||||
* New addresses are inserted; already seen ones get `last_seen` refreshed.
|
||||
|
||||
Reference in new issue
Block a user