This project collects CIDR prefixes for specified Autonomous Systems (AS) from the RIPE NCC API and resolves IP addresses for specified FQDNs. It accumulates these addresses over time, maintaining a history of discovered prefixes. It also provides a FastAPI-based HTTP interface to retrieve the collected data.
The system consists of **two independent processes**: the collector daemon (`collector_daemon.py`, runs the schedule) and the API server (`api_server.py`, serves data and edits the config). They communicate only through files: the collected addresses live in the SQLite database `ripe.db`, the rest in JSON (`config.json`, `status.json`). See section 9.
> **Quick start:** the whole system (API + collector) runs with `docker compose up -d`, see section 8. Sections 1-4 describe the manual installation (venv + systemd/OpenRC), which remains supported.
> **Upgrading from a version where the scheduler lived in the API:** running only `ripe-api` is no longer enough - without the `ripe-collector` service nothing is collected. `GET /health` reports `collector_alive: false` / `degraded` in that case.
## 1. Preparation and Installation
### Prerequisites
- Python 3.8+
-`pip` and `venv`
### Installation Steps
1.**Clone the repository** (or copy the files) to your desired location, e.g., `/opt/ripe_collector`.
Optional `allow_non_global_ips` (default `false`) - by default only public addresses resolved from FQDNs are stored; loopback, private, link-local and unspecified addresses (`127.0.0.1`, `10.x`, `0.0.0.0`, ...) are ignored and logged. Set `true` for internal names.
The image (`Dockerfile.test`, `python:3.11-slim`) contains only the code and dev dependencies - production data (`ripe.db`, `data.json`, `fqdn_data.json`) is excluded via `.dockerignore`.
6. **Knowledge graph** (optional): `/graphify .` (Claude Code skill) builds `graphify-out/` - `graph.html` (interactive), `GRAPH_REPORT.md`, `graph.json` - from code and `docs/`; refresh with `/graphify . --update` after changes. The directory is excluded from git and from Docker images.
7. **Repository:** `main` branch, remote `origin` = `https://artstore.rxmsk.ru/ayurishchev/ripe-cidr-collector.git`. Committed: code, tests, Docker files, `config.json` (initial sources), `.env.example`, `docs/`. Not committed (`.gitignore`): `venv/`, `.env`, databases `*.db*`, collected data `data.json` / `fqdn_data.json`, `graphify-out/`, service files. Every change follows the plan -> implementation -> summary flow in `docs/` and gets its own commit.
`requirements.txt` / `requirements-dev.txt` list direct dependencies with exact versions (`==`); `constraints.txt` is the full `pip freeze` (including transitive packages) of the image the tests pass on. Both Dockerfiles install with `-c constraints.txt`. To update: change the versions in a container, run the tests, regenerate `constraints.txt` (`docker run --rm --entrypoint pip <test-image> freeze > constraints.txt`) and commit all three files together.
The collector normally runs as the **daemon** `collector_daemon.py` (see sections 3-4): it keeps the schedule from `config.json` (`schedule.asn`, `schedule.fqdn`) and picks up changes made via `POST /schedule` within 30 seconds, without a restart. Only one daemon instance can run at a time (a second one exits with an error).
The API runs by default on port `8000`. It allows retrieving the collected data in a flat JSON list.
**Security model:** read endpoints (`/addresses`, `/schedule` GET, `/health`) are open, since consumers (routers, firewalls) usually can only do a plain GET - restrict access to port 8000 with a firewall. `POST /schedule` requires the header `X-API-Key: <RIPE_API_TOKEN>`; if the `RIPE_API_TOKEN` environment variable is not set, `POST` is disabled (503). If the data files are unreadable, the API answers `503`.
### Base URL
`http://<server-ip>:8000`
### Endpoint: Get Addresses
**GET** `/addresses`
Retrieves the list of collected IP addresses/CIDRs.
| `format` | string | No | `json` | Output format: `json`, `nftables`, `mikrotik`, `bird`, `frr` (see below). |
| `ip_version` | string | No | `all` | `4`, `6` or `all`. |
| `aggregate` | bool | No | `false` | Collapse overlapping/adjacent prefixes (`/23` + `/24` -> `/23`); a host IP inside a wider prefix is dropped. Applied after `cidr` and `fqdn` sources are merged. |
| `name` | string | No | `ripe` | Name of the list/set in generated configs (`[A-Za-z][A-Za-z0-9_]{0,31}`). |
With the defaults the response is the same flat JSON list as before. Other formats return `text/plain` scripts that are safe to re-apply (they replace the previous list).
#### Example 1: Get All Addresses (Default)
**Request:**
```bash
curl -X GET "http://localhost:8000/addresses"
```
**Response (JSON):**
```json
[
"142.250.1.1",
"149.154.160.0/22",
"149.154.160.0/23",
"2001:4860:4860::8888",
"91.108.4.0/22"
]
```
#### Example 2: Get Only CIDRs (from ASNs)
**Request:**
```bash
curl -X GET "http://localhost:8000/addresses?type=cidr"
```
**Response (JSON):**
```json
[
"149.154.160.0/22",
"149.154.160.0/23",
"91.108.4.0/22"
]
```
#### Example 3: Get Only Resolved IPs (from FQDNs)
**Request:**
```bash
curl -X GET "http://localhost:8000/addresses?type=fqdn"
```
**Response (JSON):**
```json
[
"142.250.1.1",
"2001:4860:4860::8888"
]
```
#### Ready-to-use configuration formats
| `format` | Result | How to apply |
| :--- | :--- | :--- |
| `nftables` | table `inet <name>` with interval sets `<name>_v4` / `<name>_v6` (`auto-merge`); other objects of the table are untouched | `curl -s "$URL/addresses?format=nftables" \| nft -f -` |
| `mikrotik` | `/ip firewall address-list` and `/ipv6 firewall address-list` named `<name>` (old entries are removed first, so there is a short window with an empty list) | `/tool fetch url="$URL/addresses?format=mikrotik" dst-path=ripe.rsc` then `/import ripe.rsc` |
| `bird` | BIRD 2 prefix sets `define <NAME>_V4 = [...]`, `<NAME>_V6` for use in filters (`net ~ RIPE_V4`) | save to a file, `include` it, `birdc configure` |
A version without addresses is omitted from the script. The `bird` and `nftables` outputs were syntax-checked with `bird -p` and `nft -c`.
### Endpoint: Changes since the last sync (diff)
**GET** `/addresses/diff?since=<cursor | time>[&type=cidr|fqdn|all][&ip_version=all|4|6]` - only what was added and removed, instead of the full list. Read access is open, like `/addresses`.
- `since` is a **cursor** from the previous answer (recommended: exact, independent of clocks) or an ISO 8601 time (no time zone = UTC; write `+` in a URL as `%2B`). Time is compared with millisecond precision, borders are inclusive, so an entry may be reported twice - repeating it is harmless. A cursor is at most 30 ASCII digits; anything else that is not an ISO 8601 time (including out-of-range dates) is `400`.
- The result is the net effect: an address added and removed (or the other way) within the interval is not reported; an address that another source still holds is not reported as removed. Both CIDRs and IPs of FQDNs count. Output is JSON only, no aggregation.
- The first sync: take the full `/addresses` list and the cursor from its response header `X-Changes-Cursor` (read before the data), then call `/addresses/diff?since=<that cursor>` regularly.
- `400` - invalid `since`; `410 Gone` - `since` is older than the journal (see `changes_retention_days`) or the cursor does not belong to this database: fetch the full `/addresses` list and continue from the new `cursor`.
### Endpoint: Manage Schedule
**GET** `/schedule`
Returns the current cron schedules.
**POST** `/schedule` (requires `X-API-Key`)
Updates the schedule for a specific collector type.
```bash
curl -X POST http://localhost:8000/schedule -H "X-API-Key: $RIPE_API_TOKEN" \
*Note: `type` can be `asn`, `fqdn` or `backup` (database backup, see section 9). The change is written to `config.json`; the collector daemon applies it within 30 seconds (`applied_within_seconds` in the response). An invalid cron expression is rejected by the API (400) and ignored by the daemon.*
The lists of monitored sources are stored in `config.json` and can be managed through the API. Reading is open; changing requires the `X-API-Key` header (same token as `POST /schedule`).
| `POST /asns` `{"asn": 62041}` | Add an ASN (`1..4294967295`). `201` if added, `200` if it was already there. |
| `POST /fqdns` `{"fqdn": "example.com"}` | Add a domain (lower-cased, trailing dot removed; IP addresses and invalid labels are rejected with `422`). |
| `DELETE /asns/{asn}?purge=false`, `DELETE /fqdns/{fqdn}?purge=false` | Remove a source (`404` if unknown). |
```bash
curl -X POST http://localhost:8000/asns -H "X-API-Key: $RIPE_API_TOKEN" \
- A new source is collected on the next scheduled run of the daemon; to collect it right away call `POST /collect` (see below).
- **What happens to the data of a removed source:** by default it is kept and its addresses expire by `ttl_days` (the collector keeps ageing entries of sources that are no longer configured and deletes the entry when it is empty). With `purge=true` the collected addresses are deleted immediately and disappear from `/addresses`. With `ttl_days = 0` nothing expires - use `purge=true`.
- The CLI commands (`add`, `remove`, `add-fqdn`, `remove-fqdn`) use the same locked, atomic config update, so they are safe to use alongside the API.
### Endpoint: Collect now
**POST** `/collect` (requires `X-API-Key`) - starts a collection immediately instead of waiting for the schedule.
```bash
curl -X POST http://localhost:8000/collect -H "X-API-Key: $RIPE_API_TOKEN" \
Body is optional: `type` is `asn`, `fqdn` or `all` (default). The API does not collect itself: it passes the request to the collector daemon through the file `collect_request.json`, which the daemon checks every 5 seconds, so the collection starts within ~5 s. Answer `202` lists the requested types; follow the progress in `GET /health` (`running`, `last_run`, `last_finished`, `last_error` of each job).
- Repeated requests are merged into one; if a collection of that type is already running (scheduled or manual) the new start is skipped.
- If the collector daemon is not running (no fresh heartbeat), the answer is `503` and nothing is queued.
Reports the state of the collector daemon (read from `status.json`): `collector_alive`, the cron / `running` / last run / `last_finished` / last error / next run of each job, the number of stored addresses and `last_restore` (`null`, or the record of the last automatic restore of the database from a backup) and `db_recreated` (`null`, or `{at, pending}` after the database was recreated without a backup; `pending: true` makes `status` `degraded`), see section 9. The daemon writes a heartbeat every 30 seconds; `collector_alive` is `false` if it is older than 120 seconds or the daemon never ran. `status` is `ok` only if the daemon is alive and no job failed in its last run; otherwise `degraded` (HTTP code is still 200).
The collector script supports running modes independently:
```bash
# Run both (Default)
python3 cidr_collector.py run
# Run only ASN collection
python3 cidr_collector.py run --mode asn
# Run only FQDN collection
python3 cidr_collector.py run --mode fqdn
```
---
## 7. Internal Logic & Architecture
### Collector Logic
When the collector runs (whether manually or via schedule):
1. **Instantiation**: Creates a new instance of `CIDRCollector` or `FQDNCollector`. This forces a fresh read of `config.json`, ensuring any added ASNs/FQDNs are immediately processed.
2. **Fetching**:
* **ASN**: Queries RIPE NCC API (`stat.ripe.net`).
* **FQDN**: Uses Python's `socket.getaddrinfo` to resolve A and AAAA records. Non-public addresses are dropped (see `allow_non_global_ips`); if nothing is left the run is treated like a DNS failure (nothing is deleted).
3. **Merge in one transaction**: the fetched addresses are merged into the SQLite table `addresses` (`db.py`) in a single write transaction, so readers (the API) never see a half-updated state.
4. **Accumulation with TTL**: Each address has `first_seen`/`last_seen`.
* New addresses are inserted; already seen ones get `last_seen` refreshed.
* Addresses not seen for longer than `ttl_days` are removed - but only after a *successful* fetch. A RIPE/DNS failure never deletes anything.
* Sources that are no longer configured are not fetched any more; their addresses simply expire by TTL.
5. **Persistence**: SQLite in WAL mode - the API reads while the collector writes. A corrupted database file is renamed to `ripe.db.corrupt-<timestamp>` and the API answers `503` instead of serving wrong data.
### Scheduler Logic
`collector_daemon.py` uses `APScheduler` (`BlockingScheduler`) in its own process; the API has no scheduler.
1. **Startup**: the daemon takes an exclusive lock (`collector.daemon.lock`, single instance), loads the `schedule` block from `config.json` and creates three independent jobs (`asn_job`, `fqdn_job`, `backup_job`; defaults `0 2 * **`, `0 3 ** *` and `30 4 * * *`). Overlapping runs of the same job are not allowed.
2. **Runtime updates (POST /schedule)**: the API validates the cron expression and writes `config.json` atomically. Every 30 seconds the daemon compares the `schedule` block with the active one and reschedules the changed job (`reschedule_job`). An invalid cron expression is logged and ignored, the previous schedule stays.
3. **Status**: at the start and end of every run and every 30 seconds the daemon atomically writes `status.json` (jobs state + `updated_at` heartbeat); `GET /health` reads it.
**Manual runs**: every 5 seconds the daemon checks `collect_request.json` (written by `POST /collect`) and starts the requested collections as one-off jobs; one run per type at a time.
4. **Concurrency**: a running job completes normally when the schedule changes; the new schedule applies to the next calculated run time. On `SIGTERM` the daemon exits after the running collection finishes. Restarting the API does not affect collection.
One image (`Dockerfile`), two services: `api` (uvicorn) and `collector` (`collector_daemon.py`). They share the named volume `ripe_data` mounted at `/data` (`RIPE_DATA_DIR`), which holds `ripe.db` (with `-wal`/`-shm`), `backups/` (database copies, see section 9), `config.json`, `status.json` and lock files. Containers run as non-root (uid 10001) with a read-only root filesystem, dropped capabilities and rotated logs. Both have healthchecks (`healthcheck.py`).
# set RIPE_API_TOKEN (openssl rand -hex 32) and, if needed, TZ / API_PORT
docker compose up -d
docker compose ps # both services become "healthy" within about a minute
docker compose logs -f collector
```
Then add sources through the API (section 5), e.g. `POST /asns` with the token. With an empty volume the lists are empty.
### Settings (`.env`)
| Variable | Default | Description |
| :--- | :--- | :--- |
| `RIPE_API_TOKEN` | - | Token for changing requests (`X-API-Key`). Without it they are disabled (503). |
| `TZ` | `UTC` | Time zone in which the cron schedules are evaluated (e.g. `Europe/Moscow`). |
| `API_PORT` | `8000` | Host port of the API. |
### Migrating existing data into the volume
Run once, from the directory with your current `config.json`, `data.json`, `fqdn_data.json`, before the first `up` (the JSON data files are imported into `ripe.db` automatically on the first start, see section 9):
```bash
docker compose create
docker volume ls | grep ripe_data # the volume is named <project>_ripe_data, e.g. ripe_cidr_collector_ripe_data
docker run --rm -v <project>_ripe_data:/data -v "$PWD":/src:ro alpine \
docker compose build && docker compose up -d # update after code changes
docker compose stop collector # graceful: a running collection finishes (up to 60s)
docker compose down # stop; data stays in the volume (add -v to delete it)
```
### Notes and risks
- **Port 8000 is published without TLS**, so `X-API-Key` travels in clear text. Restrict access with a firewall or put a TLS reverse proxy in front (bind the port to `127.0.0.1` by changing `ports` in `docker-compose.yml`).
- Dependencies are pinned (see "Dependency versions" below), so rebuilds give the same libraries. The base image `python:3.11-slim` is not pinned by digest: Python patch releases arrive on rebuild.
- Files written by the services in the volume (`config.json`, `status.json`) have mode `600`; both services run as the same user.
- `docker compose` uses the image name `ripe-cidr-collector`; `Dockerfile.test` is used only for running the tests (section 1, step 5).
---
## 9. Data Storage (SQLite)
Collected addresses are stored in `ripe.db` (in the project directory, or in `RIPE_DATA_DIR` - the `/data` volume in Docker). `config.json` (sources, schedule, `ttl_days`) and `status.json` (collector heartbeat) stay JSON.
```sql
CREATE TABLE addresses (
kind TEXT NOT NULL, -- 'asn' or 'fqdn'
source TEXT NOT NULL, -- '62041' or 'example.com'
value TEXT NOT NULL, -- prefix or IP
first_seen TEXT NOT NULL, -- ISO time
last_seen TEXT NOT NULL,
PRIMARY KEY (kind, source, value)
);
```
The schema version is stored in `PRAGMA user_version`. The database runs in WAL mode (files `ripe.db-wal`, `ripe.db-shm` next to it); API and collector (also two containers sharing one local volume) can work with it concurrently. Do not place it on a network file system.
### Change journal
For `/addresses/diff` the database keeps the tables `changes(id, ts, kind, value, action add|del)` and `meta` (the journal "horizon"). SQL triggers on `addresses` write to `changes`, so every path (collection, TTL, `purge`, removal of a source) is covered: `add` - the value appeared and no other source had it, `del` - the last entry of the value is gone. `ts` is UTC. The import from JSON is not written to the journal. Old records are deleted by the collector (`changes_retention_days`); the schema version is 2 (a version 1 database is upgraded automatically on the first start, data is kept).
Inspect the data:
```bash
sqlite3 ripe.db "SELECT kind, source, count(*), max(last_seen) FROM addresses GROUP BY 1, 2"
```
### Migration from the JSON storage
Automatic: on the first start of any process (API, collector or CLI) `data.json` and `fqdn_data.json` are imported into `ripe.db` in one transaction (concurrent starts are safe). Existing `first_seen`/`last_seen` are preserved; data in the oldest format without them gets `last_seen` = migration time (TTL starts counting from then). The originals are **not deleted** but renamed to `data.json.migrated-<timestamp>` and `fqdn_data.json.migrated-<timestamp>`.
### Rollback to the JSON version
Stop both services, rename the `*.migrated-*` files back to `data.json` / `fqdn_data.json`, remove `ripe.db*`, start the previous version of the code. Addresses collected after the migration are lost in that case.
**Backup job.** The collector daemon runs the job `backup` (default `30 4 * * *`, change it with `POST /schedule` and `"type": "backup"` or in `schedule.backup`). It makes an online copy of `ripe.db` (safe while the services run), checks the live database and the copy with `PRAGMA quick_check`, and keeps the last `backup_keep` copies (default 7; older ones are deleted only after a new copy succeeded). If the live database fails the check or the copy is bad, no copy is written, the older copies stay, and `GET /health` shows `degraded` (`jobs.backup.last_error`).
Copies are named `ripe-<UTC time>.db` and stored in `RIPE_BACKUP_DIR` (default `<RIPE_DATA_DIR>/backups`, i.e. `/data/backups` in Docker). **By default they are on the same volume as the database**: this protects against a corrupted file, not against losing the volume. For that, mount a separate volume/host directory and set `RIPE_BACKUP_DIR` to it, or copy the directory elsewhere regularly (e.g. `docker compose cp collector:/data/backups ./backups`).
**Automatic restore.** If `ripe.db` cannot be opened as a database (any process: API, daemon, CLI), the file is moved to `ripe.db.corrupt-<timestamp>` and the newest copy that passes the integrity check is put in its place; concurrent processes are serialized with a lock. The event is logged (ERROR), written to `last_restore.json` and shown in `GET /health` as `last_restore`. Data collected after that copy is lost; the collector gathers it again on the next runs. If there is no valid copy, a new empty database is created and the marker file `db_recreated.json` is written. **While the data is not gathered again, `GET /addresses` and `GET /addresses/diff` answer `503` with `Retry-After: 300`** instead of an empty list (a consumer could take it for the truth and wipe its rules); a type without configured sources (e.g. no FQDNs) is not blocked. The collector removes the marker after a run when every configured type has data again; to accept an empty result earlier, delete `db_recreated.json` by hand. `GET /health` shows `degraded` and `db_recreated` (`at`, `pending`) meanwhile. A fresh installation (no database yet) and a manually deleted database file are not treated as a loss and return an empty list until the first collection.
- The change journal goes back with the copy, so after a restore all cursors and times issued earlier answer `410` on `/addresses/diff` (the client makes a full download). The journal counter is shifted by 1,000,000 for that (a heuristic: it assumes fewer changes than that between two copies).
- Only corruption detected **when the database is opened** is restored automatically. Damage inside the file shows up as `503` on reads and is caught by the `backup` job (`quick_check`); restore it by hand: stop both services, keep the damaged `ripe.db*`, copy the chosen `backups/ripe-*.db` to `ripe.db`, start the services.
Manual consistent copy: `sqlite3 ripe.db ".backup ripe.db.bak"` (do not copy `ripe.db` alone while the services are running - part of the data may still be in the `-wal` file).
