ripe-cidr-collector/README.md

# RIPE AS CIDR & FQDN IP Collector

This project collects CIDR prefixes for specified Autonomous Systems (AS) from the RIPE NCC API and resolves IP addresses for specified FQDNs. It accumulates these addresses over time, maintaining a history of discovered prefixes. It also provides a FastAPI-based HTTP interface to retrieve the collected data.

The system consists of **two independent processes**: the collector daemon (`collector_daemon.py`, runs the schedule) and the API server (`api_server.py`, serves data and edits the config). They communicate only through files: the collected addresses live in the SQLite database `ripe.db`, the rest in JSON (`config.json`, `status.json`). See section 9.

> **Quick start:** the whole system (API + collector) runs with `docker compose up -d`, see section 8. Sections 1-4 describe the manual installation (venv + systemd/OpenRC), which remains supported.

> **Upgrading from a version where the scheduler lived in the API:** running only `ripe-api` is no longer enough - without the `ripe-collector` service nothing is collected. `GET /health` reports `collector_alive: false` / `degraded` in that case.

## 1. Preparation and Installation

### Prerequisites
- Python 3.8+
- `pip` and `venv`

### Installation Steps
1.  **Clone the repository** (or copy the files) to your desired location, e.g., `/opt/ripe_collector`.
    ```bash
    mkdir -p /opt/ripe_collector
    cd /opt/ripe_collector
    # Copy files: *.py (api_server, cidr_collector, collector_daemon, db, formatters, storage, healthcheck),
    # requirements.txt, constraints.txt, config.json
    ```

2.  **Create a Virtual Environment**:
    ```bash
    python3 -m venv venv
    ```

3.  **Install Dependencies**:
    ```bash
    source venv/bin/activate
    pip install -c constraints.txt -r requirements.txt   # exact versions, see "Dependency versions" below
    deactivate
    ```

4.  **Initial Configuration**:
    Edit `config.json` to set your initial ASNs and FQDNs.
    ```json
    {
        "asns": [62041],
        "fqdns": ["google.com"],
        "ttl_days": 90
    }
    ```
    `ttl_days` - how long an address is kept after it was last seen (default `90`, `0` = keep forever).
    Optional `source_failure_threshold` (default `3`) - how many runs in a row a source may fail before `GET /health` reports `degraded` (see the sources status below).
    Optional `ripestat_sourceapp` (default `ripe-cidr-collector`) - the application name sent to RIPEstat as `sourceapp`; RIPEstat asks clients to identify themselves, you can add a contact (`my-collector admin@example.org`).
    Optional `allow_non_global_ips` (default `false`) - by default only public addresses resolved from FQDNs are stored; loopback, private, link-local and unspecified addresses (`127.0.0.1`, `10.x`, `0.0.0.0`, ...) are ignored and logged. Set `true` for internal names.
    Optional `backup_keep` - how many database backups to keep (default `7`); the backup cron is `schedule.backup` (default `30 4 * * *`), see section 9.
    Optional `changes_retention_days` - how long the change journal for `/addresses/diff` is kept (default `30`, `0` = forever).

5.  **Tests** (optional) run in a container, not in the local `venv`:
    ```bash
    docker build -f Dockerfile.test -t ripe-collector-test .
    docker run --rm ripe-collector-test
    ```
    The image (`Dockerfile.test`, `python:3.11-slim`) contains only the code and dev dependencies - production data (`ripe.db`, `data.json`, `fqdn_data.json`) is excluded via `.dockerignore`.

6.  **Knowledge graph** (optional): `/graphify .` (Claude Code skill) builds `graphify-out/` - `graph.html` (interactive), `GRAPH_REPORT.md`, `graph.json` - from code and `docs/`; refresh with `/graphify . --update` after changes. The directory is excluded from git and from Docker images.

7.  **Repository:** `main` branch, remote `origin` = `https://artstore.rxmsk.ru/ayurishchev/ripe-cidr-collector.git`. Committed: code, tests, Docker files, `config.json` (initial sources), `.env.example`, `docs/`. Not committed (`.gitignore`): `venv/`, `.env`, databases `*.db*`, collected data `data.json` / `fqdn_data.json`, `graphify-out/`, service files. Every change follows the plan -> implementation -> summary flow in `docs/` and gets its own commit.


### Dependency versions
`requirements.txt` / `requirements-dev.txt` list direct dependencies with exact versions (`==`); `constraints.txt` is the full `pip freeze` (including transitive packages) of the image the tests pass on. Both Dockerfiles install with `-c constraints.txt`. To update: change the versions in a container, run the tests, regenerate `constraints.txt` (`docker run --rm --entrypoint pip <test-image> freeze > constraints.txt`) and commit all three files together.

---

## 2. Running the Collector

The collector normally runs as the **daemon** `collector_daemon.py` (see sections 3-4): it keeps the schedule from `config.json` (`schedule.asn`, `schedule.fqdn`) and picks up changes made via `POST /schedule` within 30 seconds, without a restart. Only one daemon instance can run at a time (a second one exits with an error).

```bash
/opt/ripe_collector/venv/bin/python3 /opt/ripe_collector/collector_daemon.py
```

The script `cidr_collector.py` is used for manual runs and for managing the lists (see section 6).

### Manual Run
```bash
/opt/ripe_collector/venv/bin/python3 /opt/ripe_collector/cidr_collector.py run
```

### Alternative: cron instead of the daemon
If you prefer cron, run the collector once per day (do not combine with the daemon - it is redundant). To run daily at 02:00 AM:

> Run the job as the same user as the API (`crontab -u ripe -e`), otherwise data files may end up with the wrong owner.

1.  Open crontab:
    ```bash
    crontab -e
    ```
2.  Add the line:
    ```cron
    0 2 * * * /opt/ripe_collector/venv/bin/python3 /opt/ripe_collector/cidr_collector.py run >> /var/log/ripe_collector.log 2>&1
    ```

---

## 3. Application Setup: Systemd (Ubuntu, Debian)

This section describes how to run two system services: the **API Server** (`api_server.py`) and the **collector daemon** (`collector_daemon.py`).

### Create Service File
Create a dedicated user and the token file (the token protects `POST /schedule`):

```bash
useradd --system --home /opt/ripe_collector --shell /usr/sbin/nologin ripe
chown -R ripe: /opt/ripe_collector
echo "RIPE_API_TOKEN=$(openssl rand -hex 32)" > /etc/ripe-api.env
chmod 600 /etc/ripe-api.env
```

Create `/etc/systemd/system/ripe-api.service`:

```ini
[Unit]
Description=RIPE CIDR Collector API
After=network.target

[Service]
User=ripe
EnvironmentFile=/etc/ripe-api.env
WorkingDirectory=/opt/ripe_collector
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/ripe_collector
ExecStart=/opt/ripe_collector/venv/bin/uvicorn api_server:app --host 0.0.0.0 --port 8000
Restart=always

[Install]
WantedBy=multi-user.target
```

Create `/etc/systemd/system/ripe-collector.service` (the daemon does not need the token):

```ini
[Unit]
Description=RIPE CIDR Collector daemon
After=network.target

[Service]
User=ripe
WorkingDirectory=/opt/ripe_collector
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/opt/ripe_collector
ExecStart=/opt/ripe_collector/venv/bin/python collector_daemon.py
Restart=always

[Install]
WantedBy=multi-user.target
```

### Enable and Start
```bash
# Reload systemd
sudo systemctl daemon-reload

# Enable services to start on boot and start them immediately
sudo systemctl enable --now ripe-collector ripe-api

# Check status
sudo systemctl status ripe-collector ripe-api
```

---

## 4. Application Setup: RC-Script (Alpine Linux)

For Alpine Linux using OpenRC.

### Create Init Script
Create `/etc/init.d/ripe-api`:

```sh
#!/sbin/openrc-run

name="ripe-api"
description="RIPE CIDR Collector API"
command="/opt/ripe_collector/venv/bin/uvicorn"
# --host and --port and module:app passed as arguments
command_args="api_server:app --host 0.0.0.0 --port 8000"
command_background="yes"
pidfile="/run/${RC_SVCNAME}.pid"
directory="/opt/ripe_collector"

depend() {
    need net
}
```

Add the token and a non-root user (`/etc/conf.d/ripe-api` is read by OpenRC):

```sh
adduser -S -h /opt/ripe_collector ripe
chown -R ripe /opt/ripe_collector
echo 'export RIPE_API_TOKEN="<random-token>"' > /etc/conf.d/ripe-api
chmod 600 /etc/conf.d/ripe-api
```
and add `command_user="ripe"` to the init script.

Create the collector daemon script `/etc/init.d/ripe-collector`:

```sh
#!/sbin/openrc-run

name="ripe-collector"
description="RIPE CIDR Collector daemon"
command="/opt/ripe_collector/venv/bin/python"
command_args="collector_daemon.py"
command_background="yes"
command_user="ripe"
pidfile="/run/${RC_SVCNAME}.pid"
directory="/opt/ripe_collector"

depend() {
    need net
}
```

### Make Executable
```bash
chmod +x /etc/init.d/ripe-api /etc/init.d/ripe-collector
```

### Enable and Start
```bash
# Add to default runlevel
rc-update add ripe-collector default
rc-update add ripe-api default

# Start services
service ripe-collector start
service ripe-api start

# Check status
service ripe-collector status
service ripe-api status
```

---

## 5. API Usage Documentation

The API runs by default on port `8000`. It allows retrieving the collected data in a flat JSON list.

**Security model:** read endpoints (`/addresses`, `/schedule` GET, `/health`) are open, since consumers (routers, firewalls) usually can only do a plain GET - restrict access to port 8000 with a firewall. `POST /schedule` requires the header `X-API-Key: <RIPE_API_TOKEN>`; if the `RIPE_API_TOKEN` environment variable is not set, `POST` is disabled (503). If the data files are unreadable, the API answers `503`.

### Base URL
`http://<server-ip>:8000`

### Endpoint: Get Addresses
**GET** `/addresses`

Retrieves the list of collected IP addresses/CIDRs.

| Parameter | Type | Required | Default | Description |
| :--- | :--- | :--- | :--- | :--- |
| `type` | string | No | `all` | Filter by source type. Options: `cidr` (ASNs only), `fqdn` (Domains only), `all` (Both). |
| `format` | string | No | `json` | Output format: `json`, `nftables`, `mikrotik`, `bird`, `frr` (see below). |
| `ip_version` | string | No | `all` | `4`, `6` or `all`. |
| `aggregate` | bool | No | `false` | Collapse overlapping/adjacent prefixes (`/23` + `/24` -> `/23`); a host IP inside a wider prefix is dropped. Applied after `cidr` and `fqdn` sources are merged. |
| `name` | string | No | `ripe` | Name of the list/set in generated configs (`[A-Za-z][A-Za-z0-9_]{0,31}`). |

With the defaults the response is the same flat JSON list as before. Other formats return `text/plain` scripts that are safe to re-apply (they replace the previous list).

#### Example 1: Get All Addresses (Default)

**Request:**
```bash
curl -X GET "http://localhost:8000/addresses"
```

**Response (JSON):**
```json
[
  "142.250.1.1",
  "149.154.160.0/22",
  "149.154.160.0/23",
  "2001:4860:4860::8888",
  "91.108.4.0/22"
]
```

#### Example 2: Get Only CIDRs (from ASNs)

**Request:**
```bash
curl -X GET "http://localhost:8000/addresses?type=cidr"
```

**Response (JSON):**
```json
[
  "149.154.160.0/22",
  "149.154.160.0/23",
  "91.108.4.0/22"
]
```

#### Example 3: Get Only Resolved IPs (from FQDNs)

**Request:**
```bash
curl -X GET "http://localhost:8000/addresses?type=fqdn"
```

**Response (JSON):**
```json
[
  "142.250.1.1",
  "2001:4860:4860::8888"
]
```

#### Ready-to-use configuration formats

| `format` | Result | How to apply |
| :--- | :--- | :--- |
| `nftables` | table `inet <name>` with interval sets `<name>_v4` / `<name>_v6` (`auto-merge`); other objects of the table are untouched | `curl -s "$URL/addresses?format=nftables" \| nft -f -` |
| `mikrotik` | `/ip firewall address-list` and `/ipv6 firewall address-list` named `<name>` (old entries are removed first, so there is a short window with an empty list) | `/tool fetch url="$URL/addresses?format=mikrotik" dst-path=ripe.rsc` then `/import ripe.rsc` |
| `bird` | BIRD 2 prefix sets `define <NAME>_V4 = [...]`, `<NAME>_V6` for use in filters (`net ~ RIPE_V4`) | save to a file, `include` it, `birdc configure` |
| `frr` | `ip prefix-list <name>_v4` / `ipv6 prefix-list <name>_v6` with `permit` entries | `curl -s "$URL/addresses?format=frr" > ripe.conf && vtysh -f ripe.conf` |

Examples:
```bash
# IPv4 only, aggregated, as an nftables set named "tg"
curl -s "http://localhost:8000/addresses?format=nftables&ip_version=4&aggregate=true&name=tg"
```
A version without addresses is omitted from the script. The `bird` and `nftables` outputs were syntax-checked with `bird -p` and `nft -c`.

### Endpoint: Changes since the last sync (diff)
**GET** `/addresses/diff?since=<cursor | time>[&type=cidr|fqdn|all][&ip_version=all|4|6]` - only what was added and removed, instead of the full list. Read access is open, like `/addresses`.

```bash
curl "http://localhost:8000/addresses/diff?since=0"
# {"since":"0","now":"2026-09-21T04:09:44Z","cursor":42,"added":["3.0.0.0/24"],"removed":["2.0.0.0/24"]}
curl "http://localhost:8000/addresses/diff?since=42"      # next sync: use the returned cursor
```
- `since` is a **cursor** from the previous answer (recommended: exact, independent of clocks) or an ISO 8601 time (no time zone = UTC; write `+` in a URL as `%2B`). Time is compared with millisecond precision, borders are inclusive, so an entry may be reported twice - repeating it is harmless. A cursor is at most 30 ASCII digits; anything else that is not an ISO 8601 time (including out-of-range dates) is `400`.
- The result is the net effect: an address added and removed (or the other way) within the interval is not reported; an address that another source still holds is not reported as removed. Both CIDRs and IPs of FQDNs count. Output is JSON only, no aggregation.
- The first sync: take the full `/addresses` list and the cursor from its response header `X-Changes-Cursor` (taken in the same database snapshot as the data, so the two match exactly), then call `/addresses/diff?since=<that cursor>` regularly.
- `400` - invalid `since`; `410 Gone` - `since` is older than the journal (see `changes_retention_days`) or the cursor does not belong to this database: fetch the full `/addresses` list and continue from the new `cursor`.

### Endpoint: Manage Schedule
**GET** `/schedule`
Returns the current cron schedules.

**POST** `/schedule` (requires `X-API-Key`)
Updates the schedule for a specific collector type.
```bash
curl -X POST http://localhost:8000/schedule -H "X-API-Key: $RIPE_API_TOKEN" \
     -H "Content-Type: application/json" -d '{"type": "asn", "cron": "*/15 * * * *"}'
```
Body:
```json
{
    "type": "asn", 
    "cron": "*/15 * * * *"
}
```
*Note: `type` can be `asn`, `fqdn` or `backup` (database backup, see section 9). The change is written to `config.json`; the collector daemon applies it within 30 seconds (`applied_within_seconds` in the response). An invalid cron expression is rejected by the API (400) and ignored by the daemon.*

### Endpoints: Manage ASNs and FQDNs
The lists of monitored sources are stored in `config.json` and can be managed through the API. Reading is open; changing requires the `X-API-Key` header (same token as `POST /schedule`).

| Method and path | Description |
| :--- | :--- |
| `GET /asns`, `GET /fqdns` | Current lists: `{"asns": [...]}`, `{"fqdns": [...]}` |
| `POST /asns` `{"asn": 62041}` | Add an ASN (`1..4294967295`). `201` if added, `200` if it was already there. |
| `POST /fqdns` `{"fqdn": "example.com"}` | Add a domain (lower-cased, trailing dot removed; IP addresses and invalid labels are rejected with `422`). |
| `DELETE /asns/{asn}?purge=false`, `DELETE /fqdns/{fqdn}?purge=false` | Remove a source (`404` if unknown). |

```bash
curl -X POST http://localhost:8000/asns -H "X-API-Key: $RIPE_API_TOKEN" \
     -H "Content-Type: application/json" -d '{"asn": 62041}'
curl -X DELETE "http://localhost:8000/fqdns/example.com?purge=true" -H "X-API-Key: $RIPE_API_TOKEN"
```

- A new source is collected on the next scheduled run of the daemon; to collect it right away call `POST /collect` (see below).
- **What happens to the data of a removed source:** by default it is kept and its addresses expire by `ttl_days` (the collector keeps ageing entries of sources that are no longer configured and deletes the entry when it is empty). With `purge=true` the collected addresses are deleted immediately and disappear from `/addresses`. With `ttl_days = 0` nothing expires - use `purge=true`.
- The CLI commands (`add`, `remove`, `add-fqdn`, `remove-fqdn`) use the same locked, atomic config update, so they are safe to use alongside the API.

### Endpoint: Collect now
**POST** `/collect` (requires `X-API-Key`) - starts a collection immediately instead of waiting for the schedule.

```bash
curl -X POST http://localhost:8000/collect -H "X-API-Key: $RIPE_API_TOKEN" \
     -H "Content-Type: application/json" -d '{"type": "asn"}'
```
Body is optional: `type` is `asn`, `fqdn` or `all` (default). The API does not collect itself: it passes the request to the collector daemon through the file `collect_request.json`, which the daemon checks every 5 seconds, so the collection starts within ~5 s. Answer `202` lists the requested types; follow the progress in `GET /health` (`running`, `last_run`, `last_finished`, `last_error` of each job).

- Repeated requests are merged into one; if a collection of that type is already running (scheduled or manual) the new start is skipped.
- If the collector daemon is not running (no fresh heartbeat), the answer is `503` and nothing is queued.

### Endpoint: Health
**GET** `/health`
Reports the state of the collector daemon (read from `status.json`): `collector_alive`, the cron / `running` / last run / `last_finished` / last error / next run of each job, the number of stored addresses and `last_restore` (`null`, or `{at, backup}` - time and file name of the last automatic restore from a backup; directories and the error text stay in `last_restore.json`) and `db_recreated` (`null`, or `{at, pending}` after the database was recreated without a backup; `pending: true` makes `status` `degraded`), see section 9. The daemon writes a heartbeat every 30 seconds; `collector_alive` is `false` if it is older than 120 seconds or the daemon never ran. `status` is `ok` only if the daemon is alive and no job failed in its last run; otherwise `degraded` (HTTP code is still 200).

`sources` shows the state of the data sources: for `asn` and `fqdn` the number of configured sources (`total`) and the list of **failing** ones (`failing`: `source`, `failures`, `last_success`, `error_kind`). A source is failing when its last `source_failure_threshold` runs in a row failed (default 3, so a single RIPEstat hiccup does not turn the status yellow); any failing source makes `status` `degraded`. `error_kind` is a short category without details (`timeout`, `connection`, `http_429`, `http_4xx`, `http_5xx`, `invalid_response`, `dns`, `no_global_addresses`, `unknown`); the full error text stays in the collector log.

### Endpoint: Sources status
**GET** `/sources` (open) - the state of every configured source: `kind`, `source`, `addresses` (how many values it gave), `last_attempt`, `last_success`, `failures` (runs failed in a row), `error_kind`, plus `failure_threshold`. A source that was never polled has empty state; a source that "succeeds" but gives `addresses: 0` is easy to spot here. The state is kept in the database (table `source_status`), survives restarts and is included in backups; it is removed with the source (`purge` or the next collection after removal).

---

## 6. Advanced CLI Usage

The collector script supports running modes independently:

```bash
# Run both (Default)
python3 cidr_collector.py run

# Run only ASN collection
python3 cidr_collector.py run --mode asn

# Run only FQDN collection
python3 cidr_collector.py run --mode fqdn
```

---

## 7. Internal Logic & Architecture

### Collector Logic
When the collector runs (whether manually or via schedule):
1.  **Instantiation**: Creates a new instance of `CIDRCollector` or `FQDNCollector`. This forces a fresh read of `config.json`, ensuring any added ASNs/FQDNs are immediately processed.
2.  **Fetching**:
    *   **ASN**: Queries RIPE NCC API (`stat.ripe.net`) with the `sourceapp` parameter. Connection failures and codes 429/500/502/503/504 are retried up to 3 times with growing pauses (0, 2, 4 s; a `Retry-After` pause is capped at 30 s), 10 s per attempt - at worst about 50 s per ASN. If all attempts fail the ASN is skipped for this run and nothing is deleted; the failure is recorded for `GET /health` and `GET /sources`. Every run ends with one summary line in the log (`ASN collection finished: 6 sources, 5 ok, 1 failed, +12/-3 prefixes`).
    *   **FQDN**: Uses Python's `socket.getaddrinfo` to resolve A and AAAA records. Non-public addresses are dropped (see `allow_non_global_ips`); if nothing is left the run is treated like a DNS failure (nothing is deleted).
3.  **Merge in one transaction**: the fetched addresses are merged into the SQLite table `addresses` (`db.py`) in a single write transaction, so readers (the API) never see a half-updated state.
4.  **Accumulation with TTL**: Each address has `first_seen`/`last_seen`.
    *   New addresses are inserted; already seen ones get `last_seen` refreshed.
    *   Addresses not seen for longer than `ttl_days` are removed - but only after a *successful* fetch. A RIPE/DNS failure never deletes anything.
    *   Sources that are no longer configured are not fetched any more; their addresses simply expire by TTL.
5.  **Persistence**: SQLite in WAL mode - the API reads while the collector writes. A corrupted database file is renamed to `ripe.db.corrupt-<timestamp>` and the API answers `503` instead of serving wrong data.

### Scheduler Logic
`collector_daemon.py` uses `APScheduler` (`BlockingScheduler`) in its own process; the API has no scheduler.

1.  **Startup**: the daemon takes an exclusive lock (`collector.daemon.lock`, single instance), loads the `schedule` block from `config.json` and creates three independent jobs (`asn_job`, `fqdn_job`, `backup_job`; defaults `0 2 * * *`, `0 3 * * *` and `30 4 * * *`). Overlapping runs of the same job are not allowed.
2.  **Runtime updates (POST /schedule)**: the API validates the cron expression and writes `config.json` atomically. Every 30 seconds the daemon compares the `schedule` block with the active one and reschedules the changed job (`reschedule_job`). An invalid cron expression is logged and ignored, the previous schedule stays.
3.  **Status**: at the start and end of every run and every 30 seconds the daemon atomically writes `status.json` (jobs state + `updated_at` heartbeat); `GET /health` reads it.
    **Manual runs**: every 5 seconds the daemon checks `collect_request.json` (written by `POST /collect`) and starts the requested collections as one-off jobs; one run per type at a time.
4.  **Concurrency**: a running job completes normally when the schedule changes; the new schedule applies to the next calculated run time. On `SIGTERM` the daemon exits after the running collection finishes. Restarting the API does not affect collection.

---

## 8. Application Setup: Docker Compose

One image (`Dockerfile`), two services: `api` (uvicorn) and `collector` (`collector_daemon.py`). They share the named volume `ripe_data` mounted at `/data` (`RIPE_DATA_DIR`), which holds `ripe.db` (with `-wal`/`-shm`), `backups/` (database copies, see section 9), `config.json`, `status.json` and lock files. Containers run as non-root (uid 10001) with a read-only root filesystem, dropped capabilities and rotated logs. Both have healthchecks (`healthcheck.py`).

### Start
```bash
cp .env.example .env
# set RIPE_API_TOKEN (openssl rand -hex 32) and, if needed, TZ / API_PORT
docker compose up -d
docker compose ps          # both services become "healthy" within about a minute
docker compose logs -f collector
```
Then add sources through the API (section 5), e.g. `POST /asns` with the token. With an empty volume the lists are empty.

### Settings (`.env`)
| Variable | Default | Description |
| :--- | :--- | :--- |
| `RIPE_API_TOKEN` | - | Token for changing requests (`X-API-Key`). Without it they are disabled (503). |
| `TZ` | `UTC` | Time zone in which the cron schedules are evaluated (e.g. `Europe/Moscow`). |
| `API_PORT` | `8000` | Host port of the API. |

### Migrating existing data into the volume
Run once, from the directory with your current `config.json`, `data.json`, `fqdn_data.json`, before the first `up` (the JSON data files are imported into `ripe.db` automatically on the first start, see section 9):
```bash
docker compose create
docker volume ls | grep ripe_data      # the volume is named <project>_ripe_data, e.g. ripe_cidr_collector_ripe_data
docker run --rm -v <project>_ripe_data:/data -v "$PWD":/src:ro alpine \
    sh -c 'cp /src/config.json /src/data.json /src/fqdn_data.json /data/ && chown 10001 /data/*.json'
docker compose up -d
```

### Operations
```bash
docker compose build && docker compose up -d   # update after code changes
docker compose stop collector                  # graceful: a running collection finishes (up to 60s)
docker compose down                            # stop; data stays in the volume (add -v to delete it)
```

### Notes and risks
- **Port 8000 is published without TLS**, so `X-API-Key` travels in clear text. Restrict access with a firewall or put a TLS reverse proxy in front (bind the port to `127.0.0.1` by changing `ports` in `docker-compose.yml`).
- The image copies all `*.py` modules from the project root and imports the main ones during the build, so a missing or broken module fails the build instead of the start.
- Dependencies are pinned (see "Dependency versions" below), so rebuilds give the same libraries. The base image `python:3.11-slim` is not pinned by digest: Python patch releases arrive on rebuild.
- Files written by the services in the volume (`config.json`, `status.json`) have mode `600`; both services run as the same user.
- `docker compose` uses the image name `ripe-cidr-collector`; `Dockerfile.test` is used only for running the tests (section 1, step 5).

---

## 9. Data Storage (SQLite)

Collected addresses are stored in `ripe.db` (in the project directory, or in `RIPE_DATA_DIR` - the `/data` volume in Docker). `config.json` (sources, schedule, `ttl_days`) and `status.json` (collector heartbeat) stay JSON.

```sql
CREATE TABLE addresses (
    kind       TEXT NOT NULL,   -- 'asn' or 'fqdn'
    source     TEXT NOT NULL,   -- '62041' or 'example.com'
    value      TEXT NOT NULL,   -- prefix or IP
    first_seen TEXT NOT NULL,   -- ISO time
    last_seen  TEXT NOT NULL,
    PRIMARY KEY (kind, source, value)
);
```
The schema version is stored in `PRAGMA user_version`. The database runs in WAL mode (files `ripe.db-wal`, `ripe.db-shm` next to it); API and collector (also two containers sharing one local volume) can work with it concurrently. Do not place it on a network file system.

### Change journal
For `/addresses/diff` the database keeps the tables `changes(id, ts, kind, value, action add|del)` and `meta` (the journal "horizon"). SQL triggers on `addresses` write to `changes`, so every path (collection, TTL, `purge`, removal of a source) is covered: `add` - the value appeared and no other source had it, `del` - the last entry of the value is gone. `ts` is UTC. The import from JSON is not written to the journal. Old records are deleted by the collector (`changes_retention_days`); the schema version is 3 (an older database is upgraded automatically on the first start, data is kept). Version 3 adds the table `source_status(kind, source, last_attempt, last_success, failures, error_kind)`.

Inspect the data:
```bash
sqlite3 ripe.db "SELECT kind, source, count(*), max(last_seen) FROM addresses GROUP BY 1, 2"
```

### Migration from the JSON storage
Automatic: on the first start of any process (API, collector or CLI) `data.json` and `fqdn_data.json` are imported into `ripe.db` in one transaction (concurrent starts are safe). Existing `first_seen`/`last_seen` are preserved; data in the oldest format without them gets `last_seen` = migration time (TTL starts counting from then). The originals are **not deleted** but renamed to `data.json.migrated-<timestamp>` and `fqdn_data.json.migrated-<timestamp>`.

### Rollback to the JSON version
Stop both services, rename the `*.migrated-*` files back to `data.json` / `fqdn_data.json`, remove `ripe.db*`, start the previous version of the code. Addresses collected after the migration are lost in that case.

### Backup and automatic restore
**Backup job.** The collector daemon runs the job `backup` (default `30 4 * * *`, change it with `POST /schedule` and `"type": "backup"` or in `schedule.backup`). It makes an online copy of `ripe.db` (safe while the services run), checks the live database and the copy with `PRAGMA quick_check`, and keeps the last `backup_keep` copies (default 7; older ones are deleted only after a new copy succeeded). An empty database is not copied while sources are configured (a warning is logged, the job stays successful, older copies are kept): an empty copy would push good ones out of the rotation and restoring it would give an empty list. If the live database fails the check or the copy is bad, no copy is written, the older copies stay, and `GET /health` shows `degraded` (`jobs.backup.last_error`).

Copies are named `ripe-<UTC time>.db` and stored in `RIPE_BACKUP_DIR` (default `<RIPE_DATA_DIR>/backups`, i.e. `/data/backups` in Docker). **By default they are on the same volume as the database**: this protects against a corrupted file, not against losing the volume. For that, mount a separate volume/host directory and set `RIPE_BACKUP_DIR` to it, or copy the directory elsewhere regularly (e.g. `docker compose cp collector:/data/backups ./backups`).

**Automatic restore.** If `ripe.db` cannot be opened as a database (any process: API, daemon, CLI), the file is moved to `ripe.db.corrupt-<timestamp>` and the newest copy that passes the integrity check is put in its place; concurrent processes are serialized with a lock. The event is logged (ERROR), written to `last_restore.json` and shown in `GET /health` as `last_restore`. Data collected after that copy is lost; the collector gathers it again on the next runs. If the newest valid copy has no data although sources are configured, it is restored and the same marker is written. If there is no valid copy, a new empty database is created and the marker file `db_recreated.json` is written. **While the data is not gathered again, `GET /addresses` and `GET /addresses/diff` answer `503` with `Retry-After: 300`** instead of an empty list (a consumer could take it for the truth and wipe its rules); a type without configured sources (e.g. no FQDNs) is not blocked. The collector removes the marker after a run when every configured type has data again; to accept an empty result earlier, delete `db_recreated.json` by hand. `GET /health` shows `degraded` and `db_recreated` (`at`, `pending`) meanwhile. A fresh installation (no database yet) and a manually deleted database file are not treated as a loss and return an empty list until the first collection.

- The change journal goes back with the copy, so after a restore all cursors and times issued earlier answer `410` on `/addresses/diff` (the client makes a full download). The journal counter is shifted by 1,000,000 for that (a heuristic: it assumes fewer changes than that between two copies).
- Only corruption detected **when the database is opened** is restored automatically. Damage inside the file shows up as `503` on reads and is caught by the `backup` job (`quick_check`); restore it by hand: stop both services, keep the damaged `ripe.db*`, copy the chosen `backups/ripe-*.db` to `ripe.db`, start the services.

Manual consistent copy: `sqlite3 ripe.db ".backup ripe.db.bak"` (do not copy `ripe.db` alone while the services are running - part of the data may still be in the `-wal` file).