Grigo/AISHub
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

closed TG-1; git was inited;

Browse Source
This commit is contained in:
grigo
2026-05-04 08:13:38 +03:00
commit bcf20fcb04
105 changed files with 6592 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+156
View File
@@ -0,0 +1,156 @@
# ais_hub
Central AIS/GPS/radio telemetry aggregation service for embedded Linux.
`ais_hub` is a Python 3.11 asyncio service. It ingests raw AIS (UDP), GPS
(UART) and radio-telemetry (UDP) streams, reassembles multi-fragment AIS
sentences, maintains a single authoritative in-memory state keyed by MMSI,
writes history to SQLite (WAL + retention), and publishes data to external
processes (BLE server, web UI, SPI bridge) via stable interfaces.
No BLE, web UI or MCU code lives here; those are separate processes that
consume the interfaces below.
## Interfaces
Полный JSON-контракт с примерами для фронтенда — в **[docs/API.md](docs/API.md)**.
Короткая сводка ниже.
### HTTP (REST, `GET` unless noted)
| Path | Purpose |
| --- | --- |
| `/api/v1/health` | Liveness + uptime + version. |
| `/api/v1/stats` | Counters, gauges, uptime. |
| `/api/v1/ownship` | Latest GPS fix. |
| `/api/v1/vessels` | All merged targets. Query: `since`, `limit`. |
| `/api/v1/vessels/{mmsi}` | One merged target. |
| `/api/v1/base_stations` | AIS base station snapshots. |
| `/api/v1/atons` | AtoN snapshots. |
| `/api/v1/slots` | TDMA slot occupancy + per-slot detail from AIS-catcher. |
| `/api/v1/radio` | Latest RSSI pair (channel A/B) from AIS-catcher. |
| `/api/v1/logs` | **Service logs tail**, from in-memory ring. Query: `level`, `limit`, `since`. |
| `/api/v1/nmea/tail` | **Raw NMEA tail** (separate from `/logs`). Query: `source=ais\|gps\|radio`, `limit`. |
| `/api/v1/history/positions` | AIS dynamic history. Query: `mmsi`, `from`, `to`, `limit`. |
| `POST /api/v1/tx` | Inject NMEA sentence(s) into the TX outbox. Body: `{"payload": "!AIVDM,..."}` or `{"nmea": ["...", "..."]}`. |
### WebSocket
- `GET /ws` — on connect, clients receive a full `state.snapshot` event
(ownship, vessels, base_stations, atons, slots, stats). Then a live
stream of `Event` envelopes:
- `ownship.update`, `target.update`, `base_station.update`,
`aton.update`, `radio.update`, `slots.update`, `slots.detail`,
`signal.update`, `stats.update`.
- Each client has its own bounded queue; slow consumers drop oldest
events and the `ws_dropped` counter increments.
### Local UDP
| Endpoint | Direction | Contract |
| --- | --- | --- |
| `127.0.0.1:7001` | ais_hub → external | JSON `Event`, one event per datagram, dropped if > `max_datagram_bytes`. |
| `127.0.0.1:6007` | ais_hub → SPI bridge | Raw NMEA fan-out, one sentence per datagram (`<line>\r\n`). Untouched from ingest. |
| `127.0.0.1:6010` | ais_hub → SPI bridge | **TX publish outbox**. One NMEA sentence per datagram. Fed by `POST /api/v1/tx` and internal emitters. |
ais_hub does **not** listen on `:6010`; it is publish-only.
## Layout
- `src/ais_hub/ingest/` — AIS UDP / GPS UART / radio UDP listeners, raw tap.
- `src/ais_hub/parser/` — NMEA utils, AIS multi-fragment assembler, GPS,
radio telemetry parsers.
- `src/ais_hub/core/` — domain models, in-memory state, eager per-MMSI
merge, stats counters, internal pub/sub bus.
- `src/ais_hub/storage/` — aiosqlite DB (WAL), batched writer with retry,
retention cleanup, read queries for REST.
- `src/ais_hub/publish/` — aiohttp REST + WS server, UDP publishers,
bounded queue helpers with drop-oldest policy.
- `src/ais_hub/app.py` — orchestrator (supervisors, graceful shutdown).
## Install
```bash
pip install -e .
```
Runtime dependencies: `aiohttp`, `pyais`, `pyserial-asyncio`, `aiosqlite`,
`PyYAML`.
## Run
```bash
ais_hub --config config/config.example.yaml
```
### Environment variable overrides
Env vars override any YAML value. Prefix `AIS_HUB_`, level separator
`__`, upper-case. Examples:
```bash
AIS_HUB_INGEST__AIS_UDP__PORT=4011
AIS_HUB_STORAGE__PATH=/data/ais_hub.db
AIS_HUB_STORAGE__STORE_RAW_NMEA=true
AIS_HUB_LOGGING__LEVEL=DEBUG
```
## Deploy on embedded Linux (systemd)
1. Create user and directories:
```bash
useradd -r -s /sbin/nologin ais_hub
install -d -o ais_hub -g ais_hub /var/lib/ais_hub /var/log/ais_hub /etc/ais_hub
```
2. Install package (system-wide or in a venv the unit points at):
```bash
pip install .
```
3. Copy config:
```bash
install -m 0644 config/config.example.yaml /etc/ais_hub/config.yaml
```
4. Install unit and enable:
```bash
install -m 0644 systemd/ais_hub.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now ais_hub
journalctl -u ais_hub -f
```
For UART access, add the `ais_hub` user to the serial/dialout group or
grant access via a udev rule on `/dev/ttyS1`.
## Resilience notes
- Each subsystem runs under a supervisor task: on exception it is logged
and restarted with exponential backoff; the process itself never dies
because of a single subsystem failure.
- All inter-subsystem queues are bounded with drop-oldest. Per-channel
drop counters are exposed via `/api/v1/stats`:
- `parser_in_dropped`, `udp_nmea_dropped`,
- `storage_in_dropped`, `tx_outbox_dropped`,
- `bus_dropped`, `ws_dropped`.
- SQLite transient errors are retried up to 3 times with exponential
backoff; on persistent failure the batch is dropped, the writer keeps
draining the queue, and REST history may return an error while the
rest of the service continues to serve live data.
- Multi-fragment AIS is keyed by
`(source_tag, talker, channel, seq_id, total_fragments)`. Fragment
ordering is strictly validated; any deviation resets the per-key
buffer and increments `ais_fragment_errors`. Stale buffers are GC'd
after 60 s (`ais_fragment_timeouts`).
## Tests
```bash
pip install -e .[dev]
pytest
```
config/config.example.yaml
+82
View File
@@ -0,0 +1,82 @@
# ais_hub configuration (example).
# All fields can be overridden via env vars with prefix AIS_HUB_ and
# "__" as level separator, e.g. AIS_HUB_INGEST__AIS_UDP__PORT=4011.
ingest:
ais_udp:
host: 0.0.0.0
port: 4001
gps_uart:
device: /dev/ttyS1
baud: 38400
enabled: true
radio_udp:
host: 0.0.0.0
port: 4010
# AIS-catcher Mini binary telemetry (see ais-mini.conf slot_udp_*/rssi_udp_*/event_udp_*).
# Set enabled: true and point AIS-catcher's *_udp_host/port here (defaults match its defaults).
aiscatcher_udp:
enabled: false
host: 127.0.0.1
slot_port: 10111
slot_detail_port: 10112
rssi_port: 10113
event_port: 10114
publish:
http:
host: 0.0.0.0
port: 8080
udp_events:
host: 127.0.0.1
port: 7001
max_datagram_bytes: 1400
udp_nmea:
host: 127.0.0.1
port: 6007
udp_tx_outbox:
host: 127.0.0.1
port: 6010
nmea_tail:
in_memory_ring: 500
storage:
path: /var/lib/ais_hub/ais_hub.db
store_raw_nmea: false
retention:
ais_dynamic_days: 7
gps_fix_days: 7
raw_nmea_days: 2
radio_telemetry_days: 7
writer:
batch_size: 200
flush_interval_ms: 500
retention_interval_sec: 3600
queues:
parser_in: 2048
ws_client: 512
udp_events: 1024
storage_in: 4096
tx_outbox: 512
parser:
ais:
# Ignore checksum completely (even if present and invalid).
# If enabled, this overrides allow_checksumless.
ignore_checksum: false
# Some real-world NMEA forwarders omit the *HH checksum on !AIVDM/!AIVDO.
# OpenCPN is often tolerant; ais_hub is strict by default.
allow_checksumless: false
# Some multi-fragment AIS sentences come with an empty seq_id (field 3).
# Enabling this uses a best-effort heuristic to reassemble those fragments.
allow_multipart_without_seq_id: false
# TTL for in-flight multipart reassembly buffers.
multipart_ttl_sec: 60.0
logging:
level: INFO
file: /var/log/ais_hub/ais_hub.log
max_bytes: 10485760
backup_count: 5
json: true
docs/API.md
+630
View File
@@ -0,0 +1,630 @@
# ais_hub API reference
Полный контракт всех публичных интерфейсов `ais_hub` для веб-морды / SPI-bridge / BLE-сервера.
- **Base URL (HTTP/WS):** `http://<host>:8081` (порт из `publish.http.port`).
- **Авторизация:** нет (сервис привязан к localhost / доверенной сети).
- **Кодировка:** `UTF-8`. Все JSON — одна строка либо pretty, клиент не должен полагаться на форматирование.
- **Timestamps:** все поля `ts` — **Unix epoch в секундах**, `float`. Поля в миллисекундах (от AIS-catcher) явно называются `*_unix_ms`.
- **Nullability:** любое поле с неизвестным значением = `null`, даже если тип в таблице int/float.
- **Стабильность:** REST пути версионированы (`/api/v1/*`). Новые поля могут добавляться без breaking change — клиент должен игнорировать неизвестные ключи.
---
## 1. HTTP REST
### 1.1 `GET /api/v1/health`
Liveness probe. Всегда 200, если процесс жив.
```json
{
"status": "ok",
"version": "0.1.0",
"uptime_sec": 12345.678
}
```
---
### 1.2 `GET /api/v1/stats`
Все внутренние счётчики и gauges. Полезно для мониторинга и диагностики.
```json
{
"uptime_sec": 12345.678,
"started_at": 1700000000.123,
"counters": {
"ais_udp_datagrams": 10423,
"ais_udp_malformed": 2,
"ais_reports": 9812,
"ais_msg_1": 4200,
"ais_msg_3": 1800,
"ais_msg_5": 410,
"ais_msg_18": 3200,
"ais_msg_24": 202,
"ais_fragment_errors": 0,
"ais_fragment_timeouts": 0,
"parser_errors": 1,
"parser_checksum_errors": 0,
"state_ownship_updates": 1234,
"state_targets_new": 120,
"state_targets_updates": 9500,
"state_base_station_updates": 0,
"state_aton_updates": 0,
"state_rssi_updates": 45200,
"state_slot_bitmap_updates": 12,
"state_slot_detail_updates": 12,
"state_signal_events": 8800,
"state_warmup_vessels": 42,
"storage_batches": 950,
"storage_rows": 31200,
"storage_errors": 0,
"udp_events_sent": 123456,
"udp_events_oversize": 3,
"supervisor_restarts_ingest_ais_udp": 1,
"ws_clients_total": 4
},
"gauges": {
"ws_clients": 1,
"storage_last_batch": 200
}
}
```
**Ключевые counters для UI**
- `ais_msg_<N>` — сколько сообщений каждого типа декодировано (1/2/3/4/5/18/19/21/24/27 и т.д.).
- `ais_fragment_*` — проблемы мультифрагментной сборки.
- `state_targets_new` — количество впервые увиденных MMSI.
- `state_warmup_vessels` — сколько судов поднято из SQLite при старте (persistence).
- `supervisor_restarts_*` — если не 0, какой-то subsystem падает.
---
### 1.3 `GET /api/v1/ownship`
Последний GPS-fix (собственное судно).
```json
{
"ts": 1700000000.123,
"lat": 59.123456,
"lon": 10.987654,
"sog": 5.4,
"cog": 128.5,
"alt": 12.3,
"fix_quality": 1,
"sats": 9,
"hdop": 0.8
}
```
| Поле | Тип | Описание |
| --- | --- | --- |
| `ts` | float | Unix-время последнего обновления. 0 если fix ещё не пришёл. |
| `lat`, `lon` | float \| null | Широта/долгота в градусах (WGS-84). |
| `sog` | float \| null | Speed over ground, узлы. |
| `cog` | float \| null | Course over ground, градусы от истинного севера. |
| `alt` | float \| null | Высота над эллипсоидом, метры (GGA). |
| `fix_quality` | int \| null | NMEA GGA fix quality (0=нет, 1=GPS, 2=DGPS…). |
| `sats` | int \| null | Количество спутников в решении (GGA). |
| `hdop` | float \| null | Horizontal dilution of precision. |
---
### 1.4 `GET /api/v1/vessels`
Все известные MMSI, отсортированные по `last_seen` убыванию.
**Query-параметры**
| Параметр | Тип | По умолчанию | Описание |
| --- | --- | --- | --- |
| `since` | float (unix ts) | — | Вернуть только target'ы с `last_seen >= since`. |
| `limit` | int | без лимита | Срез первых N (после сортировки). |
**Response** — массив объектов `MergedTarget` (см. 1.5).
---
### 1.5 `GET /api/v1/vessels/{mmsi}`
Один target. `mmsi` в URL — целое число.
**Коды ответа**
- `200` — найдено;
- `400` — невалидный MMSI;
- `404` — `{"error": "not found", "mmsi": <mmsi>}`.
```json
{
"mmsi": 506140446,
"name": "CELESTIAL STAR",
"callsign": "V7AB3",
"imo": 9123456,
"ship_type": 70,
"dims": { "a": 180, "b": 30, "c": 16, "d": 16 },
"voyage": {
"eta": "05-20 12:00",
"draught": 7.5,
"destination": "ROTTERDAM"
},
"dynamic": {
"lat": 59.912345,
"lon": 10.754321,
"sog": 12.4,
"cog": 210.3,
"heading": 208,
"nav_status": 0,
"rot": -1.2
},
"signal": {
"last_db": -72.5,
"last_ts": 1700000000.123,
"last_slot": 1487,
"last_channel": "A"
},
"last_static_ts": 1699990000.0,
"last_dynamic_ts": 1700000000.123,
"last_seen": 1700000000.123,
"sources": ["ais_udp:0.0.0.0:5005:127.0.0.1"],
"msg_types": [5, 18, 24]
}
```
| Блок | Поле | Тип | Описание |
| --- | --- | --- | --- |
| root | `mmsi` | int | MMSI (9 цифр для судов, другие диапазоны — base stations / AtoN / SAR и т.д.). |
| root | `name` | str \| null | Имя судна (type 5 / 24A). Trim + trailing `@` удалены. |
| root | `callsign` | str \| null | Callsign (type 5 / 24B). |
| root | `imo` | int \| null | IMO номер (type 5). `0` нормализуется в `null` при наличии. |
| root | `ship_type` | int \| null | Ship/cargo type 0–99 (всегда плоский int, не enum). |
| `dims` | `a/b/c/d` | int \| null | Расстояния от GPS-антенны: `a`=нос, `b`=корма, `c`=левый борт, `d`=правый борт; метры. Длина = a+b, ширина = c+d. |
| `voyage` | `eta` | str \| null | ETA в формате `MM-DD HH:MM` (как отдаёт pyais). |
| `voyage` | `draught` | float \| null | Осадка, метры. |
| `voyage` | `destination` | str \| null | Порт назначения (до 20 символов). |
| `dynamic` | `lat`, `lon` | float \| null | Позиция, градусы WGS-84. |
| `dynamic` | `sog` | float \| null | Скорость, узлы (0…102.2). |
| `dynamic` | `cog` | float \| null | Course over ground, градусы (0…359.9). |
| `dynamic` | `heading` | float \| null | Heading, градусы (0…359, 511=unknown → `null`). |
| `dynamic` | `nav_status` | int \| null | Navigational status (0–15). |
| `dynamic` | `rot` | float \| null | Rate of turn, град/мин. |
| `signal` | `last_db` | float \| null | Уровень сигнала последнего декода (от AIS-catcher decode events, port 10114). |
| `signal` | `last_ts` | float \| null | Unix-время последнего decode event. |
| `signal` | `last_slot` | int \| null | TDMA-слот (0…2249). |
| `signal` | `last_channel` | str \| null | `"A"` или `"B"`. |
| root | `last_static_ts` | float | Последний приход type-5 / 24 для этого MMSI. 0 если не было. |
| root | `last_dynamic_ts` | float | Последний приход type-1/2/3/18/19/27. |
| root | `last_seen` | float | `max(last_static_ts, last_dynamic_ts, last_signal_ts)`. |
| root | `sources` | str[] | Отсортированный список source-тегов, с которых приходили данные (см. формат в разделе "Источники"). |
| root | `msg_types` | int[] | Отсортированный список типов AIS-сообщений, наблюдавшихся для этого MMSI. |
---
### 1.6 `GET /api/v1/base_stations`
AIS base stations (msg 4/11).
```json
[
{ "mmsi": 2570001, "ts": 1700000000.0, "lat": 59.9, "lon": 10.7, "epfd": 1 }
]
```
---
### 1.7 `GET /api/v1/atons`
Aids-to-Navigation (msg 21).
```json
[
{
"mmsi": 992570001,
"ts": 1700000000.0,
"lat": 59.9, "lon": 10.7,
"type": 3,
"name": "FLAKFORTET",
"virtual": false
}
]
```
| Поле | Тип | Описание |
| --- | --- | --- |
| `type` | int \| null | AtoN type (1–31). |
| `virtual` | bool \| null | `true` = виртуальный AtoN. |
---
### 1.8 `GET /api/v1/slots`
Снапшот TDMA-слотов и occupancy (наполняется из AIS-catcher, порты 10111/10112).
```json
{
"per_channel": {},
"occupancy": {
"A": {
"ts": 1700000060.0,
"utc_minute": 28333334,
"slots_total": 2250,
"occupied_count": 437,
"occupied_fraction": 0.1942,
"slot0_unix_ms": 1700000040000,
"first_occupied_unix_ms": 1700000041123
},
"B": { ... }
},
"detail": {
"A": {
"ts": 1700000060.0,
"utc_minute": 28333334,
"slot0_unix_ms": 1700000040000,
"entries": [
{ "slot": 42, "level_db": -72.5 },
{ "slot": 1487, "level_db": -80.1 }
]
}
}
}
```
- `per_channel` — зарезервировано (пусто в текущей версии).
- `occupancy` — загрузка эфира. Обновляется раз в минуту.
- `detail` — список занятых слотов с уровнем сигнала. Обновляется раз в минуту. Может отсутствовать или быть с пустым `entries`.
---
### 1.9 `GET /api/v1/radio`
Мгновенная мощность каналов A/B (от AIS-catcher RSSI, порт 10113, ~10 Гц).
```json
{
"power": {
"ts": 1700000000.123,
"power_a_db": -42.25,
"power_b_db": -55.75
}
}
```
Если RSSI ни разу не приходил — `"power": {}`.
---
### 1.10 `GET /api/v1/logs`
Service logs tail из in-memory ring buffer (не raw NMEA).
**Query-параметры**
| Параметр | Тип | По умолчанию | Описание |
| --- | --- | --- | --- |
| `level` | str | — | Фильтр: `DEBUG`/`INFO`/`WARNING`/`ERROR`/`CRITICAL` (точный match). |
| `limit` | int | 200 | Сколько последних записей вернуть. |
| `since` | float | — | Unix-ts; вернуть записи с `ts >= since`. |
```json
[
{
"ts": 1700000000.123,
"level": "INFO",
"logger": "ais_hub.app",
"message": "warm start: 42 vessels, 0 base_stations, 0 atons from sqlite"
},
{
"ts": 1700000001.234,
"level": "WARNING",
"logger": "ais_hub.ingest.ais_udp",
"message": "ais_udp: cannot bind 0.0.0.0:5005 — already in use..."
}
]
```
---
### 1.11 `GET /api/v1/nmea/tail`
Raw NMEA tail — отдельно от `/logs`. Сперва отдаёт in-memory ring buffer (всегда ведётся), при необходимости фоллбэчит в SQLite `raw_nmea` (если `storage.store_raw_nmea: true`).
**Query-параметры**
| Параметр | Тип | По умолчанию | Описание |
| --- | --- | --- | --- |
| `source` | str | — | Фильтр: `ais`, `gps`, `radio`. |
| `limit` | int | 200 | Сколько последних строк. |
```json
[
{
"ts": 1700000000.123,
"source": "ais_udp:0.0.0.0:5005:127.0.0.1",
"kind": "ais",
"line": "!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F"
},
{
"ts": 1700000000.456,
"source": "gps_uart:/dev/ttyS1",
"kind": "gps",
"line": "$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,,,*7A"
}
]
```
---
### 1.12 `GET /api/v1/history/positions`
История AIS dynamic для конкретного MMSI из SQLite.
**Query-параметры**
| Параметр | Тип | Обязательный | Описание |
| --- | --- | --- | --- |
| `mmsi` | int | да | MMSI. |
| `from` | float (unix ts) | нет | Нижняя граница по `ts`. |
| `to` | float (unix ts) | нет | Верхняя граница по `ts`. |
| `limit` | int | нет | По умолчанию 1000, максимум 100000. |
**Коды**
- `200` — массив точек (может быть пустой).
- `400` — нет/невалидный `mmsi`.
- `503` — `{"error": "history storage disabled"}`, если `storage.path` пуст.
```json
[
{
"ts": 1700000000.123,
"lat": 59.912345, "lon": 10.754321,
"sog": 12.4, "cog": 210.3,
"heading": 208, "nav_status": 0, "rot": -1.2,
"msg_type": 18
}
]
```
Отсортировано по `ts` **убыванию**.
---
### 1.13 `POST /api/v1/tx`
Инжекция NMEA-строк в TX outbox (UDP 127.0.0.1:6010 → SPI bridge).
**Body (JSON)** — одна из двух форм:
```json
{ "payload": "!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F" }
```
или
```json
{ "nmea": [
"!AIVDM,1,1,,A,...,0*2F",
"$GPRMC,...,*7A"
] }
```
**Response (200)**
```json
{
"queued": 1,
"rejected": [],
"queue_depth": 3
}
```
**Response (400)** — невалидный JSON или нет `payload`/`nmea`.
**Response (503)** — outbox переполнен:
```json
{
"error": "outbox full",
"accepted": ["!AIVDM,..."],
"rejected": []
}
```
Каждая строка валидируется: структура NMEA + XOR-чексумма `*HH`. Невалидные попадают в `rejected: [{"line": "...", "reason": "invalid nmea or checksum"}]`, валидные — в outbox.
---
## 2. WebSocket `/ws`
**URL:** `ws://<host>:8081/ws`
**Subprotocol:** нет.
**Heartbeat:** 30 секунд (aiohttp ping, клиент должен отвечать на pong).
**Направление:** publish-only от сервера; сообщения от клиента игнорируются.
### 2.1 Формат конверта
Все сообщения — JSON одной строкой:
```json
{ "type": "<event_type>", "ts": 1700000000.123, "data": { ... } }
```
### 2.2 Первое сообщение после коннекта — `state.snapshot`
```json
{
"type": "state.snapshot",
"ts": 1700000000.123,
"data": {
"ownship": { ... }, // см. GET /api/v1/ownship
"vessels": [ ... ], // см. GET /api/v1/vessels
"base_stations": [ ... ], // см. GET /api/v1/base_stations
"atons": [ ... ], // см. GET /api/v1/atons
"slots": { ... }, // см. GET /api/v1/slots
"stats": { ... } // см. GET /api/v1/stats
}
}
```
Клиент инициализирует своё состояние из снапшота, затем применяет живые updates.
### 2.3 Живые события
| `type` | Когда публикуется | `data` |
| --- | --- | --- |
| `ownship.update` | На каждый GPS-fix с новыми полями | Объект, идентичный `GET /api/v1/ownship`. |
| `target.update` | На каждое AIS dynamic/static для MMSI, которое что-то изменило | Полный `MergedTarget` (см. 1.5). |
| `base_station.update` | Type 4/11 | `{mmsi, ts, lat, lon, epfd}`. |
| `aton.update` | Type 21 | `{mmsi, ts, lat, lon, type, name, virtual}`. |
| `radio.update` (JSON) | Радиотелеметрия в JSON (port `radio_udp`) | `{channel, rssi, snr, slot, raw}`. |
| `radio.update` (RSSI) | От AIS-catcher RSSI (10113, ~10 Гц) | `{source: "aiscatcher_rssi", power_a_db, power_b_db}`. Отличайте по `data.source`. |
| `slots.update` | Bitmap (10111, ~1/мин) | `{channel, ts, utc_minute, slots_total, occupied_count, occupied_fraction, slot0_unix_ms, first_occupied_unix_ms, bitmap_hex}`. |
| `slots.detail` | Detail (10112, ~1/мин) | `{channel, ts, utc_minute, slot0_unix_ms, entries:[{slot, level_db}]}`. |
| `signal.update` | Пачка decode events (10114) | `{channel, events:[{unix_ms, slot, mmsi, level_db}]}`. |
| `stats.update` | Раз в 5 секунд | Полный snapshot stats (см. 1.2). |
**Правила апдейта** клиента:
- `target.update`: merge по `mmsi`; не терять предыдущие поля (новый пакет с `null` в static не перезаписывает старые имя/IMO — backend этим занимается через `COALESCE` в SQLite и `merge.apply`).
- `signal.update`: обновлять sparkline сигнала; `target.update` уже содержит `signal` блок, но `signal.update` приходит чаще (с каждой успешной декодой пакета).
- Порядок событий не гарантируется быть идеально хронологическим между разными типами, но в пределах одного `type` он монотонный.
### 2.4 Обратное давление
У каждого WS-клиента внутренняя очередь размера `queues.ws_client` (по умолчанию 512). Медленный клиент приводит к drop-oldest; счётчик — `stats.counters.ws_dropped`.
---
## 3. UDP интерфейсы (localhost only)
Все UDP-сокеты слушают/шлют только на `127.0.0.1`.
### 3.1 UDP JSON events — `127.0.0.1:7001` (publish, out)
- Контракт: **одно событие = одна UDP-датаграмма**.
- Формат: JSON-сериализованный `Event` envelope (такой же, как в WS, но без `state.snapshot`).
- Максимальный размер датаграммы: `publish.udp_events.max_datagram_bytes` (1400 по умолчанию). Превышение → drop + `stats.counters.udp_events_oversize`.
- Подходит для локальных читателей (BLE-сервер, SPI bridge), которым не нужен full-duplex.
Пример одной датаграммы:
```json
{"type":"target.update","ts":1700000000.123,"data":{"mmsi":506140446,...}}
```
### 3.2 UDP raw NMEA fan-out — `127.0.0.1:6007` (publish, out)
- Контракт: **одна NMEA-строка = одна UDP-датаграмма**, завершённая `\r\n`.
- Формат: как было получено по UDP/UART (после валидации структуры, но сохраняется исходная `*HH`).
- Используется SPI bridge'ем для сквозного форвардинга в MCU.
Пример одной датаграммы:
```
!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F\r\n
```
### 3.3 UDP TX outbox — `127.0.0.1:6010` (publish, out)
- Контракт: **одна NMEA-строка = одна UDP-датаграмма**, `\r\n` в конце.
- Источник: `POST /api/v1/tx`, внутренние эмиттеры.
- Это **не ingest**: сервис только шлёт наружу. Для приёма NMEA использовать `127.0.0.1:5005` (AIS UDP) или UART.
### 3.4 Ingest AIS UDP — `<ingest.ais_udp>`
- Порт в конфиге (по умолчанию 4001, у вас на устройстве 5005 — совпадает с AIS-catcher).
- Формат входа: **одна или несколько NMEA-строк в одной датаграмме**, разделитель `\r\n`/`\n`.
- Сервис декодирует AIVDM/AIVDO и мультифрагментные.
### 3.5 Ingest Radio UDP (JSON) — `<ingest.radio_udp>`
- Порт по умолчанию 4010.
- Формат: одна датаграмма = один JSON-объект с полями `{channel, rssi, snr, slot, ...}` — любые ключи сохраняются в `raw`.
### 3.6 Ingest AIS-catcher binary — `<ingest.aiscatcher_udp>`
Четыре независимых UDP-листенера (все BE, binary, подробные layout'ы — в `src/ais_hub/parser/aiscatcher.py`):
| Порт по умолчанию | Что | Длина |
| --- | --- | --- |
| 10111 | Slot bitmap occupancy (per minute) | 315 Б |
| 10112 | Slot detail (per-slot levels) | 15 + 6×N |
| 10113 | RSSI IQ pair (~10 Гц) | 8 Б |
| 10114 | Decode events (mmsi/slot/level) | 3 + 18×N |
В REST/WS эти потоки отображаются как `/api/v1/radio`, `/api/v1/slots`, `radio.update`, `slots.update`, `slots.detail`, `signal.update`.
---
## 4. Источники (`source` / `sources`)
Формат строки источника в `RawFrame.source`, `sources` array у target'а, в `GET /api/v1/nmea/tail`:
| Prefix | Пример | Смысл |
| --- | --- | --- |
| `ais_udp:<listener>:<sender_ip>` | `ais_udp:0.0.0.0:5005:127.0.0.1` | NMEA пришёл по UDP. Sender-порт намеренно не включается (ephemeral, скачет). |
| `gps_uart:<device>` | `gps_uart:/dev/ttyS1` | NMEA с UART. |
| `radio_udp:<listener>:<sender_ip>` | `radio_udp:0.0.0.0:4010:127.0.0.1` | JSON radio telemetry. |
| `ownship` | `ownship` | Используется в WriteJob для `gps_fix` в БД. |
---
## 5. Коды ответов
| HTTP | Когда |
| --- | --- |
| `200` | OK |
| `400` | Невалидные query-параметры / JSON-тело / MMSI. |
| `404` | Unknown MMSI в `/api/v1/vessels/{mmsi}`. |
| `500` | SQLite query провалился. |
| `503` | Storage disabled (`history/positions`) или TX outbox full. |
---
## 6. Пример клиента на TypeScript
```ts
type Event = { type: string; ts: number; data: any };
async function loadInitial() {
const [ownship, vessels, stats] = await Promise.all([
fetch("/api/v1/ownship").then(r => r.json()),
fetch("/api/v1/vessels?limit=500").then(r => r.json()),
fetch("/api/v1/stats").then(r => r.json()),
]);
return { ownship, vessels, stats };
}
function connectWs(onEvent: (ev: Event) => void) {
const ws = new WebSocket(`ws://${location.host}/ws`);
ws.onmessage = (m) => onEvent(JSON.parse(m.data));
ws.onclose = () => setTimeout(() => connectWs(onEvent), 2000);
return ws;
}
```
Стандартная схема для web UI:
1. Загрузить `GET /api/v1/vessels` (для первичного рендера карты).
2. Открыть `/ws` — получить `state.snapshot`, заменить состояние им.
3. Применять `target.update`/`ownship.update`/и т.д. поверх.
4. `signal.update` и `radio.update` (RSSI) — для графиков / индикатора сигнала.
5. `slots.update` / `slots.detail` — для визуализации загрузки эфира.
---
## 7. Persistence при рестарте
При старте сервис автоматически загружает из SQLite:
- `vessel_static` → все `MergedTarget.<name|callsign|imo|ship_type|dim_*|eta|draught|destination>`, а также `last_static_ts`. Dynamic (lat/lon/sog/cog/heading/nav_status/rot) не восстанавливается — придёт с первым новым type-1/2/3/18.
- `base_station` → `State.base_stations`.
- `aton` → `State.atons`.
В логе при успешном warm start:
```
INFO warm start: 42 vessels, 0 base_stations, 0 atons from sqlite
```
И счётчики `stats.counters.state_warmup_*`.
Таким образом, сценарий «принял 5-е и 24-е, перезапустил, принял 18-е» теперь отдаст target с заполненными именем/IMO/размерениями сразу.
pyproject.toml
+39
View File
@@ -0,0 +1,39 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "ais_hub"
version = "0.1.0"
description = "Central AIS/GPS/radio telemetry aggregation service for embedded Linux."
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Proprietary" }
authors = [{ name = "AisHub" }]
dependencies = [
"aiohttp>=3.9,<4",
"pyais>=2.6,<3",
"pyserial-asyncio>=0.6,<1",
"aiosqlite>=0.19,<1",
"PyYAML>=6,<7",
]
[project.optional-dependencies]
dev = [
"pytest>=8",
"pytest-asyncio>=0.23",
"pytest-cov>=4",
]
[project.scripts]
ais_hub = "ais_hub.__main__:main"
[tool.setuptools.packages.find]
where = ["src"]
[tool.setuptools.package-data]
ais_hub = ["py.typed"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
src/ais_hub/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""ais_hub: central AIS/GPS/radio telemetry aggregation service."""
from .version import __version__
__all__ = ["__version__"]
src/ais_hub/__main__.py
+48
View File
@@ -0,0 +1,48 @@
"""CLI entry point for ais_hub."""
from __future__ import annotations
import argparse
import asyncio
import logging
import sys
from pathlib import Path
from .app import Application
from .config import Config, load_config
from .logging_setup import setup_logging
from .version import __version__
log = logging.getLogger(__name__)
def _parse_args(argv: list[str]) -> argparse.Namespace:
p = argparse.ArgumentParser(prog="ais_hub", description="AIS/GPS/radio aggregation service")
p.add_argument("--config", "-c", type=Path, required=False, help="Path to YAML config file")
p.add_argument("--version", action="version", version=f"ais_hub {__version__}")
return p.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = _parse_args(argv if argv is not None else sys.argv[1:])
try:
cfg: Config = load_config(args.config)
except Exception as exc:
print(f"config error: {exc}", file=sys.stderr)
return 2
setup_logging(cfg.logging)
log.info("starting ais_hub %s", __version__)
try:
asyncio.run(Application(cfg).run())
except KeyboardInterrupt:
log.info("interrupted")
return 0
except Exception:
log.exception("fatal error")
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
src/ais_hub/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/__pycache__/app.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/__pycache__/config.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/__pycache__/logging_setup.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/__pycache__/version.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/app.py
+376
View File
@@ -0,0 +1,376 @@
"""Application orchestrator: wires subsystems, runs supervisors, handles shutdown."""
from __future__ import annotations
import asyncio
import logging
import signal
import time
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable
import aiosqlite
from aiohttp import web
from .config import Config
from .core.bus import EventBus
from .core.models import AisReport, Event, GpsFix, RadioReport, RawFrame, TxMessage
from .core.state import State
from .core.stats import Stats
from .ingest.ais_udp import AisUdpIngest
from .ingest.aiscatcher_udp import AisCatcherUdpIngest
from .ingest.base import AddressInUseError, RawTap
from .ingest.gps_uart import GpsUartIngest
from .ingest.radio_udp import RadioUdpIngest
from .parser.ais import AisAssembler
from .parser.gps import GpsParser
from .parser.nmea_utils import parse_sentence
from .parser.radio import RadioParser
from .publish import rest as rest_module
from .publish import ws as ws_module
from .publish.queues import put_drop_oldest
from .publish.udp_events import UdpEventsPublisher
from .publish.udp_nmea import UdpNmeaPublisher
from .publish.udp_tx_outbox import UdpTxOutboxPublisher
from .storage import db as storage_db
from .storage import queries as storage_queries
from .storage import retention as storage_retention
from .storage.writer import StorageEventSink, Writer, _RawNmeaRow
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Shared app context
# ---------------------------------------------------------------------------
@dataclass
class Context:
cfg: Config
stats: Stats
bus: EventBus
state: State
raw_tap: RawTap
tx_outbox: "asyncio.Queue[TxMessage]"
db: aiosqlite.Connection | None = None
storage_sink: StorageEventSink | None = None
# ---------------------------------------------------------------------------
# Supervisor
# ---------------------------------------------------------------------------
async def _supervise(
name: str,
factory: Callable[[], Awaitable[None]],
stats: Stats,
shutdown: asyncio.Event,
*,
initial_backoff: float = 1.0,
max_backoff: float = 30.0,
) -> None:
"""Run ``factory`` and restart on errors with exponential backoff.
Exits cleanly when ``shutdown`` is set.
"""
backoff = initial_backoff
while not shutdown.is_set():
try:
await factory()
# Normal return also counts as "done"; for long-running tasks this
# should not happen, but we don't try to restart healthy exits.
return
except asyncio.CancelledError:
raise
except AddressInUseError as exc:
# Expected runtime condition: port collision. Log compactly and
# keep retrying; no traceback spam.
stats.incr(f"supervisor_restarts_{name}")
log.warning("subsystem %s: %s (retry in %.1fs)", name, exc, backoff)
try:
await asyncio.wait_for(shutdown.wait(), timeout=backoff)
return
except asyncio.TimeoutError:
pass
backoff = min(backoff * 2, max_backoff)
except Exception:
stats.incr(f"supervisor_restarts_{name}")
log.exception("subsystem %s crashed; restarting in %.1fs", name, backoff)
try:
await asyncio.wait_for(shutdown.wait(), timeout=backoff)
return
except asyncio.TimeoutError:
pass
backoff = min(backoff * 2, max_backoff)
# ---------------------------------------------------------------------------
# Parser task
# ---------------------------------------------------------------------------
async def _parser_task(
parser_in: "asyncio.Queue[RawFrame]",
state: State,
stats: Stats,
cfg: Config,
) -> None:
"""Consume RawFrames, dispatch to AIS/GPS/radio parsers, update state."""
ais = AisAssembler(
stats=stats,
ttl_sec=cfg.parser.ais.multipart_ttl_sec,
ignore_checksum=cfg.parser.ais.ignore_checksum,
allow_checksumless=cfg.parser.ais.allow_checksumless,
allow_multipart_without_seq_id=cfg.parser.ais.allow_multipart_without_seq_id,
)
gps = GpsParser(stats=stats)
radio = RadioParser(stats=stats)
last_gc = time.monotonic()
while True:
frame = await parser_in.get()
try:
if frame.kind == "ais":
reports = ais.feed(frame.source, frame.line, ts=frame.ts)
for rep in reports:
state.apply_ais(rep)
elif frame.kind == "gps":
fix = gps.feed(frame.source, frame.line, ts=frame.ts)
if fix is not None:
state.apply_gps(fix)
elif frame.kind == "radio":
report = radio.feed(frame.source, frame.line, ts=frame.ts)
if report is not None:
state.apply_radio(report)
except Exception:
stats.incr("parser_task_errors")
log.exception("parser task error on kind=%s source=%s", frame.kind, frame.source)
now = time.monotonic()
if now - last_gc > 5.0:
ais.gc(frame.ts)
last_gc = now
# ---------------------------------------------------------------------------
# Stats emitter
# ---------------------------------------------------------------------------
async def _stats_emitter(bus: EventBus, stats: Stats, period: float = 5.0) -> None:
while True:
await asyncio.sleep(period)
snap = stats.snapshot()
bus.publish(Event(type="stats.update", ts=time.time(), data=snap))
# ---------------------------------------------------------------------------
# Raw-NMEA -> storage bridge (optional, when store_raw_nmea=True)
# ---------------------------------------------------------------------------
async def _raw_to_storage_task(
raw_queue: "asyncio.Queue[RawFrame]",
storage_in: "asyncio.Queue[Any]",
stats: Stats,
) -> None:
while True:
frame = await raw_queue.get()
put_drop_oldest(storage_in, _RawNmeaRow(frame=frame), stats, "storage_in_dropped")
# ---------------------------------------------------------------------------
# Application
# ---------------------------------------------------------------------------
class Application:
def __init__(self, cfg: Config) -> None:
self._cfg = cfg
self._stats = Stats()
self._bus = EventBus(stats=self._stats, default_maxsize=1024)
self._state = State(bus=self._bus, stats=self._stats)
self._raw_tap = RawTap(
stats=self._stats,
ring_size=cfg.publish.nmea_tail.in_memory_ring,
)
self._shutdown = asyncio.Event()
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
async def run(self) -> None:
loop = asyncio.get_running_loop()
# Install signal handlers where supported (POSIX).
for sig in (signal.SIGTERM, signal.SIGINT):
try:
loop.add_signal_handler(sig, self._request_shutdown)
except (NotImplementedError, RuntimeError):
# Windows / test environments.
pass
parser_in: asyncio.Queue[RawFrame] = asyncio.Queue(maxsize=self._cfg.queues.parser_in)
udp_nmea_queue: asyncio.Queue[RawFrame] = asyncio.Queue(maxsize=max(512, self._cfg.queues.parser_in // 2))
storage_in: asyncio.Queue[Any] = asyncio.Queue(maxsize=self._cfg.queues.storage_in)
tx_outbox: asyncio.Queue[TxMessage] = asyncio.Queue(maxsize=self._cfg.queues.tx_outbox)
# Subscribe the UDP NMEA fan-out to the raw tap.
self._raw_tap.subscribe(udp_nmea_queue, "udp_nmea_dropped")
# Optional raw-NMEA -> storage feed.
raw_storage_queue: asyncio.Queue[RawFrame] | None = None
if self._cfg.storage.store_raw_nmea:
raw_storage_queue = asyncio.Queue(maxsize=self._cfg.queues.storage_in)
self._raw_tap.subscribe(raw_storage_queue, "storage_in_dropped")
# Open SQLite (optional).
db_conn: aiosqlite.Connection | None = None
if self._cfg.storage.path:
try:
db_conn = await storage_db.connect(self._cfg.storage.path)
log.info("sqlite opened: %s", self._cfg.storage.path)
except Exception:
log.exception("sqlite open failed; continuing without history")
db_conn = None
# Warm start: preload persisted static/base/aton rows into state so
# a restart retains known vessel names/dims/etc. even before the
# first type-5/24 re-arrives over the air.
if db_conn is not None:
try:
vessels = await storage_queries.load_vessel_static(db_conn)
base_stations = await storage_queries.load_base_stations(db_conn)
atons = await storage_queries.load_atons(db_conn)
counts = self._state.warmup(
vessels=vessels,
base_stations=base_stations,
atons=atons,
)
log.info(
"warm start: %d vessels, %d base_stations, %d atons from sqlite",
counts["vessels"], counts["base_stations"], counts["atons"],
)
self._stats.incr("state_warmup_vessels", counts["vessels"])
self._stats.incr("state_warmup_base_stations", counts["base_stations"])
self._stats.incr("state_warmup_atons", counts["atons"])
except Exception:
log.exception("warm start failed; continuing with empty state")
storage_sink: StorageEventSink | None = None
if db_conn is not None:
storage_sink = StorageEventSink(
bus=self._bus,
state=self._state,
stats=self._stats,
out_queue=storage_in,
store_raw_nmea=self._cfg.storage.store_raw_nmea,
)
ctx = Context(
cfg=self._cfg,
stats=self._stats,
bus=self._bus,
state=self._state,
raw_tap=self._raw_tap,
tx_outbox=tx_outbox,
db=db_conn,
storage_sink=storage_sink,
)
# Build aiohttp app for REST + WS.
aio_app = web.Application()
aio_app["ctx"] = ctx
rest_module.register_routes(aio_app)
ws_module.register_routes(aio_app)
runner = web.AppRunner(aio_app)
await runner.setup()
site = web.TCPSite(runner, host=self._cfg.publish.http.host, port=self._cfg.publish.http.port)
await site.start()
log.info("http listening on %s:%d", self._cfg.publish.http.host, self._cfg.publish.http.port)
tasks: list[asyncio.Task[Any]] = []
def spawn(name: str, factory: Callable[[], Awaitable[None]]) -> None:
t = asyncio.create_task(
_supervise(name, factory, self._stats, self._shutdown),
name=name,
)
tasks.append(t)
# Parser
spawn("parser", lambda: _parser_task(parser_in, self._state, self._stats, self._cfg))
# Stats emitter
spawn("stats_emitter", lambda: _stats_emitter(self._bus, self._stats, period=5.0))
# Ingest
ais_udp = AisUdpIngest(self._cfg.ingest.ais_udp,
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
radio_udp = RadioUdpIngest(self._cfg.ingest.radio_udp,
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
gps_uart = GpsUartIngest(self._cfg.ingest.gps_uart,
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
spawn("ingest_ais_udp", ais_udp.run)
spawn("ingest_radio_udp", radio_udp.run)
spawn("ingest_gps_uart", gps_uart.run)
if self._cfg.ingest.aiscatcher_udp.enabled:
aiscatcher = AisCatcherUdpIngest(
self._cfg.ingest.aiscatcher_udp,
state=self._state,
stats=self._stats,
)
spawn("ingest_aiscatcher_udp", aiscatcher.run)
# Publishers
udp_events = UdpEventsPublisher(self._cfg.publish.udp_events, self._bus, self._stats)
udp_nmea = UdpNmeaPublisher(self._cfg.publish.udp_nmea, self._stats, udp_nmea_queue)
udp_tx = UdpTxOutboxPublisher(self._cfg.publish.udp_tx_outbox, self._stats, tx_outbox)
spawn("udp_events", udp_events.run)
spawn("udp_nmea", udp_nmea.run)
spawn("udp_tx_outbox", udp_tx.run)
# Storage
if db_conn is not None and storage_sink is not None:
writer = Writer(db_conn, self._cfg.storage, self._stats, storage_in)
spawn("storage_writer", writer.run)
spawn("storage_sink", storage_sink.run)
spawn("storage_retention",
lambda: storage_retention.run_retention(db_conn, self._cfg.storage, self._stats))
if raw_storage_queue is not None:
spawn("raw_to_storage",
lambda: _raw_to_storage_task(raw_storage_queue, storage_in, self._stats))
# Wait for shutdown.
try:
await self._shutdown.wait()
finally:
log.info("shutting down ais_hub")
for t in tasks:
t.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
try:
await site.stop()
except Exception:
pass
try:
await runner.cleanup()
except Exception:
pass
await storage_db.close(db_conn)
log.info("ais_hub stopped")
def _request_shutdown(self) -> None:
if not self._shutdown.is_set():
log.info("shutdown signal received")
self._shutdown.set()
__all__ = ["Application"]
src/ais_hub/config.py
+272
View File
@@ -0,0 +1,272 @@
"""Configuration loader: YAML file + environment variable overrides.
Env-var convention:
AIS_HUB_<PATH> where path components are joined by "__".
Example:
AIS_HUB_INGEST__AIS_UDP__PORT=4011
-> cfg.ingest.ais_udp.port = 4011
AIS_HUB_LOGGING__LEVEL=DEBUG
-> cfg.logging.level = "DEBUG"
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field, fields, is_dataclass
from pathlib import Path
from typing import Any
import yaml
ENV_PREFIX = "AIS_HUB_"
ENV_SEP = "__"
# ---------------------------------------------------------------------------
# Dataclasses mirroring config.example.yaml structure
# ---------------------------------------------------------------------------
@dataclass
class AisUdpCfg:
host: str = "0.0.0.0"
port: int = 4001
@dataclass
class GpsUartCfg:
device: str = "/dev/ttyS1"
baud: int = 38400
enabled: bool = True
@dataclass
class RadioUdpCfg:
host: str = "0.0.0.0"
port: int = 4010
@dataclass
class AisCatcherUdpCfg:
"""Four binary telemetry streams from AIS-catcher Mini.
Ports mirror ``ais-mini.conf``:
* ``slot_port`` — 315-byte slot bitmap per minute
* ``slot_detail_port`` — per-slot levels per minute
* ``rssi_port`` — 8-byte RSSI pair at ~10 Hz
* ``event_port`` — decode events (ts, slot, mmsi, level)
"""
enabled: bool = False
host: str = "127.0.0.1"
slot_port: int = 10111
slot_detail_port: int = 10112
rssi_port: int = 10113
event_port: int = 10114
@dataclass
class IngestCfg:
ais_udp: AisUdpCfg = field(default_factory=AisUdpCfg)
gps_uart: GpsUartCfg = field(default_factory=GpsUartCfg)
radio_udp: RadioUdpCfg = field(default_factory=RadioUdpCfg)
aiscatcher_udp: AisCatcherUdpCfg = field(default_factory=AisCatcherUdpCfg)
@dataclass
class HttpCfg:
host: str = "0.0.0.0"
port: int = 8080
@dataclass
class UdpEventsCfg:
host: str = "127.0.0.1"
port: int = 7001
max_datagram_bytes: int = 1400
@dataclass
class UdpNmeaCfg:
host: str = "127.0.0.1"
port: int = 6007
@dataclass
class UdpTxOutboxCfg:
host: str = "127.0.0.1"
port: int = 6010
@dataclass
class NmeaTailCfg:
in_memory_ring: int = 500
@dataclass
class PublishCfg:
http: HttpCfg = field(default_factory=HttpCfg)
udp_events: UdpEventsCfg = field(default_factory=UdpEventsCfg)
udp_nmea: UdpNmeaCfg = field(default_factory=UdpNmeaCfg)
udp_tx_outbox: UdpTxOutboxCfg = field(default_factory=UdpTxOutboxCfg)
nmea_tail: NmeaTailCfg = field(default_factory=NmeaTailCfg)
@dataclass
class RetentionCfg:
ais_dynamic_days: int = 7
gps_fix_days: int = 7
raw_nmea_days: int = 2
radio_telemetry_days: int = 7
@dataclass
class WriterCfg:
batch_size: int = 200
flush_interval_ms: int = 500
@dataclass
class StorageCfg:
path: str = "/var/lib/ais_hub/ais_hub.db"
store_raw_nmea: bool = False
retention: RetentionCfg = field(default_factory=RetentionCfg)
writer: WriterCfg = field(default_factory=WriterCfg)
retention_interval_sec: int = 3600
@dataclass
class QueuesCfg:
parser_in: int = 2048
ws_client: int = 512
udp_events: int = 1024
storage_in: int = 4096
tx_outbox: int = 512
@dataclass
class LoggingCfg:
level: str = "INFO"
file: str | None = "/var/log/ais_hub/ais_hub.log"
max_bytes: int = 10 * 1024 * 1024
backup_count: int = 5
json: bool = True
@dataclass
class AisParserCfg:
"""AIS parser tolerance knobs.
Defaults are strict (drop checksumless/invalid and multipart without seq_id).
Some real-world NMEA forwarders omit these fields; enabling tolerances may
increase decoded target count at the cost of potential false merges when
streams are heavily interleaved.
"""
# Ignore checksum completely (even if present and invalid).
# If enabled, this overrides allow_checksumless.
ignore_checksum: bool = False
allow_checksumless: bool = False
allow_multipart_without_seq_id: bool = False
multipart_ttl_sec: float = 60.0
@dataclass
class ParserCfg:
ais: AisParserCfg = field(default_factory=AisParserCfg)
@dataclass
class Config:
ingest: IngestCfg = field(default_factory=IngestCfg)
publish: PublishCfg = field(default_factory=PublishCfg)
storage: StorageCfg = field(default_factory=StorageCfg)
queues: QueuesCfg = field(default_factory=QueuesCfg)
parser: ParserCfg = field(default_factory=ParserCfg)
logging: LoggingCfg = field(default_factory=LoggingCfg)
# ---------------------------------------------------------------------------
# Loading helpers
# ---------------------------------------------------------------------------
def _coerce(value: str, hint: Any) -> Any:
"""Coerce a string (from env var) into a field type.
``hint`` may be a concrete type (``int``, ``str`` …), a string
annotation (e.g. ``"int"``, ``"int | None"``) when
``from __future__ import annotations`` is active, or ``None`` to
trigger best-effort auto-detection based on ``value``.
"""
if isinstance(hint, str):
h = hint.lower().replace(" ", "")
if "bool" in h:
hint = bool
elif "int" in h:
hint = int
elif "float" in h:
hint = float
else:
hint = str
if hint is bool:
v = value.strip().lower()
if v in ("1", "true", "yes", "on"):
return True
if v in ("0", "false", "no", "off"):
return False
raise ValueError(f"invalid bool value: {value!r}")
if hint is int:
return int(value)
if hint is float:
return float(value)
# Fallback: keep as string.
return value
def _apply_dict(dc: Any, data: dict[str, Any]) -> None:
"""Recursively apply a YAML-derived dict onto a dataclass instance."""
for f in fields(dc):
if f.name not in data:
continue
incoming = data[f.name]
current = getattr(dc, f.name)
if is_dataclass(current) and isinstance(incoming, dict):
_apply_dict(current, incoming)
else:
setattr(dc, f.name, incoming)
def _apply_env(dc: Any, path: tuple[str, ...] = ()) -> None:
"""Walk dataclass tree and apply AIS_HUB_* env overrides."""
for f in fields(dc):
current = getattr(dc, f.name)
sub_path = path + (f.name,)
if is_dataclass(current):
_apply_env(current, sub_path)
continue
env_name = ENV_PREFIX + ENV_SEP.join(sub_path).upper()
if env_name in os.environ:
raw = os.environ[env_name]
try:
setattr(dc, f.name, _coerce(raw, f.type))
except Exception as exc:
raise ValueError(f"env {env_name}: {exc}") from exc
def load_config(path: Path | str | None) -> Config:
"""Load configuration from a YAML file (optional) and apply env overrides."""
cfg = Config()
if path is not None:
p = Path(path)
if not p.exists():
raise FileNotFoundError(f"config file not found: {p}")
with p.open("r", encoding="utf-8") as fh:
raw = yaml.safe_load(fh) or {}
if not isinstance(raw, dict):
raise ValueError(f"config file must be a YAML mapping, got {type(raw).__name__}")
_apply_dict(cfg, raw)
_apply_env(cfg)
return cfg
src/ais_hub/core/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Core domain: in-memory state, eager MMSI merge, stats, event bus."""
src/ais_hub/core/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/__pycache__/bus.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/__pycache__/merge.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/__pycache__/models.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/__pycache__/state.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/__pycache__/stats.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/core/bus.py
+66
View File
@@ -0,0 +1,66 @@
"""Internal pub/sub event bus.
Subscribers are given a bounded ``asyncio.Queue`` of ``Event`` envelopes.
If a subscriber queue overflows, the *oldest* event is dropped (drop-oldest)
and the ``bus_dropped`` counter is incremented in stats.
Subscribers are expected to drain their queues in their own task. The bus
never blocks on a slow subscriber.
"""
from __future__ import annotations
import asyncio
import logging
from typing import TYPE_CHECKING
from .models import Event
if TYPE_CHECKING:
from .stats import Stats
log = logging.getLogger(__name__)
class EventBus:
"""Fan-out pub/sub hub for ``Event`` envelopes."""
def __init__(self, stats: "Stats | None" = None, default_maxsize: int = 1024) -> None:
self._subs: list[asyncio.Queue[Event]] = []
self._stats = stats
self._default_maxsize = default_maxsize
self._lock = asyncio.Lock()
def subscribe(self, maxsize: int | None = None) -> asyncio.Queue[Event]:
"""Register a new subscriber queue. Caller owns the queue."""
q: asyncio.Queue[Event] = asyncio.Queue(maxsize=maxsize or self._default_maxsize)
self._subs.append(q)
return q
def unsubscribe(self, q: asyncio.Queue[Event]) -> None:
try:
self._subs.remove(q)
except ValueError:
pass
def publish(self, event: Event) -> None:
"""Non-blocking publish; drops oldest on a subscriber if its queue is full."""
for q in list(self._subs):
while True:
try:
q.put_nowait(event)
break
except asyncio.QueueFull:
try:
q.get_nowait()
except asyncio.QueueEmpty:
# Race; try again.
continue
if self._stats is not None:
self._stats.incr("bus_dropped")
# Retry put after dropping one item.
continue
@property
def subscribers(self) -> int:
return len(self._subs)
src/ais_hub/core/merge.py
+52
View File
@@ -0,0 +1,52 @@
"""Eager merge of AIS reports into ``MergedTarget`` objects."""
from __future__ import annotations
from .models import AisReport, MergedTarget
DYNAMIC_FIELDS = ("lat", "lon", "sog", "cog", "heading", "nav_status", "rot")
STATIC_FIELDS = (
"name", "callsign", "imo", "ship_type",
"dim_a", "dim_b", "dim_c", "dim_d",
"eta", "draught", "destination",
)
def apply(target: MergedTarget, report: AisReport) -> bool:
"""Merge ``report`` into ``target`` in place. Return True if changed.
- Dynamic fields are overwritten only when ``report.ts >= last_dynamic_ts``.
- Static fields are overwritten only when ``report.ts >= last_static_ts``.
- Msg 24A (name) and 24B (callsign/dims) both feed into the same
static slot; the timestamp guards preserve the latest value per
field but do not clobber a still-relevant name.
"""
changed = False
if report.ts > target.last_seen:
target.last_seen = report.ts
changed = True
target.sources.add(report.source)
target.msg_types.add(report.msg_type)
d = report.data
if report.kind == "dynamic" and report.ts >= target.last_dynamic_ts:
for f in DYNAMIC_FIELDS:
if f in d:
setattr(target, f, d[f])
changed = True
target.last_dynamic_ts = report.ts
elif report.kind == "static" and report.ts >= target.last_static_ts:
for f in STATIC_FIELDS:
if f in d and d[f] not in (None, ""):
setattr(target, f, d[f])
changed = True
target.last_static_ts = report.ts
return changed
__all__ = ["apply"]
src/ais_hub/core/models.py
+279
View File
@@ -0,0 +1,279 @@
"""Domain dataclasses used across ingest/parser/core/storage/publish."""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Literal
# ---------------------------------------------------------------------------
# Raw NMEA frame (ingest -> parser / raw tap)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class RawFrame:
"""A single raw NMEA line as received from an ingest channel."""
ts: float # unix epoch seconds (monotonic-ish)
source: str # e.g. "ais_udp:192.168.1.2:4001", "gps_uart:/dev/ttyS1"
kind: Literal["ais", "gps", "radio"] # rough classification by ingest
line: str # NMEA sentence with no trailing CRLF
# ---------------------------------------------------------------------------
# AIS reports (parser -> core)
# ---------------------------------------------------------------------------
AisKind = Literal[
"dynamic", # msg 1/2/3/18/19/27 - position report
"static", # msg 5/24 - static/voyage
"base_station", # msg 4 - base station report
"aton", # msg 21 - AtoN report
"other",
]
@dataclass(slots=True)
class AisReport:
"""Normalized AIS message payload."""
ts: float
source: str
kind: AisKind
mmsi: int
msg_type: int
channel: str | None = None
raw: str | None = None # original sentence(s) joined with \n
data: dict[str, Any] = field(default_factory=dict)
# ---------------------------------------------------------------------------
# GPS fix (parser -> core)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class GpsFix:
ts: float
source: str
lat: float | None = None
lon: float | None = None
sog: float | None = None # speed over ground, knots
cog: float | None = None # course over ground, degrees true
alt: float | None = None # meters
fix_quality: int | None = None # GGA field 6
sats: int | None = None
hdop: float | None = None
# raw sentence kind(s) that contributed to this fix (e.g. "RMC", "GGA")
sentences: tuple[str, ...] = ()
# ---------------------------------------------------------------------------
# Radio telemetry (parser -> core)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class RadioReport:
ts: float
source: str
channel: str | None = None
rssi: float | None = None
snr: float | None = None
slot: int | None = None
raw: dict[str, Any] = field(default_factory=dict)
# ---------------------------------------------------------------------------
# Merged target state (core.state)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class MergedTarget:
"""Single authoritative snapshot per MMSI, kept up to date by core.merge."""
mmsi: int
# identity / static
name: str | None = None
callsign: str | None = None
imo: int | None = None
ship_type: int | None = None
dim_a: int | None = None
dim_b: int | None = None
dim_c: int | None = None
dim_d: int | None = None
# voyage
eta: str | None = None
draught: float | None = None
destination: str | None = None
# dynamic
lat: float | None = None
lon: float | None = None
sog: float | None = None
cog: float | None = None
heading: float | None = None
nav_status: int | None = None
rot: float | None = None
# signal (from AIS-catcher decode events, channel 10114)
last_signal_db: float | None = None
last_signal_ts: float = 0.0
last_signal_slot: int | None = None
last_signal_channel: str | None = None
# bookkeeping
last_static_ts: float = 0.0
last_dynamic_ts: float = 0.0
last_seen: float = 0.0
sources: set[str] = field(default_factory=set)
msg_types: set[int] = field(default_factory=set)
def to_dict(self) -> dict[str, Any]:
return {
"mmsi": self.mmsi,
"name": self.name,
"callsign": self.callsign,
"imo": self.imo,
"ship_type": self.ship_type,
"dims": {
"a": self.dim_a,
"b": self.dim_b,
"c": self.dim_c,
"d": self.dim_d,
},
"voyage": {
"eta": self.eta,
"draught": self.draught,
"destination": self.destination,
},
"dynamic": {
"lat": self.lat,
"lon": self.lon,
"sog": self.sog,
"cog": self.cog,
"heading": self.heading,
"nav_status": self.nav_status,
"rot": self.rot,
},
"signal": {
"last_db": self.last_signal_db,
"last_ts": self.last_signal_ts or None,
"last_slot": self.last_signal_slot,
"last_channel": self.last_signal_channel,
},
"last_static_ts": self.last_static_ts,
"last_dynamic_ts": self.last_dynamic_ts,
"last_seen": self.last_seen,
"sources": sorted(self.sources),
"msg_types": sorted(self.msg_types),
}
@dataclass(slots=True)
class BaseStation:
mmsi: int
ts: float = 0.0
lat: float | None = None
lon: float | None = None
epfd: int | None = None
def to_dict(self) -> dict[str, Any]:
return {
"mmsi": self.mmsi,
"ts": self.ts,
"lat": self.lat,
"lon": self.lon,
"epfd": self.epfd,
}
@dataclass(slots=True)
class AtoN:
mmsi: int
ts: float = 0.0
lat: float | None = None
lon: float | None = None
aton_type: int | None = None
name: str | None = None
virtual: bool | None = None
def to_dict(self) -> dict[str, Any]:
return {
"mmsi": self.mmsi,
"ts": self.ts,
"lat": self.lat,
"lon": self.lon,
"type": self.aton_type,
"name": self.name,
"virtual": self.virtual,
}
@dataclass(slots=True)
class Ownship:
"""Latest GPS fix (treated as own-ship position)."""
ts: float = 0.0
lat: float | None = None
lon: float | None = None
sog: float | None = None
cog: float | None = None
alt: float | None = None
fix_quality: int | None = None
sats: int | None = None
hdop: float | None = None
def to_dict(self) -> dict[str, Any]:
return {
"ts": self.ts,
"lat": self.lat,
"lon": self.lon,
"sog": self.sog,
"cog": self.cog,
"alt": self.alt,
"fix_quality": self.fix_quality,
"sats": self.sats,
"hdop": self.hdop,
}
# ---------------------------------------------------------------------------
# Event envelope (core.bus -> publishers / storage)
# ---------------------------------------------------------------------------
EventType = Literal[
"ownship.update",
"target.update",
"base_station.update",
"aton.update",
"radio.update",
"stats.update",
]
@dataclass(slots=True)
class Event:
"""Stable event envelope published on the internal pub/sub bus."""
type: str
ts: float
data: dict[str, Any]
def to_dict(self) -> dict[str, Any]:
return {"type": self.type, "ts": self.ts, "data": self.data}
# ---------------------------------------------------------------------------
# TX outbox message (publish/udp_tx_outbox)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class TxMessage:
"""One NMEA sentence queued for transmission via the SPI bridge outbox."""
ts: float
line: str # full NMEA sentence, incl. "!"/"$" and "*HH"
origin: str = "internal" # e.g. "rest:POST /api/v1/tx", "emitter:ownship"
src/ais_hub/core/state.py
+358
View File
@@ -0,0 +1,358 @@
"""Central in-memory state: ownship, merged targets, base stations, AtoNs.
The state is the single source of truth. Parser tasks call ``apply_*``
methods, which:
1. Update the in-memory snapshot (eager merge for targets).
2. Publish a corresponding ``Event`` on the event bus.
3. Increment stats counters.
REST reads directly from snapshot methods (O(1) per target).
"""
from __future__ import annotations
import logging
import time
from typing import Any
from . import merge
from ..parser.aiscatcher import (
RssiIq,
SignalEventBatch,
SlotBitmap,
SlotDetail,
)
from .bus import EventBus
from .models import (
AisReport,
AtoN,
BaseStation,
Event,
GpsFix,
MergedTarget,
Ownship,
RadioReport,
)
from .stats import Stats
log = logging.getLogger(__name__)
class State:
"""In-memory aggregated state of the service."""
def __init__(self, bus: EventBus, stats: Stats) -> None:
self._bus = bus
self._stats = stats
self.ownship: Ownship = Ownship()
self.targets: dict[int, MergedTarget] = {}
self.base_stations: dict[int, BaseStation] = {}
self.atons: dict[int, AtoN] = {}
# TDMA slot map: channel -> slot -> {"mmsi": int, "ts": float, ...}
self.slots: dict[str, dict[int, dict[str, Any]]] = {}
self.last_radio: RadioReport | None = None
# AIS-catcher telemetry (set by apply_rssi / apply_slot_* / apply_signal_events).
# Each of the dicts below has channel ('A'/'B') as top-level key where applicable.
self.radio_power: dict[str, Any] = {} # {"ts":..., "power_a_db":..., "power_b_db":...}
self.slot_occupancy: dict[str, dict[str, Any]] = {} # ch -> {"ts","utc_minute","occupied_count","slots_total", ...}
self.slot_detail: dict[str, dict[str, Any]] = {} # ch -> {"ts","utc_minute","entries":[...]}
# ----- mutation API ---------------------------------------------------
def apply_gps(self, fix: GpsFix) -> None:
if fix.lat is not None:
self.ownship.lat = fix.lat
if fix.lon is not None:
self.ownship.lon = fix.lon
if fix.sog is not None:
self.ownship.sog = fix.sog
if fix.cog is not None:
self.ownship.cog = fix.cog
if fix.alt is not None:
self.ownship.alt = fix.alt
if fix.fix_quality is not None:
self.ownship.fix_quality = fix.fix_quality
if fix.sats is not None:
self.ownship.sats = fix.sats
if fix.hdop is not None:
self.ownship.hdop = fix.hdop
self.ownship.ts = fix.ts
self._stats.incr("state_ownship_updates")
self._bus.publish(Event(type="ownship.update", ts=fix.ts, data=self.ownship.to_dict()))
def apply_ais(self, report: AisReport) -> None:
if report.kind == "base_station":
self._apply_base_station(report)
return
if report.kind == "aton":
self._apply_aton(report)
return
if report.kind in ("dynamic", "static"):
self._apply_target(report)
return
# "other": still record source/msg type for stats.
self._stats.incr("state_ais_other")
def _apply_target(self, report: AisReport) -> None:
tgt = self.targets.get(report.mmsi)
created = False
if tgt is None:
tgt = MergedTarget(mmsi=report.mmsi)
self.targets[report.mmsi] = tgt
created = True
self._stats.incr("state_targets_new")
changed = merge.apply(tgt, report)
if created or changed:
self._stats.incr("state_targets_updates")
self._bus.publish(Event(type="target.update", ts=report.ts, data=tgt.to_dict()))
def _apply_base_station(self, report: AisReport) -> None:
bs = self.base_stations.get(report.mmsi)
if bs is None:
bs = BaseStation(mmsi=report.mmsi)
self.base_stations[report.mmsi] = bs
if "lat" in report.data:
bs.lat = report.data["lat"]
if "lon" in report.data:
bs.lon = report.data["lon"]
if "epfd" in report.data:
bs.epfd = report.data["epfd"]
bs.ts = report.ts
self._stats.incr("state_base_station_updates")
self._bus.publish(Event(type="base_station.update", ts=report.ts, data=bs.to_dict()))
def _apply_aton(self, report: AisReport) -> None:
a = self.atons.get(report.mmsi)
if a is None:
a = AtoN(mmsi=report.mmsi)
self.atons[report.mmsi] = a
for src, dst in (("lat", "lat"), ("lon", "lon"),
("aton_type", "aton_type"), ("name", "name"),
("virtual", "virtual")):
if src in report.data and report.data[src] is not None:
setattr(a, dst, report.data[src])
a.ts = report.ts
self._stats.incr("state_aton_updates")
self._bus.publish(Event(type="aton.update", ts=report.ts, data=a.to_dict()))
def apply_radio(self, report: RadioReport) -> None:
self.last_radio = report
if report.channel and report.slot is not None:
by_slot = self.slots.setdefault(report.channel, {})
by_slot[report.slot] = {
"ts": report.ts,
"rssi": report.rssi,
"snr": report.snr,
}
self._stats.incr("state_radio_updates")
self._bus.publish(Event(
type="radio.update",
ts=report.ts,
data={
"channel": report.channel,
"rssi": report.rssi,
"snr": report.snr,
"slot": report.slot,
"raw": report.raw,
},
))
# ----- AIS-catcher telemetry -----------------------------------------
def apply_rssi_iq(self, rssi: RssiIq, ts: float) -> None:
"""Store the latest RSSI pair and publish ``radio.update``."""
self.radio_power = {
"ts": ts,
"power_a_db": rssi.power_a_db,
"power_b_db": rssi.power_b_db,
}
self._stats.incr("state_rssi_updates")
self._bus.publish(Event(
type="radio.update",
ts=ts,
data={
"source": "aiscatcher_rssi",
"power_a_db": rssi.power_a_db,
"power_b_db": rssi.power_b_db,
},
))
def apply_slot_bitmap(self, snap: SlotBitmap, ts: float) -> None:
self.slot_occupancy[snap.channel] = {
"ts": ts,
"utc_minute": snap.utc_minute,
"slots_total": snap.slots_total,
"occupied_count": snap.occupied_count,
"occupied_fraction": snap.occupied_fraction(),
"slot0_unix_ms": snap.slot0_unix_ms,
"first_occupied_unix_ms": snap.first_occupied_unix_ms or None,
}
self._stats.incr("state_slot_bitmap_updates")
self._bus.publish(Event(
type="slots.update",
ts=ts,
data={
"channel": snap.channel,
**self.slot_occupancy[snap.channel],
"bitmap_hex": snap.bitmap.hex(),
},
))
def apply_slot_detail(self, detail: SlotDetail, ts: float) -> None:
entries = [{"slot": e.slot, "level_db": e.level_db} for e in detail.entries]
self.slot_detail[detail.channel] = {
"ts": ts,
"utc_minute": detail.utc_minute,
"slot0_unix_ms": detail.slot0_unix_ms,
"entries": entries,
}
self._stats.incr("state_slot_detail_updates")
self._bus.publish(Event(
type="slots.detail",
ts=ts,
data={
"channel": detail.channel,
**self.slot_detail[detail.channel],
},
))
def apply_signal_events(self, batch: SignalEventBatch, ts: float) -> None:
"""Update per-MMSI signal levels from AIS-catcher decode events."""
for ev in batch.events:
tgt = self.targets.get(ev.mmsi)
if tgt is None:
tgt = MergedTarget(mmsi=ev.mmsi)
self.targets[ev.mmsi] = tgt
self._stats.incr("state_targets_new")
ev_ts = ev.unix_ms / 1000.0 if ev.unix_ms else ts
if ev_ts >= tgt.last_signal_ts:
tgt.last_signal_ts = ev_ts
tgt.last_signal_db = ev.level_db
tgt.last_signal_slot = ev.slot
tgt.last_signal_channel = batch.channel
if tgt.last_seen < ev_ts:
tgt.last_seen = ev_ts
self._stats.incr("state_signal_events", len(batch.events))
if batch.events:
self._bus.publish(Event(
type="signal.update",
ts=ts,
data={
"channel": batch.channel,
"events": [
{
"unix_ms": e.unix_ms,
"slot": e.slot,
"mmsi": e.mmsi,
"level_db": e.level_db,
}
for e in batch.events
],
},
))
# ----- warm start from SQLite ----------------------------------------
def warmup(
self,
*,
vessels: list[dict[str, Any]] | None = None,
base_stations: list[dict[str, Any]] | None = None,
atons: list[dict[str, Any]] | None = None,
) -> dict[str, int]:
"""Populate in-memory state from previously persisted rows.
No events are published; this is intended to run once at startup
before ingest tasks begin. Returns counts for logging/stats.
"""
counts = {"vessels": 0, "base_stations": 0, "atons": 0}
for row in vessels or ():
mmsi = int(row["mmsi"])
tgt = self.targets.get(mmsi)
if tgt is None:
tgt = MergedTarget(mmsi=mmsi)
self.targets[mmsi] = tgt
tgt.name = row.get("name") or tgt.name
tgt.callsign = row.get("callsign") or tgt.callsign
if row.get("imo") is not None:
tgt.imo = row["imo"]
if row.get("ship_type") is not None:
tgt.ship_type = row["ship_type"]
for k in ("dim_a", "dim_b", "dim_c", "dim_d"):
if row.get(k) is not None:
setattr(tgt, k, row[k])
if row.get("eta"):
tgt.eta = row["eta"]
if row.get("draught") is not None:
tgt.draught = row["draught"]
if row.get("destination"):
tgt.destination = row["destination"]
updated_at = float(row.get("updated_at") or 0.0)
if updated_at and updated_at > tgt.last_static_ts:
tgt.last_static_ts = updated_at
counts["vessels"] += 1
for row in base_stations or ():
mmsi = int(row["mmsi"])
bs = self.base_stations.get(mmsi) or BaseStation(mmsi=mmsi)
bs.ts = float(row.get("ts") or 0.0)
bs.lat = row.get("lat")
bs.lon = row.get("lon")
bs.epfd = row.get("epfd")
self.base_stations[mmsi] = bs
counts["base_stations"] += 1
for row in atons or ():
mmsi = int(row["mmsi"])
a = self.atons.get(mmsi) or AtoN(mmsi=mmsi)
a.ts = float(row.get("ts") or 0.0)
a.lat = row.get("lat")
a.lon = row.get("lon")
a.aton_type = row.get("aton_type")
a.name = row.get("name")
a.virtual = row.get("virtual")
self.atons[mmsi] = a
counts["atons"] += 1
return counts
# ----- read API -------------------------------------------------------
def snapshot_vessels(self, since: float | None = None, limit: int | None = None) -> list[dict[str, Any]]:
items = self.targets.values()
if since is not None:
items = (t for t in items if t.last_seen >= since)
out = [t.to_dict() for t in items]
out.sort(key=lambda d: d["last_seen"], reverse=True)
if limit is not None and limit > 0:
out = out[:limit]
return out
def get_vessel(self, mmsi: int) -> dict[str, Any] | None:
t = self.targets.get(mmsi)
return t.to_dict() if t is not None else None
def snapshot_base_stations(self) -> list[dict[str, Any]]:
return [b.to_dict() for b in self.base_stations.values()]
def snapshot_atons(self) -> list[dict[str, Any]]:
return [a.to_dict() for a in self.atons.values()]
def snapshot_slots(self) -> dict[str, Any]:
"""Return TDMA slot / occupancy view for REST."""
return {
"per_channel": self.slots, # legacy slot->mmsi map (currently unused)
"occupancy": self.slot_occupancy,
"detail": self.slot_detail,
}
def snapshot_ownship(self) -> dict[str, Any]:
return self.ownship.to_dict()
def snapshot_radio(self) -> dict[str, Any]:
return {"power": self.radio_power}
__all__ = ["State"]
src/ais_hub/core/stats.py
+44
View File
@@ -0,0 +1,44 @@
"""Global counters & simple rate samples."""
from __future__ import annotations
import time
from collections import defaultdict
from dataclasses import dataclass, field
from threading import Lock
from typing import Any
@dataclass
class Stats:
"""Thread/async-safe counter container.
Counters are plain integers keyed by name. Gauges hold instantaneous
numeric values. Queue depth samples are reported via ``set_gauge``.
"""
counters: dict[str, int] = field(default_factory=lambda: defaultdict(int))
gauges: dict[str, float] = field(default_factory=dict)
started_at: float = field(default_factory=time.time)
_lock: Lock = field(default_factory=Lock, repr=False)
def incr(self, name: str, n: int = 1) -> None:
with self._lock:
self.counters[name] += n
def set_gauge(self, name: str, value: float) -> None:
with self._lock:
self.gauges[name] = value
def snapshot(self) -> dict[str, Any]:
with self._lock:
now = time.time()
return {
"uptime_sec": round(now - self.started_at, 3),
"started_at": self.started_at,
"counters": dict(self.counters),
"gauges": dict(self.gauges),
}
__all__ = ["Stats"]
src/ais_hub/ingest/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Ingest subsystem: AIS UDP, GPS UART, radio UDP listeners."""
src/ais_hub/ingest/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/__pycache__/ais_udp.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/__pycache__/aiscatcher_udp.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/__pycache__/base.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/__pycache__/gps_uart.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/__pycache__/radio_udp.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/ingest/ais_udp.py
+94
View File
@@ -0,0 +1,94 @@
"""AIS NMEA UDP ingest listener.
Binds a UDP socket on the configured host:port and decodes each received
datagram as one or more ASCII NMEA lines. One datagram may contain several
sentences separated by CR/LF.
"""
from __future__ import annotations
import asyncio
import logging
from typing import Any
from ..config import AisUdpCfg
from ..core.stats import Stats
from ..parser.nmea_utils import classify_kind, parse_sentence, split_lines
from .base import IngestBase, RawTap, open_udp_listener
log = logging.getLogger(__name__)
class _AisUdpProtocol(asyncio.DatagramProtocol):
def __init__(self, ingest: "AisUdpIngest") -> None:
self._ingest = ingest
def datagram_received(self, data: bytes, addr: Any) -> None:
ip, _port = (addr[0], addr[1]) if addr else ("?", 0)
# Sender port is ephemeral and may change between fragments of the
# same AIS multi-part message; only include IP in the source tag so
# the multi-fragment assembler can correlate frag 1 and frag 2.
suffix = f":{ip}"
self._ingest._stats.incr("ais_udp_datagrams")
for line in split_lines(data):
sentence = parse_sentence(line)
if sentence is None:
self._ingest._stats.incr("ais_udp_malformed")
# Still publish the raw line so SPI bridge sees everything.
self._ingest.emit("ais", line, source_suffix=suffix)
continue
kind = classify_kind(sentence.talker, sentence.start)
self._ingest.emit(kind if kind in ("ais", "gps") else "ais", line, source_suffix=suffix)
def error_received(self, exc: Exception) -> None:
log.warning("ais_udp error_received: %s", exc)
class AisUdpIngest(IngestBase):
"""Async UDP listener producing ``RawFrame`` items."""
def __init__(
self,
cfg: AisUdpCfg,
*,
parser_in: "asyncio.Queue",
raw_tap: RawTap,
stats: Stats,
) -> None:
super().__init__(
source_prefix=f"ais_udp:{cfg.host}:{cfg.port}",
parser_in=parser_in,
raw_tap=raw_tap,
stats=stats,
parser_drop_counter="parser_in_dropped",
)
self._cfg = cfg
self._transport: asyncio.DatagramTransport | None = None
async def start(self) -> None:
self._transport = await open_udp_listener(
self._cfg.host,
self._cfg.port,
lambda: _AisUdpProtocol(self),
name="ais_udp",
)
log.info("ais_udp listening on %s:%d", self._cfg.host, self._cfg.port)
async def stop(self) -> None:
if self._transport is not None:
self._transport.close()
self._transport = None
async def run(self) -> None:
await self.start()
try:
# Listener runs inside asyncio; just sleep forever until cancelled.
while True:
await asyncio.sleep(3600)
except asyncio.CancelledError:
raise
finally:
await self.stop()
__all__ = ["AisUdpIngest"]
src/ais_hub/ingest/aiscatcher_udp.py
+146
View File
@@ -0,0 +1,146 @@
"""AIS-catcher Mini binary UDP telemetry ingest.
Four independent UDP listeners, each bound to its own port on ``host``:
* ``slot_port`` (default 10111) — 315-byte slot occupancy bitmap
* ``slot_detail_port`` (default 10112) — per-slot levels
* ``rssi_port`` (default 10113) — 8-byte RSSI pair @ ~10 Hz
* ``event_port`` (default 10114) — decode events (mmsi, slot, level)
Each datagram is decoded synchronously inside the protocol callback and
forwarded directly to ``State``; no parser queue is involved. The event
bus handles downstream fan-out (WebSocket / UDP events / storage sink).
These sockets intentionally do **not** publish to ``RawTap`` because
the payloads are binary, not NMEA.
"""
from __future__ import annotations
import asyncio
import logging
import time
from typing import Any, Callable
from ..config import AisCatcherUdpCfg
from ..core.state import State
from ..core.stats import Stats
from ..parser.aiscatcher import (
decode_rssi_iq,
decode_signal_events,
decode_slot_bitmap,
decode_slot_detail,
)
from .base import open_udp_listener
log = logging.getLogger(__name__)
class _BinaryUdpProtocol(asyncio.DatagramProtocol):
def __init__(
self,
handler: Callable[[bytes], None],
name: str,
stats: Stats,
) -> None:
self._handler = handler
self._name = name
self._stats = stats
def datagram_received(self, data: bytes, addr: Any) -> None:
self._stats.incr(f"{self._name}_datagrams")
try:
self._handler(data)
except Exception:
self._stats.incr(f"{self._name}_errors")
log.exception("%s handler failed", self._name)
def error_received(self, exc: Exception) -> None:
log.warning("%s error_received: %s", self._name, exc)
class AisCatcherUdpIngest:
"""Owns four UDP sockets; each one dispatches into ``State``."""
def __init__(
self,
cfg: AisCatcherUdpCfg,
*,
state: State,
stats: Stats,
) -> None:
self._cfg = cfg
self._state = state
self._stats = stats
self._transports: list[asyncio.DatagramTransport] = []
# --- per-port handlers ------------------------------------------------
def _on_bitmap(self, data: bytes) -> None:
snap = decode_slot_bitmap(data)
if snap is None:
self._stats.incr("aiscatcher_slot_bitmap_malformed")
return
self._state.apply_slot_bitmap(snap, ts=time.time())
def _on_detail(self, data: bytes) -> None:
detail = decode_slot_detail(data)
if detail is None:
self._stats.incr("aiscatcher_slot_detail_malformed")
return
self._state.apply_slot_detail(detail, ts=time.time())
def _on_rssi(self, data: bytes) -> None:
rssi = decode_rssi_iq(data)
if rssi is None:
self._stats.incr("aiscatcher_rssi_malformed")
return
self._state.apply_rssi_iq(rssi, ts=time.time())
def _on_events(self, data: bytes) -> None:
batch = decode_signal_events(data)
if batch is None:
self._stats.incr("aiscatcher_events_malformed")
return
self._state.apply_signal_events(batch, ts=time.time())
# --- lifecycle --------------------------------------------------------
async def start(self) -> None:
host = self._cfg.host
listeners = [
("aiscatcher_slot", self._cfg.slot_port, self._on_bitmap),
("aiscatcher_slot_detail", self._cfg.slot_detail_port, self._on_detail),
("aiscatcher_rssi", self._cfg.rssi_port, self._on_rssi),
("aiscatcher_events", self._cfg.event_port, self._on_events),
]
for name, port, handler in listeners:
transport = await open_udp_listener(
host,
port,
lambda n=name, h=handler: _BinaryUdpProtocol(h, n, self._stats),
name=name,
)
self._transports.append(transport)
log.info("%s listening on %s:%d", name, host, port)
async def stop(self) -> None:
for t in self._transports:
try:
t.close()
except Exception:
pass
self._transports.clear()
async def run(self) -> None:
await self.start()
try:
while True:
await asyncio.sleep(3600)
except asyncio.CancelledError:
raise
finally:
await self.stop()
__all__ = ["AisCatcherUdpIngest"]
src/ais_hub/ingest/base.py
+130
View File
@@ -0,0 +1,130 @@
"""Shared ingest glue: RawFrame fan-out to parser queue + raw NMEA tap."""
from __future__ import annotations
import asyncio
import collections
import errno
import logging
import socket
import time
from typing import Any, Callable, Deque
from ..core.models import RawFrame
from ..core.stats import Stats
from ..publish.queues import put_drop_oldest
log = logging.getLogger(__name__)
class AddressInUseError(RuntimeError):
"""Raised when a UDP ingest port is already bound by another process.
The supervisor logs this at WARNING (not ERROR with traceback) and
keeps retrying with backoff until the port is free.
"""
async def open_udp_listener(
host: str,
port: int,
protocol_factory: Callable[[], asyncio.DatagramProtocol],
*,
reuse_address: bool = True,
name: str = "udp",
) -> asyncio.DatagramTransport:
"""Bind a UDP socket with SO_REUSEADDR and attach it to the event loop.
``SO_REUSEADDR`` helps the listener rebind quickly after a previous
crash of the same process. It does **not** let the socket coexist
with another process that is actively listening on the same port;
in that case ``AddressInUseError`` is raised with a clear message.
"""
loop = asyncio.get_running_loop()
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setblocking(False)
try:
sock.bind((host, port))
except OSError as exc:
sock.close()
if exc.errno in (errno.EADDRINUSE, errno.EACCES):
raise AddressInUseError(
f"{name}: cannot bind {host}:{port} — already in use by another process. "
f"Check with: ss -lunp 'sport = :{port}' / lsof -iUDP:{port}"
) from exc
raise
transport, _ = await loop.create_datagram_endpoint(protocol_factory, sock=sock)
return transport
class RawTap:
"""Fan-out of raw NMEA lines to UDP fan-out / storage / in-memory ring.
Consumers register their own queue (bounded) via ``subscribe``. The tap
publishes each raw line to every subscriber with drop-oldest semantics
to isolate slow consumers.
A small in-memory ring buffer is also maintained here so
``GET /api/v1/nmea/tail`` can return recent raw sentences without going
to SQLite.
"""
def __init__(self, stats: Stats | None = None, ring_size: int = 500) -> None:
self._subs: list[tuple[asyncio.Queue[RawFrame], str]] = []
self._stats = stats
self._ring: Deque[RawFrame] = collections.deque(maxlen=max(0, ring_size))
def subscribe(self, queue: "asyncio.Queue[RawFrame]", drop_counter: str) -> None:
self._subs.append((queue, drop_counter))
def unsubscribe(self, queue: "asyncio.Queue[RawFrame]") -> None:
self._subs = [(q, c) for (q, c) in self._subs if q is not queue]
def publish(self, frame: RawFrame) -> None:
if self._ring.maxlen:
self._ring.append(frame)
for q, counter in self._subs:
put_drop_oldest(q, frame, self._stats, counter)
def tail(self, limit: int, kind: str | None = None) -> list[RawFrame]:
out: list[RawFrame] = []
for fr in self._ring:
if kind and fr.kind != kind:
continue
out.append(fr)
if limit > 0:
out = out[-limit:]
return out
class IngestBase:
"""Common helper to classify and emit a raw NMEA line from any source."""
def __init__(
self,
*,
source_prefix: str,
parser_in: "asyncio.Queue[RawFrame]",
raw_tap: RawTap,
stats: Stats,
parser_drop_counter: str,
) -> None:
self._source_prefix = source_prefix
self._parser_in = parser_in
self._raw_tap = raw_tap
self._stats = stats
self._parser_drop_counter = parser_drop_counter
def emit(self, kind: str, line: str, source_suffix: str = "") -> None:
"""Stamp, count, and fan out one raw NMEA line."""
if not line:
return
source = f"{self._source_prefix}{source_suffix}" if source_suffix else self._source_prefix
frame = RawFrame(ts=time.time(), source=source, kind=kind, line=line) # type: ignore[arg-type]
self._stats.incr(f"rx_lines_{kind}")
self._raw_tap.publish(frame)
put_drop_oldest(self._parser_in, frame, self._stats, self._parser_drop_counter)
__all__ = ["IngestBase", "RawTap"]
src/ais_hub/ingest/gps_uart.py
+99
View File
@@ -0,0 +1,99 @@
"""GPS UART ingest: reads NMEA lines from a serial device.
Uses ``pyserial-asyncio`` when available. Lines are read in line-buffered
mode (CRLF terminated). Failures are logged; the supervisor retries the
task after backoff.
"""
from __future__ import annotations
import asyncio
import logging
from typing import Any
from ..config import GpsUartCfg
from ..core.stats import Stats
from ..parser.nmea_utils import split_lines
from .base import IngestBase, RawTap
log = logging.getLogger(__name__)
class GpsUartIngest(IngestBase):
def __init__(
self,
cfg: GpsUartCfg,
*,
parser_in: "asyncio.Queue",
raw_tap: RawTap,
stats: Stats,
) -> None:
super().__init__(
source_prefix=f"gps_uart:{cfg.device}",
parser_in=parser_in,
raw_tap=raw_tap,
stats=stats,
parser_drop_counter="parser_in_dropped",
)
self._cfg = cfg
self._reader: Any = None
self._writer: Any = None
async def run(self) -> None:
if not self._cfg.enabled:
log.info("gps_uart disabled in config")
while True:
await asyncio.sleep(3600)
try:
import serial_asyncio # type: ignore
except Exception:
log.error("serial_asyncio is not installed; gps_uart disabled")
while True:
await asyncio.sleep(3600)
try:
reader, writer = await serial_asyncio.open_serial_connection(
url=self._cfg.device,
baudrate=self._cfg.baud,
)
except Exception:
log.exception("gps_uart: failed to open %s@%d", self._cfg.device, self._cfg.baud)
raise
self._reader, self._writer = reader, writer
log.info("gps_uart open %s @ %d baud", self._cfg.device, self._cfg.baud)
try:
while True:
try:
chunk = await reader.readuntil(b"\n")
except asyncio.IncompleteReadError as exc:
# Device EOF; treat as transient and raise for supervisor.
if exc.partial:
for line in split_lines(exc.partial):
self.emit("gps", line)
raise RuntimeError("gps_uart EOF")
except asyncio.LimitOverrunError:
# Overly long line without newline; drain it.
chunk = await reader.read(4096)
for line in split_lines(chunk):
self.emit("gps", line)
self._stats.incr("gps_uart_lines")
except asyncio.CancelledError:
raise
finally:
try:
if writer is not None:
writer.close()
try:
await writer.wait_closed()
except Exception:
pass
except Exception:
pass
self._reader = None
self._writer = None
__all__ = ["GpsUartIngest"]
src/ais_hub/ingest/radio_udp.py
+86
View File
@@ -0,0 +1,86 @@
"""Radio telemetry UDP ingest listener.
Each datagram carries a JSON object (see ``parser.radio``). The listener
does not parse it here; it only stamps a ``RawFrame`` with ``kind="radio"``
and hands it to the parser queue. The raw NMEA tap receives the datagram
too (as a line), so SPI bridge can observe radio traffic if wanted.
"""
from __future__ import annotations
import asyncio
import logging
from typing import Any
from ..config import RadioUdpCfg
from ..core.stats import Stats
from .base import IngestBase, RawTap, open_udp_listener
log = logging.getLogger(__name__)
class _RadioUdpProtocol(asyncio.DatagramProtocol):
def __init__(self, ingest: "RadioUdpIngest") -> None:
self._ingest = ingest
def datagram_received(self, data: bytes, addr: Any) -> None:
ip, port = (addr[0], addr[1]) if addr else ("?", 0)
self._ingest._stats.incr("radio_udp_datagrams")
# Radio telemetry is JSON; tap publishes it as-is (stripped).
try:
line = data.decode("utf-8", errors="replace").strip()
except Exception:
return
if not line:
return
self._ingest.emit("radio", line, source_suffix=f":{ip}:{port}")
def error_received(self, exc: Exception) -> None:
log.warning("radio_udp error_received: %s", exc)
class RadioUdpIngest(IngestBase):
def __init__(
self,
cfg: RadioUdpCfg,
*,
parser_in: "asyncio.Queue",
raw_tap: RawTap,
stats: Stats,
) -> None:
super().__init__(
source_prefix=f"radio_udp:{cfg.host}:{cfg.port}",
parser_in=parser_in,
raw_tap=raw_tap,
stats=stats,
parser_drop_counter="parser_in_dropped",
)
self._cfg = cfg
self._transport: asyncio.DatagramTransport | None = None
async def start(self) -> None:
self._transport = await open_udp_listener(
self._cfg.host,
self._cfg.port,
lambda: _RadioUdpProtocol(self),
name="radio_udp",
)
log.info("radio_udp listening on %s:%d", self._cfg.host, self._cfg.port)
async def stop(self) -> None:
if self._transport is not None:
self._transport.close()
self._transport = None
async def run(self) -> None:
await self.start()
try:
while True:
await asyncio.sleep(3600)
except asyncio.CancelledError:
raise
finally:
await self.stop()
__all__ = ["RadioUdpIngest"]
src/ais_hub/logging_setup.py
+143
View File
@@ -0,0 +1,143 @@
"""Structured logging setup: JSON formatter + rotating file + stderr.
Also keeps an in-memory ring buffer of recent log records so the REST
endpoint ``/api/v1/logs`` can tail them without reading the file from
disk on every request.
"""
from __future__ import annotations
import collections
import json
import logging
import logging.handlers
import os
import sys
import time
from pathlib import Path
from typing import Any, Deque
from .config import LoggingCfg
# Ring buffer of recent formatted log records. Populated by MemoryRingHandler.
# Each item is a dict with keys: ts, level, logger, message.
LOG_RING: Deque[dict[str, Any]] = collections.deque(maxlen=1000)
class JsonFormatter(logging.Formatter):
"""Minimal JSON log formatter with stable keys."""
def format(self, record: logging.LogRecord) -> str: # noqa: D401
payload: dict[str, Any] = {
"ts": round(record.created, 3),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
}
if record.exc_info:
payload["exc"] = self.formatException(record.exc_info)
if record.stack_info:
payload["stack"] = record.stack_info
# Attach extra attributes set via logger.log(..., extra={...}).
for key, val in record.__dict__.items():
if key in _STD_RECORD_FIELDS or key.startswith("_"):
continue
try:
json.dumps(val) # probe serializability
except Exception:
val = repr(val)
payload[key] = val
return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
_STD_RECORD_FIELDS = {
"name", "msg", "args", "levelname", "levelno", "pathname", "filename",
"module", "exc_info", "exc_text", "stack_info", "lineno", "funcName",
"created", "msecs", "relativeCreated", "thread", "threadName",
"processName", "process", "message", "asctime", "taskName",
}
class MemoryRingHandler(logging.Handler):
"""Logging handler that appends formatted records to ``LOG_RING``."""
def emit(self, record: logging.LogRecord) -> None:
try:
LOG_RING.append({
"ts": round(record.created, 3),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
})
except Exception:
# Never let logging crash the service.
pass
def setup_logging(cfg: LoggingCfg) -> None:
"""Configure root logger according to ``cfg``."""
root = logging.getLogger()
# Remove any pre-existing handlers to allow idempotent re-setup (tests).
for h in list(root.handlers):
root.removeHandler(h)
level = getattr(logging, cfg.level.upper(), logging.INFO)
root.setLevel(level)
formatter: logging.Formatter
if cfg.json:
formatter = JsonFormatter()
else:
formatter = logging.Formatter(
fmt="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
# stderr handler (always).
stderr_h = logging.StreamHandler(stream=sys.stderr)
stderr_h.setFormatter(formatter)
root.addHandler(stderr_h)
# Rotating file handler if configured.
if cfg.file:
try:
Path(cfg.file).parent.mkdir(parents=True, exist_ok=True)
file_h = logging.handlers.RotatingFileHandler(
cfg.file,
maxBytes=cfg.max_bytes,
backupCount=cfg.backup_count,
encoding="utf-8",
)
file_h.setFormatter(formatter)
root.addHandler(file_h)
except OSError as exc:
# Do not fail startup because of log-file permission issues.
sys.stderr.write(f"warning: cannot open log file {cfg.file}: {exc}\n")
# In-memory ring for /api/v1/logs.
ring_h = MemoryRingHandler()
root.addHandler(ring_h)
# Quiet a few noisy libraries by default.
logging.getLogger("aiohttp.access").setLevel(logging.WARNING)
def tail_logs(limit: int = 200, level: str | None = None, since: float | None = None) -> list[dict[str, Any]]:
"""Return up to ``limit`` recent log records from the in-memory ring.
``level`` filters by severity (case-insensitive). ``since`` is a unix
timestamp lower bound.
"""
lvl = level.upper() if level else None
out: list[dict[str, Any]] = []
for rec in LOG_RING:
if lvl and rec["level"] != lvl:
continue
if since is not None and rec["ts"] < since:
continue
out.append(rec)
if limit > 0:
out = out[-limit:]
return out
__all__ = ["setup_logging", "tail_logs", "LOG_RING"]
src/ais_hub/parser/__init__.py
+1
View File
@@ -0,0 +1 @@
"""NMEA parsers: shared utils, AIS assembler, GPS, radio telemetry."""
src/ais_hub/parser/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/__pycache__/ais.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/__pycache__/aiscatcher.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/__pycache__/gps.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/__pycache__/nmea_utils.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/__pycache__/radio.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/parser/ais.py
+382
View File
@@ -0,0 +1,382 @@
"""AIS multi-fragment assembler + pyais decoder + normalization.
Fragment key is a strengthened composite:
(source_tag, talker, channel, seq_id, total_fragments)
- ``source_tag`` comes from the ingest layer (e.g. "ais_udp:192.168.1.2:34567").
Two separate AIS receivers can reuse the same ``seq_id`` independently,
so we must key by source to avoid cross-contamination.
- ``talker`` is the talker id (``AIVDM`` / ``AIVDO`` / ``BSVDM`` / ...).
- ``channel`` is the AIS channel (``A`` or ``B`` or empty).
- ``seq_id`` is the sequential message id field (field 3).
- ``total_fragments`` guards against a stale in-flight key being reused
with a different total.
Rules enforced on arrival:
- Fragment numbers must progress strictly ``1, 2, ..., total`` without
gaps or duplicates. Any violation clears the buffer for that key and
increments ``ais_fragment_errors``.
- Buffers older than ``TTL_SEC`` are evicted and counted as timeouts.
"""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass, field
from typing import Any, Iterable
from ..core.models import AisReport
from ..core.stats import Stats
from .nmea_utils import NmeaSentence, parse_sentence
log = logging.getLogger(__name__)
TTL_SEC = 60.0
DYNAMIC_TYPES = {1, 2, 3, 18, 19, 27}
STATIC_TYPES = {5, 24}
BASE_TYPES = {4, 11}
ATON_TYPES = {21}
FragmentKey = tuple[str, str, str, str, int] # (source, talker, channel, seq, total)
@dataclass(slots=True)
class _Buffer:
expected: int # next fragment_number we expect (1..total)
total: int
lines: list[str] = field(default_factory=list)
created_at: float = 0.0
def _classify(msg_type: int) -> str:
if msg_type in DYNAMIC_TYPES:
return "dynamic"
if msg_type in STATIC_TYPES:
return "static"
if msg_type in BASE_TYPES:
return "base_station"
if msg_type in ATON_TYPES:
return "aton"
return "other"
def _safe_int(v: Any) -> int | None:
"""Coerce a value to plain ``int``.
This deliberately flattens ``IntEnum`` / ``Enum.value`` instances
(pyais uses them for ``ship_type``, ``epfd``, ``status`` etc.) into
plain ints so downstream JSON serialization and equality checks
stay stable across pyais versions.
"""
if v is None:
return None
try:
return int(v)
except (TypeError, ValueError):
return None
def _safe_float(v: Any) -> float | None:
try:
f = float(v)
except (TypeError, ValueError):
return None
# pyais uses +/-91 / +/-181 as "unavailable" markers in some types.
return f
class AisAssembler:
"""Reassemble multi-fragment AIS sentences and decode them.
Instances are *not* thread-safe; use one per async task.
"""
def __init__(
self,
*,
stats: Stats | None = None,
ttl_sec: float = TTL_SEC,
ignore_checksum: bool = False,
allow_checksumless: bool = False,
allow_multipart_without_seq_id: bool = False,
) -> None:
self._buffers: dict[FragmentKey, _Buffer] = {}
self._noseq_buffers: dict[tuple[str, str, str, int], _Buffer] = {}
self._stats = stats
self._ttl = ttl_sec
self._ignore_checksum = ignore_checksum
self._allow_checksumless = allow_checksumless
self._allow_multipart_without_seq_id = allow_multipart_without_seq_id
# -- public API --------------------------------------------------------
def feed(self, source_tag: str, line: str, ts: float | None = None) -> list[AisReport]:
"""Accept one raw NMEA AIS line. Return 0+ decoded reports.
``ts`` defaults to ``time.time()`` and is stamped on the produced
reports.
"""
if ts is None:
ts = time.time()
self._gc(ts)
sentence = parse_sentence(line)
if sentence is None:
self._incr("parser_errors")
return []
if not self._ignore_checksum and not sentence.checksum_ok:
# Allow checksumless (no "*HH") sentences when configured, but still
# reject sentences with a present-yet-invalid checksum.
if self._allow_checksumless and "*" not in sentence.raw:
pass
else:
self._incr("parser_checksum_errors")
return []
if sentence.start != "!":
# Non-encapsulated, not AIS.
return []
if len(sentence.fields) < 5:
self._incr("parser_errors")
return []
try:
total = int(sentence.fields[0]) if sentence.fields[0] else 1
frag_no = int(sentence.fields[1]) if sentence.fields[1] else 1
except ValueError:
self._incr("parser_errors")
return []
seq_id = sentence.fields[2] or ""
channel = sentence.fields[3] or ""
# Single-fragment fast path: no multipart state needed.
if total == 1 and frag_no == 1:
return self._decode([sentence.raw], ts, source_tag, channel)
# Multi-fragment validation.
if total < 1 or not (1 <= frag_no <= total):
self._incr("ais_fragment_errors")
return []
if not seq_id:
if not self._allow_multipart_without_seq_id:
# Multi-fragment without seq_id is not reliably groupable.
self._incr("ais_fragment_errors")
return []
return self._feed_noseq(source_tag, sentence, ts, total, frag_no, channel)
key: FragmentKey = (source_tag, sentence.talker, channel, seq_id, total)
buf = self._buffers.get(key)
if buf is None:
if frag_no != 1:
# Mid-stream fragment without a leading fragment is unusable.
self._incr("ais_fragment_errors")
return []
buf = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
self._buffers[key] = buf
return []
# Existing buffer: expect strictly increasing fragment_number.
if frag_no != buf.expected or total != buf.total:
self._incr("ais_fragment_errors")
self._buffers.pop(key, None)
# If the violating fragment itself is a fresh start (frag_no==1),
# seed a new buffer so we don't lose it.
if frag_no == 1:
self._buffers[key] = _Buffer(
expected=2, total=total, lines=[sentence.raw], created_at=ts,
)
return []
buf.lines.append(sentence.raw)
buf.expected += 1
if len(buf.lines) == buf.total:
self._buffers.pop(key, None)
return self._decode(buf.lines, ts, source_tag, channel)
return []
def gc(self, now: float | None = None) -> None:
"""Evict buffers older than TTL (exposed for tests)."""
self._gc(now if now is not None else time.time())
# -- internals ---------------------------------------------------------
def _gc(self, now: float) -> None:
dead: list[FragmentKey] = []
cutoff = now - self._ttl
for key, buf in self._buffers.items():
if buf.created_at < cutoff:
dead.append(key)
for k in dead:
self._buffers.pop(k, None)
self._incr("ais_fragment_timeouts")
dead2: list[tuple[str, str, str, int]] = []
for key, buf in self._noseq_buffers.items():
if buf.created_at < cutoff:
dead2.append(key)
for k in dead2:
self._noseq_buffers.pop(k, None)
self._incr("ais_fragment_timeouts")
def _feed_noseq(
self,
source_tag: str,
sentence: NmeaSentence,
ts: float,
total: int,
frag_no: int,
channel: str,
) -> list[AisReport]:
"""Best-effort multipart reassembly for sentences with empty seq_id.
We keep at most one active buffer per (source, talker, channel, total).
This works well when the upstream does not interleave multiple
checksumless/no-seq multipart messages on the same channel.
"""
key2 = (source_tag, sentence.talker, channel, total)
buf = self._noseq_buffers.get(key2)
if buf is None:
if frag_no != 1:
self._incr("ais_fragment_errors")
return []
self._noseq_buffers[key2] = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
return []
if frag_no != buf.expected or total != buf.total:
self._incr("ais_fragment_errors")
self._noseq_buffers.pop(key2, None)
if frag_no == 1:
self._noseq_buffers[key2] = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
return []
buf.lines.append(sentence.raw)
buf.expected += 1
if len(buf.lines) == buf.total:
self._noseq_buffers.pop(key2, None)
return self._decode(buf.lines, ts, source_tag, channel)
return []
def _incr(self, name: str, n: int = 1) -> None:
if self._stats is not None:
self._stats.incr(name, n)
def _decode(
self, lines: list[str], ts: float, source_tag: str, channel: str,
) -> list[AisReport]:
"""Invoke pyais on the collected lines and produce one AisReport."""
try:
# Import lazily so unit tests that don't need pyais can run.
from pyais import decode as pyais_decode # type: ignore
except Exception: # pragma: no cover
self._incr("parser_errors")
log.exception("pyais import failed")
return []
try:
decoded = pyais_decode(*[ln.encode("ascii", errors="replace") for ln in lines])
except Exception:
self._incr("parser_errors")
log.debug("pyais decode failed for %d lines", len(lines), exc_info=True)
return []
try:
payload: dict[str, Any] = decoded.asdict() # type: ignore[attr-defined]
except Exception:
try:
payload = dict(decoded) # type: ignore[arg-type]
except Exception:
self._incr("parser_errors")
return []
msg_type = _safe_int(payload.get("msg_type") or payload.get("type"))
mmsi = _safe_int(payload.get("mmsi"))
if msg_type is None or mmsi is None:
self._incr("parser_errors")
return []
kind = _classify(msg_type)
normalized = _normalize_payload(payload, kind, msg_type)
report = AisReport(
ts=ts,
source=source_tag,
kind=kind, # type: ignore[arg-type]
mmsi=mmsi,
msg_type=msg_type,
channel=channel or None,
raw="\n".join(lines),
data=normalized,
)
self._incr("ais_reports")
self._incr(f"ais_msg_{msg_type}")
return [report]
# ---------------------------------------------------------------------------
# Normalization helpers
# ---------------------------------------------------------------------------
def _normalize_payload(payload: dict[str, Any], kind: str, msg_type: int) -> dict[str, Any]:
"""Map pyais fields into a stable subset used by core/publish."""
out: dict[str, Any] = {"msg_type": msg_type}
# Common dynamic fields.
for src, dst in (
("lat", "lat"), ("lon", "lon"),
("speed", "sog"), ("course", "cog"),
("heading", "heading"),
("status", "nav_status"), ("nav_status", "nav_status"),
("turn", "rot"), ("rot", "rot"),
):
if src in payload and payload[src] is not None:
if dst in ("lat", "lon", "sog", "cog", "heading", "rot"):
out[dst] = _safe_float(payload[src])
else:
out[dst] = _safe_int(payload[src])
# Static fields.
for src, dst in (
("shipname", "name"), ("name", "name"),
("callsign", "callsign"),
("imo", "imo"),
("ship_type", "ship_type"),
("to_bow", "dim_a"), ("to_stern", "dim_b"),
("to_port", "dim_c"), ("to_starboard", "dim_d"),
("destination", "destination"),
("draught", "draught"),
("eta", "eta"),
("epfd", "epfd"),
):
if src in payload and payload[src] is not None:
val = payload[src]
if dst == "name" or dst == "callsign" or dst == "destination":
out[dst] = str(val).strip().rstrip("@").strip() or None
elif dst == "draught":
out[dst] = _safe_float(val)
elif dst == "eta":
# pyais may return a string, datetime, or nested struct.
out[dst] = str(val)
else:
# Covers IntEnum (ShipType/EpfdType/NavigationStatus) and plain ints.
out[dst] = _safe_int(val)
# AtoN specifics.
if kind == "aton":
if "aid_type" in payload and payload["aid_type"] is not None:
out["aton_type"] = _safe_int(payload["aid_type"])
if "virtual_aid" in payload:
out["virtual"] = bool(payload["virtual_aid"])
if "name" in payload and payload["name"] is not None:
out["name"] = str(payload["name"]).strip().rstrip("@").strip() or None
return out
__all__ = ["AisAssembler", "AisReport"]
src/ais_hub/parser/aiscatcher.py
+217
View File
@@ -0,0 +1,217 @@
"""Binary decoders for AIS-catcher Mini UDP telemetry.
Four independent datagram families, all big-endian:
1. Slot occupancy bitmap (``slot_udp_*``), fixed 315 bytes.
2. Slot detail (``slot_detail_udp_*``), variable length.
3. RSSI IQ (``rssi_udp_*``), fixed 8 bytes, ~10 Hz.
4. Decode events (``event_udp_*``), variable length.
All helpers are pure functions that return dataclasses. Any framing error
returns ``None`` so ingest can count malformed packets without raising.
"""
from __future__ import annotations
import struct
from dataclasses import dataclass, field
__all__ = [
"SlotBitmap",
"SlotDetail",
"SlotLevel",
"RssiIq",
"SignalEvent",
"SignalEventBatch",
"decode_slot_bitmap",
"decode_slot_detail",
"decode_rssi_iq",
"decode_signal_events",
]
# ---------------------------------------------------------------------------
# Shared helpers
# ---------------------------------------------------------------------------
def _channel(byte: int) -> str:
"""Map a channel byte to ``"A"``/``"B"``/``"?"``."""
if byte in (0x41, 0x42): # 'A' / 'B'
return chr(byte)
return "?"
# ---------------------------------------------------------------------------
# 1) Slot bitmap (315 bytes)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class SlotBitmap:
channel: str
utc_minute: int # floor(epoch_ms / 60000)
slots_total: int # always 2250
occupied_count: int
noise_floor: float # reserved (0)
threshold: float # reserved (0)
slot0_unix_ms: int
first_occupied_unix_ms: int
bitmap: bytes # 282 bytes, LSB-first per byte
def occupied_fraction(self) -> float:
if self.slots_total <= 0:
return 0.0
return self.occupied_count / self.slots_total
# Fixed header layout: 1 + 4 + 2 + 2 + 4 + 4 + 8 + 8 = 33 bytes.
_BITMAP_HEADER = struct.Struct(">BIHHffQQ")
_BITMAP_TOTAL_LEN = 33 + 282 # 315
def decode_slot_bitmap(data: bytes) -> SlotBitmap | None:
if len(data) != _BITMAP_TOTAL_LEN:
return None
try:
ch_b, utc_minute, slots_total, occupied_count, nf, thr, slot0, first_occ = \
_BITMAP_HEADER.unpack_from(data, 0)
except struct.error:
return None
return SlotBitmap(
channel=_channel(ch_b),
utc_minute=utc_minute,
slots_total=slots_total,
occupied_count=occupied_count,
noise_floor=nf,
threshold=thr,
slot0_unix_ms=slot0,
first_occupied_unix_ms=first_occ,
bitmap=bytes(data[33:315]),
)
# ---------------------------------------------------------------------------
# 2) Slot detail (variable)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class SlotLevel:
slot: int
level_db: float
@dataclass(slots=True)
class SlotDetail:
channel: str
utc_minute: int
slot0_unix_ms: int
entries: list[SlotLevel] = field(default_factory=list)
# Header: 1 + 4 + 8 + 2 = 15 bytes.
_DETAIL_HEADER = struct.Struct(">BIQH")
_DETAIL_ENTRY = struct.Struct(">Hf") # 6 bytes per entry
_DETAIL_HEADER_LEN = 15
_DETAIL_ENTRY_LEN = 6
def decode_slot_detail(data: bytes) -> SlotDetail | None:
if len(data) < _DETAIL_HEADER_LEN:
return None
try:
ch_b, utc_minute, slot0, count = _DETAIL_HEADER.unpack_from(data, 0)
except struct.error:
return None
expected = _DETAIL_HEADER_LEN + count * _DETAIL_ENTRY_LEN
if len(data) < expected:
return None
entries: list[SlotLevel] = []
off = _DETAIL_HEADER_LEN
for _ in range(count):
try:
slot_num, level = _DETAIL_ENTRY.unpack_from(data, off)
except struct.error:
return None
entries.append(SlotLevel(slot=slot_num, level_db=level))
off += _DETAIL_ENTRY_LEN
return SlotDetail(
channel=_channel(ch_b),
utc_minute=utc_minute,
slot0_unix_ms=slot0,
entries=entries,
)
# ---------------------------------------------------------------------------
# 3) RSSI IQ (8 bytes)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class RssiIq:
power_a_db: float
power_b_db: float
_RSSI = struct.Struct(">ff")
def decode_rssi_iq(data: bytes) -> RssiIq | None:
if len(data) != 8:
return None
try:
a, b = _RSSI.unpack(data)
except struct.error:
return None
return RssiIq(power_a_db=a, power_b_db=b)
# ---------------------------------------------------------------------------
# 4) Decode events (variable)
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class SignalEvent:
unix_ms: int
slot: int
mmsi: int
level_db: float
@dataclass(slots=True)
class SignalEventBatch:
channel: str
events: list[SignalEvent] = field(default_factory=list)
# Header: 1 + 2 = 3 bytes.
_EVENTS_HEADER = struct.Struct(">BH")
_EVENT_ENTRY = struct.Struct(">QHIf") # 18 bytes per entry
_EVENTS_HEADER_LEN = 3
_EVENT_ENTRY_LEN = 18
def decode_signal_events(data: bytes) -> SignalEventBatch | None:
if len(data) < _EVENTS_HEADER_LEN:
return None
try:
ch_b, count = _EVENTS_HEADER.unpack_from(data, 0)
except struct.error:
return None
expected = _EVENTS_HEADER_LEN + count * _EVENT_ENTRY_LEN
if len(data) < expected:
return None
events: list[SignalEvent] = []
off = _EVENTS_HEADER_LEN
for _ in range(count):
try:
unix_ms, slot, mmsi, level = _EVENT_ENTRY.unpack_from(data, off)
except struct.error:
return None
events.append(SignalEvent(unix_ms=unix_ms, slot=slot, mmsi=mmsi, level_db=level))
off += _EVENT_ENTRY_LEN
return SignalEventBatch(channel=_channel(ch_b), events=events)
src/ais_hub/parser/gps.py
+245
View File
@@ -0,0 +1,245 @@
"""GPS NMEA 0183 parser for RMC / GGA / VTG / GSA / GSV.
Only fields actually used downstream are extracted; the rest is ignored.
GSA / GSV are tracked lightly to surface fix quality indicators.
"""
from __future__ import annotations
import logging
import time
from typing import Any
from ..core.models import GpsFix
from ..core.stats import Stats
from .nmea_utils import NmeaSentence, parse_sentence
log = logging.getLogger(__name__)
KNOTS_PER_KMH = 1.0 / 1.852
def _to_float(v: str | None) -> float | None:
if v is None or v == "":
return None
try:
return float(v)
except ValueError:
return None
def _to_int(v: str | None) -> int | None:
if v is None or v == "":
return None
try:
return int(v)
except ValueError:
return None
def _parse_lat(raw: str, hemi: str) -> float | None:
"""NMEA latitude ddmm.mmmm + N/S -> decimal degrees."""
if not raw or not hemi:
return None
try:
deg = int(raw[:2])
minutes = float(raw[2:])
except (ValueError, IndexError):
return None
val = deg + minutes / 60.0
if hemi.upper() == "S":
val = -val
return val
def _parse_lon(raw: str, hemi: str) -> float | None:
"""NMEA longitude dddmm.mmmm + E/W -> decimal degrees."""
if not raw or not hemi:
return None
try:
deg = int(raw[:3])
minutes = float(raw[3:])
except (ValueError, IndexError):
return None
val = deg + minutes / 60.0
if hemi.upper() == "W":
val = -val
return val
class GpsParser:
"""Parse GPS NMEA sentences and emit ``GpsFix`` snapshots.
The parser holds a small amount of state so fields from RMC and GGA can
be merged into a single, richer ``GpsFix`` object on each update.
"""
def __init__(self, stats: Stats | None = None) -> None:
self._stats = stats
self._fix = GpsFix(ts=0.0, source="")
self._last_source: str = ""
def feed(self, source: str, line: str, ts: float | None = None) -> GpsFix | None:
"""Parse a single GPS NMEA line. Return a fresh ``GpsFix`` on success."""
if ts is None:
ts = time.time()
sentence = parse_sentence(line)
if sentence is None:
self._incr("parser_errors")
return None
if not sentence.checksum_ok:
self._incr("parser_checksum_errors")
return None
if sentence.start != "$":
return None
talker = sentence.talker
kind = talker[2:] if len(talker) >= 5 else talker # strip GP/GN/GL/...
fields = sentence.fields
updated = False
if kind == "RMC":
updated = self._apply_rmc(fields) or updated
elif kind == "GGA":
updated = self._apply_gga(fields) or updated
elif kind == "VTG":
updated = self._apply_vtg(fields) or updated
elif kind == "GSA":
updated = self._apply_gsa(fields) or updated
elif kind == "GSV":
# GSV only provides satellite count; use field 3 (total sats in view).
if len(fields) >= 3:
sats = _to_int(fields[2])
if sats is not None:
self._fix.sats = sats
updated = True
else:
return None
if not updated:
return None
self._fix.ts = ts
self._fix.source = source
sentences = set(self._fix.sentences)
sentences.add(kind)
self._fix.sentences = tuple(sorted(sentences))
self._incr("gps_fixes")
# Return a shallow copy so callers can freeze the moment in time.
return GpsFix(
ts=self._fix.ts,
source=self._fix.source,
lat=self._fix.lat,
lon=self._fix.lon,
sog=self._fix.sog,
cog=self._fix.cog,
alt=self._fix.alt,
fix_quality=self._fix.fix_quality,
sats=self._fix.sats,
hdop=self._fix.hdop,
sentences=self._fix.sentences,
)
# -- sentence applicators ---------------------------------------------
def _apply_rmc(self, f: tuple[str, ...]) -> bool:
# RMC: time, status, lat, N/S, lon, E/W, sog(knots), cog, date, magvar, E/W, [mode]
if len(f) < 8:
return False
status = f[1] if len(f) > 1 else ""
if status and status.upper() != "A":
# "V" = navigation receiver warning; ignore position but still update
# speed/course if present.
pass
lat = _parse_lat(f[2], f[3])
lon = _parse_lon(f[4], f[5])
sog = _to_float(f[6])
cog = _to_float(f[7])
updated = False
if lat is not None:
self._fix.lat = lat
updated = True
if lon is not None:
self._fix.lon = lon
updated = True
if sog is not None:
self._fix.sog = sog
updated = True
if cog is not None:
self._fix.cog = cog
updated = True
return updated
def _apply_gga(self, f: tuple[str, ...]) -> bool:
# GGA: time, lat, N/S, lon, E/W, fix_quality, sats, hdop, alt, M, ...
if len(f) < 9:
return False
lat = _parse_lat(f[1], f[2])
lon = _parse_lon(f[3], f[4])
quality = _to_int(f[5])
sats = _to_int(f[6])
hdop = _to_float(f[7])
alt = _to_float(f[8])
updated = False
if lat is not None:
self._fix.lat = lat
updated = True
if lon is not None:
self._fix.lon = lon
updated = True
if quality is not None:
self._fix.fix_quality = quality
updated = True
if sats is not None:
self._fix.sats = sats
updated = True
if hdop is not None:
self._fix.hdop = hdop
updated = True
if alt is not None:
self._fix.alt = alt
updated = True
return updated
def _apply_vtg(self, f: tuple[str, ...]) -> bool:
# VTG: cog_true, T, cog_mag, M, sog_knots, N, sog_kmh, K, [mode]
if len(f) < 8:
return False
cog_true = _to_float(f[0])
sog_knots = _to_float(f[4])
sog_kmh = _to_float(f[6])
updated = False
if cog_true is not None:
self._fix.cog = cog_true
updated = True
if sog_knots is not None:
self._fix.sog = sog_knots
updated = True
elif sog_kmh is not None:
self._fix.sog = sog_kmh * KNOTS_PER_KMH
updated = True
return updated
def _apply_gsa(self, f: tuple[str, ...]) -> bool:
# GSA: mode, fix_type, sat1..sat12, pdop, hdop, vdop
if len(f) < 17:
return False
# fix_type: 1=no fix, 2=2D, 3=3D
fix_type = _to_int(f[1])
hdop = _to_float(f[15])
updated = False
if fix_type is not None:
self._fix.fix_quality = fix_type
updated = True
if hdop is not None:
self._fix.hdop = hdop
updated = True
return updated
def _incr(self, name: str, n: int = 1) -> None:
if self._stats is not None:
self._stats.incr(name, n)
__all__ = ["GpsParser"]
src/ais_hub/parser/nmea_utils.py
+136
View File
@@ -0,0 +1,136 @@
"""NMEA 0183 helpers: checksum, tokenization, line validation.
An NMEA 0183 sentence looks like::
!AIVDM,1,1,,A,15M67FC000G?ufbE`FepT@3n00Sa,0*5B
$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
- The sentence starts with ``!`` (encapsulated, e.g. AIS) or ``$`` (plain).
- Everything between the leading character and ``*`` is the payload used
for the XOR checksum.
- ``*HH`` is a 2-digit hex checksum of the XOR of the payload bytes.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Iterable
@dataclass(slots=True, frozen=True)
class NmeaSentence:
"""Parsed structural view of a single NMEA sentence."""
raw: str # original line, no trailing CRLF
start: str # "!" or "$"
talker: str # e.g. "AIVDM", "GPRMC", "BSVDM", "GNGGA"
fields: tuple[str, ...] # comma-separated fields after the talker
checksum_ok: bool # whether the line had a valid *HH checksum
def strip_eol(line: str) -> str:
"""Remove trailing CR/LF if present."""
if line.endswith("\r\n"):
return line[:-2]
if line.endswith("\n") or line.endswith("\r"):
return line[:-1]
return line
def compute_checksum(payload: str) -> int:
"""XOR checksum of every byte in ``payload`` (no start char, no '*')."""
cs = 0
for ch in payload.encode("ascii", errors="replace"):
cs ^= ch
return cs
def parse_sentence(line: str) -> NmeaSentence | None:
"""Parse a single NMEA line; return None for clearly-malformed input.
The ``checksum_ok`` flag indicates whether the ``*HH`` checksum matched.
Lines without a checksum segment are returned with ``checksum_ok=False``.
"""
s = strip_eol(line).strip()
if not s:
return None
if s[0] not in ("!", "$"):
return None
if len(s) < 4:
return None
# Split payload / checksum.
star = s.rfind("*")
checksum_ok = False
if star != -1 and len(s) - star >= 3:
payload = s[1:star]
csum_str = s[star + 1 : star + 3]
try:
csum = int(csum_str, 16)
except ValueError:
csum = -1
checksum_ok = csum == compute_checksum(payload)
body = payload
else:
body = s[1:]
checksum_ok = False
parts = body.split(",")
if not parts or not parts[0]:
return None
talker = parts[0]
fields = tuple(parts[1:])
return NmeaSentence(raw=s, start=s[0], talker=talker, fields=fields, checksum_ok=checksum_ok)
def split_lines(chunk: bytes | str) -> list[str]:
"""Split a raw chunk into NMEA lines, dropping empty lines.
UART/UDP may deliver several sentences in one read; handles CRLF/LF/CR.
"""
if isinstance(chunk, (bytes, bytearray)):
try:
text = chunk.decode("ascii", errors="replace")
except Exception:
text = chunk.decode("latin-1", errors="replace")
else:
text = chunk
out: list[str] = []
for part in text.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
p = part.strip()
if p:
out.append(p)
return out
def classify_kind(talker: str, start: str) -> str:
"""Rough classifier used by ingest to tag a raw frame.
Returns one of: ``"ais"``, ``"gps"``, ``"other"``.
"""
if start == "!":
# AIVDM / AIVDO / BSVDM / ANVDM / etc.
if talker.endswith("VDM") or talker.endswith("VDO"):
return "ais"
return "other"
# $-prefixed sentences are typically from GNSS receivers.
if talker.startswith(("GP", "GN", "GL", "GA", "BD", "GB", "II")):
return "gps"
return "other"
def lines_from_iter(source: Iterable[bytes | str]) -> Iterable[str]:
for chunk in source:
for ln in split_lines(chunk):
yield ln
__all__ = [
"NmeaSentence",
"compute_checksum",
"parse_sentence",
"split_lines",
"classify_kind",
"strip_eol",
"lines_from_iter",
]
src/ais_hub/parser/radio.py
+79
View File
@@ -0,0 +1,79 @@
"""Radio telemetry parser.
The radio telemetry stream is expected to arrive as UDP datagrams carrying
JSON objects describing TDMA slot / RSSI / SNR data. The exact schema is
defined by the radio front-end; this parser is intentionally permissive.
"""
from __future__ import annotations
import json
import logging
import time
from typing import Any
from ..core.models import RadioReport
from ..core.stats import Stats
log = logging.getLogger(__name__)
class RadioParser:
"""Decode one UDP datagram of radio telemetry."""
def __init__(self, stats: Stats | None = None) -> None:
self._stats = stats
def feed(self, source: str, data: bytes | str, ts: float | None = None) -> RadioReport | None:
if ts is None:
ts = time.time()
try:
if isinstance(data, (bytes, bytearray)):
text = data.decode("utf-8", errors="replace")
else:
text = data
payload = json.loads(text)
except Exception:
if self._stats is not None:
self._stats.incr("radio_parser_errors")
return None
if not isinstance(payload, dict):
if self._stats is not None:
self._stats.incr("radio_parser_errors")
return None
report = RadioReport(
ts=ts,
source=source,
channel=_as_str(payload.get("channel")),
rssi=_as_float(payload.get("rssi")),
snr=_as_float(payload.get("snr")),
slot=_as_int(payload.get("slot")),
raw=payload,
)
if self._stats is not None:
self._stats.incr("radio_reports")
return report
def _as_float(v: Any) -> float | None:
try:
return float(v) if v is not None else None
except (TypeError, ValueError):
return None
def _as_int(v: Any) -> int | None:
try:
return int(v) if v is not None else None
except (TypeError, ValueError):
return None
def _as_str(v: Any) -> str | None:
if v is None:
return None
return str(v)
__all__ = ["RadioParser"]
src/ais_hub/publish/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Publish subsystem: REST + WS, UDP JSON events, UDP raw NMEA fan-out, UDP TX outbox."""
src/ais_hub/publish/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/queues.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/rest.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/udp_events.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/udp_nmea.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/udp_tx_outbox.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/__pycache__/ws.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/publish/queues.py
+47
View File
@@ -0,0 +1,47 @@
"""Bounded asyncio queue helpers with a drop-oldest overflow policy.
Whenever a put_nowait fails with ``QueueFull``, one item is popped from the
head of the queue and the put is retried. Each drop is counted in the
optional ``Stats`` instance under a caller-supplied counter name.
"""
from __future__ import annotations
import asyncio
from typing import TypeVar
from ..core.stats import Stats
T = TypeVar("T")
def put_drop_oldest(
queue: "asyncio.Queue[T]",
item: T,
stats: Stats | None = None,
drop_counter: str | None = None,
) -> bool:
"""Put ``item`` into ``queue``; drop the oldest item on overflow.
Returns True if the item was stored, False only if the queue has no
capacity at all (maxsize=0 semantics, which would block — guarded here).
"""
while True:
try:
queue.put_nowait(item)
return True
except asyncio.QueueFull:
try:
queue.get_nowait()
except asyncio.QueueEmpty:
return False
if stats is not None and drop_counter is not None:
stats.incr(drop_counter)
def make_queue(maxsize: int) -> "asyncio.Queue[object]":
"""Create a bounded queue with the configured maxsize."""
return asyncio.Queue(maxsize=max(1, maxsize))
__all__ = ["put_drop_oldest", "make_queue"]
src/ais_hub/publish/rest.py
+253
View File
@@ -0,0 +1,253 @@
"""REST endpoints under ``/api/v1/*``."""
from __future__ import annotations
import asyncio
import logging
import time
from typing import Any
from aiohttp import web
from ..core.models import TxMessage
from ..logging_setup import tail_logs
from ..parser.nmea_utils import parse_sentence
from ..storage import queries
from ..version import __version__
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _ctx(request: web.Request):
return request.app["ctx"]
def _json(data: Any, status: int = 200) -> web.Response:
return web.json_response(data, status=status)
def _bad_request(reason: str) -> web.Response:
return _json({"error": reason}, status=400)
# ---------------------------------------------------------------------------
# GET handlers
# ---------------------------------------------------------------------------
async def health(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json({
"status": "ok",
"version": __version__,
"uptime_sec": round(time.time() - ctx.stats.started_at, 3),
})
async def stats(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json(ctx.stats.snapshot())
async def ownship(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json(ctx.state.snapshot_ownship())
async def vessels(request: web.Request) -> web.Response:
ctx = _ctx(request)
since = _opt_float(request.query.get("since"))
limit = _opt_int(request.query.get("limit"))
return _json(ctx.state.snapshot_vessels(since=since, limit=limit))
async def vessel_by_mmsi(request: web.Request) -> web.Response:
ctx = _ctx(request)
try:
mmsi = int(request.match_info["mmsi"])
except ValueError:
return _bad_request("invalid mmsi")
v = ctx.state.get_vessel(mmsi)
if v is None:
return _json({"error": "not found", "mmsi": mmsi}, status=404)
return _json(v)
async def base_stations(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json(ctx.state.snapshot_base_stations())
async def atons(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json(ctx.state.snapshot_atons())
async def slots(request: web.Request) -> web.Response:
ctx = _ctx(request)
return _json(ctx.state.snapshot_slots())
async def radio(request: web.Request) -> web.Response:
"""Latest RSSI pair + optional per-channel last detail."""
ctx = _ctx(request)
return _json(ctx.state.snapshot_radio())
async def logs(request: web.Request) -> web.Response:
"""Service log tail. NOT raw NMEA; see /nmea/tail for that."""
level = request.query.get("level")
limit = _opt_int(request.query.get("limit")) or 200
since = _opt_float(request.query.get("since"))
return _json(tail_logs(limit=limit, level=level, since=since))
async def nmea_tail_handler(request: web.Request) -> web.Response:
"""Raw NMEA tail. Reads from in-memory ring first; falls back to SQLite."""
ctx = _ctx(request)
source = request.query.get("source") # "ais" | "gps" | "radio"
limit = _opt_int(request.query.get("limit")) or 200
# In-memory ring (always maintained by the raw tap).
frames = ctx.raw_tap.tail(limit=limit, kind=source)
if frames:
return _json([
{"ts": f.ts, "source": f.source, "kind": f.kind, "line": f.line}
for f in frames
])
# Fall back to SQLite raw_nmea if enabled.
if ctx.db is not None:
try:
rows = await queries.nmea_tail(ctx.db, source_kind=source, limit=limit)
return _json(rows)
except Exception:
log.exception("nmea_tail DB query failed")
return _json([])
async def history_positions(request: web.Request) -> web.Response:
ctx = _ctx(request)
if ctx.db is None:
return _json({"error": "history storage disabled"}, status=503)
mmsi_raw = request.query.get("mmsi")
if not mmsi_raw:
return _bad_request("missing mmsi")
try:
mmsi = int(mmsi_raw)
except ValueError:
return _bad_request("invalid mmsi")
ts_from = _opt_float(request.query.get("from"))
ts_to = _opt_float(request.query.get("to"))
limit = _opt_int(request.query.get("limit")) or 1000
try:
rows = await queries.history_positions(
ctx.db, mmsi, ts_from=ts_from, ts_to=ts_to, limit=limit,
)
except Exception:
log.exception("history_positions failed for mmsi=%d", mmsi)
return _json({"error": "query failed"}, status=500)
return _json(rows)
# ---------------------------------------------------------------------------
# POST handlers
# ---------------------------------------------------------------------------
async def tx(request: web.Request) -> web.Response:
"""Inject NMEA sentence(s) into the TX outbox for the SPI bridge."""
ctx = _ctx(request)
try:
payload = await request.json()
except Exception:
return _bad_request("invalid json body")
lines: list[str] = []
if isinstance(payload, dict):
if "payload" in payload and isinstance(payload["payload"], str):
lines = [payload["payload"]]
elif "nmea" in payload and isinstance(payload["nmea"], list):
lines = [str(x) for x in payload["nmea"] if isinstance(x, str)]
if not lines:
return _bad_request("expected {'payload': str} or {'nmea': [str, ...]}")
accepted: list[str] = []
rejected: list[dict[str, str]] = []
for raw in lines:
s = parse_sentence(raw)
if s is None or not s.checksum_ok:
rejected.append({"line": raw, "reason": "invalid nmea or checksum"})
continue
msg = TxMessage(ts=time.time(), line=s.raw, origin="rest:POST /api/v1/tx")
try:
ctx.tx_outbox.put_nowait(msg)
accepted.append(s.raw)
except asyncio.QueueFull:
ctx.stats.incr("tx_outbox_dropped_rest")
return _json(
{"error": "outbox full", "accepted": accepted, "rejected": rejected},
status=503,
)
ctx.stats.incr("tx_submitted", len(accepted))
return _json({
"queued": len(accepted),
"rejected": rejected,
"queue_depth": ctx.tx_outbox.qsize(),
})
# ---------------------------------------------------------------------------
# Utils
# ---------------------------------------------------------------------------
def _opt_int(v: str | None) -> int | None:
if v is None or v == "":
return None
try:
return int(v)
except ValueError:
return None
def _opt_float(v: str | None) -> float | None:
if v is None or v == "":
return None
try:
return float(v)
except ValueError:
return None
# ---------------------------------------------------------------------------
# Route registration
# ---------------------------------------------------------------------------
def register_routes(app: web.Application) -> None:
app.router.add_get("/api/v1/health", health)
app.router.add_get("/api/v1/stats", stats)
app.router.add_get("/api/v1/ownship", ownship)
app.router.add_get("/api/v1/vessels", vessels)
app.router.add_get(r"/api/v1/vessels/{mmsi:\d+}", vessel_by_mmsi)
app.router.add_get("/api/v1/base_stations", base_stations)
app.router.add_get("/api/v1/atons", atons)
app.router.add_get("/api/v1/slots", slots)
app.router.add_get("/api/v1/radio", radio)
app.router.add_get("/api/v1/logs", logs)
app.router.add_get("/api/v1/nmea/tail", nmea_tail_handler)
app.router.add_get("/api/v1/history/positions", history_positions)
app.router.add_post("/api/v1/tx", tx)
__all__ = ["register_routes"]
src/ais_hub/publish/udp_events.py
+73
View File
@@ -0,0 +1,73 @@
"""UDP JSON event publisher on ``127.0.0.1:7001`` (configurable).
Contract: one serialized ``Event`` per UDP datagram. Events larger than
``max_datagram_bytes`` are dropped and counted in
``stats.udp_events_oversize`` (no fragmentation).
"""
from __future__ import annotations
import asyncio
import json
import logging
from typing import Any
from ..config import UdpEventsCfg
from ..core.bus import EventBus
from ..core.models import Event
from ..core.stats import Stats
log = logging.getLogger(__name__)
class UdpEventsPublisher:
def __init__(self, cfg: UdpEventsCfg, bus: EventBus, stats: Stats) -> None:
self._cfg = cfg
self._bus = bus
self._stats = stats
self._transport: asyncio.DatagramTransport | None = None
self._sub: asyncio.Queue[Event] | None = None
async def run(self) -> None:
loop = asyncio.get_running_loop()
# Ephemeral sender socket; we only *send* to a fixed target.
transport, _ = await loop.create_datagram_endpoint(
lambda: asyncio.DatagramProtocol(),
remote_addr=(self._cfg.host, self._cfg.port),
)
self._transport = transport
self._sub = self._bus.subscribe(maxsize=1024)
log.info("udp_events publisher -> %s:%d", self._cfg.host, self._cfg.port)
try:
while True:
ev = await self._sub.get()
self._send(ev)
except asyncio.CancelledError:
raise
finally:
if self._sub is not None:
self._bus.unsubscribe(self._sub)
self._sub = None
if self._transport is not None:
self._transport.close()
self._transport = None
def _send(self, ev: Event) -> None:
try:
blob = json.dumps(ev.to_dict(), ensure_ascii=False, separators=(",", ":")).encode("utf-8")
except Exception:
self._stats.incr("udp_events_serialize_errors")
return
if len(blob) > self._cfg.max_datagram_bytes:
self._stats.incr("udp_events_oversize")
return
try:
assert self._transport is not None
self._transport.sendto(blob)
self._stats.incr("udp_events_sent")
except Exception:
self._stats.incr("udp_events_send_errors")
log.debug("udp_events send failed", exc_info=True)
__all__ = ["UdpEventsPublisher"]
src/ais_hub/publish/udp_nmea.py
+68
View File
@@ -0,0 +1,68 @@
"""UDP raw NMEA fan-out on ``127.0.0.1:6007`` (configurable).
Contract: one NMEA sentence = one UDP datagram.
Sentence is sent as ``<line>\\r\\n`` in ASCII/UTF-8, including the leading
``!`` or ``$`` and the trailing ``*HH`` checksum. No concatenation or
fragmentation is ever performed.
"""
from __future__ import annotations
import asyncio
import logging
from ..config import UdpNmeaCfg
from ..core.models import RawFrame
from ..core.stats import Stats
from .queues import put_drop_oldest
log = logging.getLogger(__name__)
class UdpNmeaPublisher:
def __init__(
self,
cfg: UdpNmeaCfg,
stats: Stats,
queue: "asyncio.Queue[RawFrame]",
) -> None:
self._cfg = cfg
self._stats = stats
self._queue = queue
self._transport: asyncio.DatagramTransport | None = None
async def run(self) -> None:
loop = asyncio.get_running_loop()
transport, _ = await loop.create_datagram_endpoint(
lambda: asyncio.DatagramProtocol(),
remote_addr=(self._cfg.host, self._cfg.port),
)
self._transport = transport
log.info("udp_nmea fan-out -> %s:%d", self._cfg.host, self._cfg.port)
try:
while True:
frame = await self._queue.get()
self._send(frame)
except asyncio.CancelledError:
raise
finally:
if self._transport is not None:
self._transport.close()
self._transport = None
def _send(self, frame: RawFrame) -> None:
line = frame.line
if not line:
return
# Preserve newline-terminated wire format expected by SPI bridge.
payload = (line + "\r\n").encode("ascii", errors="replace")
try:
assert self._transport is not None
self._transport.sendto(payload)
self._stats.incr("udp_nmea_sent")
except Exception:
self._stats.incr("udp_nmea_send_errors")
log.debug("udp_nmea send failed", exc_info=True)
__all__ = ["UdpNmeaPublisher"]
src/ais_hub/publish/udp_tx_outbox.py
+71
View File
@@ -0,0 +1,71 @@
"""UDP TX outbox publisher on ``127.0.0.1:6010`` (configurable).
This is a **publish-only** channel. ais_hub writes NMEA sentences here
that the SPI bridge should transmit. Content sources:
1. REST ``POST /api/v1/tx`` — external injection (from BLE/web, etc).
2. Internal emitters (e.g. ownship AIS position reports).
Contract: one NMEA sentence = one UDP datagram (``<line>\\r\\n``, ASCII).
Queue is bounded with drop-oldest; drops are counted in
``stats.tx_outbox_dropped``.
"""
from __future__ import annotations
import asyncio
import logging
from ..config import UdpTxOutboxCfg
from ..core.models import TxMessage
from ..core.stats import Stats
log = logging.getLogger(__name__)
class UdpTxOutboxPublisher:
def __init__(
self,
cfg: UdpTxOutboxCfg,
stats: Stats,
outbox: "asyncio.Queue[TxMessage]",
) -> None:
self._cfg = cfg
self._stats = stats
self._outbox = outbox
self._transport: asyncio.DatagramTransport | None = None
async def run(self) -> None:
loop = asyncio.get_running_loop()
transport, _ = await loop.create_datagram_endpoint(
lambda: asyncio.DatagramProtocol(),
remote_addr=(self._cfg.host, self._cfg.port),
)
self._transport = transport
log.info("udp_tx_outbox -> %s:%d", self._cfg.host, self._cfg.port)
try:
while True:
msg = await self._outbox.get()
self._send(msg)
except asyncio.CancelledError:
raise
finally:
if self._transport is not None:
self._transport.close()
self._transport = None
def _send(self, msg: TxMessage) -> None:
line = msg.line
if not line:
return
payload = (line + "\r\n").encode("ascii", errors="replace")
try:
assert self._transport is not None
self._transport.sendto(payload)
self._stats.incr("tx_outbox_sent")
except Exception:
self._stats.incr("tx_outbox_send_errors")
log.debug("tx_outbox send failed", exc_info=True)
__all__ = ["UdpTxOutboxPublisher"]
src/ais_hub/publish/ws.py
+93
View File
@@ -0,0 +1,93 @@
"""WebSocket hub.
- One aiohttp WebSocket handler under ``/ws``.
- Each client gets its own bounded queue subscribed to the event bus.
- On connect, the client receives a snapshot of the current state
(ownship + vessels + base_stations + atons + slots + stats) as a single
``state.snapshot`` event, then a continuous stream of bus events.
- Slow consumers: drop-oldest policy on their queue + ``ws_dropped``.
"""
from __future__ import annotations
import asyncio
import json
import logging
import time
from typing import TYPE_CHECKING
from aiohttp import WSMsgType, web
from ..core.models import Event
log = logging.getLogger(__name__)
async def ws_handler(request: web.Request) -> web.WebSocketResponse:
ctx = request.app["ctx"]
ws = web.WebSocketResponse(heartbeat=30.0, max_msg_size=4 * 1024 * 1024)
await ws.prepare(request)
client_queue: asyncio.Queue[Event] = ctx.bus.subscribe(maxsize=ctx.cfg.queues.ws_client)
ctx.stats.incr("ws_clients_total")
ctx.stats.set_gauge("ws_clients", float(ctx.bus.subscribers))
# Send initial snapshot.
try:
snapshot = {
"ownship": ctx.state.snapshot_ownship(),
"vessels": ctx.state.snapshot_vessels(),
"base_stations": ctx.state.snapshot_base_stations(),
"atons": ctx.state.snapshot_atons(),
"slots": ctx.state.snapshot_slots(),
"stats": ctx.stats.snapshot(),
}
await ws.send_json({"type": "state.snapshot", "ts": time.time(), "data": snapshot})
except Exception:
log.exception("ws initial snapshot failed")
sender = asyncio.create_task(_sender(ws, client_queue, ctx), name="ws.sender")
try:
async for msg in ws:
# The WS is publish-only from our side; ignore client messages
# except PING/CLOSE which aiohttp handles automatically.
if msg.type == WSMsgType.ERROR:
log.warning("ws error: %s", ws.exception())
break
finally:
sender.cancel()
try:
await sender
except Exception:
pass
ctx.bus.unsubscribe(client_queue)
ctx.stats.set_gauge("ws_clients", float(ctx.bus.subscribers))
return ws
async def _sender(ws: web.WebSocketResponse, queue: asyncio.Queue[Event], ctx) -> None:
try:
while not ws.closed:
try:
ev = await queue.get()
except asyncio.CancelledError:
raise
try:
await ws.send_str(json.dumps(ev.to_dict(), ensure_ascii=False))
except ConnectionResetError:
return
except Exception:
ctx.stats.incr("ws_send_errors")
log.debug("ws send failed", exc_info=True)
return
except asyncio.CancelledError:
raise
def register_routes(app: web.Application) -> None:
app.router.add_get("/ws", ws_handler)
__all__ = ["register_routes", "ws_handler"]
src/ais_hub/py.typed
View File
src/ais_hub/storage/__init__.py
+1
View File
@@ -0,0 +1 @@
"""SQLite storage: schema, batched writer, retention, read queries."""
src/ais_hub/storage/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/storage/__pycache__/db.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/storage/__pycache__/queries.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/storage/__pycache__/retention.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/storage/__pycache__/writer.cpython-313.pyc
BIN
View File
Binary file not shown.
src/ais_hub/storage/db.py
+162
View File
@@ -0,0 +1,162 @@
"""aiosqlite connection helper: WAL pragmas, schema bootstrap, migrations."""
from __future__ import annotations
import logging
import os
from pathlib import Path
from typing import Any
import aiosqlite
log = logging.getLogger(__name__)
SCHEMA_VERSION = 1
SCHEMA_SQL: tuple[str, ...] = (
"""
CREATE TABLE IF NOT EXISTS schema_meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
)
""",
"""
CREATE TABLE IF NOT EXISTS vessel_static (
mmsi INTEGER PRIMARY KEY,
name TEXT,
callsign TEXT,
imo INTEGER,
ship_type INTEGER,
dim_a INTEGER,
dim_b INTEGER,
dim_c INTEGER,
dim_d INTEGER,
eta TEXT,
draught REAL,
destination TEXT,
updated_at REAL NOT NULL
)
""",
"""
CREATE TABLE IF NOT EXISTS ais_dynamic (
id INTEGER PRIMARY KEY AUTOINCREMENT,
mmsi INTEGER NOT NULL,
ts REAL NOT NULL,
lat REAL,
lon REAL,
sog REAL,
cog REAL,
heading REAL,
nav_status INTEGER,
rot REAL,
raw_msg_type INTEGER
)
""",
"CREATE INDEX IF NOT EXISTS ix_ais_dynamic_mmsi_ts ON ais_dynamic(mmsi, ts)",
"CREATE INDEX IF NOT EXISTS ix_ais_dynamic_ts ON ais_dynamic(ts)",
"""
CREATE TABLE IF NOT EXISTS gps_fix (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts REAL NOT NULL,
lat REAL,
lon REAL,
sog REAL,
cog REAL,
alt REAL,
fix_quality INTEGER,
sats INTEGER,
hdop REAL
)
""",
"CREATE INDEX IF NOT EXISTS ix_gps_fix_ts ON gps_fix(ts)",
"""
CREATE TABLE IF NOT EXISTS raw_nmea (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts REAL NOT NULL,
source TEXT NOT NULL,
kind TEXT NOT NULL,
line TEXT NOT NULL
)
""",
"CREATE INDEX IF NOT EXISTS ix_raw_nmea_ts ON raw_nmea(ts)",
"CREATE INDEX IF NOT EXISTS ix_raw_nmea_kind_ts ON raw_nmea(kind, ts)",
"""
CREATE TABLE IF NOT EXISTS base_station (
mmsi INTEGER PRIMARY KEY,
ts REAL NOT NULL,
lat REAL,
lon REAL,
epfd INTEGER,
updated_at REAL NOT NULL
)
""",
"""
CREATE TABLE IF NOT EXISTS aton (
mmsi INTEGER PRIMARY KEY,
ts REAL NOT NULL,
lat REAL,
lon REAL,
aton_type INTEGER,
name TEXT,
virtual INTEGER,
updated_at REAL NOT NULL
)
""",
"""
CREATE TABLE IF NOT EXISTS radio_telemetry (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts REAL NOT NULL,
channel TEXT,
rssi REAL,
snr REAL,
slot INTEGER,
raw_json TEXT
)
""",
"CREATE INDEX IF NOT EXISTS ix_radio_telemetry_ts ON radio_telemetry(ts)",
)
async def connect(path: str) -> aiosqlite.Connection:
"""Open the SQLite DB with WAL pragmas and ensure the schema exists."""
p = Path(path)
p.parent.mkdir(parents=True, exist_ok=True)
conn = await aiosqlite.connect(path, timeout=5.0)
# Pragmas — WAL + sane defaults for embedded devices.
await conn.execute("PRAGMA journal_mode=WAL")
await conn.execute("PRAGMA synchronous=NORMAL")
await conn.execute("PRAGMA busy_timeout=5000")
await conn.execute("PRAGMA foreign_keys=ON")
await conn.execute("PRAGMA temp_store=MEMORY")
await _ensure_schema(conn)
return conn
async def _ensure_schema(conn: aiosqlite.Connection) -> None:
for stmt in SCHEMA_SQL:
await conn.execute(stmt)
await conn.execute(
"INSERT OR REPLACE INTO schema_meta(key, value) VALUES('version', ?)",
(str(SCHEMA_VERSION),),
)
await conn.commit()
async def close(conn: aiosqlite.Connection | None) -> None:
if conn is None:
return
try:
await conn.commit()
except Exception:
pass
try:
await conn.close()
except Exception:
pass
__all__ = ["connect", "close", "SCHEMA_VERSION", "SCHEMA_SQL"]
src/ais_hub/storage/queries.py
+123
View File
@@ -0,0 +1,123 @@
"""Read-only queries used by the REST layer."""
from __future__ import annotations
import logging
from typing import Any
import aiosqlite
log = logging.getLogger(__name__)
async def history_positions(
conn: aiosqlite.Connection,
mmsi: int,
ts_from: float | None = None,
ts_to: float | None = None,
limit: int = 1000,
) -> list[dict[str, Any]]:
"""Return AIS dynamic position history for ``mmsi``, ordered by ts desc."""
where = ["mmsi = ?"]
args: list[Any] = [mmsi]
if ts_from is not None:
where.append("ts >= ?")
args.append(ts_from)
if ts_to is not None:
where.append("ts <= ?")
args.append(ts_to)
sql = (
"SELECT ts, lat, lon, sog, cog, heading, nav_status, rot, raw_msg_type "
"FROM ais_dynamic WHERE " + " AND ".join(where)
+ " ORDER BY ts DESC LIMIT ?"
)
args.append(max(1, min(limit, 100_000)))
async with conn.execute(sql, args) as cur:
rows = await cur.fetchall()
return [
{
"ts": r[0], "lat": r[1], "lon": r[2],
"sog": r[3], "cog": r[4], "heading": r[5],
"nav_status": r[6], "rot": r[7], "msg_type": r[8],
}
for r in rows
]
async def nmea_tail(
conn: aiosqlite.Connection,
source_kind: str | None = None,
limit: int = 200,
) -> list[dict[str, Any]]:
"""Return most recent raw NMEA rows (if persistence enabled)."""
where = []
args: list[Any] = []
if source_kind:
where.append("kind = ?")
args.append(source_kind)
sql = "SELECT ts, source, kind, line FROM raw_nmea"
if where:
sql += " WHERE " + " AND ".join(where)
sql += " ORDER BY ts DESC LIMIT ?"
args.append(max(1, min(limit, 10_000)))
async with conn.execute(sql, args) as cur:
rows = await cur.fetchall()
return [{"ts": r[0], "source": r[1], "kind": r[2], "line": r[3]} for r in rows]
async def load_vessel_static(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
"""Load all persisted vessel static/voyage rows (used for warm start)."""
sql = (
"SELECT mmsi, name, callsign, imo, ship_type, "
" dim_a, dim_b, dim_c, dim_d, "
" eta, draught, destination, updated_at "
"FROM vessel_static"
)
async with conn.execute(sql) as cur:
rows = await cur.fetchall()
return [
{
"mmsi": r[0], "name": r[1], "callsign": r[2], "imo": r[3],
"ship_type": r[4], "dim_a": r[5], "dim_b": r[6],
"dim_c": r[7], "dim_d": r[8],
"eta": r[9], "draught": r[10], "destination": r[11],
"updated_at": r[12],
}
for r in rows
]
async def load_base_stations(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
sql = "SELECT mmsi, ts, lat, lon, epfd FROM base_station"
async with conn.execute(sql) as cur:
rows = await cur.fetchall()
return [
{"mmsi": r[0], "ts": r[1], "lat": r[2], "lon": r[3], "epfd": r[4]}
for r in rows
]
async def load_atons(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
sql = "SELECT mmsi, ts, lat, lon, aton_type, name, virtual FROM aton"
async with conn.execute(sql) as cur:
rows = await cur.fetchall()
return [
{
"mmsi": r[0], "ts": r[1], "lat": r[2], "lon": r[3],
"aton_type": r[4], "name": r[5],
"virtual": None if r[6] is None else bool(r[6]),
}
for r in rows
]
__all__ = [
"history_positions",
"nmea_tail",
"load_vessel_static",
"load_base_stations",
"load_atons",
]
src/ais_hub/storage/retention.py
+64
View File
@@ -0,0 +1,64 @@
"""Periodic retention cleanup for history tables."""
from __future__ import annotations
import asyncio
import logging
import sqlite3
import time
import aiosqlite
from ..config import StorageCfg
from ..core.stats import Stats
log = logging.getLogger(__name__)
async def run_retention(conn: aiosqlite.Connection, cfg: StorageCfg, stats: Stats) -> None:
"""Background task: periodically delete rows older than retention limits."""
interval = max(60, cfg.retention_interval_sec)
while True:
try:
await asyncio.sleep(interval)
await cleanup_once(conn, cfg, stats)
except asyncio.CancelledError:
raise
except Exception:
log.exception("retention sweep failed")
stats.incr("retention_errors")
async def cleanup_once(conn: aiosqlite.Connection, cfg: StorageCfg, stats: Stats) -> None:
"""Run one retention sweep; caller ensures this is not re-entrant."""
now = time.time()
rows_total = 0
targets = (
("ais_dynamic", cfg.retention.ais_dynamic_days),
("gps_fix", cfg.retention.gps_fix_days),
("raw_nmea", cfg.retention.raw_nmea_days),
("radio_telemetry", cfg.retention.radio_telemetry_days),
)
for table, days in targets:
if days <= 0:
continue
cutoff = now - days * 86400.0
try:
cur = await conn.execute(f"DELETE FROM {table} WHERE ts < ?", (cutoff,))
await conn.commit()
rows_total += cur.rowcount or 0
stats.incr(f"retention_deleted_{table}", cur.rowcount or 0)
except sqlite3.OperationalError as exc:
log.warning("retention: transient error on %s: %s", table, exc)
stats.incr("retention_transient_errors")
except Exception:
log.exception("retention: error deleting from %s", table)
stats.incr("retention_errors")
if rows_total > 0:
log.info("retention sweep removed %d rows", rows_total)
stats.incr("retention_sweeps")
__all__ = ["run_retention", "cleanup_once"]
src/ais_hub/storage/writer.py
+405
View File
@@ -0,0 +1,405 @@
"""Batched async SQLite writer.
Accepts write jobs on a bounded ``asyncio.Queue``. Jobs are pulled and
grouped by target table; flushed either on ``batch_size`` items or every
``flush_interval_ms`` milliseconds, whichever comes first.
Transient errors (``sqlite3.OperationalError``: "database is locked",
"disk I/O error", etc.) are retried with exponential backoff up to
``_MAX_RETRIES`` attempts; persistent failures are logged and the batch
is dropped so the writer keeps draining the queue.
"""
from __future__ import annotations
import asyncio
import json
import logging
import sqlite3
import time
from dataclasses import dataclass
from typing import Any
import aiosqlite
from ..config import StorageCfg
from ..core.bus import EventBus
from ..core.models import AisReport, AtoN, BaseStation, Event, GpsFix, MergedTarget, RadioReport, RawFrame
from ..core.state import State
from ..core.stats import Stats
from ..publish.queues import put_drop_oldest
log = logging.getLogger(__name__)
_MAX_RETRIES = 3
# ---------------------------------------------------------------------------
# Write jobs
# ---------------------------------------------------------------------------
@dataclass(slots=True)
class _VesselStaticRow:
mmsi: int
target: MergedTarget
ts: float
@dataclass(slots=True)
class _AisDynamicRow:
mmsi: int
ts: float
lat: float | None
lon: float | None
sog: float | None
cog: float | None
heading: float | None
nav_status: int | None
rot: float | None
msg_type: int
@dataclass(slots=True)
class _GpsFixRow:
fix: GpsFix
@dataclass(slots=True)
class _RawNmeaRow:
frame: RawFrame
@dataclass(slots=True)
class _BaseStationRow:
bs: BaseStation
@dataclass(slots=True)
class _AtoNRow:
aton: AtoN
@dataclass(slots=True)
class _RadioRow:
report: RadioReport
WriteJob = (
_VesselStaticRow | _AisDynamicRow | _GpsFixRow | _RawNmeaRow
| _BaseStationRow | _AtoNRow | _RadioRow
)
# ---------------------------------------------------------------------------
# Writer
# ---------------------------------------------------------------------------
class Writer:
"""Batched SQLite writer task."""
def __init__(
self,
conn: aiosqlite.Connection,
cfg: StorageCfg,
stats: Stats,
queue: "asyncio.Queue[WriteJob]",
) -> None:
self._conn = conn
self._cfg = cfg
self._stats = stats
self._queue = queue
async def run(self) -> None:
batch: list[WriteJob] = []
flush_interval = max(0.05, self._cfg.writer.flush_interval_ms / 1000.0)
batch_size = max(1, self._cfg.writer.batch_size)
while True:
try:
first = await asyncio.wait_for(self._queue.get(), timeout=flush_interval)
except asyncio.TimeoutError:
continue
except asyncio.CancelledError:
await self._flush(batch)
raise
batch.append(first)
deadline = time.monotonic() + flush_interval
while len(batch) < batch_size:
remaining = deadline - time.monotonic()
if remaining <= 0:
break
try:
job = await asyncio.wait_for(self._queue.get(), timeout=remaining)
except asyncio.TimeoutError:
break
batch.append(job)
await self._flush(batch)
batch.clear()
async def _flush(self, batch: list[WriteJob]) -> None:
if not batch:
return
self._stats.set_gauge("storage_last_batch", float(len(batch)))
for attempt in range(1, _MAX_RETRIES + 1):
try:
await self._write_batch(batch)
self._stats.incr("storage_batches")
self._stats.incr("storage_rows", len(batch))
return
except sqlite3.OperationalError as exc:
self._stats.incr("storage_transient_errors")
log.warning("sqlite transient error (attempt %d/%d): %s",
attempt, _MAX_RETRIES, exc)
await asyncio.sleep(0.1 * (2 ** (attempt - 1)))
except Exception:
self._stats.incr("storage_errors")
log.exception("sqlite write batch failed; dropping %d rows", len(batch))
return
self._stats.incr("storage_dropped_batches")
log.error("sqlite: giving up on batch of %d rows after %d retries",
len(batch), _MAX_RETRIES)
async def _write_batch(self, batch: list[WriteJob]) -> None:
# Group by table for efficient multi-row inserts.
static_rows: list[tuple] = []
dynamic_rows: list[tuple] = []
gps_rows: list[tuple] = []
raw_rows: list[tuple] = []
base_rows: list[tuple] = []
aton_rows: list[tuple] = []
radio_rows: list[tuple] = []
for job in batch:
if isinstance(job, _VesselStaticRow):
t = job.target
static_rows.append((
t.mmsi, t.name, t.callsign, t.imo, t.ship_type,
t.dim_a, t.dim_b, t.dim_c, t.dim_d,
t.eta, t.draught, t.destination, job.ts,
))
elif isinstance(job, _AisDynamicRow):
dynamic_rows.append((
job.mmsi, job.ts, job.lat, job.lon, job.sog, job.cog,
job.heading, job.nav_status, job.rot, job.msg_type,
))
elif isinstance(job, _GpsFixRow):
f = job.fix
gps_rows.append((
f.ts, f.lat, f.lon, f.sog, f.cog, f.alt,
f.fix_quality, f.sats, f.hdop,
))
elif isinstance(job, _RawNmeaRow):
fr = job.frame
raw_rows.append((fr.ts, fr.source, fr.kind, fr.line))
elif isinstance(job, _BaseStationRow):
b = job.bs
base_rows.append((b.mmsi, b.ts, b.lat, b.lon, b.epfd, b.ts))
elif isinstance(job, _AtoNRow):
a = job.aton
aton_rows.append((
a.mmsi, a.ts, a.lat, a.lon, a.aton_type,
a.name, 1 if a.virtual else 0 if a.virtual is not None else None,
a.ts,
))
elif isinstance(job, _RadioRow):
r = job.report
radio_rows.append((
r.ts, r.channel, r.rssi, r.snr, r.slot,
json.dumps(r.raw, ensure_ascii=False) if r.raw else None,
))
async with self._conn.cursor() as cur:
if static_rows:
await cur.executemany(
"""INSERT INTO vessel_static
(mmsi, name, callsign, imo, ship_type,
dim_a, dim_b, dim_c, dim_d,
eta, draught, destination, updated_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
ON CONFLICT(mmsi) DO UPDATE SET
name=COALESCE(excluded.name, vessel_static.name),
callsign=COALESCE(excluded.callsign, vessel_static.callsign),
imo=COALESCE(excluded.imo, vessel_static.imo),
ship_type=COALESCE(excluded.ship_type, vessel_static.ship_type),
dim_a=COALESCE(excluded.dim_a, vessel_static.dim_a),
dim_b=COALESCE(excluded.dim_b, vessel_static.dim_b),
dim_c=COALESCE(excluded.dim_c, vessel_static.dim_c),
dim_d=COALESCE(excluded.dim_d, vessel_static.dim_d),
eta=COALESCE(excluded.eta, vessel_static.eta),
draught=COALESCE(excluded.draught, vessel_static.draught),
destination=COALESCE(excluded.destination, vessel_static.destination),
updated_at=excluded.updated_at
""",
static_rows,
)
if dynamic_rows:
await cur.executemany(
"""INSERT INTO ais_dynamic
(mmsi, ts, lat, lon, sog, cog, heading, nav_status, rot, raw_msg_type)
VALUES (?,?,?,?,?,?,?,?,?,?)""",
dynamic_rows,
)
if gps_rows:
await cur.executemany(
"""INSERT INTO gps_fix
(ts, lat, lon, sog, cog, alt, fix_quality, sats, hdop)
VALUES (?,?,?,?,?,?,?,?,?)""",
gps_rows,
)
if raw_rows:
await cur.executemany(
"INSERT INTO raw_nmea (ts, source, kind, line) VALUES (?,?,?,?)",
raw_rows,
)
if base_rows:
await cur.executemany(
"""INSERT INTO base_station (mmsi, ts, lat, lon, epfd, updated_at)
VALUES (?,?,?,?,?,?)
ON CONFLICT(mmsi) DO UPDATE SET
ts=excluded.ts, lat=excluded.lat, lon=excluded.lon,
epfd=excluded.epfd, updated_at=excluded.updated_at""",
base_rows,
)
if aton_rows:
await cur.executemany(
"""INSERT INTO aton (mmsi, ts, lat, lon, aton_type, name, virtual, updated_at)
VALUES (?,?,?,?,?,?,?,?)
ON CONFLICT(mmsi) DO UPDATE SET
ts=excluded.ts, lat=excluded.lat, lon=excluded.lon,
aton_type=excluded.aton_type, name=excluded.name,
virtual=excluded.virtual, updated_at=excluded.updated_at""",
aton_rows,
)
if radio_rows:
await cur.executemany(
"""INSERT INTO radio_telemetry (ts, channel, rssi, snr, slot, raw_json)
VALUES (?,?,?,?,?,?)""",
radio_rows,
)
await self._conn.commit()
# ---------------------------------------------------------------------------
# Event-bus sink: converts Events back into WriteJobs
# ---------------------------------------------------------------------------
class StorageEventSink:
"""Subscribe to ``EventBus`` and enqueue write jobs.
This keeps the writer decoupled from ingest/parser; it uses the same
``core.state`` snapshots that publishers see, so anything stored is
consistent with what external processes observe.
"""
def __init__(
self,
bus: EventBus,
state: State,
stats: Stats,
out_queue: "asyncio.Queue[WriteJob]",
store_raw_nmea: bool,
) -> None:
self._bus = bus
self._state = state
self._stats = stats
self._out = out_queue
self._store_raw = store_raw_nmea
self._sub = bus.subscribe(maxsize=4096)
async def run(self) -> None:
try:
while True:
ev: Event = await self._sub.get()
self._dispatch(ev)
except asyncio.CancelledError:
raise
finally:
self._bus.unsubscribe(self._sub)
def enqueue_raw(self, frame: RawFrame) -> None:
if not self._store_raw:
return
put_drop_oldest(self._out, _RawNmeaRow(frame=frame),
self._stats, "storage_in_dropped")
def _dispatch(self, ev: Event) -> None:
try:
if ev.type == "target.update":
self._on_target(ev)
elif ev.type == "ownship.update":
self._on_ownship(ev)
elif ev.type == "base_station.update":
self._on_base_station(ev)
elif ev.type == "aton.update":
self._on_aton(ev)
elif ev.type == "radio.update":
self._on_radio(ev)
except Exception:
log.exception("storage sink dispatch failed for event type=%s", ev.type)
def _put(self, job: WriteJob) -> None:
put_drop_oldest(self._out, job, self._stats, "storage_in_dropped")
def _on_target(self, ev: Event) -> None:
mmsi = int(ev.data["mmsi"])
target = self._state.targets.get(mmsi)
if target is None:
return
# Write static row (upsert) so dims/name/etc. are persisted.
self._put(_VesselStaticRow(mmsi=mmsi, target=target, ts=ev.ts))
# Append a dynamic row (only when dynamic fields were updated,
# otherwise last_dynamic_ts == ev.ts would mean we already have a fix).
if target.last_dynamic_ts == ev.ts:
self._put(_AisDynamicRow(
mmsi=mmsi,
ts=ev.ts,
lat=target.lat,
lon=target.lon,
sog=target.sog,
cog=target.cog,
heading=target.heading,
nav_status=target.nav_status,
rot=target.rot,
msg_type=max(target.msg_types) if target.msg_types else 0,
))
def _on_ownship(self, ev: Event) -> None:
d = ev.data
fix = GpsFix(
ts=ev.ts, source="ownship",
lat=d.get("lat"), lon=d.get("lon"),
sog=d.get("sog"), cog=d.get("cog"), alt=d.get("alt"),
fix_quality=d.get("fix_quality"), sats=d.get("sats"), hdop=d.get("hdop"),
)
self._put(_GpsFixRow(fix=fix))
def _on_base_station(self, ev: Event) -> None:
mmsi = int(ev.data["mmsi"])
bs = self._state.base_stations.get(mmsi)
if bs is not None:
self._put(_BaseStationRow(bs=bs))
def _on_aton(self, ev: Event) -> None:
mmsi = int(ev.data["mmsi"])
a = self._state.atons.get(mmsi)
if a is not None:
self._put(_AtoNRow(aton=a))
def _on_radio(self, ev: Event) -> None:
d = ev.data
self._put(_RadioRow(report=RadioReport(
ts=ev.ts, source="bus",
channel=d.get("channel"),
rssi=d.get("rssi"), snr=d.get("snr"), slot=d.get("slot"),
raw=d.get("raw") or {},
)))
__all__ = ["Writer", "StorageEventSink", "WriteJob"]
src/ais_hub/version.py
+1
View File
@@ -0,0 +1 @@
__version__ = "0.1.0"
systemd/ais_hub.service
+28
View File
@@ -0,0 +1,28 @@
[Unit]
Description=ais_hub AIS/GPS/radio telemetry aggregation service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=ais_hub
Group=ais_hub
ExecStart=/usr/bin/python3 -m ais_hub --config /etc/ais_hub/config.yaml
Restart=on-failure
RestartSec=3
KillSignal=SIGTERM
TimeoutStopSec=15
LimitNOFILE=4096
# Hardening (embedded Linux friendly defaults; remove if they conflict with
# UART device access).
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/ais_hub /var/log/ais_hub
# For UART access, user must be in the serial/dialout group or a udev rule
# should grant access to /dev/ttyS1.
[Install]
WantedBy=multi-user.target
tests/__init__.py
View File
tests/__pycache__/__init__.cpython-313.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/conftest.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_ais_assembler.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_ais_type5_integration.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_aiscatcher_parser.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_aiscatcher_state.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_config.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_gps_parser.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_merge.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_nmea_utils.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_queues_bus.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_rest_smoke.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_retention.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/__pycache__/test_state_warmup.cpython-313-pytest-9.0.3.pyc
BIN
View File
Binary file not shown.
tests/conftest.py
+7
View File
@@ -0,0 +1,7 @@
import sys
from pathlib import Path
_ROOT = Path(__file__).resolve().parents[1]
_SRC = _ROOT / "src"
if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
tests/test_ais_assembler.py
+115
View File
@@ -0,0 +1,115 @@
"""Multi-fragment AIS assembler: strict-order and strengthened-key tests.
These tests exercise the assembler logic only up to fragment validation.
Actual decoding with pyais is covered by a separate integration check
if pyais is importable.
"""
from __future__ import annotations
import pytest
from ais_hub.core.stats import Stats
from ais_hub.parser.ais import AisAssembler
SRC_A = "ais_udp:10.0.0.1:5000"
SRC_B = "ais_udp:10.0.0.2:5000"
def _frag(total: int, n: int, seq: str, channel: str = "A", payload: str = "abcd", fill: str = "0") -> str:
# Compute checksum so parse_sentence accepts the line.
body = f"AIVDM,{total},{n},{seq},{channel},{payload},{fill}"
cs = 0
for ch in body.encode("ascii"):
cs ^= ch
return f"!{body}*{cs:02X}"
def test_single_fragment_fast_path_attempts_decode():
"""A single-fragment sentence is passed to pyais; decode may fail, but
the assembler must not keep any buffers for it."""
stats = Stats()
a = AisAssembler(stats=stats)
line = _frag(1, 1, "")
a.feed(SRC_A, line)
# No multi-fragment buffers.
assert len(a._buffers) == 0
def test_multi_fragment_happy_path_clears_buffer():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 1, "3"))
assert len(a._buffers) == 1
a.feed(SRC_A, _frag(2, 2, "3"))
# After the final fragment arrives, the buffer is removed (regardless
# of whether pyais decodes successfully).
assert len(a._buffers) == 0
def test_out_of_order_fragment_triggers_error_and_reset():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(3, 1, "7"))
a.feed(SRC_A, _frag(3, 3, "7")) # skipped fragment 2
assert stats.counters["ais_fragment_errors"] >= 1
def test_duplicate_fragment_is_error():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 1, "9"))
a.feed(SRC_A, _frag(2, 1, "9")) # duplicate
assert stats.counters["ais_fragment_errors"] >= 1
def test_mid_stream_fragment_without_leading_is_error():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 2, "4"))
assert stats.counters["ais_fragment_errors"] >= 1
assert len(a._buffers) == 0
def test_strengthened_key_isolates_sources():
"""Two sources reusing the same seq_id must not cross-contaminate."""
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 1, "5"))
a.feed(SRC_B, _frag(2, 1, "5"))
assert len(a._buffers) == 2 # two independent buffers
assert stats.counters.get("ais_fragment_errors", 0) == 0
def test_multi_fragment_without_seq_id_is_error():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 1, "")) # empty seq_id on a multi-fragment
assert stats.counters["ais_fragment_errors"] >= 1
def test_total_mismatch_on_followup_is_error():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, _frag(2, 1, "2"))
# Fragment claims total=3 now; must be treated as a fresh start or error.
a.feed(SRC_A, _frag(3, 2, "2"))
assert stats.counters["ais_fragment_errors"] >= 1
def test_ttl_gc_evicts_stale_buffers():
stats = Stats()
a = AisAssembler(stats=stats, ttl_sec=1.0)
a.feed(SRC_A, _frag(2, 1, "8"), ts=1000.0)
assert len(a._buffers) == 1
a.gc(now=1002.0)
assert len(a._buffers) == 0
assert stats.counters["ais_fragment_timeouts"] >= 1
def test_bad_checksum_ignored_with_counter():
stats = Stats()
a = AisAssembler(stats=stats)
a.feed(SRC_A, "!AIVDM,1,1,,A,x,0*00") # deliberately wrong checksum
assert stats.counters["parser_checksum_errors"] >= 1
tests/test_ais_type5_integration.py
+80
View File
@@ -0,0 +1,80 @@
"""End-to-end check for AIS multi-fragment type-5 (static/voyage) decoding.
Verifies that the assembler -> pyais -> normalization -> state -> MergedTarget
chain fills in ``name``/``callsign``/``imo``/``ship_type``/dims/draught/destination.
"""
from __future__ import annotations
import pytest
from ais_hub.core.bus import EventBus
from ais_hub.core.state import State
from ais_hub.core.stats import Stats
from ais_hub.parser.ais import AisAssembler
from ais_hub.parser.nmea_utils import compute_checksum
pyais = pytest.importorskip("pyais")
def _with_csum(body: str) -> str:
return f"!{body}*{compute_checksum(body):02X}"
# Classic pyais demo payload for MT.MITCHELL (MMSI 369190000).
FRAG_1_BODY = (
"AIVDM,2,1,1,A,55P5TL01VIaAL@7WKO@mBplU@<PDhh000000001S;AJ::4A80?4i@E53,0"
)
FRAG_2_BODY = "AIVDM,2,2,1,A,1@0000000000000,2"
def _state() -> State:
stats = Stats()
bus = EventBus(stats=stats, default_maxsize=64)
return State(bus=bus, stats=stats)
def test_type5_two_fragments_populates_static_fields():
state = _state()
stats = Stats()
asm = AisAssembler(stats=stats)
src = "ais_udp:0.0.0.0:5005:127.0.0.1"
for ln in (_with_csum(FRAG_1_BODY), _with_csum(FRAG_2_BODY)):
for rep in asm.feed(src, ln, ts=1000.0):
state.apply_ais(rep)
assert stats.counters.get("ais_msg_5", 0) == 1
v = state.get_vessel(369190000)
assert v is not None
assert v["name"] == "MT.MITCHELL"
assert v["callsign"] == "WDA9674"
assert v["imo"] == 6710932
# ship_type must be a plain int (IntEnum flattened by normalization).
assert isinstance(v["ship_type"], int)
assert v["ship_type"] == 99
assert v["dims"] == {"a": 90, "b": 90, "c": 10, "d": 10}
assert v["voyage"]["destination"] == "SEATTLE"
assert v["voyage"]["draught"] == pytest.approx(6.0)
assert 5 in v["msg_types"]
def test_type5_fragments_with_different_sender_ports_still_merge():
"""Sender port in source_tag changing mid-message must not break assembly.
The AIS UDP ingest only includes the sender IP (not the ephemeral port)
in the source tag, so fragments from the same receiver reach the same
assembler buffer even if the kernel picked different source ports.
"""
state = _state()
stats = Stats()
asm = AisAssembler(stats=stats)
# Same receiver IP -> same source_tag for both fragments.
src = "ais_udp:0.0.0.0:5005:127.0.0.1"
for ln in (_with_csum(FRAG_1_BODY), _with_csum(FRAG_2_BODY)):
for rep in asm.feed(src, ln, ts=1500.0):
state.apply_ais(rep)
assert state.get_vessel(369190000) is not None
tests/test_aiscatcher_parser.py
+167
View File
@@ -0,0 +1,167 @@
"""Binary decoder tests for AIS-catcher Mini UDP telemetry.
The expected byte layouts come from ais-mini's UDP comment block; any
layout drift (field order, endianness, sizes) must fail here before the
ingest runs on real hardware.
"""
from __future__ import annotations
import struct
import pytest
from ais_hub.parser.aiscatcher import (
decode_rssi_iq,
decode_signal_events,
decode_slot_bitmap,
decode_slot_detail,
)
# ---------------------------------------------------------------------------
# Slot bitmap (315 bytes)
# ---------------------------------------------------------------------------
def _build_bitmap(
*,
channel: bytes = b"A",
utc_minute: int = 28_279_310,
slots_total: int = 2250,
occupied: int = 437,
noise: float = 0.0,
threshold: float = 0.0,
slot0_ms: int = 28_279_310 * 60_000,
first_occ_ms: int = 28_279_310 * 60_000 + 123,
bitmap: bytes | None = None,
) -> bytes:
if bitmap is None:
bitmap = bytes(282)
assert len(channel) == 1
assert len(bitmap) == 282
head = struct.pack(
">BIHHffQQ",
channel[0], utc_minute, slots_total, occupied,
noise, threshold, slot0_ms, first_occ_ms,
)
assert len(head) == 33
return head + bitmap
def test_bitmap_decodes_roundtrip():
bm = bytes(range(256)) + bytes(282 - 256)
data = _build_bitmap(channel=b"B", occupied=1234, bitmap=bm)
snap = decode_slot_bitmap(data)
assert snap is not None
assert snap.channel == "B"
assert snap.utc_minute == 28_279_310
assert snap.slots_total == 2250
assert snap.occupied_count == 1234
assert snap.slot0_unix_ms == 28_279_310 * 60_000
assert snap.first_occupied_unix_ms == 28_279_310 * 60_000 + 123
assert snap.bitmap == bm
# occupied_fraction is derived
assert snap.occupied_fraction() == pytest.approx(1234 / 2250)
def test_bitmap_rejects_wrong_size():
data = _build_bitmap()
assert decode_slot_bitmap(data[:-1]) is None
assert decode_slot_bitmap(data + b"\x00") is None
def test_bitmap_unknown_channel_byte_marked_questionmark():
data = _build_bitmap(channel=b"X")
snap = decode_slot_bitmap(data)
assert snap is not None
assert snap.channel == "?"
# ---------------------------------------------------------------------------
# Slot detail
# ---------------------------------------------------------------------------
def test_slot_detail_roundtrip():
entries = [(0, -85.5), (100, -72.125), (2249, -99.0)]
head = struct.pack(">BIQH", ord(b"A"), 28_279_310, 28_279_310 * 60_000, len(entries))
body = b"".join(struct.pack(">Hf", slot, lvl) for slot, lvl in entries)
detail = decode_slot_detail(head + body)
assert detail is not None
assert detail.channel == "A"
assert detail.utc_minute == 28_279_310
assert detail.slot0_unix_ms == 28_279_310 * 60_000
assert [(e.slot, round(e.level_db, 4)) for e in detail.entries] == [
(0, -85.5),
(100, -72.125),
(2249, -99.0),
]
def test_slot_detail_rejects_short_body():
head = struct.pack(">BIQH", ord(b"A"), 1, 2, 3)
# only 1 full entry provided where 3 claimed
body = struct.pack(">Hf", 10, -70.0)
assert decode_slot_detail(head + body) is None
def test_slot_detail_empty_is_valid():
head = struct.pack(">BIQH", ord(b"B"), 1, 2, 0)
detail = decode_slot_detail(head)
assert detail is not None
assert detail.entries == []
# ---------------------------------------------------------------------------
# RSSI IQ (8 bytes)
# ---------------------------------------------------------------------------
def test_rssi_decodes_pair():
data = struct.pack(">ff", -42.25, -55.75)
rssi = decode_rssi_iq(data)
assert rssi is not None
assert rssi.power_a_db == pytest.approx(-42.25)
assert rssi.power_b_db == pytest.approx(-55.75)
def test_rssi_wrong_size():
assert decode_rssi_iq(b"\x00" * 7) is None
assert decode_rssi_iq(b"\x00" * 9) is None
# ---------------------------------------------------------------------------
# Signal events (decode markers)
# ---------------------------------------------------------------------------
def test_signal_events_roundtrip():
events = [
(1_700_000_000_000, 42, 257_123_456, -72.0),
(1_700_000_000_100, 1500, 227_000_001, -88.5),
]
head = struct.pack(">BH", ord(b"A"), len(events))
body = b"".join(struct.pack(">QHIf", ts, slot, mmsi, lvl)
for ts, slot, mmsi, lvl in events)
batch = decode_signal_events(head + body)
assert batch is not None
assert batch.channel == "A"
assert len(batch.events) == 2
assert batch.events[0].unix_ms == 1_700_000_000_000
assert batch.events[0].slot == 42
assert batch.events[0].mmsi == 257_123_456
assert batch.events[0].level_db == pytest.approx(-72.0)
assert batch.events[1].mmsi == 227_000_001
def test_signal_events_empty():
data = struct.pack(">BH", ord(b"A"), 0)
batch = decode_signal_events(data)
assert batch is not None
assert batch.events == []
def test_signal_events_rejects_short_body():
head = struct.pack(">BH", ord(b"A"), 5)
assert decode_signal_events(head + b"\x00" * 10) is None
tests/test_aiscatcher_state.py
+126
View File
@@ -0,0 +1,126 @@
"""State integration tests for AIS-catcher binary telemetry."""
from __future__ import annotations
from ais_hub.core.bus import EventBus
from ais_hub.core.state import State
from ais_hub.core.stats import Stats
from ais_hub.parser.aiscatcher import (
RssiIq,
SignalEvent,
SignalEventBatch,
SlotBitmap,
SlotDetail,
SlotLevel,
)
def _make_state() -> State:
stats = Stats()
bus = EventBus(stats=stats, default_maxsize=64)
return State(bus=bus, stats=stats)
def test_rssi_iq_updates_state_and_publishes():
state = _make_state()
sub = state._bus.subscribe(maxsize=16)
state.apply_rssi_iq(RssiIq(power_a_db=-41.5, power_b_db=-43.0), ts=123.0)
assert state.radio_power == {
"ts": 123.0,
"power_a_db": -41.5,
"power_b_db": -43.0,
}
assert not sub.empty()
ev = sub.get_nowait()
assert ev.type == "radio.update"
assert ev.data["source"] == "aiscatcher_rssi"
assert ev.data["power_a_db"] == -41.5
def test_slot_bitmap_stored_per_channel():
state = _make_state()
snap = SlotBitmap(
channel="A",
utc_minute=28279310,
slots_total=2250,
occupied_count=100,
noise_floor=0.0,
threshold=0.0,
slot0_unix_ms=28279310 * 60000,
first_occupied_unix_ms=0,
bitmap=bytes(282),
)
state.apply_slot_bitmap(snap, ts=50.0)
assert "A" in state.slot_occupancy
occ = state.slot_occupancy["A"]
assert occ["occupied_count"] == 100
assert occ["slots_total"] == 2250
assert occ["occupied_fraction"] == 100 / 2250
# snapshot_slots exposes everything for REST
snap_dict = state.snapshot_slots()
assert snap_dict["occupancy"]["A"]["occupied_count"] == 100
def test_slot_detail_stores_entries():
state = _make_state()
detail = SlotDetail(
channel="B",
utc_minute=1,
slot0_unix_ms=60000,
entries=[SlotLevel(slot=10, level_db=-70.0), SlotLevel(slot=2200, level_db=-80.0)],
)
state.apply_slot_detail(detail, ts=60.0)
stored = state.slot_detail["B"]
assert len(stored["entries"]) == 2
assert stored["entries"][0] == {"slot": 10, "level_db": -70.0}
def test_signal_event_creates_target_with_signal_fields():
state = _make_state()
batch = SignalEventBatch(
channel="A",
events=[
SignalEvent(unix_ms=1_700_000_000_000, slot=42, mmsi=257_000_001, level_db=-75.0),
],
)
state.apply_signal_events(batch, ts=1_700_000_000.0)
assert 257_000_001 in state.targets
tgt = state.targets[257_000_001]
assert tgt.last_signal_db == -75.0
assert tgt.last_signal_slot == 42
assert tgt.last_signal_channel == "A"
# last_seen must be set from signal event if it's newer
assert tgt.last_seen >= 1_700_000_000.0
# to_dict exposes signal block
d = tgt.to_dict()
assert d["signal"] == {
"last_db": -75.0,
"last_ts": 1_700_000_000.0,
"last_slot": 42,
"last_channel": "A",
}
def test_signal_event_older_does_not_overwrite():
state = _make_state()
mmsi = 111_222_333
# first, newer event
state.apply_signal_events(SignalEventBatch(
channel="A",
events=[SignalEvent(unix_ms=2_000_000_000_000, slot=1, mmsi=mmsi, level_db=-60.0)],
), ts=2_000_000_000.0)
# then, older event — must not clobber last_signal_db
state.apply_signal_events(SignalEventBatch(
channel="B",
events=[SignalEvent(unix_ms=1_000_000_000_000, slot=99, mmsi=mmsi, level_db=-90.0)],
), ts=2_000_000_001.0)
tgt = state.targets[mmsi]
assert tgt.last_signal_db == -60.0
assert tgt.last_signal_slot == 1
assert tgt.last_signal_channel == "A"
tests/test_config.py
+39
View File
@@ -0,0 +1,39 @@
import os
from pathlib import Path
from ais_hub.config import Config, load_config
def test_defaults_when_no_file():
cfg = load_config(None)
assert isinstance(cfg, Config)
assert cfg.ingest.ais_udp.port == 4001
assert cfg.publish.udp_tx_outbox.port == 6010
assert cfg.publish.udp_nmea.port == 6007
assert cfg.publish.udp_events.port == 7001
def test_yaml_overrides(tmp_path: Path):
p = tmp_path / "c.yaml"
p.write_text(
"ingest:\n"
" ais_udp:\n"
" port: 5555\n"
"publish:\n"
" http:\n"
" port: 9090\n",
encoding="utf-8",
)
cfg = load_config(p)
assert cfg.ingest.ais_udp.port == 5555
assert cfg.publish.http.port == 9090
# Untouched defaults remain.
assert cfg.publish.udp_nmea.port == 6007
def test_env_overrides(tmp_path: Path, monkeypatch):
monkeypatch.setenv("AIS_HUB_PUBLISH__HTTP__PORT", "7070")
monkeypatch.setenv("AIS_HUB_STORAGE__STORE_RAW_NMEA", "true")
cfg = load_config(None)
assert cfg.publish.http.port == 7070
assert cfg.storage.store_raw_nmea is True
tests/test_gps_parser.py
+47
View File
@@ -0,0 +1,47 @@
from ais_hub.core.stats import Stats
from ais_hub.parser.gps import GpsParser
def _withcs(body: str) -> str:
cs = 0
for ch in body.encode("ascii"):
cs ^= ch
return f"${body}*{cs:02X}"
def test_rmc_produces_lat_lon_sog_cog():
p = GpsParser(stats=Stats())
line = _withcs("GPRMC,123519,A,4807.038,N,01131.000,E,22.4,84.4,230394,3.1,W")
fix = p.feed("gps_uart", line)
assert fix is not None
assert abs(fix.lat - (48 + 7.038 / 60)) < 1e-6
assert abs(fix.lon - (11 + 31.0 / 60)) < 1e-6
assert fix.sog == 22.4
assert fix.cog == 84.4
def test_gga_supplies_fix_quality_and_hdop():
p = GpsParser(stats=Stats())
line = _withcs("GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,")
fix = p.feed("gps_uart", line)
assert fix is not None
assert fix.fix_quality == 1
assert fix.sats == 8
assert fix.hdop == 0.9
assert fix.alt == 545.4
def test_vtg_speed_from_kmh_when_knots_missing():
p = GpsParser(stats=Stats())
line = _withcs("GPVTG,054.7,T,034.4,M,,,010.0,K,A")
fix = p.feed("gps_uart", line)
assert fix is not None
assert fix.cog == 54.7
# 10 km/h ~ 5.3996 knots
assert fix.sog is not None and abs(fix.sog - 5.3996) < 1e-3
def test_bad_checksum_rejected():
p = GpsParser(stats=Stats())
fix = p.feed("gps_uart", "$GPRMC,,,,,,,,,,,,*FF")
assert fix is None
tests/test_merge.py
+63
View File
@@ -0,0 +1,63 @@
from ais_hub.core import merge
from ais_hub.core.models import AisReport, MergedTarget
def test_eager_merge_updates_dynamic_fields():
t = MergedTarget(mmsi=123)
rep = AisReport(
ts=100.0, source="src", kind="dynamic", mmsi=123, msg_type=1,
channel="A", data={"lat": 10.5, "lon": 20.5, "sog": 7.1, "cog": 180.0},
)
changed = merge.apply(t, rep)
assert changed is True
assert t.lat == 10.5
assert t.sog == 7.1
assert t.last_dynamic_ts == 100.0
assert t.last_seen == 100.0
def test_stale_dynamic_report_is_ignored():
t = MergedTarget(mmsi=123)
t.last_dynamic_ts = 200.0
t.lat = 1.0
older = AisReport(
ts=100.0, source="src", kind="dynamic", mmsi=123, msg_type=1,
data={"lat": 50.0},
)
merge.apply(t, older)
assert t.lat == 1.0 # unchanged
def test_static_and_dynamic_keep_separate_timestamps():
t = MergedTarget(mmsi=42)
dyn = AisReport(
ts=100.0, source="a", kind="dynamic", mmsi=42, msg_type=1,
data={"lat": 1.0, "lon": 2.0},
)
stat = AisReport(
ts=110.0, source="a", kind="static", mmsi=42, msg_type=5,
data={"name": "MV TEST", "callsign": "ABCD", "imo": 9999999},
)
merge.apply(t, dyn)
merge.apply(t, stat)
assert t.lat == 1.0 and t.name == "MV TEST"
assert t.last_dynamic_ts == 100.0
assert t.last_static_ts == 110.0
assert t.last_seen == 110.0
def test_msg24_name_and_callsign_combine():
t = MergedTarget(mmsi=7)
# Msg 24A -> name
merge.apply(t, AisReport(
ts=10.0, source="a", kind="static", mmsi=7, msg_type=24,
data={"name": "BOAT"},
))
# Msg 24B -> callsign + dims (later, must not wipe name)
merge.apply(t, AisReport(
ts=11.0, source="a", kind="static", mmsi=7, msg_type=24,
data={"callsign": "CALL", "dim_a": 10, "dim_b": 20},
))
assert t.name == "BOAT"
assert t.callsign == "CALL"
assert t.dim_a == 10

Some files were not shown because too many files have changed in this diff Show More