closed TG-1; git was inited;
This commit is contained in:
@@ -0,0 +1,156 @@
|
||||
# ais_hub
|
||||
|
||||
Central AIS/GPS/radio telemetry aggregation service for embedded Linux.
|
||||
|
||||
`ais_hub` is a Python 3.11 asyncio service. It ingests raw AIS (UDP), GPS
|
||||
(UART) and radio-telemetry (UDP) streams, reassembles multi-fragment AIS
|
||||
sentences, maintains a single authoritative in-memory state keyed by MMSI,
|
||||
writes history to SQLite (WAL + retention), and publishes data to external
|
||||
processes (BLE server, web UI, SPI bridge) via stable interfaces.
|
||||
|
||||
No BLE, web UI or MCU code lives here; those are separate processes that
|
||||
consume the interfaces below.
|
||||
|
||||
## Interfaces
|
||||
|
||||
Полный JSON-контракт с примерами для фронтенда — в **[docs/API.md](docs/API.md)**.
|
||||
Короткая сводка ниже.
|
||||
|
||||
### HTTP (REST, `GET` unless noted)
|
||||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `/api/v1/health` | Liveness + uptime + version. |
|
||||
| `/api/v1/stats` | Counters, gauges, uptime. |
|
||||
| `/api/v1/ownship` | Latest GPS fix. |
|
||||
| `/api/v1/vessels` | All merged targets. Query: `since`, `limit`. |
|
||||
| `/api/v1/vessels/{mmsi}` | One merged target. |
|
||||
| `/api/v1/base_stations` | AIS base station snapshots. |
|
||||
| `/api/v1/atons` | AtoN snapshots. |
|
||||
| `/api/v1/slots` | TDMA slot occupancy + per-slot detail from AIS-catcher. |
|
||||
| `/api/v1/radio` | Latest RSSI pair (channel A/B) from AIS-catcher. |
|
||||
| `/api/v1/logs` | **Service logs tail**, from in-memory ring. Query: `level`, `limit`, `since`. |
|
||||
| `/api/v1/nmea/tail` | **Raw NMEA tail** (separate from `/logs`). Query: `source=ais\|gps\|radio`, `limit`. |
|
||||
| `/api/v1/history/positions` | AIS dynamic history. Query: `mmsi`, `from`, `to`, `limit`. |
|
||||
| `POST /api/v1/tx` | Inject NMEA sentence(s) into the TX outbox. Body: `{"payload": "!AIVDM,..."}` or `{"nmea": ["...", "..."]}`. |
|
||||
|
||||
### WebSocket
|
||||
|
||||
- `GET /ws` — on connect, clients receive a full `state.snapshot` event
|
||||
(ownship, vessels, base_stations, atons, slots, stats). Then a live
|
||||
stream of `Event` envelopes:
|
||||
- `ownship.update`, `target.update`, `base_station.update`,
|
||||
`aton.update`, `radio.update`, `slots.update`, `slots.detail`,
|
||||
`signal.update`, `stats.update`.
|
||||
- Each client has its own bounded queue; slow consumers drop oldest
|
||||
events and the `ws_dropped` counter increments.
|
||||
|
||||
### Local UDP
|
||||
|
||||
| Endpoint | Direction | Contract |
|
||||
| --- | --- | --- |
|
||||
| `127.0.0.1:7001` | ais_hub → external | JSON `Event`, one event per datagram, dropped if > `max_datagram_bytes`. |
|
||||
| `127.0.0.1:6007` | ais_hub → SPI bridge | Raw NMEA fan-out, one sentence per datagram (`<line>\r\n`). Untouched from ingest. |
|
||||
| `127.0.0.1:6010` | ais_hub → SPI bridge | **TX publish outbox**. One NMEA sentence per datagram. Fed by `POST /api/v1/tx` and internal emitters. |
|
||||
|
||||
ais_hub does **not** listen on `:6010`; it is publish-only.
|
||||
|
||||
## Layout
|
||||
|
||||
- `src/ais_hub/ingest/` — AIS UDP / GPS UART / radio UDP listeners, raw tap.
|
||||
- `src/ais_hub/parser/` — NMEA utils, AIS multi-fragment assembler, GPS,
|
||||
radio telemetry parsers.
|
||||
- `src/ais_hub/core/` — domain models, in-memory state, eager per-MMSI
|
||||
merge, stats counters, internal pub/sub bus.
|
||||
- `src/ais_hub/storage/` — aiosqlite DB (WAL), batched writer with retry,
|
||||
retention cleanup, read queries for REST.
|
||||
- `src/ais_hub/publish/` — aiohttp REST + WS server, UDP publishers,
|
||||
bounded queue helpers with drop-oldest policy.
|
||||
- `src/ais_hub/app.py` — orchestrator (supervisors, graceful shutdown).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
Runtime dependencies: `aiohttp`, `pyais`, `pyserial-asyncio`, `aiosqlite`,
|
||||
`PyYAML`.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
ais_hub --config config/config.example.yaml
|
||||
```
|
||||
|
||||
### Environment variable overrides
|
||||
|
||||
Env vars override any YAML value. Prefix `AIS_HUB_`, level separator
|
||||
`__`, upper-case. Examples:
|
||||
|
||||
```bash
|
||||
AIS_HUB_INGEST__AIS_UDP__PORT=4011
|
||||
AIS_HUB_STORAGE__PATH=/data/ais_hub.db
|
||||
AIS_HUB_STORAGE__STORE_RAW_NMEA=true
|
||||
AIS_HUB_LOGGING__LEVEL=DEBUG
|
||||
```
|
||||
|
||||
## Deploy on embedded Linux (systemd)
|
||||
|
||||
1. Create user and directories:
|
||||
|
||||
```bash
|
||||
useradd -r -s /sbin/nologin ais_hub
|
||||
install -d -o ais_hub -g ais_hub /var/lib/ais_hub /var/log/ais_hub /etc/ais_hub
|
||||
```
|
||||
|
||||
2. Install package (system-wide or in a venv the unit points at):
|
||||
|
||||
```bash
|
||||
pip install .
|
||||
```
|
||||
|
||||
3. Copy config:
|
||||
|
||||
```bash
|
||||
install -m 0644 config/config.example.yaml /etc/ais_hub/config.yaml
|
||||
```
|
||||
|
||||
4. Install unit and enable:
|
||||
|
||||
```bash
|
||||
install -m 0644 systemd/ais_hub.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now ais_hub
|
||||
journalctl -u ais_hub -f
|
||||
```
|
||||
|
||||
For UART access, add the `ais_hub` user to the serial/dialout group or
|
||||
grant access via a udev rule on `/dev/ttyS1`.
|
||||
|
||||
## Resilience notes
|
||||
|
||||
- Each subsystem runs under a supervisor task: on exception it is logged
|
||||
and restarted with exponential backoff; the process itself never dies
|
||||
because of a single subsystem failure.
|
||||
- All inter-subsystem queues are bounded with drop-oldest. Per-channel
|
||||
drop counters are exposed via `/api/v1/stats`:
|
||||
- `parser_in_dropped`, `udp_nmea_dropped`,
|
||||
- `storage_in_dropped`, `tx_outbox_dropped`,
|
||||
- `bus_dropped`, `ws_dropped`.
|
||||
- SQLite transient errors are retried up to 3 times with exponential
|
||||
backoff; on persistent failure the batch is dropped, the writer keeps
|
||||
draining the queue, and REST history may return an error while the
|
||||
rest of the service continues to serve live data.
|
||||
- Multi-fragment AIS is keyed by
|
||||
`(source_tag, talker, channel, seq_id, total_fragments)`. Fragment
|
||||
ordering is strictly validated; any deviation resets the per-key
|
||||
buffer and increments `ais_fragment_errors`. Stale buffers are GC'd
|
||||
after 60 s (`ais_fragment_timeouts`).
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
pip install -e .[dev]
|
||||
pytest
|
||||
```
|
||||
@@ -0,0 +1,82 @@
|
||||
# ais_hub configuration (example).
|
||||
# All fields can be overridden via env vars with prefix AIS_HUB_ and
|
||||
# "__" as level separator, e.g. AIS_HUB_INGEST__AIS_UDP__PORT=4011.
|
||||
|
||||
ingest:
|
||||
ais_udp:
|
||||
host: 0.0.0.0
|
||||
port: 4001
|
||||
gps_uart:
|
||||
device: /dev/ttyS1
|
||||
baud: 38400
|
||||
enabled: true
|
||||
radio_udp:
|
||||
host: 0.0.0.0
|
||||
port: 4010
|
||||
# AIS-catcher Mini binary telemetry (see ais-mini.conf slot_udp_*/rssi_udp_*/event_udp_*).
|
||||
# Set enabled: true and point AIS-catcher's *_udp_host/port here (defaults match its defaults).
|
||||
aiscatcher_udp:
|
||||
enabled: false
|
||||
host: 127.0.0.1
|
||||
slot_port: 10111
|
||||
slot_detail_port: 10112
|
||||
rssi_port: 10113
|
||||
event_port: 10114
|
||||
|
||||
publish:
|
||||
http:
|
||||
host: 0.0.0.0
|
||||
port: 8080
|
||||
udp_events:
|
||||
host: 127.0.0.1
|
||||
port: 7001
|
||||
max_datagram_bytes: 1400
|
||||
udp_nmea:
|
||||
host: 127.0.0.1
|
||||
port: 6007
|
||||
udp_tx_outbox:
|
||||
host: 127.0.0.1
|
||||
port: 6010
|
||||
nmea_tail:
|
||||
in_memory_ring: 500
|
||||
|
||||
storage:
|
||||
path: /var/lib/ais_hub/ais_hub.db
|
||||
store_raw_nmea: false
|
||||
retention:
|
||||
ais_dynamic_days: 7
|
||||
gps_fix_days: 7
|
||||
raw_nmea_days: 2
|
||||
radio_telemetry_days: 7
|
||||
writer:
|
||||
batch_size: 200
|
||||
flush_interval_ms: 500
|
||||
retention_interval_sec: 3600
|
||||
|
||||
queues:
|
||||
parser_in: 2048
|
||||
ws_client: 512
|
||||
udp_events: 1024
|
||||
storage_in: 4096
|
||||
tx_outbox: 512
|
||||
|
||||
parser:
|
||||
ais:
|
||||
# Ignore checksum completely (even if present and invalid).
|
||||
# If enabled, this overrides allow_checksumless.
|
||||
ignore_checksum: false
|
||||
# Some real-world NMEA forwarders omit the *HH checksum on !AIVDM/!AIVDO.
|
||||
# OpenCPN is often tolerant; ais_hub is strict by default.
|
||||
allow_checksumless: false
|
||||
# Some multi-fragment AIS sentences come with an empty seq_id (field 3).
|
||||
# Enabling this uses a best-effort heuristic to reassemble those fragments.
|
||||
allow_multipart_without_seq_id: false
|
||||
# TTL for in-flight multipart reassembly buffers.
|
||||
multipart_ttl_sec: 60.0
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
file: /var/log/ais_hub/ais_hub.log
|
||||
max_bytes: 10485760
|
||||
backup_count: 5
|
||||
json: true
|
||||
+630
@@ -0,0 +1,630 @@
|
||||
# ais_hub API reference
|
||||
|
||||
Полный контракт всех публичных интерфейсов `ais_hub` для веб-морды / SPI-bridge / BLE-сервера.
|
||||
|
||||
- **Base URL (HTTP/WS):** `http://<host>:8081` (порт из `publish.http.port`).
|
||||
- **Авторизация:** нет (сервис привязан к localhost / доверенной сети).
|
||||
- **Кодировка:** `UTF-8`. Все JSON — одна строка либо pretty, клиент не должен полагаться на форматирование.
|
||||
- **Timestamps:** все поля `ts` — **Unix epoch в секундах**, `float`. Поля в миллисекундах (от AIS-catcher) явно называются `*_unix_ms`.
|
||||
- **Nullability:** любое поле с неизвестным значением = `null`, даже если тип в таблице int/float.
|
||||
- **Стабильность:** REST пути версионированы (`/api/v1/*`). Новые поля могут добавляться без breaking change — клиент должен игнорировать неизвестные ключи.
|
||||
|
||||
---
|
||||
|
||||
## 1. HTTP REST
|
||||
|
||||
### 1.1 `GET /api/v1/health`
|
||||
|
||||
Liveness probe. Всегда 200, если процесс жив.
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"version": "0.1.0",
|
||||
"uptime_sec": 12345.678
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.2 `GET /api/v1/stats`
|
||||
|
||||
Все внутренние счётчики и gauges. Полезно для мониторинга и диагностики.
|
||||
|
||||
```json
|
||||
{
|
||||
"uptime_sec": 12345.678,
|
||||
"started_at": 1700000000.123,
|
||||
"counters": {
|
||||
"ais_udp_datagrams": 10423,
|
||||
"ais_udp_malformed": 2,
|
||||
"ais_reports": 9812,
|
||||
"ais_msg_1": 4200,
|
||||
"ais_msg_3": 1800,
|
||||
"ais_msg_5": 410,
|
||||
"ais_msg_18": 3200,
|
||||
"ais_msg_24": 202,
|
||||
"ais_fragment_errors": 0,
|
||||
"ais_fragment_timeouts": 0,
|
||||
"parser_errors": 1,
|
||||
"parser_checksum_errors": 0,
|
||||
"state_ownship_updates": 1234,
|
||||
"state_targets_new": 120,
|
||||
"state_targets_updates": 9500,
|
||||
"state_base_station_updates": 0,
|
||||
"state_aton_updates": 0,
|
||||
"state_rssi_updates": 45200,
|
||||
"state_slot_bitmap_updates": 12,
|
||||
"state_slot_detail_updates": 12,
|
||||
"state_signal_events": 8800,
|
||||
"state_warmup_vessels": 42,
|
||||
"storage_batches": 950,
|
||||
"storage_rows": 31200,
|
||||
"storage_errors": 0,
|
||||
"udp_events_sent": 123456,
|
||||
"udp_events_oversize": 3,
|
||||
"supervisor_restarts_ingest_ais_udp": 1,
|
||||
"ws_clients_total": 4
|
||||
},
|
||||
"gauges": {
|
||||
"ws_clients": 1,
|
||||
"storage_last_batch": 200
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Ключевые counters для UI**
|
||||
- `ais_msg_<N>` — сколько сообщений каждого типа декодировано (1/2/3/4/5/18/19/21/24/27 и т.д.).
|
||||
- `ais_fragment_*` — проблемы мультифрагментной сборки.
|
||||
- `state_targets_new` — количество впервые увиденных MMSI.
|
||||
- `state_warmup_vessels` — сколько судов поднято из SQLite при старте (persistence).
|
||||
- `supervisor_restarts_*` — если не 0, какой-то subsystem падает.
|
||||
|
||||
---
|
||||
|
||||
### 1.3 `GET /api/v1/ownship`
|
||||
|
||||
Последний GPS-fix (собственное судно).
|
||||
|
||||
```json
|
||||
{
|
||||
"ts": 1700000000.123,
|
||||
"lat": 59.123456,
|
||||
"lon": 10.987654,
|
||||
"sog": 5.4,
|
||||
"cog": 128.5,
|
||||
"alt": 12.3,
|
||||
"fix_quality": 1,
|
||||
"sats": 9,
|
||||
"hdop": 0.8
|
||||
}
|
||||
```
|
||||
|
||||
| Поле | Тип | Описание |
|
||||
| --- | --- | --- |
|
||||
| `ts` | float | Unix-время последнего обновления. 0 если fix ещё не пришёл. |
|
||||
| `lat`, `lon` | float \| null | Широта/долгота в градусах (WGS-84). |
|
||||
| `sog` | float \| null | Speed over ground, узлы. |
|
||||
| `cog` | float \| null | Course over ground, градусы от истинного севера. |
|
||||
| `alt` | float \| null | Высота над эллипсоидом, метры (GGA). |
|
||||
| `fix_quality` | int \| null | NMEA GGA fix quality (0=нет, 1=GPS, 2=DGPS…). |
|
||||
| `sats` | int \| null | Количество спутников в решении (GGA). |
|
||||
| `hdop` | float \| null | Horizontal dilution of precision. |
|
||||
|
||||
---
|
||||
|
||||
### 1.4 `GET /api/v1/vessels`
|
||||
|
||||
Все известные MMSI, отсортированные по `last_seen` убыванию.
|
||||
|
||||
**Query-параметры**
|
||||
|
||||
| Параметр | Тип | По умолчанию | Описание |
|
||||
| --- | --- | --- | --- |
|
||||
| `since` | float (unix ts) | — | Вернуть только target'ы с `last_seen >= since`. |
|
||||
| `limit` | int | без лимита | Срез первых N (после сортировки). |
|
||||
|
||||
**Response** — массив объектов `MergedTarget` (см. 1.5).
|
||||
|
||||
---
|
||||
|
||||
### 1.5 `GET /api/v1/vessels/{mmsi}`
|
||||
|
||||
Один target. `mmsi` в URL — целое число.
|
||||
|
||||
**Коды ответа**
|
||||
- `200` — найдено;
|
||||
- `400` — невалидный MMSI;
|
||||
- `404` — `{"error": "not found", "mmsi": <mmsi>}`.
|
||||
|
||||
```json
|
||||
{
|
||||
"mmsi": 506140446,
|
||||
"name": "CELESTIAL STAR",
|
||||
"callsign": "V7AB3",
|
||||
"imo": 9123456,
|
||||
"ship_type": 70,
|
||||
"dims": { "a": 180, "b": 30, "c": 16, "d": 16 },
|
||||
"voyage": {
|
||||
"eta": "05-20 12:00",
|
||||
"draught": 7.5,
|
||||
"destination": "ROTTERDAM"
|
||||
},
|
||||
"dynamic": {
|
||||
"lat": 59.912345,
|
||||
"lon": 10.754321,
|
||||
"sog": 12.4,
|
||||
"cog": 210.3,
|
||||
"heading": 208,
|
||||
"nav_status": 0,
|
||||
"rot": -1.2
|
||||
},
|
||||
"signal": {
|
||||
"last_db": -72.5,
|
||||
"last_ts": 1700000000.123,
|
||||
"last_slot": 1487,
|
||||
"last_channel": "A"
|
||||
},
|
||||
"last_static_ts": 1699990000.0,
|
||||
"last_dynamic_ts": 1700000000.123,
|
||||
"last_seen": 1700000000.123,
|
||||
"sources": ["ais_udp:0.0.0.0:5005:127.0.0.1"],
|
||||
"msg_types": [5, 18, 24]
|
||||
}
|
||||
```
|
||||
|
||||
| Блок | Поле | Тип | Описание |
|
||||
| --- | --- | --- | --- |
|
||||
| root | `mmsi` | int | MMSI (9 цифр для судов, другие диапазоны — base stations / AtoN / SAR и т.д.). |
|
||||
| root | `name` | str \| null | Имя судна (type 5 / 24A). Trim + trailing `@` удалены. |
|
||||
| root | `callsign` | str \| null | Callsign (type 5 / 24B). |
|
||||
| root | `imo` | int \| null | IMO номер (type 5). `0` нормализуется в `null` при наличии. |
|
||||
| root | `ship_type` | int \| null | Ship/cargo type 0–99 (всегда плоский int, не enum). |
|
||||
| `dims` | `a/b/c/d` | int \| null | Расстояния от GPS-антенны: `a`=нос, `b`=корма, `c`=левый борт, `d`=правый борт; метры. Длина = a+b, ширина = c+d. |
|
||||
| `voyage` | `eta` | str \| null | ETA в формате `MM-DD HH:MM` (как отдаёт pyais). |
|
||||
| `voyage` | `draught` | float \| null | Осадка, метры. |
|
||||
| `voyage` | `destination` | str \| null | Порт назначения (до 20 символов). |
|
||||
| `dynamic` | `lat`, `lon` | float \| null | Позиция, градусы WGS-84. |
|
||||
| `dynamic` | `sog` | float \| null | Скорость, узлы (0…102.2). |
|
||||
| `dynamic` | `cog` | float \| null | Course over ground, градусы (0…359.9). |
|
||||
| `dynamic` | `heading` | float \| null | Heading, градусы (0…359, 511=unknown → `null`). |
|
||||
| `dynamic` | `nav_status` | int \| null | Navigational status (0–15). |
|
||||
| `dynamic` | `rot` | float \| null | Rate of turn, град/мин. |
|
||||
| `signal` | `last_db` | float \| null | Уровень сигнала последнего декода (от AIS-catcher decode events, port 10114). |
|
||||
| `signal` | `last_ts` | float \| null | Unix-время последнего decode event. |
|
||||
| `signal` | `last_slot` | int \| null | TDMA-слот (0…2249). |
|
||||
| `signal` | `last_channel` | str \| null | `"A"` или `"B"`. |
|
||||
| root | `last_static_ts` | float | Последний приход type-5 / 24 для этого MMSI. 0 если не было. |
|
||||
| root | `last_dynamic_ts` | float | Последний приход type-1/2/3/18/19/27. |
|
||||
| root | `last_seen` | float | `max(last_static_ts, last_dynamic_ts, last_signal_ts)`. |
|
||||
| root | `sources` | str[] | Отсортированный список source-тегов, с которых приходили данные (см. формат в разделе "Источники"). |
|
||||
| root | `msg_types` | int[] | Отсортированный список типов AIS-сообщений, наблюдавшихся для этого MMSI. |
|
||||
|
||||
---
|
||||
|
||||
### 1.6 `GET /api/v1/base_stations`
|
||||
|
||||
AIS base stations (msg 4/11).
|
||||
|
||||
```json
|
||||
[
|
||||
{ "mmsi": 2570001, "ts": 1700000000.0, "lat": 59.9, "lon": 10.7, "epfd": 1 }
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.7 `GET /api/v1/atons`
|
||||
|
||||
Aids-to-Navigation (msg 21).
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"mmsi": 992570001,
|
||||
"ts": 1700000000.0,
|
||||
"lat": 59.9, "lon": 10.7,
|
||||
"type": 3,
|
||||
"name": "FLAKFORTET",
|
||||
"virtual": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
| Поле | Тип | Описание |
|
||||
| --- | --- | --- |
|
||||
| `type` | int \| null | AtoN type (1–31). |
|
||||
| `virtual` | bool \| null | `true` = виртуальный AtoN. |
|
||||
|
||||
---
|
||||
|
||||
### 1.8 `GET /api/v1/slots`
|
||||
|
||||
Снапшот TDMA-слотов и occupancy (наполняется из AIS-catcher, порты 10111/10112).
|
||||
|
||||
```json
|
||||
{
|
||||
"per_channel": {},
|
||||
"occupancy": {
|
||||
"A": {
|
||||
"ts": 1700000060.0,
|
||||
"utc_minute": 28333334,
|
||||
"slots_total": 2250,
|
||||
"occupied_count": 437,
|
||||
"occupied_fraction": 0.1942,
|
||||
"slot0_unix_ms": 1700000040000,
|
||||
"first_occupied_unix_ms": 1700000041123
|
||||
},
|
||||
"B": { ... }
|
||||
},
|
||||
"detail": {
|
||||
"A": {
|
||||
"ts": 1700000060.0,
|
||||
"utc_minute": 28333334,
|
||||
"slot0_unix_ms": 1700000040000,
|
||||
"entries": [
|
||||
{ "slot": 42, "level_db": -72.5 },
|
||||
{ "slot": 1487, "level_db": -80.1 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `per_channel` — зарезервировано (пусто в текущей версии).
|
||||
- `occupancy` — загрузка эфира. Обновляется раз в минуту.
|
||||
- `detail` — список занятых слотов с уровнем сигнала. Обновляется раз в минуту. Может отсутствовать или быть с пустым `entries`.
|
||||
|
||||
---
|
||||
|
||||
### 1.9 `GET /api/v1/radio`
|
||||
|
||||
Мгновенная мощность каналов A/B (от AIS-catcher RSSI, порт 10113, ~10 Гц).
|
||||
|
||||
```json
|
||||
{
|
||||
"power": {
|
||||
"ts": 1700000000.123,
|
||||
"power_a_db": -42.25,
|
||||
"power_b_db": -55.75
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Если RSSI ни разу не приходил — `"power": {}`.
|
||||
|
||||
---
|
||||
|
||||
### 1.10 `GET /api/v1/logs`
|
||||
|
||||
Service logs tail из in-memory ring buffer (не raw NMEA).
|
||||
|
||||
**Query-параметры**
|
||||
|
||||
| Параметр | Тип | По умолчанию | Описание |
|
||||
| --- | --- | --- | --- |
|
||||
| `level` | str | — | Фильтр: `DEBUG`/`INFO`/`WARNING`/`ERROR`/`CRITICAL` (точный match). |
|
||||
| `limit` | int | 200 | Сколько последних записей вернуть. |
|
||||
| `since` | float | — | Unix-ts; вернуть записи с `ts >= since`. |
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ts": 1700000000.123,
|
||||
"level": "INFO",
|
||||
"logger": "ais_hub.app",
|
||||
"message": "warm start: 42 vessels, 0 base_stations, 0 atons from sqlite"
|
||||
},
|
||||
{
|
||||
"ts": 1700000001.234,
|
||||
"level": "WARNING",
|
||||
"logger": "ais_hub.ingest.ais_udp",
|
||||
"message": "ais_udp: cannot bind 0.0.0.0:5005 — already in use..."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.11 `GET /api/v1/nmea/tail`
|
||||
|
||||
Raw NMEA tail — отдельно от `/logs`. Сперва отдаёт in-memory ring buffer (всегда ведётся), при необходимости фоллбэчит в SQLite `raw_nmea` (если `storage.store_raw_nmea: true`).
|
||||
|
||||
**Query-параметры**
|
||||
|
||||
| Параметр | Тип | По умолчанию | Описание |
|
||||
| --- | --- | --- | --- |
|
||||
| `source` | str | — | Фильтр: `ais`, `gps`, `radio`. |
|
||||
| `limit` | int | 200 | Сколько последних строк. |
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ts": 1700000000.123,
|
||||
"source": "ais_udp:0.0.0.0:5005:127.0.0.1",
|
||||
"kind": "ais",
|
||||
"line": "!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F"
|
||||
},
|
||||
{
|
||||
"ts": 1700000000.456,
|
||||
"source": "gps_uart:/dev/ttyS1",
|
||||
"kind": "gps",
|
||||
"line": "$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,,,*7A"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 1.12 `GET /api/v1/history/positions`
|
||||
|
||||
История AIS dynamic для конкретного MMSI из SQLite.
|
||||
|
||||
**Query-параметры**
|
||||
|
||||
| Параметр | Тип | Обязательный | Описание |
|
||||
| --- | --- | --- | --- |
|
||||
| `mmsi` | int | да | MMSI. |
|
||||
| `from` | float (unix ts) | нет | Нижняя граница по `ts`. |
|
||||
| `to` | float (unix ts) | нет | Верхняя граница по `ts`. |
|
||||
| `limit` | int | нет | По умолчанию 1000, максимум 100000. |
|
||||
|
||||
**Коды**
|
||||
- `200` — массив точек (может быть пустой).
|
||||
- `400` — нет/невалидный `mmsi`.
|
||||
- `503` — `{"error": "history storage disabled"}`, если `storage.path` пуст.
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ts": 1700000000.123,
|
||||
"lat": 59.912345, "lon": 10.754321,
|
||||
"sog": 12.4, "cog": 210.3,
|
||||
"heading": 208, "nav_status": 0, "rot": -1.2,
|
||||
"msg_type": 18
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Отсортировано по `ts` **убыванию**.
|
||||
|
||||
---
|
||||
|
||||
### 1.13 `POST /api/v1/tx`
|
||||
|
||||
Инжекция NMEA-строк в TX outbox (UDP 127.0.0.1:6010 → SPI bridge).
|
||||
|
||||
**Body (JSON)** — одна из двух форм:
|
||||
|
||||
```json
|
||||
{ "payload": "!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F" }
|
||||
```
|
||||
|
||||
или
|
||||
|
||||
```json
|
||||
{ "nmea": [
|
||||
"!AIVDM,1,1,,A,...,0*2F",
|
||||
"$GPRMC,...,*7A"
|
||||
] }
|
||||
```
|
||||
|
||||
**Response (200)**
|
||||
|
||||
```json
|
||||
{
|
||||
"queued": 1,
|
||||
"rejected": [],
|
||||
"queue_depth": 3
|
||||
}
|
||||
```
|
||||
|
||||
**Response (400)** — невалидный JSON или нет `payload`/`nmea`.
|
||||
**Response (503)** — outbox переполнен:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": "outbox full",
|
||||
"accepted": ["!AIVDM,..."],
|
||||
"rejected": []
|
||||
}
|
||||
```
|
||||
|
||||
Каждая строка валидируется: структура NMEA + XOR-чексумма `*HH`. Невалидные попадают в `rejected: [{"line": "...", "reason": "invalid nmea or checksum"}]`, валидные — в outbox.
|
||||
|
||||
---
|
||||
|
||||
## 2. WebSocket `/ws`
|
||||
|
||||
**URL:** `ws://<host>:8081/ws`
|
||||
**Subprotocol:** нет.
|
||||
**Heartbeat:** 30 секунд (aiohttp ping, клиент должен отвечать на pong).
|
||||
**Направление:** publish-only от сервера; сообщения от клиента игнорируются.
|
||||
|
||||
### 2.1 Формат конверта
|
||||
|
||||
Все сообщения — JSON одной строкой:
|
||||
|
||||
```json
|
||||
{ "type": "<event_type>", "ts": 1700000000.123, "data": { ... } }
|
||||
```
|
||||
|
||||
### 2.2 Первое сообщение после коннекта — `state.snapshot`
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "state.snapshot",
|
||||
"ts": 1700000000.123,
|
||||
"data": {
|
||||
"ownship": { ... }, // см. GET /api/v1/ownship
|
||||
"vessels": [ ... ], // см. GET /api/v1/vessels
|
||||
"base_stations": [ ... ], // см. GET /api/v1/base_stations
|
||||
"atons": [ ... ], // см. GET /api/v1/atons
|
||||
"slots": { ... }, // см. GET /api/v1/slots
|
||||
"stats": { ... } // см. GET /api/v1/stats
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Клиент инициализирует своё состояние из снапшота, затем применяет живые updates.
|
||||
|
||||
### 2.3 Живые события
|
||||
|
||||
| `type` | Когда публикуется | `data` |
|
||||
| --- | --- | --- |
|
||||
| `ownship.update` | На каждый GPS-fix с новыми полями | Объект, идентичный `GET /api/v1/ownship`. |
|
||||
| `target.update` | На каждое AIS dynamic/static для MMSI, которое что-то изменило | Полный `MergedTarget` (см. 1.5). |
|
||||
| `base_station.update` | Type 4/11 | `{mmsi, ts, lat, lon, epfd}`. |
|
||||
| `aton.update` | Type 21 | `{mmsi, ts, lat, lon, type, name, virtual}`. |
|
||||
| `radio.update` (JSON) | Радиотелеметрия в JSON (port `radio_udp`) | `{channel, rssi, snr, slot, raw}`. |
|
||||
| `radio.update` (RSSI) | От AIS-catcher RSSI (10113, ~10 Гц) | `{source: "aiscatcher_rssi", power_a_db, power_b_db}`. Отличайте по `data.source`. |
|
||||
| `slots.update` | Bitmap (10111, ~1/мин) | `{channel, ts, utc_minute, slots_total, occupied_count, occupied_fraction, slot0_unix_ms, first_occupied_unix_ms, bitmap_hex}`. |
|
||||
| `slots.detail` | Detail (10112, ~1/мин) | `{channel, ts, utc_minute, slot0_unix_ms, entries:[{slot, level_db}]}`. |
|
||||
| `signal.update` | Пачка decode events (10114) | `{channel, events:[{unix_ms, slot, mmsi, level_db}]}`. |
|
||||
| `stats.update` | Раз в 5 секунд | Полный snapshot stats (см. 1.2). |
|
||||
|
||||
**Правила апдейта** клиента:
|
||||
- `target.update`: merge по `mmsi`; не терять предыдущие поля (новый пакет с `null` в static не перезаписывает старые имя/IMO — backend этим занимается через `COALESCE` в SQLite и `merge.apply`).
|
||||
- `signal.update`: обновлять sparkline сигнала; `target.update` уже содержит `signal` блок, но `signal.update` приходит чаще (с каждой успешной декодой пакета).
|
||||
- Порядок событий не гарантируется быть идеально хронологическим между разными типами, но в пределах одного `type` он монотонный.
|
||||
|
||||
### 2.4 Обратное давление
|
||||
|
||||
У каждого WS-клиента внутренняя очередь размера `queues.ws_client` (по умолчанию 512). Медленный клиент приводит к drop-oldest; счётчик — `stats.counters.ws_dropped`.
|
||||
|
||||
---
|
||||
|
||||
## 3. UDP интерфейсы (localhost only)
|
||||
|
||||
Все UDP-сокеты слушают/шлют только на `127.0.0.1`.
|
||||
|
||||
### 3.1 UDP JSON events — `127.0.0.1:7001` (publish, out)
|
||||
|
||||
- Контракт: **одно событие = одна UDP-датаграмма**.
|
||||
- Формат: JSON-сериализованный `Event` envelope (такой же, как в WS, но без `state.snapshot`).
|
||||
- Максимальный размер датаграммы: `publish.udp_events.max_datagram_bytes` (1400 по умолчанию). Превышение → drop + `stats.counters.udp_events_oversize`.
|
||||
- Подходит для локальных читателей (BLE-сервер, SPI bridge), которым не нужен full-duplex.
|
||||
|
||||
Пример одной датаграммы:
|
||||
|
||||
```json
|
||||
{"type":"target.update","ts":1700000000.123,"data":{"mmsi":506140446,...}}
|
||||
```
|
||||
|
||||
### 3.2 UDP raw NMEA fan-out — `127.0.0.1:6007` (publish, out)
|
||||
|
||||
- Контракт: **одна NMEA-строка = одна UDP-датаграмма**, завершённая `\r\n`.
|
||||
- Формат: как было получено по UDP/UART (после валидации структуры, но сохраняется исходная `*HH`).
|
||||
- Используется SPI bridge'ем для сквозного форвардинга в MCU.
|
||||
|
||||
Пример одной датаграммы:
|
||||
|
||||
```
|
||||
!AIVDM,1,1,,A,15MwkT001s8rFfwJh<dC<mNf0<0Q,0*2F\r\n
|
||||
```
|
||||
|
||||
### 3.3 UDP TX outbox — `127.0.0.1:6010` (publish, out)
|
||||
|
||||
- Контракт: **одна NMEA-строка = одна UDP-датаграмма**, `\r\n` в конце.
|
||||
- Источник: `POST /api/v1/tx`, внутренние эмиттеры.
|
||||
- Это **не ingest**: сервис только шлёт наружу. Для приёма NMEA использовать `127.0.0.1:5005` (AIS UDP) или UART.
|
||||
|
||||
### 3.4 Ingest AIS UDP — `<ingest.ais_udp>`
|
||||
|
||||
- Порт в конфиге (по умолчанию 4001, у вас на устройстве 5005 — совпадает с AIS-catcher).
|
||||
- Формат входа: **одна или несколько NMEA-строк в одной датаграмме**, разделитель `\r\n`/`\n`.
|
||||
- Сервис декодирует AIVDM/AIVDO и мультифрагментные.
|
||||
|
||||
### 3.5 Ingest Radio UDP (JSON) — `<ingest.radio_udp>`
|
||||
|
||||
- Порт по умолчанию 4010.
|
||||
- Формат: одна датаграмма = один JSON-объект с полями `{channel, rssi, snr, slot, ...}` — любые ключи сохраняются в `raw`.
|
||||
|
||||
### 3.6 Ingest AIS-catcher binary — `<ingest.aiscatcher_udp>`
|
||||
|
||||
Четыре независимых UDP-листенера (все BE, binary, подробные layout'ы — в `src/ais_hub/parser/aiscatcher.py`):
|
||||
|
||||
| Порт по умолчанию | Что | Длина |
|
||||
| --- | --- | --- |
|
||||
| 10111 | Slot bitmap occupancy (per minute) | 315 Б |
|
||||
| 10112 | Slot detail (per-slot levels) | 15 + 6×N |
|
||||
| 10113 | RSSI IQ pair (~10 Гц) | 8 Б |
|
||||
| 10114 | Decode events (mmsi/slot/level) | 3 + 18×N |
|
||||
|
||||
В REST/WS эти потоки отображаются как `/api/v1/radio`, `/api/v1/slots`, `radio.update`, `slots.update`, `slots.detail`, `signal.update`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Источники (`source` / `sources`)
|
||||
|
||||
Формат строки источника в `RawFrame.source`, `sources` array у target'а, в `GET /api/v1/nmea/tail`:
|
||||
|
||||
| Prefix | Пример | Смысл |
|
||||
| --- | --- | --- |
|
||||
| `ais_udp:<listener>:<sender_ip>` | `ais_udp:0.0.0.0:5005:127.0.0.1` | NMEA пришёл по UDP. Sender-порт намеренно не включается (ephemeral, скачет). |
|
||||
| `gps_uart:<device>` | `gps_uart:/dev/ttyS1` | NMEA с UART. |
|
||||
| `radio_udp:<listener>:<sender_ip>` | `radio_udp:0.0.0.0:4010:127.0.0.1` | JSON radio telemetry. |
|
||||
| `ownship` | `ownship` | Используется в WriteJob для `gps_fix` в БД. |
|
||||
|
||||
---
|
||||
|
||||
## 5. Коды ответов
|
||||
|
||||
| HTTP | Когда |
|
||||
| --- | --- |
|
||||
| `200` | OK |
|
||||
| `400` | Невалидные query-параметры / JSON-тело / MMSI. |
|
||||
| `404` | Unknown MMSI в `/api/v1/vessels/{mmsi}`. |
|
||||
| `500` | SQLite query провалился. |
|
||||
| `503` | Storage disabled (`history/positions`) или TX outbox full. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Пример клиента на TypeScript
|
||||
|
||||
```ts
|
||||
type Event = { type: string; ts: number; data: any };
|
||||
|
||||
async function loadInitial() {
|
||||
const [ownship, vessels, stats] = await Promise.all([
|
||||
fetch("/api/v1/ownship").then(r => r.json()),
|
||||
fetch("/api/v1/vessels?limit=500").then(r => r.json()),
|
||||
fetch("/api/v1/stats").then(r => r.json()),
|
||||
]);
|
||||
return { ownship, vessels, stats };
|
||||
}
|
||||
|
||||
function connectWs(onEvent: (ev: Event) => void) {
|
||||
const ws = new WebSocket(`ws://${location.host}/ws`);
|
||||
ws.onmessage = (m) => onEvent(JSON.parse(m.data));
|
||||
ws.onclose = () => setTimeout(() => connectWs(onEvent), 2000);
|
||||
return ws;
|
||||
}
|
||||
```
|
||||
|
||||
Стандартная схема для web UI:
|
||||
1. Загрузить `GET /api/v1/vessels` (для первичного рендера карты).
|
||||
2. Открыть `/ws` — получить `state.snapshot`, заменить состояние им.
|
||||
3. Применять `target.update`/`ownship.update`/и т.д. поверх.
|
||||
4. `signal.update` и `radio.update` (RSSI) — для графиков / индикатора сигнала.
|
||||
5. `slots.update` / `slots.detail` — для визуализации загрузки эфира.
|
||||
|
||||
---
|
||||
|
||||
## 7. Persistence при рестарте
|
||||
|
||||
При старте сервис автоматически загружает из SQLite:
|
||||
|
||||
- `vessel_static` → все `MergedTarget.<name|callsign|imo|ship_type|dim_*|eta|draught|destination>`, а также `last_static_ts`. Dynamic (lat/lon/sog/cog/heading/nav_status/rot) не восстанавливается — придёт с первым новым type-1/2/3/18.
|
||||
- `base_station` → `State.base_stations`.
|
||||
- `aton` → `State.atons`.
|
||||
|
||||
В логе при успешном warm start:
|
||||
|
||||
```
|
||||
INFO warm start: 42 vessels, 0 base_stations, 0 atons from sqlite
|
||||
```
|
||||
|
||||
И счётчики `stats.counters.state_warmup_*`.
|
||||
|
||||
Таким образом, сценарий «принял 5-е и 24-е, перезапустил, принял 18-е» теперь отдаст target с заполненными именем/IMO/размерениями сразу.
|
||||
@@ -0,0 +1,39 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ais_hub"
|
||||
version = "0.1.0"
|
||||
description = "Central AIS/GPS/radio telemetry aggregation service for embedded Linux."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
license = { text = "Proprietary" }
|
||||
authors = [{ name = "AisHub" }]
|
||||
dependencies = [
|
||||
"aiohttp>=3.9,<4",
|
||||
"pyais>=2.6,<3",
|
||||
"pyserial-asyncio>=0.6,<1",
|
||||
"aiosqlite>=0.19,<1",
|
||||
"PyYAML>=6,<7",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8",
|
||||
"pytest-asyncio>=0.23",
|
||||
"pytest-cov>=4",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
ais_hub = "ais_hub.__main__:main"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
ais_hub = ["py.typed"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
@@ -0,0 +1,5 @@
|
||||
"""ais_hub: central AIS/GPS/radio telemetry aggregation service."""
|
||||
|
||||
from .version import __version__
|
||||
|
||||
__all__ = ["__version__"]
|
||||
@@ -0,0 +1,48 @@
|
||||
"""CLI entry point for ais_hub."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .app import Application
|
||||
from .config import Config, load_config
|
||||
from .logging_setup import setup_logging
|
||||
from .version import __version__
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
p = argparse.ArgumentParser(prog="ais_hub", description="AIS/GPS/radio aggregation service")
|
||||
p.add_argument("--config", "-c", type=Path, required=False, help="Path to YAML config file")
|
||||
p.add_argument("--version", action="version", version=f"ais_hub {__version__}")
|
||||
return p.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parse_args(argv if argv is not None else sys.argv[1:])
|
||||
try:
|
||||
cfg: Config = load_config(args.config)
|
||||
except Exception as exc:
|
||||
print(f"config error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
setup_logging(cfg.logging)
|
||||
log.info("starting ais_hub %s", __version__)
|
||||
try:
|
||||
asyncio.run(Application(cfg).run())
|
||||
except KeyboardInterrupt:
|
||||
log.info("interrupted")
|
||||
return 0
|
||||
except Exception:
|
||||
log.exception("fatal error")
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,376 @@
|
||||
"""Application orchestrator: wires subsystems, runs supervisors, handles shutdown."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import signal
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Awaitable, Callable
|
||||
|
||||
import aiosqlite
|
||||
from aiohttp import web
|
||||
|
||||
from .config import Config
|
||||
from .core.bus import EventBus
|
||||
from .core.models import AisReport, Event, GpsFix, RadioReport, RawFrame, TxMessage
|
||||
from .core.state import State
|
||||
from .core.stats import Stats
|
||||
from .ingest.ais_udp import AisUdpIngest
|
||||
from .ingest.aiscatcher_udp import AisCatcherUdpIngest
|
||||
from .ingest.base import AddressInUseError, RawTap
|
||||
from .ingest.gps_uart import GpsUartIngest
|
||||
from .ingest.radio_udp import RadioUdpIngest
|
||||
from .parser.ais import AisAssembler
|
||||
from .parser.gps import GpsParser
|
||||
from .parser.nmea_utils import parse_sentence
|
||||
from .parser.radio import RadioParser
|
||||
from .publish import rest as rest_module
|
||||
from .publish import ws as ws_module
|
||||
from .publish.queues import put_drop_oldest
|
||||
from .publish.udp_events import UdpEventsPublisher
|
||||
from .publish.udp_nmea import UdpNmeaPublisher
|
||||
from .publish.udp_tx_outbox import UdpTxOutboxPublisher
|
||||
from .storage import db as storage_db
|
||||
from .storage import queries as storage_queries
|
||||
from .storage import retention as storage_retention
|
||||
from .storage.writer import StorageEventSink, Writer, _RawNmeaRow
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared app context
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Context:
|
||||
cfg: Config
|
||||
stats: Stats
|
||||
bus: EventBus
|
||||
state: State
|
||||
raw_tap: RawTap
|
||||
tx_outbox: "asyncio.Queue[TxMessage]"
|
||||
db: aiosqlite.Connection | None = None
|
||||
storage_sink: StorageEventSink | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Supervisor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _supervise(
|
||||
name: str,
|
||||
factory: Callable[[], Awaitable[None]],
|
||||
stats: Stats,
|
||||
shutdown: asyncio.Event,
|
||||
*,
|
||||
initial_backoff: float = 1.0,
|
||||
max_backoff: float = 30.0,
|
||||
) -> None:
|
||||
"""Run ``factory`` and restart on errors with exponential backoff.
|
||||
|
||||
Exits cleanly when ``shutdown`` is set.
|
||||
"""
|
||||
backoff = initial_backoff
|
||||
while not shutdown.is_set():
|
||||
try:
|
||||
await factory()
|
||||
# Normal return also counts as "done"; for long-running tasks this
|
||||
# should not happen, but we don't try to restart healthy exits.
|
||||
return
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except AddressInUseError as exc:
|
||||
# Expected runtime condition: port collision. Log compactly and
|
||||
# keep retrying; no traceback spam.
|
||||
stats.incr(f"supervisor_restarts_{name}")
|
||||
log.warning("subsystem %s: %s (retry in %.1fs)", name, exc, backoff)
|
||||
try:
|
||||
await asyncio.wait_for(shutdown.wait(), timeout=backoff)
|
||||
return
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
backoff = min(backoff * 2, max_backoff)
|
||||
except Exception:
|
||||
stats.incr(f"supervisor_restarts_{name}")
|
||||
log.exception("subsystem %s crashed; restarting in %.1fs", name, backoff)
|
||||
try:
|
||||
await asyncio.wait_for(shutdown.wait(), timeout=backoff)
|
||||
return
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
backoff = min(backoff * 2, max_backoff)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parser task
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _parser_task(
|
||||
parser_in: "asyncio.Queue[RawFrame]",
|
||||
state: State,
|
||||
stats: Stats,
|
||||
cfg: Config,
|
||||
) -> None:
|
||||
"""Consume RawFrames, dispatch to AIS/GPS/radio parsers, update state."""
|
||||
ais = AisAssembler(
|
||||
stats=stats,
|
||||
ttl_sec=cfg.parser.ais.multipart_ttl_sec,
|
||||
ignore_checksum=cfg.parser.ais.ignore_checksum,
|
||||
allow_checksumless=cfg.parser.ais.allow_checksumless,
|
||||
allow_multipart_without_seq_id=cfg.parser.ais.allow_multipart_without_seq_id,
|
||||
)
|
||||
gps = GpsParser(stats=stats)
|
||||
radio = RadioParser(stats=stats)
|
||||
last_gc = time.monotonic()
|
||||
|
||||
while True:
|
||||
frame = await parser_in.get()
|
||||
try:
|
||||
if frame.kind == "ais":
|
||||
reports = ais.feed(frame.source, frame.line, ts=frame.ts)
|
||||
for rep in reports:
|
||||
state.apply_ais(rep)
|
||||
elif frame.kind == "gps":
|
||||
fix = gps.feed(frame.source, frame.line, ts=frame.ts)
|
||||
if fix is not None:
|
||||
state.apply_gps(fix)
|
||||
elif frame.kind == "radio":
|
||||
report = radio.feed(frame.source, frame.line, ts=frame.ts)
|
||||
if report is not None:
|
||||
state.apply_radio(report)
|
||||
except Exception:
|
||||
stats.incr("parser_task_errors")
|
||||
log.exception("parser task error on kind=%s source=%s", frame.kind, frame.source)
|
||||
|
||||
now = time.monotonic()
|
||||
if now - last_gc > 5.0:
|
||||
ais.gc(frame.ts)
|
||||
last_gc = now
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Stats emitter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _stats_emitter(bus: EventBus, stats: Stats, period: float = 5.0) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(period)
|
||||
snap = stats.snapshot()
|
||||
bus.publish(Event(type="stats.update", ts=time.time(), data=snap))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Raw-NMEA -> storage bridge (optional, when store_raw_nmea=True)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _raw_to_storage_task(
|
||||
raw_queue: "asyncio.Queue[RawFrame]",
|
||||
storage_in: "asyncio.Queue[Any]",
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
while True:
|
||||
frame = await raw_queue.get()
|
||||
put_drop_oldest(storage_in, _RawNmeaRow(frame=frame), stats, "storage_in_dropped")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Application
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Application:
|
||||
def __init__(self, cfg: Config) -> None:
|
||||
self._cfg = cfg
|
||||
self._stats = Stats()
|
||||
self._bus = EventBus(stats=self._stats, default_maxsize=1024)
|
||||
self._state = State(bus=self._bus, stats=self._stats)
|
||||
self._raw_tap = RawTap(
|
||||
stats=self._stats,
|
||||
ring_size=cfg.publish.nmea_tail.in_memory_ring,
|
||||
)
|
||||
self._shutdown = asyncio.Event()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
async def run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
# Install signal handlers where supported (POSIX).
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
try:
|
||||
loop.add_signal_handler(sig, self._request_shutdown)
|
||||
except (NotImplementedError, RuntimeError):
|
||||
# Windows / test environments.
|
||||
pass
|
||||
|
||||
parser_in: asyncio.Queue[RawFrame] = asyncio.Queue(maxsize=self._cfg.queues.parser_in)
|
||||
udp_nmea_queue: asyncio.Queue[RawFrame] = asyncio.Queue(maxsize=max(512, self._cfg.queues.parser_in // 2))
|
||||
storage_in: asyncio.Queue[Any] = asyncio.Queue(maxsize=self._cfg.queues.storage_in)
|
||||
tx_outbox: asyncio.Queue[TxMessage] = asyncio.Queue(maxsize=self._cfg.queues.tx_outbox)
|
||||
|
||||
# Subscribe the UDP NMEA fan-out to the raw tap.
|
||||
self._raw_tap.subscribe(udp_nmea_queue, "udp_nmea_dropped")
|
||||
|
||||
# Optional raw-NMEA -> storage feed.
|
||||
raw_storage_queue: asyncio.Queue[RawFrame] | None = None
|
||||
if self._cfg.storage.store_raw_nmea:
|
||||
raw_storage_queue = asyncio.Queue(maxsize=self._cfg.queues.storage_in)
|
||||
self._raw_tap.subscribe(raw_storage_queue, "storage_in_dropped")
|
||||
|
||||
# Open SQLite (optional).
|
||||
db_conn: aiosqlite.Connection | None = None
|
||||
if self._cfg.storage.path:
|
||||
try:
|
||||
db_conn = await storage_db.connect(self._cfg.storage.path)
|
||||
log.info("sqlite opened: %s", self._cfg.storage.path)
|
||||
except Exception:
|
||||
log.exception("sqlite open failed; continuing without history")
|
||||
db_conn = None
|
||||
|
||||
# Warm start: preload persisted static/base/aton rows into state so
|
||||
# a restart retains known vessel names/dims/etc. even before the
|
||||
# first type-5/24 re-arrives over the air.
|
||||
if db_conn is not None:
|
||||
try:
|
||||
vessels = await storage_queries.load_vessel_static(db_conn)
|
||||
base_stations = await storage_queries.load_base_stations(db_conn)
|
||||
atons = await storage_queries.load_atons(db_conn)
|
||||
counts = self._state.warmup(
|
||||
vessels=vessels,
|
||||
base_stations=base_stations,
|
||||
atons=atons,
|
||||
)
|
||||
log.info(
|
||||
"warm start: %d vessels, %d base_stations, %d atons from sqlite",
|
||||
counts["vessels"], counts["base_stations"], counts["atons"],
|
||||
)
|
||||
self._stats.incr("state_warmup_vessels", counts["vessels"])
|
||||
self._stats.incr("state_warmup_base_stations", counts["base_stations"])
|
||||
self._stats.incr("state_warmup_atons", counts["atons"])
|
||||
except Exception:
|
||||
log.exception("warm start failed; continuing with empty state")
|
||||
|
||||
storage_sink: StorageEventSink | None = None
|
||||
if db_conn is not None:
|
||||
storage_sink = StorageEventSink(
|
||||
bus=self._bus,
|
||||
state=self._state,
|
||||
stats=self._stats,
|
||||
out_queue=storage_in,
|
||||
store_raw_nmea=self._cfg.storage.store_raw_nmea,
|
||||
)
|
||||
|
||||
ctx = Context(
|
||||
cfg=self._cfg,
|
||||
stats=self._stats,
|
||||
bus=self._bus,
|
||||
state=self._state,
|
||||
raw_tap=self._raw_tap,
|
||||
tx_outbox=tx_outbox,
|
||||
db=db_conn,
|
||||
storage_sink=storage_sink,
|
||||
)
|
||||
|
||||
# Build aiohttp app for REST + WS.
|
||||
aio_app = web.Application()
|
||||
aio_app["ctx"] = ctx
|
||||
rest_module.register_routes(aio_app)
|
||||
ws_module.register_routes(aio_app)
|
||||
|
||||
runner = web.AppRunner(aio_app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, host=self._cfg.publish.http.host, port=self._cfg.publish.http.port)
|
||||
await site.start()
|
||||
log.info("http listening on %s:%d", self._cfg.publish.http.host, self._cfg.publish.http.port)
|
||||
|
||||
tasks: list[asyncio.Task[Any]] = []
|
||||
|
||||
def spawn(name: str, factory: Callable[[], Awaitable[None]]) -> None:
|
||||
t = asyncio.create_task(
|
||||
_supervise(name, factory, self._stats, self._shutdown),
|
||||
name=name,
|
||||
)
|
||||
tasks.append(t)
|
||||
|
||||
# Parser
|
||||
spawn("parser", lambda: _parser_task(parser_in, self._state, self._stats, self._cfg))
|
||||
|
||||
# Stats emitter
|
||||
spawn("stats_emitter", lambda: _stats_emitter(self._bus, self._stats, period=5.0))
|
||||
|
||||
# Ingest
|
||||
ais_udp = AisUdpIngest(self._cfg.ingest.ais_udp,
|
||||
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
|
||||
radio_udp = RadioUdpIngest(self._cfg.ingest.radio_udp,
|
||||
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
|
||||
gps_uart = GpsUartIngest(self._cfg.ingest.gps_uart,
|
||||
parser_in=parser_in, raw_tap=self._raw_tap, stats=self._stats)
|
||||
spawn("ingest_ais_udp", ais_udp.run)
|
||||
spawn("ingest_radio_udp", radio_udp.run)
|
||||
spawn("ingest_gps_uart", gps_uart.run)
|
||||
|
||||
if self._cfg.ingest.aiscatcher_udp.enabled:
|
||||
aiscatcher = AisCatcherUdpIngest(
|
||||
self._cfg.ingest.aiscatcher_udp,
|
||||
state=self._state,
|
||||
stats=self._stats,
|
||||
)
|
||||
spawn("ingest_aiscatcher_udp", aiscatcher.run)
|
||||
|
||||
# Publishers
|
||||
udp_events = UdpEventsPublisher(self._cfg.publish.udp_events, self._bus, self._stats)
|
||||
udp_nmea = UdpNmeaPublisher(self._cfg.publish.udp_nmea, self._stats, udp_nmea_queue)
|
||||
udp_tx = UdpTxOutboxPublisher(self._cfg.publish.udp_tx_outbox, self._stats, tx_outbox)
|
||||
spawn("udp_events", udp_events.run)
|
||||
spawn("udp_nmea", udp_nmea.run)
|
||||
spawn("udp_tx_outbox", udp_tx.run)
|
||||
|
||||
# Storage
|
||||
if db_conn is not None and storage_sink is not None:
|
||||
writer = Writer(db_conn, self._cfg.storage, self._stats, storage_in)
|
||||
spawn("storage_writer", writer.run)
|
||||
spawn("storage_sink", storage_sink.run)
|
||||
spawn("storage_retention",
|
||||
lambda: storage_retention.run_retention(db_conn, self._cfg.storage, self._stats))
|
||||
if raw_storage_queue is not None:
|
||||
spawn("raw_to_storage",
|
||||
lambda: _raw_to_storage_task(raw_storage_queue, storage_in, self._stats))
|
||||
|
||||
# Wait for shutdown.
|
||||
try:
|
||||
await self._shutdown.wait()
|
||||
finally:
|
||||
log.info("shutting down ais_hub")
|
||||
for t in tasks:
|
||||
t.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
try:
|
||||
await site.stop()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await runner.cleanup()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
await storage_db.close(db_conn)
|
||||
log.info("ais_hub stopped")
|
||||
|
||||
def _request_shutdown(self) -> None:
|
||||
if not self._shutdown.is_set():
|
||||
log.info("shutdown signal received")
|
||||
self._shutdown.set()
|
||||
|
||||
|
||||
__all__ = ["Application"]
|
||||
@@ -0,0 +1,272 @@
|
||||
"""Configuration loader: YAML file + environment variable overrides.
|
||||
|
||||
Env-var convention:
|
||||
AIS_HUB_<PATH> where path components are joined by "__".
|
||||
Example:
|
||||
AIS_HUB_INGEST__AIS_UDP__PORT=4011
|
||||
-> cfg.ingest.ais_udp.port = 4011
|
||||
AIS_HUB_LOGGING__LEVEL=DEBUG
|
||||
-> cfg.logging.level = "DEBUG"
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field, fields, is_dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
ENV_PREFIX = "AIS_HUB_"
|
||||
ENV_SEP = "__"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dataclasses mirroring config.example.yaml structure
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class AisUdpCfg:
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 4001
|
||||
|
||||
|
||||
@dataclass
|
||||
class GpsUartCfg:
|
||||
device: str = "/dev/ttyS1"
|
||||
baud: int = 38400
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class RadioUdpCfg:
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 4010
|
||||
|
||||
|
||||
@dataclass
|
||||
class AisCatcherUdpCfg:
|
||||
"""Four binary telemetry streams from AIS-catcher Mini.
|
||||
|
||||
Ports mirror ``ais-mini.conf``:
|
||||
* ``slot_port`` — 315-byte slot bitmap per minute
|
||||
* ``slot_detail_port`` — per-slot levels per minute
|
||||
* ``rssi_port`` — 8-byte RSSI pair at ~10 Hz
|
||||
* ``event_port`` — decode events (ts, slot, mmsi, level)
|
||||
"""
|
||||
|
||||
enabled: bool = False
|
||||
host: str = "127.0.0.1"
|
||||
slot_port: int = 10111
|
||||
slot_detail_port: int = 10112
|
||||
rssi_port: int = 10113
|
||||
event_port: int = 10114
|
||||
|
||||
|
||||
@dataclass
|
||||
class IngestCfg:
|
||||
ais_udp: AisUdpCfg = field(default_factory=AisUdpCfg)
|
||||
gps_uart: GpsUartCfg = field(default_factory=GpsUartCfg)
|
||||
radio_udp: RadioUdpCfg = field(default_factory=RadioUdpCfg)
|
||||
aiscatcher_udp: AisCatcherUdpCfg = field(default_factory=AisCatcherUdpCfg)
|
||||
|
||||
|
||||
@dataclass
|
||||
class HttpCfg:
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 8080
|
||||
|
||||
|
||||
@dataclass
|
||||
class UdpEventsCfg:
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 7001
|
||||
max_datagram_bytes: int = 1400
|
||||
|
||||
|
||||
@dataclass
|
||||
class UdpNmeaCfg:
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 6007
|
||||
|
||||
|
||||
@dataclass
|
||||
class UdpTxOutboxCfg:
|
||||
host: str = "127.0.0.1"
|
||||
port: int = 6010
|
||||
|
||||
|
||||
@dataclass
|
||||
class NmeaTailCfg:
|
||||
in_memory_ring: int = 500
|
||||
|
||||
|
||||
@dataclass
|
||||
class PublishCfg:
|
||||
http: HttpCfg = field(default_factory=HttpCfg)
|
||||
udp_events: UdpEventsCfg = field(default_factory=UdpEventsCfg)
|
||||
udp_nmea: UdpNmeaCfg = field(default_factory=UdpNmeaCfg)
|
||||
udp_tx_outbox: UdpTxOutboxCfg = field(default_factory=UdpTxOutboxCfg)
|
||||
nmea_tail: NmeaTailCfg = field(default_factory=NmeaTailCfg)
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetentionCfg:
|
||||
ais_dynamic_days: int = 7
|
||||
gps_fix_days: int = 7
|
||||
raw_nmea_days: int = 2
|
||||
radio_telemetry_days: int = 7
|
||||
|
||||
|
||||
@dataclass
|
||||
class WriterCfg:
|
||||
batch_size: int = 200
|
||||
flush_interval_ms: int = 500
|
||||
|
||||
|
||||
@dataclass
|
||||
class StorageCfg:
|
||||
path: str = "/var/lib/ais_hub/ais_hub.db"
|
||||
store_raw_nmea: bool = False
|
||||
retention: RetentionCfg = field(default_factory=RetentionCfg)
|
||||
writer: WriterCfg = field(default_factory=WriterCfg)
|
||||
retention_interval_sec: int = 3600
|
||||
|
||||
|
||||
@dataclass
|
||||
class QueuesCfg:
|
||||
parser_in: int = 2048
|
||||
ws_client: int = 512
|
||||
udp_events: int = 1024
|
||||
storage_in: int = 4096
|
||||
tx_outbox: int = 512
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoggingCfg:
|
||||
level: str = "INFO"
|
||||
file: str | None = "/var/log/ais_hub/ais_hub.log"
|
||||
max_bytes: int = 10 * 1024 * 1024
|
||||
backup_count: int = 5
|
||||
json: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class AisParserCfg:
|
||||
"""AIS parser tolerance knobs.
|
||||
|
||||
Defaults are strict (drop checksumless/invalid and multipart without seq_id).
|
||||
Some real-world NMEA forwarders omit these fields; enabling tolerances may
|
||||
increase decoded target count at the cost of potential false merges when
|
||||
streams are heavily interleaved.
|
||||
"""
|
||||
|
||||
# Ignore checksum completely (even if present and invalid).
|
||||
# If enabled, this overrides allow_checksumless.
|
||||
ignore_checksum: bool = False
|
||||
allow_checksumless: bool = False
|
||||
allow_multipart_without_seq_id: bool = False
|
||||
multipart_ttl_sec: float = 60.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParserCfg:
|
||||
ais: AisParserCfg = field(default_factory=AisParserCfg)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
ingest: IngestCfg = field(default_factory=IngestCfg)
|
||||
publish: PublishCfg = field(default_factory=PublishCfg)
|
||||
storage: StorageCfg = field(default_factory=StorageCfg)
|
||||
queues: QueuesCfg = field(default_factory=QueuesCfg)
|
||||
parser: ParserCfg = field(default_factory=ParserCfg)
|
||||
logging: LoggingCfg = field(default_factory=LoggingCfg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Loading helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _coerce(value: str, hint: Any) -> Any:
|
||||
"""Coerce a string (from env var) into a field type.
|
||||
|
||||
``hint`` may be a concrete type (``int``, ``str`` …), a string
|
||||
annotation (e.g. ``"int"``, ``"int | None"``) when
|
||||
``from __future__ import annotations`` is active, or ``None`` to
|
||||
trigger best-effort auto-detection based on ``value``.
|
||||
"""
|
||||
if isinstance(hint, str):
|
||||
h = hint.lower().replace(" ", "")
|
||||
if "bool" in h:
|
||||
hint = bool
|
||||
elif "int" in h:
|
||||
hint = int
|
||||
elif "float" in h:
|
||||
hint = float
|
||||
else:
|
||||
hint = str
|
||||
|
||||
if hint is bool:
|
||||
v = value.strip().lower()
|
||||
if v in ("1", "true", "yes", "on"):
|
||||
return True
|
||||
if v in ("0", "false", "no", "off"):
|
||||
return False
|
||||
raise ValueError(f"invalid bool value: {value!r}")
|
||||
if hint is int:
|
||||
return int(value)
|
||||
if hint is float:
|
||||
return float(value)
|
||||
# Fallback: keep as string.
|
||||
return value
|
||||
|
||||
|
||||
def _apply_dict(dc: Any, data: dict[str, Any]) -> None:
|
||||
"""Recursively apply a YAML-derived dict onto a dataclass instance."""
|
||||
for f in fields(dc):
|
||||
if f.name not in data:
|
||||
continue
|
||||
incoming = data[f.name]
|
||||
current = getattr(dc, f.name)
|
||||
if is_dataclass(current) and isinstance(incoming, dict):
|
||||
_apply_dict(current, incoming)
|
||||
else:
|
||||
setattr(dc, f.name, incoming)
|
||||
|
||||
|
||||
def _apply_env(dc: Any, path: tuple[str, ...] = ()) -> None:
|
||||
"""Walk dataclass tree and apply AIS_HUB_* env overrides."""
|
||||
for f in fields(dc):
|
||||
current = getattr(dc, f.name)
|
||||
sub_path = path + (f.name,)
|
||||
if is_dataclass(current):
|
||||
_apply_env(current, sub_path)
|
||||
continue
|
||||
env_name = ENV_PREFIX + ENV_SEP.join(sub_path).upper()
|
||||
if env_name in os.environ:
|
||||
raw = os.environ[env_name]
|
||||
try:
|
||||
setattr(dc, f.name, _coerce(raw, f.type))
|
||||
except Exception as exc:
|
||||
raise ValueError(f"env {env_name}: {exc}") from exc
|
||||
|
||||
|
||||
def load_config(path: Path | str | None) -> Config:
|
||||
"""Load configuration from a YAML file (optional) and apply env overrides."""
|
||||
cfg = Config()
|
||||
if path is not None:
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
raise FileNotFoundError(f"config file not found: {p}")
|
||||
with p.open("r", encoding="utf-8") as fh:
|
||||
raw = yaml.safe_load(fh) or {}
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"config file must be a YAML mapping, got {type(raw).__name__}")
|
||||
_apply_dict(cfg, raw)
|
||||
_apply_env(cfg)
|
||||
return cfg
|
||||
@@ -0,0 +1 @@
|
||||
"""Core domain: in-memory state, eager MMSI merge, stats, event bus."""
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,66 @@
|
||||
"""Internal pub/sub event bus.
|
||||
|
||||
Subscribers are given a bounded ``asyncio.Queue`` of ``Event`` envelopes.
|
||||
If a subscriber queue overflows, the *oldest* event is dropped (drop-oldest)
|
||||
and the ``bus_dropped`` counter is incremented in stats.
|
||||
|
||||
Subscribers are expected to drain their queues in their own task. The bus
|
||||
never blocks on a slow subscriber.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from .models import Event
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EventBus:
|
||||
"""Fan-out pub/sub hub for ``Event`` envelopes."""
|
||||
|
||||
def __init__(self, stats: "Stats | None" = None, default_maxsize: int = 1024) -> None:
|
||||
self._subs: list[asyncio.Queue[Event]] = []
|
||||
self._stats = stats
|
||||
self._default_maxsize = default_maxsize
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def subscribe(self, maxsize: int | None = None) -> asyncio.Queue[Event]:
|
||||
"""Register a new subscriber queue. Caller owns the queue."""
|
||||
q: asyncio.Queue[Event] = asyncio.Queue(maxsize=maxsize or self._default_maxsize)
|
||||
self._subs.append(q)
|
||||
return q
|
||||
|
||||
def unsubscribe(self, q: asyncio.Queue[Event]) -> None:
|
||||
try:
|
||||
self._subs.remove(q)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def publish(self, event: Event) -> None:
|
||||
"""Non-blocking publish; drops oldest on a subscriber if its queue is full."""
|
||||
for q in list(self._subs):
|
||||
while True:
|
||||
try:
|
||||
q.put_nowait(event)
|
||||
break
|
||||
except asyncio.QueueFull:
|
||||
try:
|
||||
q.get_nowait()
|
||||
except asyncio.QueueEmpty:
|
||||
# Race; try again.
|
||||
continue
|
||||
if self._stats is not None:
|
||||
self._stats.incr("bus_dropped")
|
||||
# Retry put after dropping one item.
|
||||
continue
|
||||
|
||||
@property
|
||||
def subscribers(self) -> int:
|
||||
return len(self._subs)
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Eager merge of AIS reports into ``MergedTarget`` objects."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .models import AisReport, MergedTarget
|
||||
|
||||
|
||||
DYNAMIC_FIELDS = ("lat", "lon", "sog", "cog", "heading", "nav_status", "rot")
|
||||
STATIC_FIELDS = (
|
||||
"name", "callsign", "imo", "ship_type",
|
||||
"dim_a", "dim_b", "dim_c", "dim_d",
|
||||
"eta", "draught", "destination",
|
||||
)
|
||||
|
||||
|
||||
def apply(target: MergedTarget, report: AisReport) -> bool:
|
||||
"""Merge ``report`` into ``target`` in place. Return True if changed.
|
||||
|
||||
- Dynamic fields are overwritten only when ``report.ts >= last_dynamic_ts``.
|
||||
- Static fields are overwritten only when ``report.ts >= last_static_ts``.
|
||||
- Msg 24A (name) and 24B (callsign/dims) both feed into the same
|
||||
static slot; the timestamp guards preserve the latest value per
|
||||
field but do not clobber a still-relevant name.
|
||||
"""
|
||||
changed = False
|
||||
|
||||
if report.ts > target.last_seen:
|
||||
target.last_seen = report.ts
|
||||
changed = True
|
||||
|
||||
target.sources.add(report.source)
|
||||
target.msg_types.add(report.msg_type)
|
||||
|
||||
d = report.data
|
||||
|
||||
if report.kind == "dynamic" and report.ts >= target.last_dynamic_ts:
|
||||
for f in DYNAMIC_FIELDS:
|
||||
if f in d:
|
||||
setattr(target, f, d[f])
|
||||
changed = True
|
||||
target.last_dynamic_ts = report.ts
|
||||
elif report.kind == "static" and report.ts >= target.last_static_ts:
|
||||
for f in STATIC_FIELDS:
|
||||
if f in d and d[f] not in (None, ""):
|
||||
setattr(target, f, d[f])
|
||||
changed = True
|
||||
target.last_static_ts = report.ts
|
||||
|
||||
return changed
|
||||
|
||||
|
||||
__all__ = ["apply"]
|
||||
@@ -0,0 +1,279 @@
|
||||
"""Domain dataclasses used across ingest/parser/core/storage/publish."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Raw NMEA frame (ingest -> parser / raw tap)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RawFrame:
|
||||
"""A single raw NMEA line as received from an ingest channel."""
|
||||
|
||||
ts: float # unix epoch seconds (monotonic-ish)
|
||||
source: str # e.g. "ais_udp:192.168.1.2:4001", "gps_uart:/dev/ttyS1"
|
||||
kind: Literal["ais", "gps", "radio"] # rough classification by ingest
|
||||
line: str # NMEA sentence with no trailing CRLF
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# AIS reports (parser -> core)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
AisKind = Literal[
|
||||
"dynamic", # msg 1/2/3/18/19/27 - position report
|
||||
"static", # msg 5/24 - static/voyage
|
||||
"base_station", # msg 4 - base station report
|
||||
"aton", # msg 21 - AtoN report
|
||||
"other",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class AisReport:
|
||||
"""Normalized AIS message payload."""
|
||||
|
||||
ts: float
|
||||
source: str
|
||||
kind: AisKind
|
||||
mmsi: int
|
||||
msg_type: int
|
||||
channel: str | None = None
|
||||
raw: str | None = None # original sentence(s) joined with \n
|
||||
data: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GPS fix (parser -> core)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class GpsFix:
|
||||
ts: float
|
||||
source: str
|
||||
lat: float | None = None
|
||||
lon: float | None = None
|
||||
sog: float | None = None # speed over ground, knots
|
||||
cog: float | None = None # course over ground, degrees true
|
||||
alt: float | None = None # meters
|
||||
fix_quality: int | None = None # GGA field 6
|
||||
sats: int | None = None
|
||||
hdop: float | None = None
|
||||
# raw sentence kind(s) that contributed to this fix (e.g. "RMC", "GGA")
|
||||
sentences: tuple[str, ...] = ()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Radio telemetry (parser -> core)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RadioReport:
|
||||
ts: float
|
||||
source: str
|
||||
channel: str | None = None
|
||||
rssi: float | None = None
|
||||
snr: float | None = None
|
||||
slot: int | None = None
|
||||
raw: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Merged target state (core.state)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class MergedTarget:
|
||||
"""Single authoritative snapshot per MMSI, kept up to date by core.merge."""
|
||||
|
||||
mmsi: int
|
||||
# identity / static
|
||||
name: str | None = None
|
||||
callsign: str | None = None
|
||||
imo: int | None = None
|
||||
ship_type: int | None = None
|
||||
dim_a: int | None = None
|
||||
dim_b: int | None = None
|
||||
dim_c: int | None = None
|
||||
dim_d: int | None = None
|
||||
# voyage
|
||||
eta: str | None = None
|
||||
draught: float | None = None
|
||||
destination: str | None = None
|
||||
# dynamic
|
||||
lat: float | None = None
|
||||
lon: float | None = None
|
||||
sog: float | None = None
|
||||
cog: float | None = None
|
||||
heading: float | None = None
|
||||
nav_status: int | None = None
|
||||
rot: float | None = None
|
||||
# signal (from AIS-catcher decode events, channel 10114)
|
||||
last_signal_db: float | None = None
|
||||
last_signal_ts: float = 0.0
|
||||
last_signal_slot: int | None = None
|
||||
last_signal_channel: str | None = None
|
||||
# bookkeeping
|
||||
last_static_ts: float = 0.0
|
||||
last_dynamic_ts: float = 0.0
|
||||
last_seen: float = 0.0
|
||||
sources: set[str] = field(default_factory=set)
|
||||
msg_types: set[int] = field(default_factory=set)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"name": self.name,
|
||||
"callsign": self.callsign,
|
||||
"imo": self.imo,
|
||||
"ship_type": self.ship_type,
|
||||
"dims": {
|
||||
"a": self.dim_a,
|
||||
"b": self.dim_b,
|
||||
"c": self.dim_c,
|
||||
"d": self.dim_d,
|
||||
},
|
||||
"voyage": {
|
||||
"eta": self.eta,
|
||||
"draught": self.draught,
|
||||
"destination": self.destination,
|
||||
},
|
||||
"dynamic": {
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"sog": self.sog,
|
||||
"cog": self.cog,
|
||||
"heading": self.heading,
|
||||
"nav_status": self.nav_status,
|
||||
"rot": self.rot,
|
||||
},
|
||||
"signal": {
|
||||
"last_db": self.last_signal_db,
|
||||
"last_ts": self.last_signal_ts or None,
|
||||
"last_slot": self.last_signal_slot,
|
||||
"last_channel": self.last_signal_channel,
|
||||
},
|
||||
"last_static_ts": self.last_static_ts,
|
||||
"last_dynamic_ts": self.last_dynamic_ts,
|
||||
"last_seen": self.last_seen,
|
||||
"sources": sorted(self.sources),
|
||||
"msg_types": sorted(self.msg_types),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class BaseStation:
|
||||
mmsi: int
|
||||
ts: float = 0.0
|
||||
lat: float | None = None
|
||||
lon: float | None = None
|
||||
epfd: int | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"ts": self.ts,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"epfd": self.epfd,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class AtoN:
|
||||
mmsi: int
|
||||
ts: float = 0.0
|
||||
lat: float | None = None
|
||||
lon: float | None = None
|
||||
aton_type: int | None = None
|
||||
name: str | None = None
|
||||
virtual: bool | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"mmsi": self.mmsi,
|
||||
"ts": self.ts,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"type": self.aton_type,
|
||||
"name": self.name,
|
||||
"virtual": self.virtual,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Ownship:
|
||||
"""Latest GPS fix (treated as own-ship position)."""
|
||||
|
||||
ts: float = 0.0
|
||||
lat: float | None = None
|
||||
lon: float | None = None
|
||||
sog: float | None = None
|
||||
cog: float | None = None
|
||||
alt: float | None = None
|
||||
fix_quality: int | None = None
|
||||
sats: int | None = None
|
||||
hdop: float | None = None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"ts": self.ts,
|
||||
"lat": self.lat,
|
||||
"lon": self.lon,
|
||||
"sog": self.sog,
|
||||
"cog": self.cog,
|
||||
"alt": self.alt,
|
||||
"fix_quality": self.fix_quality,
|
||||
"sats": self.sats,
|
||||
"hdop": self.hdop,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Event envelope (core.bus -> publishers / storage)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
EventType = Literal[
|
||||
"ownship.update",
|
||||
"target.update",
|
||||
"base_station.update",
|
||||
"aton.update",
|
||||
"radio.update",
|
||||
"stats.update",
|
||||
]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class Event:
|
||||
"""Stable event envelope published on the internal pub/sub bus."""
|
||||
|
||||
type: str
|
||||
ts: float
|
||||
data: dict[str, Any]
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {"type": self.type, "ts": self.ts, "data": self.data}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TX outbox message (publish/udp_tx_outbox)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class TxMessage:
|
||||
"""One NMEA sentence queued for transmission via the SPI bridge outbox."""
|
||||
|
||||
ts: float
|
||||
line: str # full NMEA sentence, incl. "!"/"$" and "*HH"
|
||||
origin: str = "internal" # e.g. "rest:POST /api/v1/tx", "emitter:ownship"
|
||||
@@ -0,0 +1,358 @@
|
||||
"""Central in-memory state: ownship, merged targets, base stations, AtoNs.
|
||||
|
||||
The state is the single source of truth. Parser tasks call ``apply_*``
|
||||
methods, which:
|
||||
|
||||
1. Update the in-memory snapshot (eager merge for targets).
|
||||
2. Publish a corresponding ``Event`` on the event bus.
|
||||
3. Increment stats counters.
|
||||
|
||||
REST reads directly from snapshot methods (O(1) per target).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from . import merge
|
||||
from ..parser.aiscatcher import (
|
||||
RssiIq,
|
||||
SignalEventBatch,
|
||||
SlotBitmap,
|
||||
SlotDetail,
|
||||
)
|
||||
from .bus import EventBus
|
||||
from .models import (
|
||||
AisReport,
|
||||
AtoN,
|
||||
BaseStation,
|
||||
Event,
|
||||
GpsFix,
|
||||
MergedTarget,
|
||||
Ownship,
|
||||
RadioReport,
|
||||
)
|
||||
from .stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class State:
|
||||
"""In-memory aggregated state of the service."""
|
||||
|
||||
def __init__(self, bus: EventBus, stats: Stats) -> None:
|
||||
self._bus = bus
|
||||
self._stats = stats
|
||||
self.ownship: Ownship = Ownship()
|
||||
self.targets: dict[int, MergedTarget] = {}
|
||||
self.base_stations: dict[int, BaseStation] = {}
|
||||
self.atons: dict[int, AtoN] = {}
|
||||
# TDMA slot map: channel -> slot -> {"mmsi": int, "ts": float, ...}
|
||||
self.slots: dict[str, dict[int, dict[str, Any]]] = {}
|
||||
self.last_radio: RadioReport | None = None
|
||||
# AIS-catcher telemetry (set by apply_rssi / apply_slot_* / apply_signal_events).
|
||||
# Each of the dicts below has channel ('A'/'B') as top-level key where applicable.
|
||||
self.radio_power: dict[str, Any] = {} # {"ts":..., "power_a_db":..., "power_b_db":...}
|
||||
self.slot_occupancy: dict[str, dict[str, Any]] = {} # ch -> {"ts","utc_minute","occupied_count","slots_total", ...}
|
||||
self.slot_detail: dict[str, dict[str, Any]] = {} # ch -> {"ts","utc_minute","entries":[...]}
|
||||
|
||||
# ----- mutation API ---------------------------------------------------
|
||||
|
||||
def apply_gps(self, fix: GpsFix) -> None:
|
||||
if fix.lat is not None:
|
||||
self.ownship.lat = fix.lat
|
||||
if fix.lon is not None:
|
||||
self.ownship.lon = fix.lon
|
||||
if fix.sog is not None:
|
||||
self.ownship.sog = fix.sog
|
||||
if fix.cog is not None:
|
||||
self.ownship.cog = fix.cog
|
||||
if fix.alt is not None:
|
||||
self.ownship.alt = fix.alt
|
||||
if fix.fix_quality is not None:
|
||||
self.ownship.fix_quality = fix.fix_quality
|
||||
if fix.sats is not None:
|
||||
self.ownship.sats = fix.sats
|
||||
if fix.hdop is not None:
|
||||
self.ownship.hdop = fix.hdop
|
||||
self.ownship.ts = fix.ts
|
||||
self._stats.incr("state_ownship_updates")
|
||||
self._bus.publish(Event(type="ownship.update", ts=fix.ts, data=self.ownship.to_dict()))
|
||||
|
||||
def apply_ais(self, report: AisReport) -> None:
|
||||
if report.kind == "base_station":
|
||||
self._apply_base_station(report)
|
||||
return
|
||||
if report.kind == "aton":
|
||||
self._apply_aton(report)
|
||||
return
|
||||
if report.kind in ("dynamic", "static"):
|
||||
self._apply_target(report)
|
||||
return
|
||||
# "other": still record source/msg type for stats.
|
||||
self._stats.incr("state_ais_other")
|
||||
|
||||
def _apply_target(self, report: AisReport) -> None:
|
||||
tgt = self.targets.get(report.mmsi)
|
||||
created = False
|
||||
if tgt is None:
|
||||
tgt = MergedTarget(mmsi=report.mmsi)
|
||||
self.targets[report.mmsi] = tgt
|
||||
created = True
|
||||
self._stats.incr("state_targets_new")
|
||||
changed = merge.apply(tgt, report)
|
||||
if created or changed:
|
||||
self._stats.incr("state_targets_updates")
|
||||
self._bus.publish(Event(type="target.update", ts=report.ts, data=tgt.to_dict()))
|
||||
|
||||
def _apply_base_station(self, report: AisReport) -> None:
|
||||
bs = self.base_stations.get(report.mmsi)
|
||||
if bs is None:
|
||||
bs = BaseStation(mmsi=report.mmsi)
|
||||
self.base_stations[report.mmsi] = bs
|
||||
if "lat" in report.data:
|
||||
bs.lat = report.data["lat"]
|
||||
if "lon" in report.data:
|
||||
bs.lon = report.data["lon"]
|
||||
if "epfd" in report.data:
|
||||
bs.epfd = report.data["epfd"]
|
||||
bs.ts = report.ts
|
||||
self._stats.incr("state_base_station_updates")
|
||||
self._bus.publish(Event(type="base_station.update", ts=report.ts, data=bs.to_dict()))
|
||||
|
||||
def _apply_aton(self, report: AisReport) -> None:
|
||||
a = self.atons.get(report.mmsi)
|
||||
if a is None:
|
||||
a = AtoN(mmsi=report.mmsi)
|
||||
self.atons[report.mmsi] = a
|
||||
for src, dst in (("lat", "lat"), ("lon", "lon"),
|
||||
("aton_type", "aton_type"), ("name", "name"),
|
||||
("virtual", "virtual")):
|
||||
if src in report.data and report.data[src] is not None:
|
||||
setattr(a, dst, report.data[src])
|
||||
a.ts = report.ts
|
||||
self._stats.incr("state_aton_updates")
|
||||
self._bus.publish(Event(type="aton.update", ts=report.ts, data=a.to_dict()))
|
||||
|
||||
def apply_radio(self, report: RadioReport) -> None:
|
||||
self.last_radio = report
|
||||
if report.channel and report.slot is not None:
|
||||
by_slot = self.slots.setdefault(report.channel, {})
|
||||
by_slot[report.slot] = {
|
||||
"ts": report.ts,
|
||||
"rssi": report.rssi,
|
||||
"snr": report.snr,
|
||||
}
|
||||
self._stats.incr("state_radio_updates")
|
||||
self._bus.publish(Event(
|
||||
type="radio.update",
|
||||
ts=report.ts,
|
||||
data={
|
||||
"channel": report.channel,
|
||||
"rssi": report.rssi,
|
||||
"snr": report.snr,
|
||||
"slot": report.slot,
|
||||
"raw": report.raw,
|
||||
},
|
||||
))
|
||||
|
||||
# ----- AIS-catcher telemetry -----------------------------------------
|
||||
|
||||
def apply_rssi_iq(self, rssi: RssiIq, ts: float) -> None:
|
||||
"""Store the latest RSSI pair and publish ``radio.update``."""
|
||||
self.radio_power = {
|
||||
"ts": ts,
|
||||
"power_a_db": rssi.power_a_db,
|
||||
"power_b_db": rssi.power_b_db,
|
||||
}
|
||||
self._stats.incr("state_rssi_updates")
|
||||
self._bus.publish(Event(
|
||||
type="radio.update",
|
||||
ts=ts,
|
||||
data={
|
||||
"source": "aiscatcher_rssi",
|
||||
"power_a_db": rssi.power_a_db,
|
||||
"power_b_db": rssi.power_b_db,
|
||||
},
|
||||
))
|
||||
|
||||
def apply_slot_bitmap(self, snap: SlotBitmap, ts: float) -> None:
|
||||
self.slot_occupancy[snap.channel] = {
|
||||
"ts": ts,
|
||||
"utc_minute": snap.utc_minute,
|
||||
"slots_total": snap.slots_total,
|
||||
"occupied_count": snap.occupied_count,
|
||||
"occupied_fraction": snap.occupied_fraction(),
|
||||
"slot0_unix_ms": snap.slot0_unix_ms,
|
||||
"first_occupied_unix_ms": snap.first_occupied_unix_ms or None,
|
||||
}
|
||||
self._stats.incr("state_slot_bitmap_updates")
|
||||
self._bus.publish(Event(
|
||||
type="slots.update",
|
||||
ts=ts,
|
||||
data={
|
||||
"channel": snap.channel,
|
||||
**self.slot_occupancy[snap.channel],
|
||||
"bitmap_hex": snap.bitmap.hex(),
|
||||
},
|
||||
))
|
||||
|
||||
def apply_slot_detail(self, detail: SlotDetail, ts: float) -> None:
|
||||
entries = [{"slot": e.slot, "level_db": e.level_db} for e in detail.entries]
|
||||
self.slot_detail[detail.channel] = {
|
||||
"ts": ts,
|
||||
"utc_minute": detail.utc_minute,
|
||||
"slot0_unix_ms": detail.slot0_unix_ms,
|
||||
"entries": entries,
|
||||
}
|
||||
self._stats.incr("state_slot_detail_updates")
|
||||
self._bus.publish(Event(
|
||||
type="slots.detail",
|
||||
ts=ts,
|
||||
data={
|
||||
"channel": detail.channel,
|
||||
**self.slot_detail[detail.channel],
|
||||
},
|
||||
))
|
||||
|
||||
def apply_signal_events(self, batch: SignalEventBatch, ts: float) -> None:
|
||||
"""Update per-MMSI signal levels from AIS-catcher decode events."""
|
||||
for ev in batch.events:
|
||||
tgt = self.targets.get(ev.mmsi)
|
||||
if tgt is None:
|
||||
tgt = MergedTarget(mmsi=ev.mmsi)
|
||||
self.targets[ev.mmsi] = tgt
|
||||
self._stats.incr("state_targets_new")
|
||||
ev_ts = ev.unix_ms / 1000.0 if ev.unix_ms else ts
|
||||
if ev_ts >= tgt.last_signal_ts:
|
||||
tgt.last_signal_ts = ev_ts
|
||||
tgt.last_signal_db = ev.level_db
|
||||
tgt.last_signal_slot = ev.slot
|
||||
tgt.last_signal_channel = batch.channel
|
||||
if tgt.last_seen < ev_ts:
|
||||
tgt.last_seen = ev_ts
|
||||
self._stats.incr("state_signal_events", len(batch.events))
|
||||
if batch.events:
|
||||
self._bus.publish(Event(
|
||||
type="signal.update",
|
||||
ts=ts,
|
||||
data={
|
||||
"channel": batch.channel,
|
||||
"events": [
|
||||
{
|
||||
"unix_ms": e.unix_ms,
|
||||
"slot": e.slot,
|
||||
"mmsi": e.mmsi,
|
||||
"level_db": e.level_db,
|
||||
}
|
||||
for e in batch.events
|
||||
],
|
||||
},
|
||||
))
|
||||
|
||||
# ----- warm start from SQLite ----------------------------------------
|
||||
|
||||
def warmup(
|
||||
self,
|
||||
*,
|
||||
vessels: list[dict[str, Any]] | None = None,
|
||||
base_stations: list[dict[str, Any]] | None = None,
|
||||
atons: list[dict[str, Any]] | None = None,
|
||||
) -> dict[str, int]:
|
||||
"""Populate in-memory state from previously persisted rows.
|
||||
|
||||
No events are published; this is intended to run once at startup
|
||||
before ingest tasks begin. Returns counts for logging/stats.
|
||||
"""
|
||||
counts = {"vessels": 0, "base_stations": 0, "atons": 0}
|
||||
|
||||
for row in vessels or ():
|
||||
mmsi = int(row["mmsi"])
|
||||
tgt = self.targets.get(mmsi)
|
||||
if tgt is None:
|
||||
tgt = MergedTarget(mmsi=mmsi)
|
||||
self.targets[mmsi] = tgt
|
||||
tgt.name = row.get("name") or tgt.name
|
||||
tgt.callsign = row.get("callsign") or tgt.callsign
|
||||
if row.get("imo") is not None:
|
||||
tgt.imo = row["imo"]
|
||||
if row.get("ship_type") is not None:
|
||||
tgt.ship_type = row["ship_type"]
|
||||
for k in ("dim_a", "dim_b", "dim_c", "dim_d"):
|
||||
if row.get(k) is not None:
|
||||
setattr(tgt, k, row[k])
|
||||
if row.get("eta"):
|
||||
tgt.eta = row["eta"]
|
||||
if row.get("draught") is not None:
|
||||
tgt.draught = row["draught"]
|
||||
if row.get("destination"):
|
||||
tgt.destination = row["destination"]
|
||||
updated_at = float(row.get("updated_at") or 0.0)
|
||||
if updated_at and updated_at > tgt.last_static_ts:
|
||||
tgt.last_static_ts = updated_at
|
||||
counts["vessels"] += 1
|
||||
|
||||
for row in base_stations or ():
|
||||
mmsi = int(row["mmsi"])
|
||||
bs = self.base_stations.get(mmsi) or BaseStation(mmsi=mmsi)
|
||||
bs.ts = float(row.get("ts") or 0.0)
|
||||
bs.lat = row.get("lat")
|
||||
bs.lon = row.get("lon")
|
||||
bs.epfd = row.get("epfd")
|
||||
self.base_stations[mmsi] = bs
|
||||
counts["base_stations"] += 1
|
||||
|
||||
for row in atons or ():
|
||||
mmsi = int(row["mmsi"])
|
||||
a = self.atons.get(mmsi) or AtoN(mmsi=mmsi)
|
||||
a.ts = float(row.get("ts") or 0.0)
|
||||
a.lat = row.get("lat")
|
||||
a.lon = row.get("lon")
|
||||
a.aton_type = row.get("aton_type")
|
||||
a.name = row.get("name")
|
||||
a.virtual = row.get("virtual")
|
||||
self.atons[mmsi] = a
|
||||
counts["atons"] += 1
|
||||
|
||||
return counts
|
||||
|
||||
# ----- read API -------------------------------------------------------
|
||||
|
||||
def snapshot_vessels(self, since: float | None = None, limit: int | None = None) -> list[dict[str, Any]]:
|
||||
items = self.targets.values()
|
||||
if since is not None:
|
||||
items = (t for t in items if t.last_seen >= since)
|
||||
out = [t.to_dict() for t in items]
|
||||
out.sort(key=lambda d: d["last_seen"], reverse=True)
|
||||
if limit is not None and limit > 0:
|
||||
out = out[:limit]
|
||||
return out
|
||||
|
||||
def get_vessel(self, mmsi: int) -> dict[str, Any] | None:
|
||||
t = self.targets.get(mmsi)
|
||||
return t.to_dict() if t is not None else None
|
||||
|
||||
def snapshot_base_stations(self) -> list[dict[str, Any]]:
|
||||
return [b.to_dict() for b in self.base_stations.values()]
|
||||
|
||||
def snapshot_atons(self) -> list[dict[str, Any]]:
|
||||
return [a.to_dict() for a in self.atons.values()]
|
||||
|
||||
def snapshot_slots(self) -> dict[str, Any]:
|
||||
"""Return TDMA slot / occupancy view for REST."""
|
||||
return {
|
||||
"per_channel": self.slots, # legacy slot->mmsi map (currently unused)
|
||||
"occupancy": self.slot_occupancy,
|
||||
"detail": self.slot_detail,
|
||||
}
|
||||
|
||||
def snapshot_ownship(self) -> dict[str, Any]:
|
||||
return self.ownship.to_dict()
|
||||
|
||||
def snapshot_radio(self) -> dict[str, Any]:
|
||||
return {"power": self.radio_power}
|
||||
|
||||
|
||||
__all__ = ["State"]
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Global counters & simple rate samples."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from threading import Lock
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass
|
||||
class Stats:
|
||||
"""Thread/async-safe counter container.
|
||||
|
||||
Counters are plain integers keyed by name. Gauges hold instantaneous
|
||||
numeric values. Queue depth samples are reported via ``set_gauge``.
|
||||
"""
|
||||
|
||||
counters: dict[str, int] = field(default_factory=lambda: defaultdict(int))
|
||||
gauges: dict[str, float] = field(default_factory=dict)
|
||||
started_at: float = field(default_factory=time.time)
|
||||
_lock: Lock = field(default_factory=Lock, repr=False)
|
||||
|
||||
def incr(self, name: str, n: int = 1) -> None:
|
||||
with self._lock:
|
||||
self.counters[name] += n
|
||||
|
||||
def set_gauge(self, name: str, value: float) -> None:
|
||||
with self._lock:
|
||||
self.gauges[name] = value
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
now = time.time()
|
||||
return {
|
||||
"uptime_sec": round(now - self.started_at, 3),
|
||||
"started_at": self.started_at,
|
||||
"counters": dict(self.counters),
|
||||
"gauges": dict(self.gauges),
|
||||
}
|
||||
|
||||
|
||||
__all__ = ["Stats"]
|
||||
@@ -0,0 +1 @@
|
||||
"""Ingest subsystem: AIS UDP, GPS UART, radio UDP listeners."""
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,94 @@
|
||||
"""AIS NMEA UDP ingest listener.
|
||||
|
||||
Binds a UDP socket on the configured host:port and decodes each received
|
||||
datagram as one or more ASCII NMEA lines. One datagram may contain several
|
||||
sentences separated by CR/LF.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..config import AisUdpCfg
|
||||
from ..core.stats import Stats
|
||||
from ..parser.nmea_utils import classify_kind, parse_sentence, split_lines
|
||||
from .base import IngestBase, RawTap, open_udp_listener
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _AisUdpProtocol(asyncio.DatagramProtocol):
|
||||
def __init__(self, ingest: "AisUdpIngest") -> None:
|
||||
self._ingest = ingest
|
||||
|
||||
def datagram_received(self, data: bytes, addr: Any) -> None:
|
||||
ip, _port = (addr[0], addr[1]) if addr else ("?", 0)
|
||||
# Sender port is ephemeral and may change between fragments of the
|
||||
# same AIS multi-part message; only include IP in the source tag so
|
||||
# the multi-fragment assembler can correlate frag 1 and frag 2.
|
||||
suffix = f":{ip}"
|
||||
self._ingest._stats.incr("ais_udp_datagrams")
|
||||
for line in split_lines(data):
|
||||
sentence = parse_sentence(line)
|
||||
if sentence is None:
|
||||
self._ingest._stats.incr("ais_udp_malformed")
|
||||
# Still publish the raw line so SPI bridge sees everything.
|
||||
self._ingest.emit("ais", line, source_suffix=suffix)
|
||||
continue
|
||||
kind = classify_kind(sentence.talker, sentence.start)
|
||||
self._ingest.emit(kind if kind in ("ais", "gps") else "ais", line, source_suffix=suffix)
|
||||
|
||||
def error_received(self, exc: Exception) -> None:
|
||||
log.warning("ais_udp error_received: %s", exc)
|
||||
|
||||
|
||||
class AisUdpIngest(IngestBase):
|
||||
"""Async UDP listener producing ``RawFrame`` items."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cfg: AisUdpCfg,
|
||||
*,
|
||||
parser_in: "asyncio.Queue",
|
||||
raw_tap: RawTap,
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
source_prefix=f"ais_udp:{cfg.host}:{cfg.port}",
|
||||
parser_in=parser_in,
|
||||
raw_tap=raw_tap,
|
||||
stats=stats,
|
||||
parser_drop_counter="parser_in_dropped",
|
||||
)
|
||||
self._cfg = cfg
|
||||
self._transport: asyncio.DatagramTransport | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
self._transport = await open_udp_listener(
|
||||
self._cfg.host,
|
||||
self._cfg.port,
|
||||
lambda: _AisUdpProtocol(self),
|
||||
name="ais_udp",
|
||||
)
|
||||
log.info("ais_udp listening on %s:%d", self._cfg.host, self._cfg.port)
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._transport is not None:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
try:
|
||||
# Listener runs inside asyncio; just sleep forever until cancelled.
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
await self.stop()
|
||||
|
||||
|
||||
__all__ = ["AisUdpIngest"]
|
||||
@@ -0,0 +1,146 @@
|
||||
"""AIS-catcher Mini binary UDP telemetry ingest.
|
||||
|
||||
Four independent UDP listeners, each bound to its own port on ``host``:
|
||||
|
||||
* ``slot_port`` (default 10111) — 315-byte slot occupancy bitmap
|
||||
* ``slot_detail_port`` (default 10112) — per-slot levels
|
||||
* ``rssi_port`` (default 10113) — 8-byte RSSI pair @ ~10 Hz
|
||||
* ``event_port`` (default 10114) — decode events (mmsi, slot, level)
|
||||
|
||||
Each datagram is decoded synchronously inside the protocol callback and
|
||||
forwarded directly to ``State``; no parser queue is involved. The event
|
||||
bus handles downstream fan-out (WebSocket / UDP events / storage sink).
|
||||
|
||||
These sockets intentionally do **not** publish to ``RawTap`` because
|
||||
the payloads are binary, not NMEA.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Callable
|
||||
|
||||
from ..config import AisCatcherUdpCfg
|
||||
from ..core.state import State
|
||||
from ..core.stats import Stats
|
||||
from ..parser.aiscatcher import (
|
||||
decode_rssi_iq,
|
||||
decode_signal_events,
|
||||
decode_slot_bitmap,
|
||||
decode_slot_detail,
|
||||
)
|
||||
from .base import open_udp_listener
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _BinaryUdpProtocol(asyncio.DatagramProtocol):
|
||||
def __init__(
|
||||
self,
|
||||
handler: Callable[[bytes], None],
|
||||
name: str,
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
self._handler = handler
|
||||
self._name = name
|
||||
self._stats = stats
|
||||
|
||||
def datagram_received(self, data: bytes, addr: Any) -> None:
|
||||
self._stats.incr(f"{self._name}_datagrams")
|
||||
try:
|
||||
self._handler(data)
|
||||
except Exception:
|
||||
self._stats.incr(f"{self._name}_errors")
|
||||
log.exception("%s handler failed", self._name)
|
||||
|
||||
def error_received(self, exc: Exception) -> None:
|
||||
log.warning("%s error_received: %s", self._name, exc)
|
||||
|
||||
|
||||
class AisCatcherUdpIngest:
|
||||
"""Owns four UDP sockets; each one dispatches into ``State``."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cfg: AisCatcherUdpCfg,
|
||||
*,
|
||||
state: State,
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
self._cfg = cfg
|
||||
self._state = state
|
||||
self._stats = stats
|
||||
self._transports: list[asyncio.DatagramTransport] = []
|
||||
|
||||
# --- per-port handlers ------------------------------------------------
|
||||
|
||||
def _on_bitmap(self, data: bytes) -> None:
|
||||
snap = decode_slot_bitmap(data)
|
||||
if snap is None:
|
||||
self._stats.incr("aiscatcher_slot_bitmap_malformed")
|
||||
return
|
||||
self._state.apply_slot_bitmap(snap, ts=time.time())
|
||||
|
||||
def _on_detail(self, data: bytes) -> None:
|
||||
detail = decode_slot_detail(data)
|
||||
if detail is None:
|
||||
self._stats.incr("aiscatcher_slot_detail_malformed")
|
||||
return
|
||||
self._state.apply_slot_detail(detail, ts=time.time())
|
||||
|
||||
def _on_rssi(self, data: bytes) -> None:
|
||||
rssi = decode_rssi_iq(data)
|
||||
if rssi is None:
|
||||
self._stats.incr("aiscatcher_rssi_malformed")
|
||||
return
|
||||
self._state.apply_rssi_iq(rssi, ts=time.time())
|
||||
|
||||
def _on_events(self, data: bytes) -> None:
|
||||
batch = decode_signal_events(data)
|
||||
if batch is None:
|
||||
self._stats.incr("aiscatcher_events_malformed")
|
||||
return
|
||||
self._state.apply_signal_events(batch, ts=time.time())
|
||||
|
||||
# --- lifecycle --------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
host = self._cfg.host
|
||||
listeners = [
|
||||
("aiscatcher_slot", self._cfg.slot_port, self._on_bitmap),
|
||||
("aiscatcher_slot_detail", self._cfg.slot_detail_port, self._on_detail),
|
||||
("aiscatcher_rssi", self._cfg.rssi_port, self._on_rssi),
|
||||
("aiscatcher_events", self._cfg.event_port, self._on_events),
|
||||
]
|
||||
for name, port, handler in listeners:
|
||||
transport = await open_udp_listener(
|
||||
host,
|
||||
port,
|
||||
lambda n=name, h=handler: _BinaryUdpProtocol(h, n, self._stats),
|
||||
name=name,
|
||||
)
|
||||
self._transports.append(transport)
|
||||
log.info("%s listening on %s:%d", name, host, port)
|
||||
|
||||
async def stop(self) -> None:
|
||||
for t in self._transports:
|
||||
try:
|
||||
t.close()
|
||||
except Exception:
|
||||
pass
|
||||
self._transports.clear()
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
await self.stop()
|
||||
|
||||
|
||||
__all__ = ["AisCatcherUdpIngest"]
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Shared ingest glue: RawFrame fan-out to parser queue + raw NMEA tap."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import collections
|
||||
import errno
|
||||
import logging
|
||||
import socket
|
||||
import time
|
||||
from typing import Any, Callable, Deque
|
||||
|
||||
from ..core.models import RawFrame
|
||||
from ..core.stats import Stats
|
||||
from ..publish.queues import put_drop_oldest
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AddressInUseError(RuntimeError):
|
||||
"""Raised when a UDP ingest port is already bound by another process.
|
||||
|
||||
The supervisor logs this at WARNING (not ERROR with traceback) and
|
||||
keeps retrying with backoff until the port is free.
|
||||
"""
|
||||
|
||||
|
||||
async def open_udp_listener(
|
||||
host: str,
|
||||
port: int,
|
||||
protocol_factory: Callable[[], asyncio.DatagramProtocol],
|
||||
*,
|
||||
reuse_address: bool = True,
|
||||
name: str = "udp",
|
||||
) -> asyncio.DatagramTransport:
|
||||
"""Bind a UDP socket with SO_REUSEADDR and attach it to the event loop.
|
||||
|
||||
``SO_REUSEADDR`` helps the listener rebind quickly after a previous
|
||||
crash of the same process. It does **not** let the socket coexist
|
||||
with another process that is actively listening on the same port;
|
||||
in that case ``AddressInUseError`` is raised with a clear message.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
sock.setblocking(False)
|
||||
try:
|
||||
sock.bind((host, port))
|
||||
except OSError as exc:
|
||||
sock.close()
|
||||
if exc.errno in (errno.EADDRINUSE, errno.EACCES):
|
||||
raise AddressInUseError(
|
||||
f"{name}: cannot bind {host}:{port} — already in use by another process. "
|
||||
f"Check with: ss -lunp 'sport = :{port}' / lsof -iUDP:{port}"
|
||||
) from exc
|
||||
raise
|
||||
transport, _ = await loop.create_datagram_endpoint(protocol_factory, sock=sock)
|
||||
return transport
|
||||
|
||||
|
||||
class RawTap:
|
||||
"""Fan-out of raw NMEA lines to UDP fan-out / storage / in-memory ring.
|
||||
|
||||
Consumers register their own queue (bounded) via ``subscribe``. The tap
|
||||
publishes each raw line to every subscriber with drop-oldest semantics
|
||||
to isolate slow consumers.
|
||||
|
||||
A small in-memory ring buffer is also maintained here so
|
||||
``GET /api/v1/nmea/tail`` can return recent raw sentences without going
|
||||
to SQLite.
|
||||
"""
|
||||
|
||||
def __init__(self, stats: Stats | None = None, ring_size: int = 500) -> None:
|
||||
self._subs: list[tuple[asyncio.Queue[RawFrame], str]] = []
|
||||
self._stats = stats
|
||||
self._ring: Deque[RawFrame] = collections.deque(maxlen=max(0, ring_size))
|
||||
|
||||
def subscribe(self, queue: "asyncio.Queue[RawFrame]", drop_counter: str) -> None:
|
||||
self._subs.append((queue, drop_counter))
|
||||
|
||||
def unsubscribe(self, queue: "asyncio.Queue[RawFrame]") -> None:
|
||||
self._subs = [(q, c) for (q, c) in self._subs if q is not queue]
|
||||
|
||||
def publish(self, frame: RawFrame) -> None:
|
||||
if self._ring.maxlen:
|
||||
self._ring.append(frame)
|
||||
for q, counter in self._subs:
|
||||
put_drop_oldest(q, frame, self._stats, counter)
|
||||
|
||||
def tail(self, limit: int, kind: str | None = None) -> list[RawFrame]:
|
||||
out: list[RawFrame] = []
|
||||
for fr in self._ring:
|
||||
if kind and fr.kind != kind:
|
||||
continue
|
||||
out.append(fr)
|
||||
if limit > 0:
|
||||
out = out[-limit:]
|
||||
return out
|
||||
|
||||
|
||||
class IngestBase:
|
||||
"""Common helper to classify and emit a raw NMEA line from any source."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
source_prefix: str,
|
||||
parser_in: "asyncio.Queue[RawFrame]",
|
||||
raw_tap: RawTap,
|
||||
stats: Stats,
|
||||
parser_drop_counter: str,
|
||||
) -> None:
|
||||
self._source_prefix = source_prefix
|
||||
self._parser_in = parser_in
|
||||
self._raw_tap = raw_tap
|
||||
self._stats = stats
|
||||
self._parser_drop_counter = parser_drop_counter
|
||||
|
||||
def emit(self, kind: str, line: str, source_suffix: str = "") -> None:
|
||||
"""Stamp, count, and fan out one raw NMEA line."""
|
||||
if not line:
|
||||
return
|
||||
source = f"{self._source_prefix}{source_suffix}" if source_suffix else self._source_prefix
|
||||
frame = RawFrame(ts=time.time(), source=source, kind=kind, line=line) # type: ignore[arg-type]
|
||||
self._stats.incr(f"rx_lines_{kind}")
|
||||
self._raw_tap.publish(frame)
|
||||
put_drop_oldest(self._parser_in, frame, self._stats, self._parser_drop_counter)
|
||||
|
||||
|
||||
__all__ = ["IngestBase", "RawTap"]
|
||||
@@ -0,0 +1,99 @@
|
||||
"""GPS UART ingest: reads NMEA lines from a serial device.
|
||||
|
||||
Uses ``pyserial-asyncio`` when available. Lines are read in line-buffered
|
||||
mode (CRLF terminated). Failures are logged; the supervisor retries the
|
||||
task after backoff.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..config import GpsUartCfg
|
||||
from ..core.stats import Stats
|
||||
from ..parser.nmea_utils import split_lines
|
||||
from .base import IngestBase, RawTap
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GpsUartIngest(IngestBase):
|
||||
def __init__(
|
||||
self,
|
||||
cfg: GpsUartCfg,
|
||||
*,
|
||||
parser_in: "asyncio.Queue",
|
||||
raw_tap: RawTap,
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
source_prefix=f"gps_uart:{cfg.device}",
|
||||
parser_in=parser_in,
|
||||
raw_tap=raw_tap,
|
||||
stats=stats,
|
||||
parser_drop_counter="parser_in_dropped",
|
||||
)
|
||||
self._cfg = cfg
|
||||
self._reader: Any = None
|
||||
self._writer: Any = None
|
||||
|
||||
async def run(self) -> None:
|
||||
if not self._cfg.enabled:
|
||||
log.info("gps_uart disabled in config")
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
|
||||
try:
|
||||
import serial_asyncio # type: ignore
|
||||
except Exception:
|
||||
log.error("serial_asyncio is not installed; gps_uart disabled")
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
|
||||
try:
|
||||
reader, writer = await serial_asyncio.open_serial_connection(
|
||||
url=self._cfg.device,
|
||||
baudrate=self._cfg.baud,
|
||||
)
|
||||
except Exception:
|
||||
log.exception("gps_uart: failed to open %s@%d", self._cfg.device, self._cfg.baud)
|
||||
raise
|
||||
|
||||
self._reader, self._writer = reader, writer
|
||||
log.info("gps_uart open %s @ %d baud", self._cfg.device, self._cfg.baud)
|
||||
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
chunk = await reader.readuntil(b"\n")
|
||||
except asyncio.IncompleteReadError as exc:
|
||||
# Device EOF; treat as transient and raise for supervisor.
|
||||
if exc.partial:
|
||||
for line in split_lines(exc.partial):
|
||||
self.emit("gps", line)
|
||||
raise RuntimeError("gps_uart EOF")
|
||||
except asyncio.LimitOverrunError:
|
||||
# Overly long line without newline; drain it.
|
||||
chunk = await reader.read(4096)
|
||||
for line in split_lines(chunk):
|
||||
self.emit("gps", line)
|
||||
self._stats.incr("gps_uart_lines")
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
try:
|
||||
if writer is not None:
|
||||
writer.close()
|
||||
try:
|
||||
await writer.wait_closed()
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
self._reader = None
|
||||
self._writer = None
|
||||
|
||||
|
||||
__all__ = ["GpsUartIngest"]
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Radio telemetry UDP ingest listener.
|
||||
|
||||
Each datagram carries a JSON object (see ``parser.radio``). The listener
|
||||
does not parse it here; it only stamps a ``RawFrame`` with ``kind="radio"``
|
||||
and hands it to the parser queue. The raw NMEA tap receives the datagram
|
||||
too (as a line), so SPI bridge can observe radio traffic if wanted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..config import RadioUdpCfg
|
||||
from ..core.stats import Stats
|
||||
from .base import IngestBase, RawTap, open_udp_listener
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class _RadioUdpProtocol(asyncio.DatagramProtocol):
|
||||
def __init__(self, ingest: "RadioUdpIngest") -> None:
|
||||
self._ingest = ingest
|
||||
|
||||
def datagram_received(self, data: bytes, addr: Any) -> None:
|
||||
ip, port = (addr[0], addr[1]) if addr else ("?", 0)
|
||||
self._ingest._stats.incr("radio_udp_datagrams")
|
||||
# Radio telemetry is JSON; tap publishes it as-is (stripped).
|
||||
try:
|
||||
line = data.decode("utf-8", errors="replace").strip()
|
||||
except Exception:
|
||||
return
|
||||
if not line:
|
||||
return
|
||||
self._ingest.emit("radio", line, source_suffix=f":{ip}:{port}")
|
||||
|
||||
def error_received(self, exc: Exception) -> None:
|
||||
log.warning("radio_udp error_received: %s", exc)
|
||||
|
||||
|
||||
class RadioUdpIngest(IngestBase):
|
||||
def __init__(
|
||||
self,
|
||||
cfg: RadioUdpCfg,
|
||||
*,
|
||||
parser_in: "asyncio.Queue",
|
||||
raw_tap: RawTap,
|
||||
stats: Stats,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
source_prefix=f"radio_udp:{cfg.host}:{cfg.port}",
|
||||
parser_in=parser_in,
|
||||
raw_tap=raw_tap,
|
||||
stats=stats,
|
||||
parser_drop_counter="parser_in_dropped",
|
||||
)
|
||||
self._cfg = cfg
|
||||
self._transport: asyncio.DatagramTransport | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
self._transport = await open_udp_listener(
|
||||
self._cfg.host,
|
||||
self._cfg.port,
|
||||
lambda: _RadioUdpProtocol(self),
|
||||
name="radio_udp",
|
||||
)
|
||||
log.info("radio_udp listening on %s:%d", self._cfg.host, self._cfg.port)
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._transport is not None:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
async def run(self) -> None:
|
||||
await self.start()
|
||||
try:
|
||||
while True:
|
||||
await asyncio.sleep(3600)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
await self.stop()
|
||||
|
||||
|
||||
__all__ = ["RadioUdpIngest"]
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Structured logging setup: JSON formatter + rotating file + stderr.
|
||||
|
||||
Also keeps an in-memory ring buffer of recent log records so the REST
|
||||
endpoint ``/api/v1/logs`` can tail them without reading the file from
|
||||
disk on every request.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
import json
|
||||
import logging
|
||||
import logging.handlers
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Deque
|
||||
|
||||
from .config import LoggingCfg
|
||||
|
||||
# Ring buffer of recent formatted log records. Populated by MemoryRingHandler.
|
||||
# Each item is a dict with keys: ts, level, logger, message.
|
||||
LOG_RING: Deque[dict[str, Any]] = collections.deque(maxlen=1000)
|
||||
|
||||
|
||||
class JsonFormatter(logging.Formatter):
|
||||
"""Minimal JSON log formatter with stable keys."""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str: # noqa: D401
|
||||
payload: dict[str, Any] = {
|
||||
"ts": round(record.created, 3),
|
||||
"level": record.levelname,
|
||||
"logger": record.name,
|
||||
"message": record.getMessage(),
|
||||
}
|
||||
if record.exc_info:
|
||||
payload["exc"] = self.formatException(record.exc_info)
|
||||
if record.stack_info:
|
||||
payload["stack"] = record.stack_info
|
||||
# Attach extra attributes set via logger.log(..., extra={...}).
|
||||
for key, val in record.__dict__.items():
|
||||
if key in _STD_RECORD_FIELDS or key.startswith("_"):
|
||||
continue
|
||||
try:
|
||||
json.dumps(val) # probe serializability
|
||||
except Exception:
|
||||
val = repr(val)
|
||||
payload[key] = val
|
||||
return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
|
||||
|
||||
|
||||
_STD_RECORD_FIELDS = {
|
||||
"name", "msg", "args", "levelname", "levelno", "pathname", "filename",
|
||||
"module", "exc_info", "exc_text", "stack_info", "lineno", "funcName",
|
||||
"created", "msecs", "relativeCreated", "thread", "threadName",
|
||||
"processName", "process", "message", "asctime", "taskName",
|
||||
}
|
||||
|
||||
|
||||
class MemoryRingHandler(logging.Handler):
|
||||
"""Logging handler that appends formatted records to ``LOG_RING``."""
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
LOG_RING.append({
|
||||
"ts": round(record.created, 3),
|
||||
"level": record.levelname,
|
||||
"logger": record.name,
|
||||
"message": record.getMessage(),
|
||||
})
|
||||
except Exception:
|
||||
# Never let logging crash the service.
|
||||
pass
|
||||
|
||||
|
||||
def setup_logging(cfg: LoggingCfg) -> None:
|
||||
"""Configure root logger according to ``cfg``."""
|
||||
root = logging.getLogger()
|
||||
# Remove any pre-existing handlers to allow idempotent re-setup (tests).
|
||||
for h in list(root.handlers):
|
||||
root.removeHandler(h)
|
||||
|
||||
level = getattr(logging, cfg.level.upper(), logging.INFO)
|
||||
root.setLevel(level)
|
||||
|
||||
formatter: logging.Formatter
|
||||
if cfg.json:
|
||||
formatter = JsonFormatter()
|
||||
else:
|
||||
formatter = logging.Formatter(
|
||||
fmt="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
|
||||
# stderr handler (always).
|
||||
stderr_h = logging.StreamHandler(stream=sys.stderr)
|
||||
stderr_h.setFormatter(formatter)
|
||||
root.addHandler(stderr_h)
|
||||
|
||||
# Rotating file handler if configured.
|
||||
if cfg.file:
|
||||
try:
|
||||
Path(cfg.file).parent.mkdir(parents=True, exist_ok=True)
|
||||
file_h = logging.handlers.RotatingFileHandler(
|
||||
cfg.file,
|
||||
maxBytes=cfg.max_bytes,
|
||||
backupCount=cfg.backup_count,
|
||||
encoding="utf-8",
|
||||
)
|
||||
file_h.setFormatter(formatter)
|
||||
root.addHandler(file_h)
|
||||
except OSError as exc:
|
||||
# Do not fail startup because of log-file permission issues.
|
||||
sys.stderr.write(f"warning: cannot open log file {cfg.file}: {exc}\n")
|
||||
|
||||
# In-memory ring for /api/v1/logs.
|
||||
ring_h = MemoryRingHandler()
|
||||
root.addHandler(ring_h)
|
||||
|
||||
# Quiet a few noisy libraries by default.
|
||||
logging.getLogger("aiohttp.access").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
def tail_logs(limit: int = 200, level: str | None = None, since: float | None = None) -> list[dict[str, Any]]:
|
||||
"""Return up to ``limit`` recent log records from the in-memory ring.
|
||||
|
||||
``level`` filters by severity (case-insensitive). ``since`` is a unix
|
||||
timestamp lower bound.
|
||||
"""
|
||||
lvl = level.upper() if level else None
|
||||
out: list[dict[str, Any]] = []
|
||||
for rec in LOG_RING:
|
||||
if lvl and rec["level"] != lvl:
|
||||
continue
|
||||
if since is not None and rec["ts"] < since:
|
||||
continue
|
||||
out.append(rec)
|
||||
if limit > 0:
|
||||
out = out[-limit:]
|
||||
return out
|
||||
|
||||
|
||||
__all__ = ["setup_logging", "tail_logs", "LOG_RING"]
|
||||
@@ -0,0 +1 @@
|
||||
"""NMEA parsers: shared utils, AIS assembler, GPS, radio telemetry."""
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,382 @@
|
||||
"""AIS multi-fragment assembler + pyais decoder + normalization.
|
||||
|
||||
Fragment key is a strengthened composite:
|
||||
|
||||
(source_tag, talker, channel, seq_id, total_fragments)
|
||||
|
||||
- ``source_tag`` comes from the ingest layer (e.g. "ais_udp:192.168.1.2:34567").
|
||||
Two separate AIS receivers can reuse the same ``seq_id`` independently,
|
||||
so we must key by source to avoid cross-contamination.
|
||||
- ``talker`` is the talker id (``AIVDM`` / ``AIVDO`` / ``BSVDM`` / ...).
|
||||
- ``channel`` is the AIS channel (``A`` or ``B`` or empty).
|
||||
- ``seq_id`` is the sequential message id field (field 3).
|
||||
- ``total_fragments`` guards against a stale in-flight key being reused
|
||||
with a different total.
|
||||
|
||||
Rules enforced on arrival:
|
||||
- Fragment numbers must progress strictly ``1, 2, ..., total`` without
|
||||
gaps or duplicates. Any violation clears the buffer for that key and
|
||||
increments ``ais_fragment_errors``.
|
||||
- Buffers older than ``TTL_SEC`` are evicted and counted as timeouts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Iterable
|
||||
|
||||
from ..core.models import AisReport
|
||||
from ..core.stats import Stats
|
||||
from .nmea_utils import NmeaSentence, parse_sentence
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
TTL_SEC = 60.0
|
||||
|
||||
DYNAMIC_TYPES = {1, 2, 3, 18, 19, 27}
|
||||
STATIC_TYPES = {5, 24}
|
||||
BASE_TYPES = {4, 11}
|
||||
ATON_TYPES = {21}
|
||||
|
||||
|
||||
FragmentKey = tuple[str, str, str, str, int] # (source, talker, channel, seq, total)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _Buffer:
|
||||
expected: int # next fragment_number we expect (1..total)
|
||||
total: int
|
||||
lines: list[str] = field(default_factory=list)
|
||||
created_at: float = 0.0
|
||||
|
||||
|
||||
def _classify(msg_type: int) -> str:
|
||||
if msg_type in DYNAMIC_TYPES:
|
||||
return "dynamic"
|
||||
if msg_type in STATIC_TYPES:
|
||||
return "static"
|
||||
if msg_type in BASE_TYPES:
|
||||
return "base_station"
|
||||
if msg_type in ATON_TYPES:
|
||||
return "aton"
|
||||
return "other"
|
||||
|
||||
|
||||
def _safe_int(v: Any) -> int | None:
|
||||
"""Coerce a value to plain ``int``.
|
||||
|
||||
This deliberately flattens ``IntEnum`` / ``Enum.value`` instances
|
||||
(pyais uses them for ``ship_type``, ``epfd``, ``status`` etc.) into
|
||||
plain ints so downstream JSON serialization and equality checks
|
||||
stay stable across pyais versions.
|
||||
"""
|
||||
if v is None:
|
||||
return None
|
||||
try:
|
||||
return int(v)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _safe_float(v: Any) -> float | None:
|
||||
try:
|
||||
f = float(v)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
# pyais uses +/-91 / +/-181 as "unavailable" markers in some types.
|
||||
return f
|
||||
|
||||
|
||||
class AisAssembler:
|
||||
"""Reassemble multi-fragment AIS sentences and decode them.
|
||||
|
||||
Instances are *not* thread-safe; use one per async task.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
stats: Stats | None = None,
|
||||
ttl_sec: float = TTL_SEC,
|
||||
ignore_checksum: bool = False,
|
||||
allow_checksumless: bool = False,
|
||||
allow_multipart_without_seq_id: bool = False,
|
||||
) -> None:
|
||||
self._buffers: dict[FragmentKey, _Buffer] = {}
|
||||
self._noseq_buffers: dict[tuple[str, str, str, int], _Buffer] = {}
|
||||
self._stats = stats
|
||||
self._ttl = ttl_sec
|
||||
self._ignore_checksum = ignore_checksum
|
||||
self._allow_checksumless = allow_checksumless
|
||||
self._allow_multipart_without_seq_id = allow_multipart_without_seq_id
|
||||
|
||||
# -- public API --------------------------------------------------------
|
||||
|
||||
def feed(self, source_tag: str, line: str, ts: float | None = None) -> list[AisReport]:
|
||||
"""Accept one raw NMEA AIS line. Return 0+ decoded reports.
|
||||
|
||||
``ts`` defaults to ``time.time()`` and is stamped on the produced
|
||||
reports.
|
||||
"""
|
||||
if ts is None:
|
||||
ts = time.time()
|
||||
self._gc(ts)
|
||||
|
||||
sentence = parse_sentence(line)
|
||||
if sentence is None:
|
||||
self._incr("parser_errors")
|
||||
return []
|
||||
if not self._ignore_checksum and not sentence.checksum_ok:
|
||||
# Allow checksumless (no "*HH") sentences when configured, but still
|
||||
# reject sentences with a present-yet-invalid checksum.
|
||||
if self._allow_checksumless and "*" not in sentence.raw:
|
||||
pass
|
||||
else:
|
||||
self._incr("parser_checksum_errors")
|
||||
return []
|
||||
if sentence.start != "!":
|
||||
# Non-encapsulated, not AIS.
|
||||
return []
|
||||
if len(sentence.fields) < 5:
|
||||
self._incr("parser_errors")
|
||||
return []
|
||||
|
||||
try:
|
||||
total = int(sentence.fields[0]) if sentence.fields[0] else 1
|
||||
frag_no = int(sentence.fields[1]) if sentence.fields[1] else 1
|
||||
except ValueError:
|
||||
self._incr("parser_errors")
|
||||
return []
|
||||
|
||||
seq_id = sentence.fields[2] or ""
|
||||
channel = sentence.fields[3] or ""
|
||||
|
||||
# Single-fragment fast path: no multipart state needed.
|
||||
if total == 1 and frag_no == 1:
|
||||
return self._decode([sentence.raw], ts, source_tag, channel)
|
||||
|
||||
# Multi-fragment validation.
|
||||
if total < 1 or not (1 <= frag_no <= total):
|
||||
self._incr("ais_fragment_errors")
|
||||
return []
|
||||
if not seq_id:
|
||||
if not self._allow_multipart_without_seq_id:
|
||||
# Multi-fragment without seq_id is not reliably groupable.
|
||||
self._incr("ais_fragment_errors")
|
||||
return []
|
||||
return self._feed_noseq(source_tag, sentence, ts, total, frag_no, channel)
|
||||
|
||||
key: FragmentKey = (source_tag, sentence.talker, channel, seq_id, total)
|
||||
buf = self._buffers.get(key)
|
||||
|
||||
if buf is None:
|
||||
if frag_no != 1:
|
||||
# Mid-stream fragment without a leading fragment is unusable.
|
||||
self._incr("ais_fragment_errors")
|
||||
return []
|
||||
buf = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
|
||||
self._buffers[key] = buf
|
||||
return []
|
||||
|
||||
# Existing buffer: expect strictly increasing fragment_number.
|
||||
if frag_no != buf.expected or total != buf.total:
|
||||
self._incr("ais_fragment_errors")
|
||||
self._buffers.pop(key, None)
|
||||
# If the violating fragment itself is a fresh start (frag_no==1),
|
||||
# seed a new buffer so we don't lose it.
|
||||
if frag_no == 1:
|
||||
self._buffers[key] = _Buffer(
|
||||
expected=2, total=total, lines=[sentence.raw], created_at=ts,
|
||||
)
|
||||
return []
|
||||
|
||||
buf.lines.append(sentence.raw)
|
||||
buf.expected += 1
|
||||
|
||||
if len(buf.lines) == buf.total:
|
||||
self._buffers.pop(key, None)
|
||||
return self._decode(buf.lines, ts, source_tag, channel)
|
||||
return []
|
||||
|
||||
def gc(self, now: float | None = None) -> None:
|
||||
"""Evict buffers older than TTL (exposed for tests)."""
|
||||
self._gc(now if now is not None else time.time())
|
||||
|
||||
# -- internals ---------------------------------------------------------
|
||||
|
||||
def _gc(self, now: float) -> None:
|
||||
dead: list[FragmentKey] = []
|
||||
cutoff = now - self._ttl
|
||||
for key, buf in self._buffers.items():
|
||||
if buf.created_at < cutoff:
|
||||
dead.append(key)
|
||||
for k in dead:
|
||||
self._buffers.pop(k, None)
|
||||
self._incr("ais_fragment_timeouts")
|
||||
dead2: list[tuple[str, str, str, int]] = []
|
||||
for key, buf in self._noseq_buffers.items():
|
||||
if buf.created_at < cutoff:
|
||||
dead2.append(key)
|
||||
for k in dead2:
|
||||
self._noseq_buffers.pop(k, None)
|
||||
self._incr("ais_fragment_timeouts")
|
||||
|
||||
def _feed_noseq(
|
||||
self,
|
||||
source_tag: str,
|
||||
sentence: NmeaSentence,
|
||||
ts: float,
|
||||
total: int,
|
||||
frag_no: int,
|
||||
channel: str,
|
||||
) -> list[AisReport]:
|
||||
"""Best-effort multipart reassembly for sentences with empty seq_id.
|
||||
|
||||
We keep at most one active buffer per (source, talker, channel, total).
|
||||
This works well when the upstream does not interleave multiple
|
||||
checksumless/no-seq multipart messages on the same channel.
|
||||
"""
|
||||
key2 = (source_tag, sentence.talker, channel, total)
|
||||
buf = self._noseq_buffers.get(key2)
|
||||
|
||||
if buf is None:
|
||||
if frag_no != 1:
|
||||
self._incr("ais_fragment_errors")
|
||||
return []
|
||||
self._noseq_buffers[key2] = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
|
||||
return []
|
||||
|
||||
if frag_no != buf.expected or total != buf.total:
|
||||
self._incr("ais_fragment_errors")
|
||||
self._noseq_buffers.pop(key2, None)
|
||||
if frag_no == 1:
|
||||
self._noseq_buffers[key2] = _Buffer(expected=2, total=total, lines=[sentence.raw], created_at=ts)
|
||||
return []
|
||||
|
||||
buf.lines.append(sentence.raw)
|
||||
buf.expected += 1
|
||||
if len(buf.lines) == buf.total:
|
||||
self._noseq_buffers.pop(key2, None)
|
||||
return self._decode(buf.lines, ts, source_tag, channel)
|
||||
return []
|
||||
|
||||
def _incr(self, name: str, n: int = 1) -> None:
|
||||
if self._stats is not None:
|
||||
self._stats.incr(name, n)
|
||||
|
||||
def _decode(
|
||||
self, lines: list[str], ts: float, source_tag: str, channel: str,
|
||||
) -> list[AisReport]:
|
||||
"""Invoke pyais on the collected lines and produce one AisReport."""
|
||||
try:
|
||||
# Import lazily so unit tests that don't need pyais can run.
|
||||
from pyais import decode as pyais_decode # type: ignore
|
||||
except Exception: # pragma: no cover
|
||||
self._incr("parser_errors")
|
||||
log.exception("pyais import failed")
|
||||
return []
|
||||
|
||||
try:
|
||||
decoded = pyais_decode(*[ln.encode("ascii", errors="replace") for ln in lines])
|
||||
except Exception:
|
||||
self._incr("parser_errors")
|
||||
log.debug("pyais decode failed for %d lines", len(lines), exc_info=True)
|
||||
return []
|
||||
|
||||
try:
|
||||
payload: dict[str, Any] = decoded.asdict() # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
try:
|
||||
payload = dict(decoded) # type: ignore[arg-type]
|
||||
except Exception:
|
||||
self._incr("parser_errors")
|
||||
return []
|
||||
|
||||
msg_type = _safe_int(payload.get("msg_type") or payload.get("type"))
|
||||
mmsi = _safe_int(payload.get("mmsi"))
|
||||
if msg_type is None or mmsi is None:
|
||||
self._incr("parser_errors")
|
||||
return []
|
||||
|
||||
kind = _classify(msg_type)
|
||||
normalized = _normalize_payload(payload, kind, msg_type)
|
||||
|
||||
report = AisReport(
|
||||
ts=ts,
|
||||
source=source_tag,
|
||||
kind=kind, # type: ignore[arg-type]
|
||||
mmsi=mmsi,
|
||||
msg_type=msg_type,
|
||||
channel=channel or None,
|
||||
raw="\n".join(lines),
|
||||
data=normalized,
|
||||
)
|
||||
self._incr("ais_reports")
|
||||
self._incr(f"ais_msg_{msg_type}")
|
||||
return [report]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Normalization helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _normalize_payload(payload: dict[str, Any], kind: str, msg_type: int) -> dict[str, Any]:
|
||||
"""Map pyais fields into a stable subset used by core/publish."""
|
||||
out: dict[str, Any] = {"msg_type": msg_type}
|
||||
|
||||
# Common dynamic fields.
|
||||
for src, dst in (
|
||||
("lat", "lat"), ("lon", "lon"),
|
||||
("speed", "sog"), ("course", "cog"),
|
||||
("heading", "heading"),
|
||||
("status", "nav_status"), ("nav_status", "nav_status"),
|
||||
("turn", "rot"), ("rot", "rot"),
|
||||
):
|
||||
if src in payload and payload[src] is not None:
|
||||
if dst in ("lat", "lon", "sog", "cog", "heading", "rot"):
|
||||
out[dst] = _safe_float(payload[src])
|
||||
else:
|
||||
out[dst] = _safe_int(payload[src])
|
||||
|
||||
# Static fields.
|
||||
for src, dst in (
|
||||
("shipname", "name"), ("name", "name"),
|
||||
("callsign", "callsign"),
|
||||
("imo", "imo"),
|
||||
("ship_type", "ship_type"),
|
||||
("to_bow", "dim_a"), ("to_stern", "dim_b"),
|
||||
("to_port", "dim_c"), ("to_starboard", "dim_d"),
|
||||
("destination", "destination"),
|
||||
("draught", "draught"),
|
||||
("eta", "eta"),
|
||||
("epfd", "epfd"),
|
||||
):
|
||||
if src in payload and payload[src] is not None:
|
||||
val = payload[src]
|
||||
if dst == "name" or dst == "callsign" or dst == "destination":
|
||||
out[dst] = str(val).strip().rstrip("@").strip() or None
|
||||
elif dst == "draught":
|
||||
out[dst] = _safe_float(val)
|
||||
elif dst == "eta":
|
||||
# pyais may return a string, datetime, or nested struct.
|
||||
out[dst] = str(val)
|
||||
else:
|
||||
# Covers IntEnum (ShipType/EpfdType/NavigationStatus) and plain ints.
|
||||
out[dst] = _safe_int(val)
|
||||
|
||||
# AtoN specifics.
|
||||
if kind == "aton":
|
||||
if "aid_type" in payload and payload["aid_type"] is not None:
|
||||
out["aton_type"] = _safe_int(payload["aid_type"])
|
||||
if "virtual_aid" in payload:
|
||||
out["virtual"] = bool(payload["virtual_aid"])
|
||||
if "name" in payload and payload["name"] is not None:
|
||||
out["name"] = str(payload["name"]).strip().rstrip("@").strip() or None
|
||||
|
||||
return out
|
||||
|
||||
|
||||
__all__ = ["AisAssembler", "AisReport"]
|
||||
@@ -0,0 +1,217 @@
|
||||
"""Binary decoders for AIS-catcher Mini UDP telemetry.
|
||||
|
||||
Four independent datagram families, all big-endian:
|
||||
|
||||
1. Slot occupancy bitmap (``slot_udp_*``), fixed 315 bytes.
|
||||
2. Slot detail (``slot_detail_udp_*``), variable length.
|
||||
3. RSSI IQ (``rssi_udp_*``), fixed 8 bytes, ~10 Hz.
|
||||
4. Decode events (``event_udp_*``), variable length.
|
||||
|
||||
All helpers are pure functions that return dataclasses. Any framing error
|
||||
returns ``None`` so ingest can count malformed packets without raising.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
__all__ = [
|
||||
"SlotBitmap",
|
||||
"SlotDetail",
|
||||
"SlotLevel",
|
||||
"RssiIq",
|
||||
"SignalEvent",
|
||||
"SignalEventBatch",
|
||||
"decode_slot_bitmap",
|
||||
"decode_slot_detail",
|
||||
"decode_rssi_iq",
|
||||
"decode_signal_events",
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _channel(byte: int) -> str:
|
||||
"""Map a channel byte to ``"A"``/``"B"``/``"?"``."""
|
||||
if byte in (0x41, 0x42): # 'A' / 'B'
|
||||
return chr(byte)
|
||||
return "?"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1) Slot bitmap (315 bytes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SlotBitmap:
|
||||
channel: str
|
||||
utc_minute: int # floor(epoch_ms / 60000)
|
||||
slots_total: int # always 2250
|
||||
occupied_count: int
|
||||
noise_floor: float # reserved (0)
|
||||
threshold: float # reserved (0)
|
||||
slot0_unix_ms: int
|
||||
first_occupied_unix_ms: int
|
||||
bitmap: bytes # 282 bytes, LSB-first per byte
|
||||
|
||||
def occupied_fraction(self) -> float:
|
||||
if self.slots_total <= 0:
|
||||
return 0.0
|
||||
return self.occupied_count / self.slots_total
|
||||
|
||||
|
||||
# Fixed header layout: 1 + 4 + 2 + 2 + 4 + 4 + 8 + 8 = 33 bytes.
|
||||
_BITMAP_HEADER = struct.Struct(">BIHHffQQ")
|
||||
_BITMAP_TOTAL_LEN = 33 + 282 # 315
|
||||
|
||||
|
||||
def decode_slot_bitmap(data: bytes) -> SlotBitmap | None:
|
||||
if len(data) != _BITMAP_TOTAL_LEN:
|
||||
return None
|
||||
try:
|
||||
ch_b, utc_minute, slots_total, occupied_count, nf, thr, slot0, first_occ = \
|
||||
_BITMAP_HEADER.unpack_from(data, 0)
|
||||
except struct.error:
|
||||
return None
|
||||
return SlotBitmap(
|
||||
channel=_channel(ch_b),
|
||||
utc_minute=utc_minute,
|
||||
slots_total=slots_total,
|
||||
occupied_count=occupied_count,
|
||||
noise_floor=nf,
|
||||
threshold=thr,
|
||||
slot0_unix_ms=slot0,
|
||||
first_occupied_unix_ms=first_occ,
|
||||
bitmap=bytes(data[33:315]),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2) Slot detail (variable)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SlotLevel:
|
||||
slot: int
|
||||
level_db: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SlotDetail:
|
||||
channel: str
|
||||
utc_minute: int
|
||||
slot0_unix_ms: int
|
||||
entries: list[SlotLevel] = field(default_factory=list)
|
||||
|
||||
|
||||
# Header: 1 + 4 + 8 + 2 = 15 bytes.
|
||||
_DETAIL_HEADER = struct.Struct(">BIQH")
|
||||
_DETAIL_ENTRY = struct.Struct(">Hf") # 6 bytes per entry
|
||||
_DETAIL_HEADER_LEN = 15
|
||||
_DETAIL_ENTRY_LEN = 6
|
||||
|
||||
|
||||
def decode_slot_detail(data: bytes) -> SlotDetail | None:
|
||||
if len(data) < _DETAIL_HEADER_LEN:
|
||||
return None
|
||||
try:
|
||||
ch_b, utc_minute, slot0, count = _DETAIL_HEADER.unpack_from(data, 0)
|
||||
except struct.error:
|
||||
return None
|
||||
expected = _DETAIL_HEADER_LEN + count * _DETAIL_ENTRY_LEN
|
||||
if len(data) < expected:
|
||||
return None
|
||||
entries: list[SlotLevel] = []
|
||||
off = _DETAIL_HEADER_LEN
|
||||
for _ in range(count):
|
||||
try:
|
||||
slot_num, level = _DETAIL_ENTRY.unpack_from(data, off)
|
||||
except struct.error:
|
||||
return None
|
||||
entries.append(SlotLevel(slot=slot_num, level_db=level))
|
||||
off += _DETAIL_ENTRY_LEN
|
||||
return SlotDetail(
|
||||
channel=_channel(ch_b),
|
||||
utc_minute=utc_minute,
|
||||
slot0_unix_ms=slot0,
|
||||
entries=entries,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3) RSSI IQ (8 bytes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class RssiIq:
|
||||
power_a_db: float
|
||||
power_b_db: float
|
||||
|
||||
|
||||
_RSSI = struct.Struct(">ff")
|
||||
|
||||
|
||||
def decode_rssi_iq(data: bytes) -> RssiIq | None:
|
||||
if len(data) != 8:
|
||||
return None
|
||||
try:
|
||||
a, b = _RSSI.unpack(data)
|
||||
except struct.error:
|
||||
return None
|
||||
return RssiIq(power_a_db=a, power_b_db=b)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4) Decode events (variable)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SignalEvent:
|
||||
unix_ms: int
|
||||
slot: int
|
||||
mmsi: int
|
||||
level_db: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SignalEventBatch:
|
||||
channel: str
|
||||
events: list[SignalEvent] = field(default_factory=list)
|
||||
|
||||
|
||||
# Header: 1 + 2 = 3 bytes.
|
||||
_EVENTS_HEADER = struct.Struct(">BH")
|
||||
_EVENT_ENTRY = struct.Struct(">QHIf") # 18 bytes per entry
|
||||
_EVENTS_HEADER_LEN = 3
|
||||
_EVENT_ENTRY_LEN = 18
|
||||
|
||||
|
||||
def decode_signal_events(data: bytes) -> SignalEventBatch | None:
|
||||
if len(data) < _EVENTS_HEADER_LEN:
|
||||
return None
|
||||
try:
|
||||
ch_b, count = _EVENTS_HEADER.unpack_from(data, 0)
|
||||
except struct.error:
|
||||
return None
|
||||
expected = _EVENTS_HEADER_LEN + count * _EVENT_ENTRY_LEN
|
||||
if len(data) < expected:
|
||||
return None
|
||||
events: list[SignalEvent] = []
|
||||
off = _EVENTS_HEADER_LEN
|
||||
for _ in range(count):
|
||||
try:
|
||||
unix_ms, slot, mmsi, level = _EVENT_ENTRY.unpack_from(data, off)
|
||||
except struct.error:
|
||||
return None
|
||||
events.append(SignalEvent(unix_ms=unix_ms, slot=slot, mmsi=mmsi, level_db=level))
|
||||
off += _EVENT_ENTRY_LEN
|
||||
return SignalEventBatch(channel=_channel(ch_b), events=events)
|
||||
@@ -0,0 +1,245 @@
|
||||
"""GPS NMEA 0183 parser for RMC / GGA / VTG / GSA / GSV.
|
||||
|
||||
Only fields actually used downstream are extracted; the rest is ignored.
|
||||
GSA / GSV are tracked lightly to surface fix quality indicators.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from ..core.models import GpsFix
|
||||
from ..core.stats import Stats
|
||||
from .nmea_utils import NmeaSentence, parse_sentence
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
KNOTS_PER_KMH = 1.0 / 1.852
|
||||
|
||||
|
||||
def _to_float(v: str | None) -> float | None:
|
||||
if v is None or v == "":
|
||||
return None
|
||||
try:
|
||||
return float(v)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _to_int(v: str | None) -> int | None:
|
||||
if v is None or v == "":
|
||||
return None
|
||||
try:
|
||||
return int(v)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _parse_lat(raw: str, hemi: str) -> float | None:
|
||||
"""NMEA latitude ddmm.mmmm + N/S -> decimal degrees."""
|
||||
if not raw or not hemi:
|
||||
return None
|
||||
try:
|
||||
deg = int(raw[:2])
|
||||
minutes = float(raw[2:])
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
val = deg + minutes / 60.0
|
||||
if hemi.upper() == "S":
|
||||
val = -val
|
||||
return val
|
||||
|
||||
|
||||
def _parse_lon(raw: str, hemi: str) -> float | None:
|
||||
"""NMEA longitude dddmm.mmmm + E/W -> decimal degrees."""
|
||||
if not raw or not hemi:
|
||||
return None
|
||||
try:
|
||||
deg = int(raw[:3])
|
||||
minutes = float(raw[3:])
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
val = deg + minutes / 60.0
|
||||
if hemi.upper() == "W":
|
||||
val = -val
|
||||
return val
|
||||
|
||||
|
||||
class GpsParser:
|
||||
"""Parse GPS NMEA sentences and emit ``GpsFix`` snapshots.
|
||||
|
||||
The parser holds a small amount of state so fields from RMC and GGA can
|
||||
be merged into a single, richer ``GpsFix`` object on each update.
|
||||
"""
|
||||
|
||||
def __init__(self, stats: Stats | None = None) -> None:
|
||||
self._stats = stats
|
||||
self._fix = GpsFix(ts=0.0, source="")
|
||||
self._last_source: str = ""
|
||||
|
||||
def feed(self, source: str, line: str, ts: float | None = None) -> GpsFix | None:
|
||||
"""Parse a single GPS NMEA line. Return a fresh ``GpsFix`` on success."""
|
||||
if ts is None:
|
||||
ts = time.time()
|
||||
sentence = parse_sentence(line)
|
||||
if sentence is None:
|
||||
self._incr("parser_errors")
|
||||
return None
|
||||
if not sentence.checksum_ok:
|
||||
self._incr("parser_checksum_errors")
|
||||
return None
|
||||
if sentence.start != "$":
|
||||
return None
|
||||
|
||||
talker = sentence.talker
|
||||
kind = talker[2:] if len(talker) >= 5 else talker # strip GP/GN/GL/...
|
||||
fields = sentence.fields
|
||||
updated = False
|
||||
|
||||
if kind == "RMC":
|
||||
updated = self._apply_rmc(fields) or updated
|
||||
elif kind == "GGA":
|
||||
updated = self._apply_gga(fields) or updated
|
||||
elif kind == "VTG":
|
||||
updated = self._apply_vtg(fields) or updated
|
||||
elif kind == "GSA":
|
||||
updated = self._apply_gsa(fields) or updated
|
||||
elif kind == "GSV":
|
||||
# GSV only provides satellite count; use field 3 (total sats in view).
|
||||
if len(fields) >= 3:
|
||||
sats = _to_int(fields[2])
|
||||
if sats is not None:
|
||||
self._fix.sats = sats
|
||||
updated = True
|
||||
else:
|
||||
return None
|
||||
|
||||
if not updated:
|
||||
return None
|
||||
|
||||
self._fix.ts = ts
|
||||
self._fix.source = source
|
||||
sentences = set(self._fix.sentences)
|
||||
sentences.add(kind)
|
||||
self._fix.sentences = tuple(sorted(sentences))
|
||||
self._incr("gps_fixes")
|
||||
|
||||
# Return a shallow copy so callers can freeze the moment in time.
|
||||
return GpsFix(
|
||||
ts=self._fix.ts,
|
||||
source=self._fix.source,
|
||||
lat=self._fix.lat,
|
||||
lon=self._fix.lon,
|
||||
sog=self._fix.sog,
|
||||
cog=self._fix.cog,
|
||||
alt=self._fix.alt,
|
||||
fix_quality=self._fix.fix_quality,
|
||||
sats=self._fix.sats,
|
||||
hdop=self._fix.hdop,
|
||||
sentences=self._fix.sentences,
|
||||
)
|
||||
|
||||
# -- sentence applicators ---------------------------------------------
|
||||
|
||||
def _apply_rmc(self, f: tuple[str, ...]) -> bool:
|
||||
# RMC: time, status, lat, N/S, lon, E/W, sog(knots), cog, date, magvar, E/W, [mode]
|
||||
if len(f) < 8:
|
||||
return False
|
||||
status = f[1] if len(f) > 1 else ""
|
||||
if status and status.upper() != "A":
|
||||
# "V" = navigation receiver warning; ignore position but still update
|
||||
# speed/course if present.
|
||||
pass
|
||||
lat = _parse_lat(f[2], f[3])
|
||||
lon = _parse_lon(f[4], f[5])
|
||||
sog = _to_float(f[6])
|
||||
cog = _to_float(f[7])
|
||||
updated = False
|
||||
if lat is not None:
|
||||
self._fix.lat = lat
|
||||
updated = True
|
||||
if lon is not None:
|
||||
self._fix.lon = lon
|
||||
updated = True
|
||||
if sog is not None:
|
||||
self._fix.sog = sog
|
||||
updated = True
|
||||
if cog is not None:
|
||||
self._fix.cog = cog
|
||||
updated = True
|
||||
return updated
|
||||
|
||||
def _apply_gga(self, f: tuple[str, ...]) -> bool:
|
||||
# GGA: time, lat, N/S, lon, E/W, fix_quality, sats, hdop, alt, M, ...
|
||||
if len(f) < 9:
|
||||
return False
|
||||
lat = _parse_lat(f[1], f[2])
|
||||
lon = _parse_lon(f[3], f[4])
|
||||
quality = _to_int(f[5])
|
||||
sats = _to_int(f[6])
|
||||
hdop = _to_float(f[7])
|
||||
alt = _to_float(f[8])
|
||||
updated = False
|
||||
if lat is not None:
|
||||
self._fix.lat = lat
|
||||
updated = True
|
||||
if lon is not None:
|
||||
self._fix.lon = lon
|
||||
updated = True
|
||||
if quality is not None:
|
||||
self._fix.fix_quality = quality
|
||||
updated = True
|
||||
if sats is not None:
|
||||
self._fix.sats = sats
|
||||
updated = True
|
||||
if hdop is not None:
|
||||
self._fix.hdop = hdop
|
||||
updated = True
|
||||
if alt is not None:
|
||||
self._fix.alt = alt
|
||||
updated = True
|
||||
return updated
|
||||
|
||||
def _apply_vtg(self, f: tuple[str, ...]) -> bool:
|
||||
# VTG: cog_true, T, cog_mag, M, sog_knots, N, sog_kmh, K, [mode]
|
||||
if len(f) < 8:
|
||||
return False
|
||||
cog_true = _to_float(f[0])
|
||||
sog_knots = _to_float(f[4])
|
||||
sog_kmh = _to_float(f[6])
|
||||
updated = False
|
||||
if cog_true is not None:
|
||||
self._fix.cog = cog_true
|
||||
updated = True
|
||||
if sog_knots is not None:
|
||||
self._fix.sog = sog_knots
|
||||
updated = True
|
||||
elif sog_kmh is not None:
|
||||
self._fix.sog = sog_kmh * KNOTS_PER_KMH
|
||||
updated = True
|
||||
return updated
|
||||
|
||||
def _apply_gsa(self, f: tuple[str, ...]) -> bool:
|
||||
# GSA: mode, fix_type, sat1..sat12, pdop, hdop, vdop
|
||||
if len(f) < 17:
|
||||
return False
|
||||
# fix_type: 1=no fix, 2=2D, 3=3D
|
||||
fix_type = _to_int(f[1])
|
||||
hdop = _to_float(f[15])
|
||||
updated = False
|
||||
if fix_type is not None:
|
||||
self._fix.fix_quality = fix_type
|
||||
updated = True
|
||||
if hdop is not None:
|
||||
self._fix.hdop = hdop
|
||||
updated = True
|
||||
return updated
|
||||
|
||||
def _incr(self, name: str, n: int = 1) -> None:
|
||||
if self._stats is not None:
|
||||
self._stats.incr(name, n)
|
||||
|
||||
|
||||
__all__ = ["GpsParser"]
|
||||
@@ -0,0 +1,136 @@
|
||||
"""NMEA 0183 helpers: checksum, tokenization, line validation.
|
||||
|
||||
An NMEA 0183 sentence looks like::
|
||||
|
||||
!AIVDM,1,1,,A,15M67FC000G?ufbE`FepT@3n00Sa,0*5B
|
||||
$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
|
||||
|
||||
- The sentence starts with ``!`` (encapsulated, e.g. AIS) or ``$`` (plain).
|
||||
- Everything between the leading character and ``*`` is the payload used
|
||||
for the XOR checksum.
|
||||
- ``*HH`` is a 2-digit hex checksum of the XOR of the payload bytes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Iterable
|
||||
|
||||
|
||||
@dataclass(slots=True, frozen=True)
|
||||
class NmeaSentence:
|
||||
"""Parsed structural view of a single NMEA sentence."""
|
||||
|
||||
raw: str # original line, no trailing CRLF
|
||||
start: str # "!" or "$"
|
||||
talker: str # e.g. "AIVDM", "GPRMC", "BSVDM", "GNGGA"
|
||||
fields: tuple[str, ...] # comma-separated fields after the talker
|
||||
checksum_ok: bool # whether the line had a valid *HH checksum
|
||||
|
||||
|
||||
def strip_eol(line: str) -> str:
|
||||
"""Remove trailing CR/LF if present."""
|
||||
if line.endswith("\r\n"):
|
||||
return line[:-2]
|
||||
if line.endswith("\n") or line.endswith("\r"):
|
||||
return line[:-1]
|
||||
return line
|
||||
|
||||
|
||||
def compute_checksum(payload: str) -> int:
|
||||
"""XOR checksum of every byte in ``payload`` (no start char, no '*')."""
|
||||
cs = 0
|
||||
for ch in payload.encode("ascii", errors="replace"):
|
||||
cs ^= ch
|
||||
return cs
|
||||
|
||||
|
||||
def parse_sentence(line: str) -> NmeaSentence | None:
|
||||
"""Parse a single NMEA line; return None for clearly-malformed input.
|
||||
|
||||
The ``checksum_ok`` flag indicates whether the ``*HH`` checksum matched.
|
||||
Lines without a checksum segment are returned with ``checksum_ok=False``.
|
||||
"""
|
||||
s = strip_eol(line).strip()
|
||||
if not s:
|
||||
return None
|
||||
if s[0] not in ("!", "$"):
|
||||
return None
|
||||
if len(s) < 4:
|
||||
return None
|
||||
|
||||
# Split payload / checksum.
|
||||
star = s.rfind("*")
|
||||
checksum_ok = False
|
||||
if star != -1 and len(s) - star >= 3:
|
||||
payload = s[1:star]
|
||||
csum_str = s[star + 1 : star + 3]
|
||||
try:
|
||||
csum = int(csum_str, 16)
|
||||
except ValueError:
|
||||
csum = -1
|
||||
checksum_ok = csum == compute_checksum(payload)
|
||||
body = payload
|
||||
else:
|
||||
body = s[1:]
|
||||
checksum_ok = False
|
||||
|
||||
parts = body.split(",")
|
||||
if not parts or not parts[0]:
|
||||
return None
|
||||
talker = parts[0]
|
||||
fields = tuple(parts[1:])
|
||||
return NmeaSentence(raw=s, start=s[0], talker=talker, fields=fields, checksum_ok=checksum_ok)
|
||||
|
||||
|
||||
def split_lines(chunk: bytes | str) -> list[str]:
|
||||
"""Split a raw chunk into NMEA lines, dropping empty lines.
|
||||
|
||||
UART/UDP may deliver several sentences in one read; handles CRLF/LF/CR.
|
||||
"""
|
||||
if isinstance(chunk, (bytes, bytearray)):
|
||||
try:
|
||||
text = chunk.decode("ascii", errors="replace")
|
||||
except Exception:
|
||||
text = chunk.decode("latin-1", errors="replace")
|
||||
else:
|
||||
text = chunk
|
||||
out: list[str] = []
|
||||
for part in text.replace("\r\n", "\n").replace("\r", "\n").split("\n"):
|
||||
p = part.strip()
|
||||
if p:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def classify_kind(talker: str, start: str) -> str:
|
||||
"""Rough classifier used by ingest to tag a raw frame.
|
||||
|
||||
Returns one of: ``"ais"``, ``"gps"``, ``"other"``.
|
||||
"""
|
||||
if start == "!":
|
||||
# AIVDM / AIVDO / BSVDM / ANVDM / etc.
|
||||
if talker.endswith("VDM") or talker.endswith("VDO"):
|
||||
return "ais"
|
||||
return "other"
|
||||
# $-prefixed sentences are typically from GNSS receivers.
|
||||
if talker.startswith(("GP", "GN", "GL", "GA", "BD", "GB", "II")):
|
||||
return "gps"
|
||||
return "other"
|
||||
|
||||
|
||||
def lines_from_iter(source: Iterable[bytes | str]) -> Iterable[str]:
|
||||
for chunk in source:
|
||||
for ln in split_lines(chunk):
|
||||
yield ln
|
||||
|
||||
|
||||
__all__ = [
|
||||
"NmeaSentence",
|
||||
"compute_checksum",
|
||||
"parse_sentence",
|
||||
"split_lines",
|
||||
"classify_kind",
|
||||
"strip_eol",
|
||||
"lines_from_iter",
|
||||
]
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Radio telemetry parser.
|
||||
|
||||
The radio telemetry stream is expected to arrive as UDP datagrams carrying
|
||||
JSON objects describing TDMA slot / RSSI / SNR data. The exact schema is
|
||||
defined by the radio front-end; this parser is intentionally permissive.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from ..core.models import RadioReport
|
||||
from ..core.stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RadioParser:
|
||||
"""Decode one UDP datagram of radio telemetry."""
|
||||
|
||||
def __init__(self, stats: Stats | None = None) -> None:
|
||||
self._stats = stats
|
||||
|
||||
def feed(self, source: str, data: bytes | str, ts: float | None = None) -> RadioReport | None:
|
||||
if ts is None:
|
||||
ts = time.time()
|
||||
try:
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
else:
|
||||
text = data
|
||||
payload = json.loads(text)
|
||||
except Exception:
|
||||
if self._stats is not None:
|
||||
self._stats.incr("radio_parser_errors")
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
if self._stats is not None:
|
||||
self._stats.incr("radio_parser_errors")
|
||||
return None
|
||||
|
||||
report = RadioReport(
|
||||
ts=ts,
|
||||
source=source,
|
||||
channel=_as_str(payload.get("channel")),
|
||||
rssi=_as_float(payload.get("rssi")),
|
||||
snr=_as_float(payload.get("snr")),
|
||||
slot=_as_int(payload.get("slot")),
|
||||
raw=payload,
|
||||
)
|
||||
if self._stats is not None:
|
||||
self._stats.incr("radio_reports")
|
||||
return report
|
||||
|
||||
|
||||
def _as_float(v: Any) -> float | None:
|
||||
try:
|
||||
return float(v) if v is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _as_int(v: Any) -> int | None:
|
||||
try:
|
||||
return int(v) if v is not None else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _as_str(v: Any) -> str | None:
|
||||
if v is None:
|
||||
return None
|
||||
return str(v)
|
||||
|
||||
|
||||
__all__ = ["RadioParser"]
|
||||
@@ -0,0 +1 @@
|
||||
"""Publish subsystem: REST + WS, UDP JSON events, UDP raw NMEA fan-out, UDP TX outbox."""
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,47 @@
|
||||
"""Bounded asyncio queue helpers with a drop-oldest overflow policy.
|
||||
|
||||
Whenever a put_nowait fails with ``QueueFull``, one item is popped from the
|
||||
head of the queue and the put is retried. Each drop is counted in the
|
||||
optional ``Stats`` instance under a caller-supplied counter name.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import TypeVar
|
||||
|
||||
from ..core.stats import Stats
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
def put_drop_oldest(
|
||||
queue: "asyncio.Queue[T]",
|
||||
item: T,
|
||||
stats: Stats | None = None,
|
||||
drop_counter: str | None = None,
|
||||
) -> bool:
|
||||
"""Put ``item`` into ``queue``; drop the oldest item on overflow.
|
||||
|
||||
Returns True if the item was stored, False only if the queue has no
|
||||
capacity at all (maxsize=0 semantics, which would block — guarded here).
|
||||
"""
|
||||
while True:
|
||||
try:
|
||||
queue.put_nowait(item)
|
||||
return True
|
||||
except asyncio.QueueFull:
|
||||
try:
|
||||
queue.get_nowait()
|
||||
except asyncio.QueueEmpty:
|
||||
return False
|
||||
if stats is not None and drop_counter is not None:
|
||||
stats.incr(drop_counter)
|
||||
|
||||
|
||||
def make_queue(maxsize: int) -> "asyncio.Queue[object]":
|
||||
"""Create a bounded queue with the configured maxsize."""
|
||||
return asyncio.Queue(maxsize=max(1, maxsize))
|
||||
|
||||
|
||||
__all__ = ["put_drop_oldest", "make_queue"]
|
||||
@@ -0,0 +1,253 @@
|
||||
"""REST endpoints under ``/api/v1/*``."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from aiohttp import web
|
||||
|
||||
from ..core.models import TxMessage
|
||||
from ..logging_setup import tail_logs
|
||||
from ..parser.nmea_utils import parse_sentence
|
||||
from ..storage import queries
|
||||
from ..version import __version__
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ctx(request: web.Request):
|
||||
return request.app["ctx"]
|
||||
|
||||
|
||||
def _json(data: Any, status: int = 200) -> web.Response:
|
||||
return web.json_response(data, status=status)
|
||||
|
||||
|
||||
def _bad_request(reason: str) -> web.Response:
|
||||
return _json({"error": reason}, status=400)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET handlers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def health(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json({
|
||||
"status": "ok",
|
||||
"version": __version__,
|
||||
"uptime_sec": round(time.time() - ctx.stats.started_at, 3),
|
||||
})
|
||||
|
||||
|
||||
async def stats(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.stats.snapshot())
|
||||
|
||||
|
||||
async def ownship(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.state.snapshot_ownship())
|
||||
|
||||
|
||||
async def vessels(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
since = _opt_float(request.query.get("since"))
|
||||
limit = _opt_int(request.query.get("limit"))
|
||||
return _json(ctx.state.snapshot_vessels(since=since, limit=limit))
|
||||
|
||||
|
||||
async def vessel_by_mmsi(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
try:
|
||||
mmsi = int(request.match_info["mmsi"])
|
||||
except ValueError:
|
||||
return _bad_request("invalid mmsi")
|
||||
v = ctx.state.get_vessel(mmsi)
|
||||
if v is None:
|
||||
return _json({"error": "not found", "mmsi": mmsi}, status=404)
|
||||
return _json(v)
|
||||
|
||||
|
||||
async def base_stations(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.state.snapshot_base_stations())
|
||||
|
||||
|
||||
async def atons(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.state.snapshot_atons())
|
||||
|
||||
|
||||
async def slots(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.state.snapshot_slots())
|
||||
|
||||
|
||||
async def radio(request: web.Request) -> web.Response:
|
||||
"""Latest RSSI pair + optional per-channel last detail."""
|
||||
ctx = _ctx(request)
|
||||
return _json(ctx.state.snapshot_radio())
|
||||
|
||||
|
||||
async def logs(request: web.Request) -> web.Response:
|
||||
"""Service log tail. NOT raw NMEA; see /nmea/tail for that."""
|
||||
level = request.query.get("level")
|
||||
limit = _opt_int(request.query.get("limit")) or 200
|
||||
since = _opt_float(request.query.get("since"))
|
||||
return _json(tail_logs(limit=limit, level=level, since=since))
|
||||
|
||||
|
||||
async def nmea_tail_handler(request: web.Request) -> web.Response:
|
||||
"""Raw NMEA tail. Reads from in-memory ring first; falls back to SQLite."""
|
||||
ctx = _ctx(request)
|
||||
source = request.query.get("source") # "ais" | "gps" | "radio"
|
||||
limit = _opt_int(request.query.get("limit")) or 200
|
||||
|
||||
# In-memory ring (always maintained by the raw tap).
|
||||
frames = ctx.raw_tap.tail(limit=limit, kind=source)
|
||||
if frames:
|
||||
return _json([
|
||||
{"ts": f.ts, "source": f.source, "kind": f.kind, "line": f.line}
|
||||
for f in frames
|
||||
])
|
||||
|
||||
# Fall back to SQLite raw_nmea if enabled.
|
||||
if ctx.db is not None:
|
||||
try:
|
||||
rows = await queries.nmea_tail(ctx.db, source_kind=source, limit=limit)
|
||||
return _json(rows)
|
||||
except Exception:
|
||||
log.exception("nmea_tail DB query failed")
|
||||
return _json([])
|
||||
|
||||
|
||||
async def history_positions(request: web.Request) -> web.Response:
|
||||
ctx = _ctx(request)
|
||||
if ctx.db is None:
|
||||
return _json({"error": "history storage disabled"}, status=503)
|
||||
|
||||
mmsi_raw = request.query.get("mmsi")
|
||||
if not mmsi_raw:
|
||||
return _bad_request("missing mmsi")
|
||||
try:
|
||||
mmsi = int(mmsi_raw)
|
||||
except ValueError:
|
||||
return _bad_request("invalid mmsi")
|
||||
|
||||
ts_from = _opt_float(request.query.get("from"))
|
||||
ts_to = _opt_float(request.query.get("to"))
|
||||
limit = _opt_int(request.query.get("limit")) or 1000
|
||||
|
||||
try:
|
||||
rows = await queries.history_positions(
|
||||
ctx.db, mmsi, ts_from=ts_from, ts_to=ts_to, limit=limit,
|
||||
)
|
||||
except Exception:
|
||||
log.exception("history_positions failed for mmsi=%d", mmsi)
|
||||
return _json({"error": "query failed"}, status=500)
|
||||
return _json(rows)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# POST handlers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def tx(request: web.Request) -> web.Response:
|
||||
"""Inject NMEA sentence(s) into the TX outbox for the SPI bridge."""
|
||||
ctx = _ctx(request)
|
||||
try:
|
||||
payload = await request.json()
|
||||
except Exception:
|
||||
return _bad_request("invalid json body")
|
||||
|
||||
lines: list[str] = []
|
||||
if isinstance(payload, dict):
|
||||
if "payload" in payload and isinstance(payload["payload"], str):
|
||||
lines = [payload["payload"]]
|
||||
elif "nmea" in payload and isinstance(payload["nmea"], list):
|
||||
lines = [str(x) for x in payload["nmea"] if isinstance(x, str)]
|
||||
if not lines:
|
||||
return _bad_request("expected {'payload': str} or {'nmea': [str, ...]}")
|
||||
|
||||
accepted: list[str] = []
|
||||
rejected: list[dict[str, str]] = []
|
||||
for raw in lines:
|
||||
s = parse_sentence(raw)
|
||||
if s is None or not s.checksum_ok:
|
||||
rejected.append({"line": raw, "reason": "invalid nmea or checksum"})
|
||||
continue
|
||||
msg = TxMessage(ts=time.time(), line=s.raw, origin="rest:POST /api/v1/tx")
|
||||
try:
|
||||
ctx.tx_outbox.put_nowait(msg)
|
||||
accepted.append(s.raw)
|
||||
except asyncio.QueueFull:
|
||||
ctx.stats.incr("tx_outbox_dropped_rest")
|
||||
return _json(
|
||||
{"error": "outbox full", "accepted": accepted, "rejected": rejected},
|
||||
status=503,
|
||||
)
|
||||
|
||||
ctx.stats.incr("tx_submitted", len(accepted))
|
||||
return _json({
|
||||
"queued": len(accepted),
|
||||
"rejected": rejected,
|
||||
"queue_depth": ctx.tx_outbox.qsize(),
|
||||
})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Utils
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _opt_int(v: str | None) -> int | None:
|
||||
if v is None or v == "":
|
||||
return None
|
||||
try:
|
||||
return int(v)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _opt_float(v: str | None) -> float | None:
|
||||
if v is None or v == "":
|
||||
return None
|
||||
try:
|
||||
return float(v)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Route registration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def register_routes(app: web.Application) -> None:
|
||||
app.router.add_get("/api/v1/health", health)
|
||||
app.router.add_get("/api/v1/stats", stats)
|
||||
app.router.add_get("/api/v1/ownship", ownship)
|
||||
app.router.add_get("/api/v1/vessels", vessels)
|
||||
app.router.add_get(r"/api/v1/vessels/{mmsi:\d+}", vessel_by_mmsi)
|
||||
app.router.add_get("/api/v1/base_stations", base_stations)
|
||||
app.router.add_get("/api/v1/atons", atons)
|
||||
app.router.add_get("/api/v1/slots", slots)
|
||||
app.router.add_get("/api/v1/radio", radio)
|
||||
app.router.add_get("/api/v1/logs", logs)
|
||||
app.router.add_get("/api/v1/nmea/tail", nmea_tail_handler)
|
||||
app.router.add_get("/api/v1/history/positions", history_positions)
|
||||
app.router.add_post("/api/v1/tx", tx)
|
||||
|
||||
|
||||
__all__ = ["register_routes"]
|
||||
@@ -0,0 +1,73 @@
|
||||
"""UDP JSON event publisher on ``127.0.0.1:7001`` (configurable).
|
||||
|
||||
Contract: one serialized ``Event`` per UDP datagram. Events larger than
|
||||
``max_datagram_bytes`` are dropped and counted in
|
||||
``stats.udp_events_oversize`` (no fragmentation).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..config import UdpEventsCfg
|
||||
from ..core.bus import EventBus
|
||||
from ..core.models import Event
|
||||
from ..core.stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class UdpEventsPublisher:
|
||||
def __init__(self, cfg: UdpEventsCfg, bus: EventBus, stats: Stats) -> None:
|
||||
self._cfg = cfg
|
||||
self._bus = bus
|
||||
self._stats = stats
|
||||
self._transport: asyncio.DatagramTransport | None = None
|
||||
self._sub: asyncio.Queue[Event] | None = None
|
||||
|
||||
async def run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
# Ephemeral sender socket; we only *send* to a fixed target.
|
||||
transport, _ = await loop.create_datagram_endpoint(
|
||||
lambda: asyncio.DatagramProtocol(),
|
||||
remote_addr=(self._cfg.host, self._cfg.port),
|
||||
)
|
||||
self._transport = transport
|
||||
self._sub = self._bus.subscribe(maxsize=1024)
|
||||
log.info("udp_events publisher -> %s:%d", self._cfg.host, self._cfg.port)
|
||||
try:
|
||||
while True:
|
||||
ev = await self._sub.get()
|
||||
self._send(ev)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
if self._sub is not None:
|
||||
self._bus.unsubscribe(self._sub)
|
||||
self._sub = None
|
||||
if self._transport is not None:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
def _send(self, ev: Event) -> None:
|
||||
try:
|
||||
blob = json.dumps(ev.to_dict(), ensure_ascii=False, separators=(",", ":")).encode("utf-8")
|
||||
except Exception:
|
||||
self._stats.incr("udp_events_serialize_errors")
|
||||
return
|
||||
if len(blob) > self._cfg.max_datagram_bytes:
|
||||
self._stats.incr("udp_events_oversize")
|
||||
return
|
||||
try:
|
||||
assert self._transport is not None
|
||||
self._transport.sendto(blob)
|
||||
self._stats.incr("udp_events_sent")
|
||||
except Exception:
|
||||
self._stats.incr("udp_events_send_errors")
|
||||
log.debug("udp_events send failed", exc_info=True)
|
||||
|
||||
|
||||
__all__ = ["UdpEventsPublisher"]
|
||||
@@ -0,0 +1,68 @@
|
||||
"""UDP raw NMEA fan-out on ``127.0.0.1:6007`` (configurable).
|
||||
|
||||
Contract: one NMEA sentence = one UDP datagram.
|
||||
Sentence is sent as ``<line>\\r\\n`` in ASCII/UTF-8, including the leading
|
||||
``!`` or ``$`` and the trailing ``*HH`` checksum. No concatenation or
|
||||
fragmentation is ever performed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from ..config import UdpNmeaCfg
|
||||
from ..core.models import RawFrame
|
||||
from ..core.stats import Stats
|
||||
from .queues import put_drop_oldest
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class UdpNmeaPublisher:
|
||||
def __init__(
|
||||
self,
|
||||
cfg: UdpNmeaCfg,
|
||||
stats: Stats,
|
||||
queue: "asyncio.Queue[RawFrame]",
|
||||
) -> None:
|
||||
self._cfg = cfg
|
||||
self._stats = stats
|
||||
self._queue = queue
|
||||
self._transport: asyncio.DatagramTransport | None = None
|
||||
|
||||
async def run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
transport, _ = await loop.create_datagram_endpoint(
|
||||
lambda: asyncio.DatagramProtocol(),
|
||||
remote_addr=(self._cfg.host, self._cfg.port),
|
||||
)
|
||||
self._transport = transport
|
||||
log.info("udp_nmea fan-out -> %s:%d", self._cfg.host, self._cfg.port)
|
||||
try:
|
||||
while True:
|
||||
frame = await self._queue.get()
|
||||
self._send(frame)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
if self._transport is not None:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
def _send(self, frame: RawFrame) -> None:
|
||||
line = frame.line
|
||||
if not line:
|
||||
return
|
||||
# Preserve newline-terminated wire format expected by SPI bridge.
|
||||
payload = (line + "\r\n").encode("ascii", errors="replace")
|
||||
try:
|
||||
assert self._transport is not None
|
||||
self._transport.sendto(payload)
|
||||
self._stats.incr("udp_nmea_sent")
|
||||
except Exception:
|
||||
self._stats.incr("udp_nmea_send_errors")
|
||||
log.debug("udp_nmea send failed", exc_info=True)
|
||||
|
||||
|
||||
__all__ = ["UdpNmeaPublisher"]
|
||||
@@ -0,0 +1,71 @@
|
||||
"""UDP TX outbox publisher on ``127.0.0.1:6010`` (configurable).
|
||||
|
||||
This is a **publish-only** channel. ais_hub writes NMEA sentences here
|
||||
that the SPI bridge should transmit. Content sources:
|
||||
|
||||
1. REST ``POST /api/v1/tx`` — external injection (from BLE/web, etc).
|
||||
2. Internal emitters (e.g. ownship AIS position reports).
|
||||
|
||||
Contract: one NMEA sentence = one UDP datagram (``<line>\\r\\n``, ASCII).
|
||||
Queue is bounded with drop-oldest; drops are counted in
|
||||
``stats.tx_outbox_dropped``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from ..config import UdpTxOutboxCfg
|
||||
from ..core.models import TxMessage
|
||||
from ..core.stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class UdpTxOutboxPublisher:
|
||||
def __init__(
|
||||
self,
|
||||
cfg: UdpTxOutboxCfg,
|
||||
stats: Stats,
|
||||
outbox: "asyncio.Queue[TxMessage]",
|
||||
) -> None:
|
||||
self._cfg = cfg
|
||||
self._stats = stats
|
||||
self._outbox = outbox
|
||||
self._transport: asyncio.DatagramTransport | None = None
|
||||
|
||||
async def run(self) -> None:
|
||||
loop = asyncio.get_running_loop()
|
||||
transport, _ = await loop.create_datagram_endpoint(
|
||||
lambda: asyncio.DatagramProtocol(),
|
||||
remote_addr=(self._cfg.host, self._cfg.port),
|
||||
)
|
||||
self._transport = transport
|
||||
log.info("udp_tx_outbox -> %s:%d", self._cfg.host, self._cfg.port)
|
||||
try:
|
||||
while True:
|
||||
msg = await self._outbox.get()
|
||||
self._send(msg)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
if self._transport is not None:
|
||||
self._transport.close()
|
||||
self._transport = None
|
||||
|
||||
def _send(self, msg: TxMessage) -> None:
|
||||
line = msg.line
|
||||
if not line:
|
||||
return
|
||||
payload = (line + "\r\n").encode("ascii", errors="replace")
|
||||
try:
|
||||
assert self._transport is not None
|
||||
self._transport.sendto(payload)
|
||||
self._stats.incr("tx_outbox_sent")
|
||||
except Exception:
|
||||
self._stats.incr("tx_outbox_send_errors")
|
||||
log.debug("tx_outbox send failed", exc_info=True)
|
||||
|
||||
|
||||
__all__ = ["UdpTxOutboxPublisher"]
|
||||
@@ -0,0 +1,93 @@
|
||||
"""WebSocket hub.
|
||||
|
||||
- One aiohttp WebSocket handler under ``/ws``.
|
||||
- Each client gets its own bounded queue subscribed to the event bus.
|
||||
- On connect, the client receives a snapshot of the current state
|
||||
(ownship + vessels + base_stations + atons + slots + stats) as a single
|
||||
``state.snapshot`` event, then a continuous stream of bus events.
|
||||
- Slow consumers: drop-oldest policy on their queue + ``ws_dropped``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from aiohttp import WSMsgType, web
|
||||
|
||||
from ..core.models import Event
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def ws_handler(request: web.Request) -> web.WebSocketResponse:
|
||||
ctx = request.app["ctx"]
|
||||
|
||||
ws = web.WebSocketResponse(heartbeat=30.0, max_msg_size=4 * 1024 * 1024)
|
||||
await ws.prepare(request)
|
||||
|
||||
client_queue: asyncio.Queue[Event] = ctx.bus.subscribe(maxsize=ctx.cfg.queues.ws_client)
|
||||
ctx.stats.incr("ws_clients_total")
|
||||
ctx.stats.set_gauge("ws_clients", float(ctx.bus.subscribers))
|
||||
|
||||
# Send initial snapshot.
|
||||
try:
|
||||
snapshot = {
|
||||
"ownship": ctx.state.snapshot_ownship(),
|
||||
"vessels": ctx.state.snapshot_vessels(),
|
||||
"base_stations": ctx.state.snapshot_base_stations(),
|
||||
"atons": ctx.state.snapshot_atons(),
|
||||
"slots": ctx.state.snapshot_slots(),
|
||||
"stats": ctx.stats.snapshot(),
|
||||
}
|
||||
await ws.send_json({"type": "state.snapshot", "ts": time.time(), "data": snapshot})
|
||||
except Exception:
|
||||
log.exception("ws initial snapshot failed")
|
||||
|
||||
sender = asyncio.create_task(_sender(ws, client_queue, ctx), name="ws.sender")
|
||||
try:
|
||||
async for msg in ws:
|
||||
# The WS is publish-only from our side; ignore client messages
|
||||
# except PING/CLOSE which aiohttp handles automatically.
|
||||
if msg.type == WSMsgType.ERROR:
|
||||
log.warning("ws error: %s", ws.exception())
|
||||
break
|
||||
finally:
|
||||
sender.cancel()
|
||||
try:
|
||||
await sender
|
||||
except Exception:
|
||||
pass
|
||||
ctx.bus.unsubscribe(client_queue)
|
||||
ctx.stats.set_gauge("ws_clients", float(ctx.bus.subscribers))
|
||||
|
||||
return ws
|
||||
|
||||
|
||||
async def _sender(ws: web.WebSocketResponse, queue: asyncio.Queue[Event], ctx) -> None:
|
||||
try:
|
||||
while not ws.closed:
|
||||
try:
|
||||
ev = await queue.get()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
try:
|
||||
await ws.send_str(json.dumps(ev.to_dict(), ensure_ascii=False))
|
||||
except ConnectionResetError:
|
||||
return
|
||||
except Exception:
|
||||
ctx.stats.incr("ws_send_errors")
|
||||
log.debug("ws send failed", exc_info=True)
|
||||
return
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
|
||||
|
||||
def register_routes(app: web.Application) -> None:
|
||||
app.router.add_get("/ws", ws_handler)
|
||||
|
||||
|
||||
__all__ = ["register_routes", "ws_handler"]
|
||||
@@ -0,0 +1 @@
|
||||
"""SQLite storage: schema, batched writer, retention, read queries."""
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,162 @@
|
||||
"""aiosqlite connection helper: WAL pragmas, schema bootstrap, migrations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import aiosqlite
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
|
||||
|
||||
SCHEMA_SQL: tuple[str, ...] = (
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS vessel_static (
|
||||
mmsi INTEGER PRIMARY KEY,
|
||||
name TEXT,
|
||||
callsign TEXT,
|
||||
imo INTEGER,
|
||||
ship_type INTEGER,
|
||||
dim_a INTEGER,
|
||||
dim_b INTEGER,
|
||||
dim_c INTEGER,
|
||||
dim_d INTEGER,
|
||||
eta TEXT,
|
||||
draught REAL,
|
||||
destination TEXT,
|
||||
updated_at REAL NOT NULL
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS ais_dynamic (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
mmsi INTEGER NOT NULL,
|
||||
ts REAL NOT NULL,
|
||||
lat REAL,
|
||||
lon REAL,
|
||||
sog REAL,
|
||||
cog REAL,
|
||||
heading REAL,
|
||||
nav_status INTEGER,
|
||||
rot REAL,
|
||||
raw_msg_type INTEGER
|
||||
)
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_ais_dynamic_mmsi_ts ON ais_dynamic(mmsi, ts)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_ais_dynamic_ts ON ais_dynamic(ts)",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS gps_fix (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts REAL NOT NULL,
|
||||
lat REAL,
|
||||
lon REAL,
|
||||
sog REAL,
|
||||
cog REAL,
|
||||
alt REAL,
|
||||
fix_quality INTEGER,
|
||||
sats INTEGER,
|
||||
hdop REAL
|
||||
)
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_gps_fix_ts ON gps_fix(ts)",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS raw_nmea (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts REAL NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
kind TEXT NOT NULL,
|
||||
line TEXT NOT NULL
|
||||
)
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_raw_nmea_ts ON raw_nmea(ts)",
|
||||
"CREATE INDEX IF NOT EXISTS ix_raw_nmea_kind_ts ON raw_nmea(kind, ts)",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS base_station (
|
||||
mmsi INTEGER PRIMARY KEY,
|
||||
ts REAL NOT NULL,
|
||||
lat REAL,
|
||||
lon REAL,
|
||||
epfd INTEGER,
|
||||
updated_at REAL NOT NULL
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS aton (
|
||||
mmsi INTEGER PRIMARY KEY,
|
||||
ts REAL NOT NULL,
|
||||
lat REAL,
|
||||
lon REAL,
|
||||
aton_type INTEGER,
|
||||
name TEXT,
|
||||
virtual INTEGER,
|
||||
updated_at REAL NOT NULL
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS radio_telemetry (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts REAL NOT NULL,
|
||||
channel TEXT,
|
||||
rssi REAL,
|
||||
snr REAL,
|
||||
slot INTEGER,
|
||||
raw_json TEXT
|
||||
)
|
||||
""",
|
||||
"CREATE INDEX IF NOT EXISTS ix_radio_telemetry_ts ON radio_telemetry(ts)",
|
||||
)
|
||||
|
||||
|
||||
async def connect(path: str) -> aiosqlite.Connection:
|
||||
"""Open the SQLite DB with WAL pragmas and ensure the schema exists."""
|
||||
p = Path(path)
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
conn = await aiosqlite.connect(path, timeout=5.0)
|
||||
# Pragmas — WAL + sane defaults for embedded devices.
|
||||
await conn.execute("PRAGMA journal_mode=WAL")
|
||||
await conn.execute("PRAGMA synchronous=NORMAL")
|
||||
await conn.execute("PRAGMA busy_timeout=5000")
|
||||
await conn.execute("PRAGMA foreign_keys=ON")
|
||||
await conn.execute("PRAGMA temp_store=MEMORY")
|
||||
|
||||
await _ensure_schema(conn)
|
||||
return conn
|
||||
|
||||
|
||||
async def _ensure_schema(conn: aiosqlite.Connection) -> None:
|
||||
for stmt in SCHEMA_SQL:
|
||||
await conn.execute(stmt)
|
||||
await conn.execute(
|
||||
"INSERT OR REPLACE INTO schema_meta(key, value) VALUES('version', ?)",
|
||||
(str(SCHEMA_VERSION),),
|
||||
)
|
||||
await conn.commit()
|
||||
|
||||
|
||||
async def close(conn: aiosqlite.Connection | None) -> None:
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
await conn.commit()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
await conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
__all__ = ["connect", "close", "SCHEMA_VERSION", "SCHEMA_SQL"]
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Read-only queries used by the REST layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import aiosqlite
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def history_positions(
|
||||
conn: aiosqlite.Connection,
|
||||
mmsi: int,
|
||||
ts_from: float | None = None,
|
||||
ts_to: float | None = None,
|
||||
limit: int = 1000,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return AIS dynamic position history for ``mmsi``, ordered by ts desc."""
|
||||
where = ["mmsi = ?"]
|
||||
args: list[Any] = [mmsi]
|
||||
if ts_from is not None:
|
||||
where.append("ts >= ?")
|
||||
args.append(ts_from)
|
||||
if ts_to is not None:
|
||||
where.append("ts <= ?")
|
||||
args.append(ts_to)
|
||||
sql = (
|
||||
"SELECT ts, lat, lon, sog, cog, heading, nav_status, rot, raw_msg_type "
|
||||
"FROM ais_dynamic WHERE " + " AND ".join(where)
|
||||
+ " ORDER BY ts DESC LIMIT ?"
|
||||
)
|
||||
args.append(max(1, min(limit, 100_000)))
|
||||
|
||||
async with conn.execute(sql, args) as cur:
|
||||
rows = await cur.fetchall()
|
||||
|
||||
return [
|
||||
{
|
||||
"ts": r[0], "lat": r[1], "lon": r[2],
|
||||
"sog": r[3], "cog": r[4], "heading": r[5],
|
||||
"nav_status": r[6], "rot": r[7], "msg_type": r[8],
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
async def nmea_tail(
|
||||
conn: aiosqlite.Connection,
|
||||
source_kind: str | None = None,
|
||||
limit: int = 200,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return most recent raw NMEA rows (if persistence enabled)."""
|
||||
where = []
|
||||
args: list[Any] = []
|
||||
if source_kind:
|
||||
where.append("kind = ?")
|
||||
args.append(source_kind)
|
||||
sql = "SELECT ts, source, kind, line FROM raw_nmea"
|
||||
if where:
|
||||
sql += " WHERE " + " AND ".join(where)
|
||||
sql += " ORDER BY ts DESC LIMIT ?"
|
||||
args.append(max(1, min(limit, 10_000)))
|
||||
|
||||
async with conn.execute(sql, args) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [{"ts": r[0], "source": r[1], "kind": r[2], "line": r[3]} for r in rows]
|
||||
|
||||
|
||||
async def load_vessel_static(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
|
||||
"""Load all persisted vessel static/voyage rows (used for warm start)."""
|
||||
sql = (
|
||||
"SELECT mmsi, name, callsign, imo, ship_type, "
|
||||
" dim_a, dim_b, dim_c, dim_d, "
|
||||
" eta, draught, destination, updated_at "
|
||||
"FROM vessel_static"
|
||||
)
|
||||
async with conn.execute(sql) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [
|
||||
{
|
||||
"mmsi": r[0], "name": r[1], "callsign": r[2], "imo": r[3],
|
||||
"ship_type": r[4], "dim_a": r[5], "dim_b": r[6],
|
||||
"dim_c": r[7], "dim_d": r[8],
|
||||
"eta": r[9], "draught": r[10], "destination": r[11],
|
||||
"updated_at": r[12],
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
async def load_base_stations(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
|
||||
sql = "SELECT mmsi, ts, lat, lon, epfd FROM base_station"
|
||||
async with conn.execute(sql) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [
|
||||
{"mmsi": r[0], "ts": r[1], "lat": r[2], "lon": r[3], "epfd": r[4]}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
async def load_atons(conn: aiosqlite.Connection) -> list[dict[str, Any]]:
|
||||
sql = "SELECT mmsi, ts, lat, lon, aton_type, name, virtual FROM aton"
|
||||
async with conn.execute(sql) as cur:
|
||||
rows = await cur.fetchall()
|
||||
return [
|
||||
{
|
||||
"mmsi": r[0], "ts": r[1], "lat": r[2], "lon": r[3],
|
||||
"aton_type": r[4], "name": r[5],
|
||||
"virtual": None if r[6] is None else bool(r[6]),
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
__all__ = [
|
||||
"history_positions",
|
||||
"nmea_tail",
|
||||
"load_vessel_static",
|
||||
"load_base_stations",
|
||||
"load_atons",
|
||||
]
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Periodic retention cleanup for history tables."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sqlite3
|
||||
import time
|
||||
|
||||
import aiosqlite
|
||||
|
||||
from ..config import StorageCfg
|
||||
from ..core.stats import Stats
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def run_retention(conn: aiosqlite.Connection, cfg: StorageCfg, stats: Stats) -> None:
|
||||
"""Background task: periodically delete rows older than retention limits."""
|
||||
interval = max(60, cfg.retention_interval_sec)
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(interval)
|
||||
await cleanup_once(conn, cfg, stats)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
log.exception("retention sweep failed")
|
||||
stats.incr("retention_errors")
|
||||
|
||||
|
||||
async def cleanup_once(conn: aiosqlite.Connection, cfg: StorageCfg, stats: Stats) -> None:
|
||||
"""Run one retention sweep; caller ensures this is not re-entrant."""
|
||||
now = time.time()
|
||||
rows_total = 0
|
||||
|
||||
targets = (
|
||||
("ais_dynamic", cfg.retention.ais_dynamic_days),
|
||||
("gps_fix", cfg.retention.gps_fix_days),
|
||||
("raw_nmea", cfg.retention.raw_nmea_days),
|
||||
("radio_telemetry", cfg.retention.radio_telemetry_days),
|
||||
)
|
||||
for table, days in targets:
|
||||
if days <= 0:
|
||||
continue
|
||||
cutoff = now - days * 86400.0
|
||||
try:
|
||||
cur = await conn.execute(f"DELETE FROM {table} WHERE ts < ?", (cutoff,))
|
||||
await conn.commit()
|
||||
rows_total += cur.rowcount or 0
|
||||
stats.incr(f"retention_deleted_{table}", cur.rowcount or 0)
|
||||
except sqlite3.OperationalError as exc:
|
||||
log.warning("retention: transient error on %s: %s", table, exc)
|
||||
stats.incr("retention_transient_errors")
|
||||
except Exception:
|
||||
log.exception("retention: error deleting from %s", table)
|
||||
stats.incr("retention_errors")
|
||||
|
||||
if rows_total > 0:
|
||||
log.info("retention sweep removed %d rows", rows_total)
|
||||
stats.incr("retention_sweeps")
|
||||
|
||||
|
||||
__all__ = ["run_retention", "cleanup_once"]
|
||||
@@ -0,0 +1,405 @@
|
||||
"""Batched async SQLite writer.
|
||||
|
||||
Accepts write jobs on a bounded ``asyncio.Queue``. Jobs are pulled and
|
||||
grouped by target table; flushed either on ``batch_size`` items or every
|
||||
``flush_interval_ms`` milliseconds, whichever comes first.
|
||||
|
||||
Transient errors (``sqlite3.OperationalError``: "database is locked",
|
||||
"disk I/O error", etc.) are retried with exponential backoff up to
|
||||
``_MAX_RETRIES`` attempts; persistent failures are logged and the batch
|
||||
is dropped so the writer keeps draining the queue.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import aiosqlite
|
||||
|
||||
from ..config import StorageCfg
|
||||
from ..core.bus import EventBus
|
||||
from ..core.models import AisReport, AtoN, BaseStation, Event, GpsFix, MergedTarget, RadioReport, RawFrame
|
||||
from ..core.state import State
|
||||
from ..core.stats import Stats
|
||||
from ..publish.queues import put_drop_oldest
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_MAX_RETRIES = 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Write jobs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _VesselStaticRow:
|
||||
mmsi: int
|
||||
target: MergedTarget
|
||||
ts: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _AisDynamicRow:
|
||||
mmsi: int
|
||||
ts: float
|
||||
lat: float | None
|
||||
lon: float | None
|
||||
sog: float | None
|
||||
cog: float | None
|
||||
heading: float | None
|
||||
nav_status: int | None
|
||||
rot: float | None
|
||||
msg_type: int
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _GpsFixRow:
|
||||
fix: GpsFix
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _RawNmeaRow:
|
||||
frame: RawFrame
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _BaseStationRow:
|
||||
bs: BaseStation
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _AtoNRow:
|
||||
aton: AtoN
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _RadioRow:
|
||||
report: RadioReport
|
||||
|
||||
|
||||
WriteJob = (
|
||||
_VesselStaticRow | _AisDynamicRow | _GpsFixRow | _RawNmeaRow
|
||||
| _BaseStationRow | _AtoNRow | _RadioRow
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Writer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Writer:
|
||||
"""Batched SQLite writer task."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
conn: aiosqlite.Connection,
|
||||
cfg: StorageCfg,
|
||||
stats: Stats,
|
||||
queue: "asyncio.Queue[WriteJob]",
|
||||
) -> None:
|
||||
self._conn = conn
|
||||
self._cfg = cfg
|
||||
self._stats = stats
|
||||
self._queue = queue
|
||||
|
||||
async def run(self) -> None:
|
||||
batch: list[WriteJob] = []
|
||||
flush_interval = max(0.05, self._cfg.writer.flush_interval_ms / 1000.0)
|
||||
batch_size = max(1, self._cfg.writer.batch_size)
|
||||
|
||||
while True:
|
||||
try:
|
||||
first = await asyncio.wait_for(self._queue.get(), timeout=flush_interval)
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
except asyncio.CancelledError:
|
||||
await self._flush(batch)
|
||||
raise
|
||||
|
||||
batch.append(first)
|
||||
deadline = time.monotonic() + flush_interval
|
||||
|
||||
while len(batch) < batch_size:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
break
|
||||
try:
|
||||
job = await asyncio.wait_for(self._queue.get(), timeout=remaining)
|
||||
except asyncio.TimeoutError:
|
||||
break
|
||||
batch.append(job)
|
||||
|
||||
await self._flush(batch)
|
||||
batch.clear()
|
||||
|
||||
async def _flush(self, batch: list[WriteJob]) -> None:
|
||||
if not batch:
|
||||
return
|
||||
self._stats.set_gauge("storage_last_batch", float(len(batch)))
|
||||
|
||||
for attempt in range(1, _MAX_RETRIES + 1):
|
||||
try:
|
||||
await self._write_batch(batch)
|
||||
self._stats.incr("storage_batches")
|
||||
self._stats.incr("storage_rows", len(batch))
|
||||
return
|
||||
except sqlite3.OperationalError as exc:
|
||||
self._stats.incr("storage_transient_errors")
|
||||
log.warning("sqlite transient error (attempt %d/%d): %s",
|
||||
attempt, _MAX_RETRIES, exc)
|
||||
await asyncio.sleep(0.1 * (2 ** (attempt - 1)))
|
||||
except Exception:
|
||||
self._stats.incr("storage_errors")
|
||||
log.exception("sqlite write batch failed; dropping %d rows", len(batch))
|
||||
return
|
||||
self._stats.incr("storage_dropped_batches")
|
||||
log.error("sqlite: giving up on batch of %d rows after %d retries",
|
||||
len(batch), _MAX_RETRIES)
|
||||
|
||||
async def _write_batch(self, batch: list[WriteJob]) -> None:
|
||||
# Group by table for efficient multi-row inserts.
|
||||
static_rows: list[tuple] = []
|
||||
dynamic_rows: list[tuple] = []
|
||||
gps_rows: list[tuple] = []
|
||||
raw_rows: list[tuple] = []
|
||||
base_rows: list[tuple] = []
|
||||
aton_rows: list[tuple] = []
|
||||
radio_rows: list[tuple] = []
|
||||
|
||||
for job in batch:
|
||||
if isinstance(job, _VesselStaticRow):
|
||||
t = job.target
|
||||
static_rows.append((
|
||||
t.mmsi, t.name, t.callsign, t.imo, t.ship_type,
|
||||
t.dim_a, t.dim_b, t.dim_c, t.dim_d,
|
||||
t.eta, t.draught, t.destination, job.ts,
|
||||
))
|
||||
elif isinstance(job, _AisDynamicRow):
|
||||
dynamic_rows.append((
|
||||
job.mmsi, job.ts, job.lat, job.lon, job.sog, job.cog,
|
||||
job.heading, job.nav_status, job.rot, job.msg_type,
|
||||
))
|
||||
elif isinstance(job, _GpsFixRow):
|
||||
f = job.fix
|
||||
gps_rows.append((
|
||||
f.ts, f.lat, f.lon, f.sog, f.cog, f.alt,
|
||||
f.fix_quality, f.sats, f.hdop,
|
||||
))
|
||||
elif isinstance(job, _RawNmeaRow):
|
||||
fr = job.frame
|
||||
raw_rows.append((fr.ts, fr.source, fr.kind, fr.line))
|
||||
elif isinstance(job, _BaseStationRow):
|
||||
b = job.bs
|
||||
base_rows.append((b.mmsi, b.ts, b.lat, b.lon, b.epfd, b.ts))
|
||||
elif isinstance(job, _AtoNRow):
|
||||
a = job.aton
|
||||
aton_rows.append((
|
||||
a.mmsi, a.ts, a.lat, a.lon, a.aton_type,
|
||||
a.name, 1 if a.virtual else 0 if a.virtual is not None else None,
|
||||
a.ts,
|
||||
))
|
||||
elif isinstance(job, _RadioRow):
|
||||
r = job.report
|
||||
radio_rows.append((
|
||||
r.ts, r.channel, r.rssi, r.snr, r.slot,
|
||||
json.dumps(r.raw, ensure_ascii=False) if r.raw else None,
|
||||
))
|
||||
|
||||
async with self._conn.cursor() as cur:
|
||||
if static_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO vessel_static
|
||||
(mmsi, name, callsign, imo, ship_type,
|
||||
dim_a, dim_b, dim_c, dim_d,
|
||||
eta, draught, destination, updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
ON CONFLICT(mmsi) DO UPDATE SET
|
||||
name=COALESCE(excluded.name, vessel_static.name),
|
||||
callsign=COALESCE(excluded.callsign, vessel_static.callsign),
|
||||
imo=COALESCE(excluded.imo, vessel_static.imo),
|
||||
ship_type=COALESCE(excluded.ship_type, vessel_static.ship_type),
|
||||
dim_a=COALESCE(excluded.dim_a, vessel_static.dim_a),
|
||||
dim_b=COALESCE(excluded.dim_b, vessel_static.dim_b),
|
||||
dim_c=COALESCE(excluded.dim_c, vessel_static.dim_c),
|
||||
dim_d=COALESCE(excluded.dim_d, vessel_static.dim_d),
|
||||
eta=COALESCE(excluded.eta, vessel_static.eta),
|
||||
draught=COALESCE(excluded.draught, vessel_static.draught),
|
||||
destination=COALESCE(excluded.destination, vessel_static.destination),
|
||||
updated_at=excluded.updated_at
|
||||
""",
|
||||
static_rows,
|
||||
)
|
||||
if dynamic_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO ais_dynamic
|
||||
(mmsi, ts, lat, lon, sog, cog, heading, nav_status, rot, raw_msg_type)
|
||||
VALUES (?,?,?,?,?,?,?,?,?,?)""",
|
||||
dynamic_rows,
|
||||
)
|
||||
if gps_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO gps_fix
|
||||
(ts, lat, lon, sog, cog, alt, fix_quality, sats, hdop)
|
||||
VALUES (?,?,?,?,?,?,?,?,?)""",
|
||||
gps_rows,
|
||||
)
|
||||
if raw_rows:
|
||||
await cur.executemany(
|
||||
"INSERT INTO raw_nmea (ts, source, kind, line) VALUES (?,?,?,?)",
|
||||
raw_rows,
|
||||
)
|
||||
if base_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO base_station (mmsi, ts, lat, lon, epfd, updated_at)
|
||||
VALUES (?,?,?,?,?,?)
|
||||
ON CONFLICT(mmsi) DO UPDATE SET
|
||||
ts=excluded.ts, lat=excluded.lat, lon=excluded.lon,
|
||||
epfd=excluded.epfd, updated_at=excluded.updated_at""",
|
||||
base_rows,
|
||||
)
|
||||
if aton_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO aton (mmsi, ts, lat, lon, aton_type, name, virtual, updated_at)
|
||||
VALUES (?,?,?,?,?,?,?,?)
|
||||
ON CONFLICT(mmsi) DO UPDATE SET
|
||||
ts=excluded.ts, lat=excluded.lat, lon=excluded.lon,
|
||||
aton_type=excluded.aton_type, name=excluded.name,
|
||||
virtual=excluded.virtual, updated_at=excluded.updated_at""",
|
||||
aton_rows,
|
||||
)
|
||||
if radio_rows:
|
||||
await cur.executemany(
|
||||
"""INSERT INTO radio_telemetry (ts, channel, rssi, snr, slot, raw_json)
|
||||
VALUES (?,?,?,?,?,?)""",
|
||||
radio_rows,
|
||||
)
|
||||
await self._conn.commit()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Event-bus sink: converts Events back into WriteJobs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class StorageEventSink:
|
||||
"""Subscribe to ``EventBus`` and enqueue write jobs.
|
||||
|
||||
This keeps the writer decoupled from ingest/parser; it uses the same
|
||||
``core.state`` snapshots that publishers see, so anything stored is
|
||||
consistent with what external processes observe.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
bus: EventBus,
|
||||
state: State,
|
||||
stats: Stats,
|
||||
out_queue: "asyncio.Queue[WriteJob]",
|
||||
store_raw_nmea: bool,
|
||||
) -> None:
|
||||
self._bus = bus
|
||||
self._state = state
|
||||
self._stats = stats
|
||||
self._out = out_queue
|
||||
self._store_raw = store_raw_nmea
|
||||
self._sub = bus.subscribe(maxsize=4096)
|
||||
|
||||
async def run(self) -> None:
|
||||
try:
|
||||
while True:
|
||||
ev: Event = await self._sub.get()
|
||||
self._dispatch(ev)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
finally:
|
||||
self._bus.unsubscribe(self._sub)
|
||||
|
||||
def enqueue_raw(self, frame: RawFrame) -> None:
|
||||
if not self._store_raw:
|
||||
return
|
||||
put_drop_oldest(self._out, _RawNmeaRow(frame=frame),
|
||||
self._stats, "storage_in_dropped")
|
||||
|
||||
def _dispatch(self, ev: Event) -> None:
|
||||
try:
|
||||
if ev.type == "target.update":
|
||||
self._on_target(ev)
|
||||
elif ev.type == "ownship.update":
|
||||
self._on_ownship(ev)
|
||||
elif ev.type == "base_station.update":
|
||||
self._on_base_station(ev)
|
||||
elif ev.type == "aton.update":
|
||||
self._on_aton(ev)
|
||||
elif ev.type == "radio.update":
|
||||
self._on_radio(ev)
|
||||
except Exception:
|
||||
log.exception("storage sink dispatch failed for event type=%s", ev.type)
|
||||
|
||||
def _put(self, job: WriteJob) -> None:
|
||||
put_drop_oldest(self._out, job, self._stats, "storage_in_dropped")
|
||||
|
||||
def _on_target(self, ev: Event) -> None:
|
||||
mmsi = int(ev.data["mmsi"])
|
||||
target = self._state.targets.get(mmsi)
|
||||
if target is None:
|
||||
return
|
||||
# Write static row (upsert) so dims/name/etc. are persisted.
|
||||
self._put(_VesselStaticRow(mmsi=mmsi, target=target, ts=ev.ts))
|
||||
# Append a dynamic row (only when dynamic fields were updated,
|
||||
# otherwise last_dynamic_ts == ev.ts would mean we already have a fix).
|
||||
if target.last_dynamic_ts == ev.ts:
|
||||
self._put(_AisDynamicRow(
|
||||
mmsi=mmsi,
|
||||
ts=ev.ts,
|
||||
lat=target.lat,
|
||||
lon=target.lon,
|
||||
sog=target.sog,
|
||||
cog=target.cog,
|
||||
heading=target.heading,
|
||||
nav_status=target.nav_status,
|
||||
rot=target.rot,
|
||||
msg_type=max(target.msg_types) if target.msg_types else 0,
|
||||
))
|
||||
|
||||
def _on_ownship(self, ev: Event) -> None:
|
||||
d = ev.data
|
||||
fix = GpsFix(
|
||||
ts=ev.ts, source="ownship",
|
||||
lat=d.get("lat"), lon=d.get("lon"),
|
||||
sog=d.get("sog"), cog=d.get("cog"), alt=d.get("alt"),
|
||||
fix_quality=d.get("fix_quality"), sats=d.get("sats"), hdop=d.get("hdop"),
|
||||
)
|
||||
self._put(_GpsFixRow(fix=fix))
|
||||
|
||||
def _on_base_station(self, ev: Event) -> None:
|
||||
mmsi = int(ev.data["mmsi"])
|
||||
bs = self._state.base_stations.get(mmsi)
|
||||
if bs is not None:
|
||||
self._put(_BaseStationRow(bs=bs))
|
||||
|
||||
def _on_aton(self, ev: Event) -> None:
|
||||
mmsi = int(ev.data["mmsi"])
|
||||
a = self._state.atons.get(mmsi)
|
||||
if a is not None:
|
||||
self._put(_AtoNRow(aton=a))
|
||||
|
||||
def _on_radio(self, ev: Event) -> None:
|
||||
d = ev.data
|
||||
self._put(_RadioRow(report=RadioReport(
|
||||
ts=ev.ts, source="bus",
|
||||
channel=d.get("channel"),
|
||||
rssi=d.get("rssi"), snr=d.get("snr"), slot=d.get("slot"),
|
||||
raw=d.get("raw") or {},
|
||||
)))
|
||||
|
||||
|
||||
__all__ = ["Writer", "StorageEventSink", "WriteJob"]
|
||||
@@ -0,0 +1 @@
|
||||
__version__ = "0.1.0"
|
||||
@@ -0,0 +1,28 @@
|
||||
[Unit]
|
||||
Description=ais_hub AIS/GPS/radio telemetry aggregation service
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ais_hub
|
||||
Group=ais_hub
|
||||
ExecStart=/usr/bin/python3 -m ais_hub --config /etc/ais_hub/config.yaml
|
||||
Restart=on-failure
|
||||
RestartSec=3
|
||||
KillSignal=SIGTERM
|
||||
TimeoutStopSec=15
|
||||
LimitNOFILE=4096
|
||||
|
||||
# Hardening (embedded Linux friendly defaults; remove if they conflict with
|
||||
# UART device access).
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths=/var/lib/ais_hub /var/log/ais_hub
|
||||
# For UART access, user must be in the serial/dialout group or a udev rule
|
||||
# should grant access to /dev/ttyS1.
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,7 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_ROOT = Path(__file__).resolve().parents[1]
|
||||
_SRC = _ROOT / "src"
|
||||
if str(_SRC) not in sys.path:
|
||||
sys.path.insert(0, str(_SRC))
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Multi-fragment AIS assembler: strict-order and strengthened-key tests.
|
||||
|
||||
These tests exercise the assembler logic only up to fragment validation.
|
||||
Actual decoding with pyais is covered by a separate integration check
|
||||
if pyais is importable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ais_hub.core.stats import Stats
|
||||
from ais_hub.parser.ais import AisAssembler
|
||||
|
||||
|
||||
SRC_A = "ais_udp:10.0.0.1:5000"
|
||||
SRC_B = "ais_udp:10.0.0.2:5000"
|
||||
|
||||
|
||||
def _frag(total: int, n: int, seq: str, channel: str = "A", payload: str = "abcd", fill: str = "0") -> str:
|
||||
# Compute checksum so parse_sentence accepts the line.
|
||||
body = f"AIVDM,{total},{n},{seq},{channel},{payload},{fill}"
|
||||
cs = 0
|
||||
for ch in body.encode("ascii"):
|
||||
cs ^= ch
|
||||
return f"!{body}*{cs:02X}"
|
||||
|
||||
|
||||
def test_single_fragment_fast_path_attempts_decode():
|
||||
"""A single-fragment sentence is passed to pyais; decode may fail, but
|
||||
the assembler must not keep any buffers for it."""
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
line = _frag(1, 1, "")
|
||||
a.feed(SRC_A, line)
|
||||
# No multi-fragment buffers.
|
||||
assert len(a._buffers) == 0
|
||||
|
||||
|
||||
def test_multi_fragment_happy_path_clears_buffer():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 1, "3"))
|
||||
assert len(a._buffers) == 1
|
||||
a.feed(SRC_A, _frag(2, 2, "3"))
|
||||
# After the final fragment arrives, the buffer is removed (regardless
|
||||
# of whether pyais decodes successfully).
|
||||
assert len(a._buffers) == 0
|
||||
|
||||
|
||||
def test_out_of_order_fragment_triggers_error_and_reset():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(3, 1, "7"))
|
||||
a.feed(SRC_A, _frag(3, 3, "7")) # skipped fragment 2
|
||||
assert stats.counters["ais_fragment_errors"] >= 1
|
||||
|
||||
|
||||
def test_duplicate_fragment_is_error():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 1, "9"))
|
||||
a.feed(SRC_A, _frag(2, 1, "9")) # duplicate
|
||||
assert stats.counters["ais_fragment_errors"] >= 1
|
||||
|
||||
|
||||
def test_mid_stream_fragment_without_leading_is_error():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 2, "4"))
|
||||
assert stats.counters["ais_fragment_errors"] >= 1
|
||||
assert len(a._buffers) == 0
|
||||
|
||||
|
||||
def test_strengthened_key_isolates_sources():
|
||||
"""Two sources reusing the same seq_id must not cross-contaminate."""
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 1, "5"))
|
||||
a.feed(SRC_B, _frag(2, 1, "5"))
|
||||
assert len(a._buffers) == 2 # two independent buffers
|
||||
assert stats.counters.get("ais_fragment_errors", 0) == 0
|
||||
|
||||
|
||||
def test_multi_fragment_without_seq_id_is_error():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 1, "")) # empty seq_id on a multi-fragment
|
||||
assert stats.counters["ais_fragment_errors"] >= 1
|
||||
|
||||
|
||||
def test_total_mismatch_on_followup_is_error():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, _frag(2, 1, "2"))
|
||||
# Fragment claims total=3 now; must be treated as a fresh start or error.
|
||||
a.feed(SRC_A, _frag(3, 2, "2"))
|
||||
assert stats.counters["ais_fragment_errors"] >= 1
|
||||
|
||||
|
||||
def test_ttl_gc_evicts_stale_buffers():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats, ttl_sec=1.0)
|
||||
a.feed(SRC_A, _frag(2, 1, "8"), ts=1000.0)
|
||||
assert len(a._buffers) == 1
|
||||
a.gc(now=1002.0)
|
||||
assert len(a._buffers) == 0
|
||||
assert stats.counters["ais_fragment_timeouts"] >= 1
|
||||
|
||||
|
||||
def test_bad_checksum_ignored_with_counter():
|
||||
stats = Stats()
|
||||
a = AisAssembler(stats=stats)
|
||||
a.feed(SRC_A, "!AIVDM,1,1,,A,x,0*00") # deliberately wrong checksum
|
||||
assert stats.counters["parser_checksum_errors"] >= 1
|
||||
@@ -0,0 +1,80 @@
|
||||
"""End-to-end check for AIS multi-fragment type-5 (static/voyage) decoding.
|
||||
|
||||
Verifies that the assembler -> pyais -> normalization -> state -> MergedTarget
|
||||
chain fills in ``name``/``callsign``/``imo``/``ship_type``/dims/draught/destination.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ais_hub.core.bus import EventBus
|
||||
from ais_hub.core.state import State
|
||||
from ais_hub.core.stats import Stats
|
||||
from ais_hub.parser.ais import AisAssembler
|
||||
from ais_hub.parser.nmea_utils import compute_checksum
|
||||
|
||||
|
||||
pyais = pytest.importorskip("pyais")
|
||||
|
||||
|
||||
def _with_csum(body: str) -> str:
|
||||
return f"!{body}*{compute_checksum(body):02X}"
|
||||
|
||||
|
||||
# Classic pyais demo payload for MT.MITCHELL (MMSI 369190000).
|
||||
FRAG_1_BODY = (
|
||||
"AIVDM,2,1,1,A,55P5TL01VIaAL@7WKO@mBplU@<PDhh000000001S;AJ::4A80?4i@E53,0"
|
||||
)
|
||||
FRAG_2_BODY = "AIVDM,2,2,1,A,1@0000000000000,2"
|
||||
|
||||
|
||||
def _state() -> State:
|
||||
stats = Stats()
|
||||
bus = EventBus(stats=stats, default_maxsize=64)
|
||||
return State(bus=bus, stats=stats)
|
||||
|
||||
|
||||
def test_type5_two_fragments_populates_static_fields():
|
||||
state = _state()
|
||||
stats = Stats()
|
||||
asm = AisAssembler(stats=stats)
|
||||
src = "ais_udp:0.0.0.0:5005:127.0.0.1"
|
||||
|
||||
for ln in (_with_csum(FRAG_1_BODY), _with_csum(FRAG_2_BODY)):
|
||||
for rep in asm.feed(src, ln, ts=1000.0):
|
||||
state.apply_ais(rep)
|
||||
|
||||
assert stats.counters.get("ais_msg_5", 0) == 1
|
||||
v = state.get_vessel(369190000)
|
||||
assert v is not None
|
||||
assert v["name"] == "MT.MITCHELL"
|
||||
assert v["callsign"] == "WDA9674"
|
||||
assert v["imo"] == 6710932
|
||||
# ship_type must be a plain int (IntEnum flattened by normalization).
|
||||
assert isinstance(v["ship_type"], int)
|
||||
assert v["ship_type"] == 99
|
||||
assert v["dims"] == {"a": 90, "b": 90, "c": 10, "d": 10}
|
||||
assert v["voyage"]["destination"] == "SEATTLE"
|
||||
assert v["voyage"]["draught"] == pytest.approx(6.0)
|
||||
assert 5 in v["msg_types"]
|
||||
|
||||
|
||||
def test_type5_fragments_with_different_sender_ports_still_merge():
|
||||
"""Sender port in source_tag changing mid-message must not break assembly.
|
||||
|
||||
The AIS UDP ingest only includes the sender IP (not the ephemeral port)
|
||||
in the source tag, so fragments from the same receiver reach the same
|
||||
assembler buffer even if the kernel picked different source ports.
|
||||
"""
|
||||
state = _state()
|
||||
stats = Stats()
|
||||
asm = AisAssembler(stats=stats)
|
||||
|
||||
# Same receiver IP -> same source_tag for both fragments.
|
||||
src = "ais_udp:0.0.0.0:5005:127.0.0.1"
|
||||
for ln in (_with_csum(FRAG_1_BODY), _with_csum(FRAG_2_BODY)):
|
||||
for rep in asm.feed(src, ln, ts=1500.0):
|
||||
state.apply_ais(rep)
|
||||
|
||||
assert state.get_vessel(369190000) is not None
|
||||
@@ -0,0 +1,167 @@
|
||||
"""Binary decoder tests for AIS-catcher Mini UDP telemetry.
|
||||
|
||||
The expected byte layouts come from ais-mini's UDP comment block; any
|
||||
layout drift (field order, endianness, sizes) must fail here before the
|
||||
ingest runs on real hardware.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
from ais_hub.parser.aiscatcher import (
|
||||
decode_rssi_iq,
|
||||
decode_signal_events,
|
||||
decode_slot_bitmap,
|
||||
decode_slot_detail,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Slot bitmap (315 bytes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_bitmap(
|
||||
*,
|
||||
channel: bytes = b"A",
|
||||
utc_minute: int = 28_279_310,
|
||||
slots_total: int = 2250,
|
||||
occupied: int = 437,
|
||||
noise: float = 0.0,
|
||||
threshold: float = 0.0,
|
||||
slot0_ms: int = 28_279_310 * 60_000,
|
||||
first_occ_ms: int = 28_279_310 * 60_000 + 123,
|
||||
bitmap: bytes | None = None,
|
||||
) -> bytes:
|
||||
if bitmap is None:
|
||||
bitmap = bytes(282)
|
||||
assert len(channel) == 1
|
||||
assert len(bitmap) == 282
|
||||
head = struct.pack(
|
||||
">BIHHffQQ",
|
||||
channel[0], utc_minute, slots_total, occupied,
|
||||
noise, threshold, slot0_ms, first_occ_ms,
|
||||
)
|
||||
assert len(head) == 33
|
||||
return head + bitmap
|
||||
|
||||
|
||||
def test_bitmap_decodes_roundtrip():
|
||||
bm = bytes(range(256)) + bytes(282 - 256)
|
||||
data = _build_bitmap(channel=b"B", occupied=1234, bitmap=bm)
|
||||
snap = decode_slot_bitmap(data)
|
||||
assert snap is not None
|
||||
assert snap.channel == "B"
|
||||
assert snap.utc_minute == 28_279_310
|
||||
assert snap.slots_total == 2250
|
||||
assert snap.occupied_count == 1234
|
||||
assert snap.slot0_unix_ms == 28_279_310 * 60_000
|
||||
assert snap.first_occupied_unix_ms == 28_279_310 * 60_000 + 123
|
||||
assert snap.bitmap == bm
|
||||
# occupied_fraction is derived
|
||||
assert snap.occupied_fraction() == pytest.approx(1234 / 2250)
|
||||
|
||||
|
||||
def test_bitmap_rejects_wrong_size():
|
||||
data = _build_bitmap()
|
||||
assert decode_slot_bitmap(data[:-1]) is None
|
||||
assert decode_slot_bitmap(data + b"\x00") is None
|
||||
|
||||
|
||||
def test_bitmap_unknown_channel_byte_marked_questionmark():
|
||||
data = _build_bitmap(channel=b"X")
|
||||
snap = decode_slot_bitmap(data)
|
||||
assert snap is not None
|
||||
assert snap.channel == "?"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Slot detail
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_slot_detail_roundtrip():
|
||||
entries = [(0, -85.5), (100, -72.125), (2249, -99.0)]
|
||||
head = struct.pack(">BIQH", ord(b"A"), 28_279_310, 28_279_310 * 60_000, len(entries))
|
||||
body = b"".join(struct.pack(">Hf", slot, lvl) for slot, lvl in entries)
|
||||
detail = decode_slot_detail(head + body)
|
||||
assert detail is not None
|
||||
assert detail.channel == "A"
|
||||
assert detail.utc_minute == 28_279_310
|
||||
assert detail.slot0_unix_ms == 28_279_310 * 60_000
|
||||
assert [(e.slot, round(e.level_db, 4)) for e in detail.entries] == [
|
||||
(0, -85.5),
|
||||
(100, -72.125),
|
||||
(2249, -99.0),
|
||||
]
|
||||
|
||||
|
||||
def test_slot_detail_rejects_short_body():
|
||||
head = struct.pack(">BIQH", ord(b"A"), 1, 2, 3)
|
||||
# only 1 full entry provided where 3 claimed
|
||||
body = struct.pack(">Hf", 10, -70.0)
|
||||
assert decode_slot_detail(head + body) is None
|
||||
|
||||
|
||||
def test_slot_detail_empty_is_valid():
|
||||
head = struct.pack(">BIQH", ord(b"B"), 1, 2, 0)
|
||||
detail = decode_slot_detail(head)
|
||||
assert detail is not None
|
||||
assert detail.entries == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RSSI IQ (8 bytes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_rssi_decodes_pair():
|
||||
data = struct.pack(">ff", -42.25, -55.75)
|
||||
rssi = decode_rssi_iq(data)
|
||||
assert rssi is not None
|
||||
assert rssi.power_a_db == pytest.approx(-42.25)
|
||||
assert rssi.power_b_db == pytest.approx(-55.75)
|
||||
|
||||
|
||||
def test_rssi_wrong_size():
|
||||
assert decode_rssi_iq(b"\x00" * 7) is None
|
||||
assert decode_rssi_iq(b"\x00" * 9) is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Signal events (decode markers)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_signal_events_roundtrip():
|
||||
events = [
|
||||
(1_700_000_000_000, 42, 257_123_456, -72.0),
|
||||
(1_700_000_000_100, 1500, 227_000_001, -88.5),
|
||||
]
|
||||
head = struct.pack(">BH", ord(b"A"), len(events))
|
||||
body = b"".join(struct.pack(">QHIf", ts, slot, mmsi, lvl)
|
||||
for ts, slot, mmsi, lvl in events)
|
||||
batch = decode_signal_events(head + body)
|
||||
assert batch is not None
|
||||
assert batch.channel == "A"
|
||||
assert len(batch.events) == 2
|
||||
assert batch.events[0].unix_ms == 1_700_000_000_000
|
||||
assert batch.events[0].slot == 42
|
||||
assert batch.events[0].mmsi == 257_123_456
|
||||
assert batch.events[0].level_db == pytest.approx(-72.0)
|
||||
assert batch.events[1].mmsi == 227_000_001
|
||||
|
||||
|
||||
def test_signal_events_empty():
|
||||
data = struct.pack(">BH", ord(b"A"), 0)
|
||||
batch = decode_signal_events(data)
|
||||
assert batch is not None
|
||||
assert batch.events == []
|
||||
|
||||
|
||||
def test_signal_events_rejects_short_body():
|
||||
head = struct.pack(">BH", ord(b"A"), 5)
|
||||
assert decode_signal_events(head + b"\x00" * 10) is None
|
||||
@@ -0,0 +1,126 @@
|
||||
"""State integration tests for AIS-catcher binary telemetry."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from ais_hub.core.bus import EventBus
|
||||
from ais_hub.core.state import State
|
||||
from ais_hub.core.stats import Stats
|
||||
from ais_hub.parser.aiscatcher import (
|
||||
RssiIq,
|
||||
SignalEvent,
|
||||
SignalEventBatch,
|
||||
SlotBitmap,
|
||||
SlotDetail,
|
||||
SlotLevel,
|
||||
)
|
||||
|
||||
|
||||
def _make_state() -> State:
|
||||
stats = Stats()
|
||||
bus = EventBus(stats=stats, default_maxsize=64)
|
||||
return State(bus=bus, stats=stats)
|
||||
|
||||
|
||||
def test_rssi_iq_updates_state_and_publishes():
|
||||
state = _make_state()
|
||||
sub = state._bus.subscribe(maxsize=16)
|
||||
|
||||
state.apply_rssi_iq(RssiIq(power_a_db=-41.5, power_b_db=-43.0), ts=123.0)
|
||||
|
||||
assert state.radio_power == {
|
||||
"ts": 123.0,
|
||||
"power_a_db": -41.5,
|
||||
"power_b_db": -43.0,
|
||||
}
|
||||
assert not sub.empty()
|
||||
ev = sub.get_nowait()
|
||||
assert ev.type == "radio.update"
|
||||
assert ev.data["source"] == "aiscatcher_rssi"
|
||||
assert ev.data["power_a_db"] == -41.5
|
||||
|
||||
|
||||
def test_slot_bitmap_stored_per_channel():
|
||||
state = _make_state()
|
||||
snap = SlotBitmap(
|
||||
channel="A",
|
||||
utc_minute=28279310,
|
||||
slots_total=2250,
|
||||
occupied_count=100,
|
||||
noise_floor=0.0,
|
||||
threshold=0.0,
|
||||
slot0_unix_ms=28279310 * 60000,
|
||||
first_occupied_unix_ms=0,
|
||||
bitmap=bytes(282),
|
||||
)
|
||||
state.apply_slot_bitmap(snap, ts=50.0)
|
||||
|
||||
assert "A" in state.slot_occupancy
|
||||
occ = state.slot_occupancy["A"]
|
||||
assert occ["occupied_count"] == 100
|
||||
assert occ["slots_total"] == 2250
|
||||
assert occ["occupied_fraction"] == 100 / 2250
|
||||
# snapshot_slots exposes everything for REST
|
||||
snap_dict = state.snapshot_slots()
|
||||
assert snap_dict["occupancy"]["A"]["occupied_count"] == 100
|
||||
|
||||
|
||||
def test_slot_detail_stores_entries():
|
||||
state = _make_state()
|
||||
detail = SlotDetail(
|
||||
channel="B",
|
||||
utc_minute=1,
|
||||
slot0_unix_ms=60000,
|
||||
entries=[SlotLevel(slot=10, level_db=-70.0), SlotLevel(slot=2200, level_db=-80.0)],
|
||||
)
|
||||
state.apply_slot_detail(detail, ts=60.0)
|
||||
stored = state.slot_detail["B"]
|
||||
assert len(stored["entries"]) == 2
|
||||
assert stored["entries"][0] == {"slot": 10, "level_db": -70.0}
|
||||
|
||||
|
||||
def test_signal_event_creates_target_with_signal_fields():
|
||||
state = _make_state()
|
||||
batch = SignalEventBatch(
|
||||
channel="A",
|
||||
events=[
|
||||
SignalEvent(unix_ms=1_700_000_000_000, slot=42, mmsi=257_000_001, level_db=-75.0),
|
||||
],
|
||||
)
|
||||
state.apply_signal_events(batch, ts=1_700_000_000.0)
|
||||
|
||||
assert 257_000_001 in state.targets
|
||||
tgt = state.targets[257_000_001]
|
||||
assert tgt.last_signal_db == -75.0
|
||||
assert tgt.last_signal_slot == 42
|
||||
assert tgt.last_signal_channel == "A"
|
||||
# last_seen must be set from signal event if it's newer
|
||||
assert tgt.last_seen >= 1_700_000_000.0
|
||||
|
||||
# to_dict exposes signal block
|
||||
d = tgt.to_dict()
|
||||
assert d["signal"] == {
|
||||
"last_db": -75.0,
|
||||
"last_ts": 1_700_000_000.0,
|
||||
"last_slot": 42,
|
||||
"last_channel": "A",
|
||||
}
|
||||
|
||||
|
||||
def test_signal_event_older_does_not_overwrite():
|
||||
state = _make_state()
|
||||
mmsi = 111_222_333
|
||||
# first, newer event
|
||||
state.apply_signal_events(SignalEventBatch(
|
||||
channel="A",
|
||||
events=[SignalEvent(unix_ms=2_000_000_000_000, slot=1, mmsi=mmsi, level_db=-60.0)],
|
||||
), ts=2_000_000_000.0)
|
||||
# then, older event — must not clobber last_signal_db
|
||||
state.apply_signal_events(SignalEventBatch(
|
||||
channel="B",
|
||||
events=[SignalEvent(unix_ms=1_000_000_000_000, slot=99, mmsi=mmsi, level_db=-90.0)],
|
||||
), ts=2_000_000_001.0)
|
||||
|
||||
tgt = state.targets[mmsi]
|
||||
assert tgt.last_signal_db == -60.0
|
||||
assert tgt.last_signal_slot == 1
|
||||
assert tgt.last_signal_channel == "A"
|
||||
@@ -0,0 +1,39 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from ais_hub.config import Config, load_config
|
||||
|
||||
|
||||
def test_defaults_when_no_file():
|
||||
cfg = load_config(None)
|
||||
assert isinstance(cfg, Config)
|
||||
assert cfg.ingest.ais_udp.port == 4001
|
||||
assert cfg.publish.udp_tx_outbox.port == 6010
|
||||
assert cfg.publish.udp_nmea.port == 6007
|
||||
assert cfg.publish.udp_events.port == 7001
|
||||
|
||||
|
||||
def test_yaml_overrides(tmp_path: Path):
|
||||
p = tmp_path / "c.yaml"
|
||||
p.write_text(
|
||||
"ingest:\n"
|
||||
" ais_udp:\n"
|
||||
" port: 5555\n"
|
||||
"publish:\n"
|
||||
" http:\n"
|
||||
" port: 9090\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
cfg = load_config(p)
|
||||
assert cfg.ingest.ais_udp.port == 5555
|
||||
assert cfg.publish.http.port == 9090
|
||||
# Untouched defaults remain.
|
||||
assert cfg.publish.udp_nmea.port == 6007
|
||||
|
||||
|
||||
def test_env_overrides(tmp_path: Path, monkeypatch):
|
||||
monkeypatch.setenv("AIS_HUB_PUBLISH__HTTP__PORT", "7070")
|
||||
monkeypatch.setenv("AIS_HUB_STORAGE__STORE_RAW_NMEA", "true")
|
||||
cfg = load_config(None)
|
||||
assert cfg.publish.http.port == 7070
|
||||
assert cfg.storage.store_raw_nmea is True
|
||||
@@ -0,0 +1,47 @@
|
||||
from ais_hub.core.stats import Stats
|
||||
from ais_hub.parser.gps import GpsParser
|
||||
|
||||
|
||||
def _withcs(body: str) -> str:
|
||||
cs = 0
|
||||
for ch in body.encode("ascii"):
|
||||
cs ^= ch
|
||||
return f"${body}*{cs:02X}"
|
||||
|
||||
|
||||
def test_rmc_produces_lat_lon_sog_cog():
|
||||
p = GpsParser(stats=Stats())
|
||||
line = _withcs("GPRMC,123519,A,4807.038,N,01131.000,E,22.4,84.4,230394,3.1,W")
|
||||
fix = p.feed("gps_uart", line)
|
||||
assert fix is not None
|
||||
assert abs(fix.lat - (48 + 7.038 / 60)) < 1e-6
|
||||
assert abs(fix.lon - (11 + 31.0 / 60)) < 1e-6
|
||||
assert fix.sog == 22.4
|
||||
assert fix.cog == 84.4
|
||||
|
||||
|
||||
def test_gga_supplies_fix_quality_and_hdop():
|
||||
p = GpsParser(stats=Stats())
|
||||
line = _withcs("GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,")
|
||||
fix = p.feed("gps_uart", line)
|
||||
assert fix is not None
|
||||
assert fix.fix_quality == 1
|
||||
assert fix.sats == 8
|
||||
assert fix.hdop == 0.9
|
||||
assert fix.alt == 545.4
|
||||
|
||||
|
||||
def test_vtg_speed_from_kmh_when_knots_missing():
|
||||
p = GpsParser(stats=Stats())
|
||||
line = _withcs("GPVTG,054.7,T,034.4,M,,,010.0,K,A")
|
||||
fix = p.feed("gps_uart", line)
|
||||
assert fix is not None
|
||||
assert fix.cog == 54.7
|
||||
# 10 km/h ~ 5.3996 knots
|
||||
assert fix.sog is not None and abs(fix.sog - 5.3996) < 1e-3
|
||||
|
||||
|
||||
def test_bad_checksum_rejected():
|
||||
p = GpsParser(stats=Stats())
|
||||
fix = p.feed("gps_uart", "$GPRMC,,,,,,,,,,,,*FF")
|
||||
assert fix is None
|
||||
@@ -0,0 +1,63 @@
|
||||
from ais_hub.core import merge
|
||||
from ais_hub.core.models import AisReport, MergedTarget
|
||||
|
||||
|
||||
def test_eager_merge_updates_dynamic_fields():
|
||||
t = MergedTarget(mmsi=123)
|
||||
rep = AisReport(
|
||||
ts=100.0, source="src", kind="dynamic", mmsi=123, msg_type=1,
|
||||
channel="A", data={"lat": 10.5, "lon": 20.5, "sog": 7.1, "cog": 180.0},
|
||||
)
|
||||
changed = merge.apply(t, rep)
|
||||
assert changed is True
|
||||
assert t.lat == 10.5
|
||||
assert t.sog == 7.1
|
||||
assert t.last_dynamic_ts == 100.0
|
||||
assert t.last_seen == 100.0
|
||||
|
||||
|
||||
def test_stale_dynamic_report_is_ignored():
|
||||
t = MergedTarget(mmsi=123)
|
||||
t.last_dynamic_ts = 200.0
|
||||
t.lat = 1.0
|
||||
older = AisReport(
|
||||
ts=100.0, source="src", kind="dynamic", mmsi=123, msg_type=1,
|
||||
data={"lat": 50.0},
|
||||
)
|
||||
merge.apply(t, older)
|
||||
assert t.lat == 1.0 # unchanged
|
||||
|
||||
|
||||
def test_static_and_dynamic_keep_separate_timestamps():
|
||||
t = MergedTarget(mmsi=42)
|
||||
dyn = AisReport(
|
||||
ts=100.0, source="a", kind="dynamic", mmsi=42, msg_type=1,
|
||||
data={"lat": 1.0, "lon": 2.0},
|
||||
)
|
||||
stat = AisReport(
|
||||
ts=110.0, source="a", kind="static", mmsi=42, msg_type=5,
|
||||
data={"name": "MV TEST", "callsign": "ABCD", "imo": 9999999},
|
||||
)
|
||||
merge.apply(t, dyn)
|
||||
merge.apply(t, stat)
|
||||
assert t.lat == 1.0 and t.name == "MV TEST"
|
||||
assert t.last_dynamic_ts == 100.0
|
||||
assert t.last_static_ts == 110.0
|
||||
assert t.last_seen == 110.0
|
||||
|
||||
|
||||
def test_msg24_name_and_callsign_combine():
|
||||
t = MergedTarget(mmsi=7)
|
||||
# Msg 24A -> name
|
||||
merge.apply(t, AisReport(
|
||||
ts=10.0, source="a", kind="static", mmsi=7, msg_type=24,
|
||||
data={"name": "BOAT"},
|
||||
))
|
||||
# Msg 24B -> callsign + dims (later, must not wipe name)
|
||||
merge.apply(t, AisReport(
|
||||
ts=11.0, source="a", kind="static", mmsi=7, msg_type=24,
|
||||
data={"callsign": "CALL", "dim_a": 10, "dim_b": 20},
|
||||
))
|
||||
assert t.name == "BOAT"
|
||||
assert t.callsign == "CALL"
|
||||
assert t.dim_a == 10
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user