screen_cast/.agents/MEMORY.md

# Project Memory — screen_cast

Last updated: Phase 8 (Android receiver app) complete and validated on a
Fairphone 6; all prior phases done. The Android app's review findings were
fixed in a follow-up pass (same day) — see "Android app review" at the
bottom of this file.

## Project state

- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30,
  app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via
  USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++
  changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`;
  Kotlin targets `compileOptions`, Java 17); no androidx (framework +
  org.json + JUnit). 25 JVM tests green. Validated end-to-end on a
  Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed
  render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in
  RUNBOOK): `INTERNET` permission is REQUIRED for NsdService;
  `DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is
  Inet6Address, not InetSocketAddress); pass the Surface to
  `MediaCodec.configure()` + `start()`; render via
  `releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
  configure (in-band SPS reconfigures); `MediaFormat.format()`/
  `KEY_MIME_TYPE` not public in API 36; NSD: the classic
  `registerService(info, flags, RegistrationListener)` API exists from API
  16 through 36 (javap-verified on the android-36 SDK) — an old reflection
  fallback targeting a never-existent "ResolutionListener" was removed as
  dead code;
  **`android._video-scaling`: C2 scales output to the Surface** → letterbox
  by sizing the TextureView to the video aspect, NOT a transform matrix
  (double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
  picks the FIRST output (eDP-1, wrong display) — use `--target window`
  (hyprland-share-picker). Activity FQN for `am start`:
  `screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package =
  namespace `screen_cast`).
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
  announces its signaling port via a threaded-poll Avahi client; senders
  browse+resolve) plus JSON session signaling (offer/answer) over TCP with
  a signaling server on the receiver (port 5005). `--send` auto-discovers
  when exactly one receiver is found; `--discover` lists receivers;
  `--peer` targets a receiver directly. `meson test` 5/5, valgrind clean.
- **Signaling is newline-delimited JSON over TCP**, not WebSocket: no WS
  library was installed and the rtp-networking skill permits plain TCP.
  The wire format (one JSON object per line: offer/answer with session id,
  codec, rtp port) is transport-agnostic; Phase 7 adds the WS dependency if
  needed.
- **Dual-stack everywhere**: this machine resolves its own services over
  IPv6 (ULA + link-local), so the UDP transport, signaling client, and both
  listeners now support both families (IPv6 sockets with IPV6_V6ONLY=0 for
  dual-stack listening; AF_UNSPEC getaddrinfo for peers). Validated over
  both 127.0.0.1 and ::1.
- **Discovery dedupe**: one entry per (service_name, signaling_port) — a
  host with many interfaces otherwise registers dozens of address variants.
- **Receiver-only builds** (`-Dsender=false`, meson option): skip the
  capture backend and sender pipeline (`SC_HAS_SENDER` guards in
  main.cpp/pipelines.cpp; `--send` refuses cleanly in such builds).
  `scripts/install-receiver.sh` uses this to install on small ARM boards
  (user's Raspberry Pi Zero 2 W receiver) without PipeWire/portal deps,
  building SDL3 from source when the distro lacks it. Requires GCC 13+
  (`<format>`). Validated on x86_64: builds, tests 5/5, --discover works,
  --send refuses with a message.
- **systemd autostart**: `systemd/screencast-receiver.service` is a
  template (`__SC_RECEIVER_BIN__`/`__SC_RECEIVER_USER__`); the install
  script substitutes and enables it (opt out: `SC_RECEIVER_SERVICE=0`),
  adding the run user to video/render/input for headless KMSDRM.
- **systemd tty trap (hit on the real Pi, reproduced locally)**: with
  StandardInput=tty + TTYPath=tty1, the service NEVER started — systemd
  blocks PRE-EXEC in acquire_terminal() waiting for a tty that the console
  session/getty already owns. Symptoms: status shows the main PID as
  "(screencast)" with Tasks:1 and ~40ms CPU, silent journal, no mDNS.
  Diagnosed by gdb-attaching the stuck process (acquire_terminal backtrace)
  after reproducing with a local transient unit. Fix: the service owns a
  dedicated free VT (tty7) + tolerant `ExecStartPre=-/usr/bin/chvt 7`;
  tty1 keeps the console. Also: `systemctl status` Tasks counts THREADS
  (a healthy receiver shows ~15), and "Console Autologin" advice was
  WRONG (it made the hang deterministic) — removed from all docs.
- **Fullscreen headless rendering**: under the KMSDRM video driver (no
  window manager) the renderer goes fullscreen automatically; --fullscreen
  forces it on desktops. Aspect is preserved via
  SDL_SetRenderLogicalPresentation(LETTERBOX) (verified: 4:3 feed on a
  3440x1440 monitor rendered with black pillarbox bars; --fullscreen
  window covered the full monitor). The service lifecycle was validated
  with a user unit: SIGTERM stop → clean exit (success), mDNS withdrawn,
  no restart.
- **mDNS operational lesson (hit during validation)**: `kill -9` on a
  process holding an avahi registration leaves stale daemon records; the
  service then browses but never resolves (timeout for every peer) until
  records expire (~75 min). `systemctl restart avahi-daemon` clears it.
  Recorded in RUNBOOK; always stop with SIGTERM.
- **Low-RAM build validated on hardware**: the Zero 2 W receiver build
  completes with the install script's temporary-swap + single-job handling
  (the user confirmed; nlohmann/json's signaling TU was the OOM trigger
  before the fix). Cross-compiling on the dev machine was considered and
  dropped — the on-Pi build works and the toolchain/container effort was
  not needed.
- **App icons wired up** (`screencast_icon/`): single 256 px PNG used
  everywhere. `meson install` ships it to
  `share/icons/hicolor/256x256/apps/screencast.png`; the GTK panel sets it
  via `set_icon_name("screencast")` (GTK4 removed the pixel-buffer window
  icon — themed names only; `gtk_icon_theme_has_icon` verified OK); the
  SDL receiver window embeds the PNG at build time (`icon_png_data.h` via
  `scripts/icon_to_header.py` custom_target) and `SDL_SetWindowIcon` with
  an optional SDL3_image dependency (no pkg-config ships with it →
  `meson.get_compiler('cpp').find_library('SDL3_image', required: false)`
  + disabler; `-DSC_HAS_WINDOW_ICON` gate; Pi builds without it still
  work). The waybar widget now shows the icon as `background-image` in
  `~/.config/waybar/style.css` (▶/⏸ text hidden; `.idle` dimmed,
  `.streaming` green tint) instead of text glyphs; backup
  `style.css.bak-20260909-*`. Lesson: icons appear in compositor
  taskbars/switchers, not title bars (GTK4 CSD and Hyprland decorations
  don't draw them), so validate via theme lookup, not title-bar
  screenshots. `sudo pacman -S sdl3_image` done (extra/sdl3_image 3.4.6).
- **GTK panel + waybar widget**: `screencast-gui` (gtkmm-4.0, behind
  `-Dgui=true`; app internals now live in the `sc_app_core` static lib so
  CLI and GUI share pipelines/session/state). `screencast waybar [--toggle]`
  prints the waybar module line and toggles streaming via SIGTERM (stop) or
  a detached re-exec spawn from `last-session.json` (start). State lives in
  `$XDG_RUNTIME_DIR/screencast/sender.json` (written by the pipeline with
  session id, receiver, bitrate, pid, start time; stale files are detected
  by pid-liveness). The waybar config was wired into the user's bar
  (`custom/screencast` before `custom/timetrack`, with backup) and the
  binaries installed to /usr/local/bin.
  - **Pi Wi-Fi hotspot** (`scripts/pi-hotspot.sh on|off|status`): NetworkManager
  AP mode (WPA2, ipv4 shared → built-in DHCP/NAT, Pi at 10.42.0.1). Takes
  over wlan0 while active; generated PSK stored in /etc/screencast-hotspot.conf.
  No application changes needed — the receiver already announces on all
  interfaces. nmcli property syntax validated against NM 1.58 with a
  disposable profile; AP bring-up itself can only be validated on the Pi.
- **Phase 7 resilience shipped** (pending Pi-side hw-decode run):
  - PLI over signaling: `SessionPli` message; the depacketizer now returns
    `DepacketizeResult{access_unit, frame_dropped}`; the receiver
    rate-limits PLIs to 1/500 ms; the sender keeps the signaling channel
    open and calls `SenderPipeline::request_keyframe()` (thread-safe
    atomic → run-loop → encoder). Validated end-to-end with a probe that
    drops a mid-keyframe packet: receiver logs "frame damaged",
    PLI arrives with the session id.
  - `RtpJitterBuffer` (rtp_packet.h): sequence reordering, 16 pkt/60 ms,
    straggler discard via serial-number arithmetic, overflow flush for
    genuine loss. Zero added latency on in-order streams.
  - Hardware decode probe in DecoderFactory (`DecoderConfig.hardware_accel`,
    default true): h264_v4l2m2m first, software fallback, `--swdecode` opts
    out. On the desktop the probe fails cleanly ("Could not find a valid
    device") and falls back; on the Pi it should pick the VideoCore m2m
    device — NEEDS THE USER'S PI RUN to confirm.
- **REAL-HARDWARE VALIDATION (desktop → Pi Zero 2 W over Wi-Fi)**: the full
  chain works on two machines: mDNS discovery → signaling negotiation →
  RTP over Wi-Fi → software H.264 decode → fullscreen KMSDRM letterboxed
  rendering on the Pi's display. Two transport lessons from that network:
  the LAN's 6to4 (2002::) addresses are unreachable between peers —
  senders now try all discovered addresses in reachability order (private
  IPv4 first; build/pi_probe.cpp in the build dir is the probe that
  validated this end-to-end against the real Pi).

- **Phase 5 (local UDP sender→receiver loopback) is implemented**:
  - `UdpRtpTransport` (src/network/udp_transport.cpp): raw POSIX sockets,
    AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop()
    closes the socket to unblock the receive jthread. ASIO was deliberately
    deferred to Phase 6 (see decisions).
  - `screencast` binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring
    SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline
    (recv→depacketize→decode→bounded 3-frame queue→render thread).
  - `SdlRenderer` (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture,
    RGBA texture upload, texture recreated on resolution change.
    `RendererFactory::create` now returns `RendererResult` (error channel
    added, mirroring codec/capture patterns).
  - Encoder change: **GLOBAL_HEADER removed** so libx264 repeats SPS/PPS
    in-band at every keyframe; a receiver now decodes from the bitstream
    alone (mid-stream join, PLI recovery-ready). `get_extradata()` is empty
    in this mode; codec round-trip test updated to match the streaming path.
  - **Burst-control fixes after the first real run** (receiver saw
    `non-existing PPS 0` forever): without VBV, a 2256x1504 IDR burst (~100s
    of KB of back-to-back FU-A packets) overflowed the ~208KB default UDP
    receive buffer; the sequence gap made drop-on-damage discard whole
    keyframes *including their in-band SPS/PPS*, so the receiver never
    recovered. Fixes: encoder VBV (`rc_max_rate = bitrate`,
    `rc_buffer_size = bitrate*2/fps` — caps any keyframe to ~2 frame periods
    of bytes), microsecond encoder time_base (kills the 25fps pts
    quantization that duplicated timestamps), and a best-effort 4MB
    SO_RCVBUF on the receive socket (kernel clamps to rmem_max).
  - **Display fixes after the receiver window never appeared**: (1) SDL must
    own the window from one thread — creating it on main and pumping/presenting
    from the render thread left the Wayland surface unmapped; the renderer now
    creates/polls/presents/destroys entirely on the render thread (with a
    start() init handshake). (2) Wayland windows are invisible until the first
    render commit, so the renderer presents a blank frame at init (visible
    black window while waiting). (3) The receiver logs `stream started (WxH)`
    on the first decoded frame and any first present failure.
  - **Phase 5 validated end-to-end** (headless, without the portal): a
    synthetic RTP feed of solid red/green frames drove the real receiver over
    localhost UDP; the window mapped, logged `stream started (320x240)`, and a
    `grim` screenshot of the window region showed the fed color (V=198,
    SAT=74 during the red feed) — decoded video visibly rendering. The user's
    real sender run showed capture→encode→send working (1 IDR + 12 P-frames,
    IDR capped at ~20KB by the VBV).
- **Automated validation**: `meson test` 4/4 — new `udp loopback` test
  encodes synthetic frames, packetizes, sends over a real localhost UDP
  socket, depacketizes, and decodes 10/10 frames with correct dimensions.
  Valgrind clean (loopback + codec tests).
- **Manual validation pending (needs the desktop)**: run the two commands in
  `docs/RUNBOOK.md` — receiver window should show the captured desktop. Tick
  `docs/PHASES.md` Phase 5 after this works.
- Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A;
  3-byte canonical start codes; drop-on-damage loss handling).
- Phase 3 capture done and validated (PipeWire/portal backend; the
  `impl_ext_end_proxy` wrong-context warnings were fixed by holding the
  thread-loop lock across all pw proxy operations).

## Decisions

- Language: C++20 with explicit modern-C++ guidelines in `AGENTS.md` and
  `cpp-meson-build/SKILL.md`.
- Build system: Meson.
- Capture: PipeWire + xdg-desktop-portal.
- Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
- H.264 encoder path: software `libx264`, low-latency settings, Annex-B output.
- **SPS/PPS are sent in-band ahead of every keyframe** (no GLOBAL_HEADER);
  the decoder starts from the bitstream alone. `DecoderConfig.extradata`
  remains available if signaling ever negotiates parameters out of band.
- Transport: RTP over UDP via raw POSIX sockets for now; **ASIO stays a
  Phase 6+ option** (the ARCHITECTURE.md dependency table places it with
  signaling/discovery). IPv4 only at the transport level for now.
- Rendering: **SDL3** (`sdl3` pkg-config, 3.4 installed); plain texture
  upload, no GPU pipeline yet.
- **Pixel-format convention trap**: FFmpeg names packed formats in memory
  byte order (AV_PIX_FMT_RGBA = R,G,B,A in memory), but SDL names 32-bit
  formats MSB-first (SDL_PIXELFORMAT_RGBA8888 = A,B,G,R in memory).
  FFmpeg RGBA data therefore needs SDL_PIXELFORMAT_ABGR8888 — using
  RGBA8888 paints the alpha byte as red (red-tinted image).
- Discovery: mDNS/Avahi.
- Namespace: `sc`.
- Module error results use per-module `std::variant<T, XError>` types
  (`CodecResult`, `CaptureResult`, `RendererResult`) since C++20 has no
  `std::expected`.

## Active blockers

None.

## Open questions

- GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
- Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate
  after software encode works; note the Pi Zero 2 W receiver use case makes
  hardware **decode** (V4L2/MMAL) the more urgent half.
- IPv6 at the transport layer — RESOLVED in Phase 6: dual-stack everywhere
  (AF_UNSPEC resolution, IPV6_V6ONLY=0 listeners); validated over both
  families.

## Forward-looking review notes (for later phases)

- `RtpTransport::start/send` return plain bools (scaffold API); error
  messages are lost — consider an error channel when signaling lands.
- Very high bitrates can still exceed even a raised receive buffer if
  `net.core.rmem_max` is low on the receiver; the encoder VBV bounds
  bursts to ~2 frame periods, so this needs `--bitrate` ≳ 100 Mbps to
  matter. Document in RUNBOOK.
- DMA-BUF-only portal streams are rejected with a clear message (hardware
  path is Phase 7).
- No negative-path tests yet (bad config, bad stride, undersized buffer).
- `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting
  encoder is ever added, prefer an explicit config flag over the heuristic.
- Region targets are rejected: the desktop portal has no region capture.
- `CaptureSession::next_frame()` returns `nullopt` on stream error without
  surfacing the reason (logged to stderr).
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
  we control never emit them, but third-party interop would need support.

### Android app review (2026-09-10) — findings FIXED same day

Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified
faithful to the C++ side source-to-source), followed by a fix pass that
landed all findings. No commit yet (user has not asked).

Fixed:

- `SignalingServer.send()` no longer throws (mirrors the C++ server, which
  ignores write failures): a broken signaling TCP no longer kills the
  `rtp-reader` thread via `requestPli()`; `peerOut` is dropped (closing the
  socket) on write failure.
- `H264Decoder.configure()` assigns the created codec before configuring,
  so a configure/start failure can no longer orphan the MediaCodec instance
  (no finalizer; scarce native slots).
- Offer-before-surface no longer configures a ByteBuffer-mode decoder: the
  offer is parked as `pendingOffer` and `attachSurface()` configures later
  (a surface-less codec can never take one — `setOutputSurface` refuses
  it, an IllegalStateException crash on the main thread). The late
  configure sends a PLI (the sender only emits IDRs when asked).
- `ReceiverActivity.onSurfaceTextureDestroyed` → `pipeline.detachSurface()`;
  the pipeline/decoder forget dead surfaces instead of configuring against
  them (frames decode unrendered until the next attach).
- The status overlay reappears: `onStatus` sets `visibility = VISIBLE`
  (it used to write into a GONE view after the first frame).
- `H264Decoder.feed()` returns false on input-queue timeout → the pipeline
  requests a PLI instead of silently dropping the frame (no flush — the
  codec is healthy).
- The depacketizer accumulates into `ByteArrayOutputStream`s instead of
  boxed `ArrayList<Int>` (was ~MB/s of Integer allocations on keyframes).
- NSD reflection fallback deleted (dead code — see the quirk note above).
- Thread-safety: all codec calls now run under `decoderLock` (MediaCodec is
  not thread-safe; feed/drain previously raced attachSurface);
  `requestPli()` is thread-safe via `pliLock`.

Validation: `gradle :app:assembleDebug :app:testDebugUnitTest` green
(25/25, full --rerun-tasks rebuild, only two pre-existing warnings);
`meson test` 5/5 unchanged (no C++ touched).

### Discovery bug found and fixed on-device (2026-09-10, same day)

The user reported the desktop sender never discovered the phone.
Root cause (found live on the Fairphone 6): `registerService(info, 0,
listener)` — Android 16's `NsdManager.checkProtocol()` rejects protocol
`0` with `IllegalArgumentException: Unsupported protocol`. The old code
swallowed it into the dead reflection fallback, and `start()` posted the
"Listening…" status AFTER advertiseNsd, overwriting the failure text — so
mDNS never advertised and Phase 8's NSD validation was only ever "no
crash" (8.5 streamed via --peer, masking it).

Fixes in `ReceiverPipeline.advertiseNsd()`/`start()`:
- pass `NsdManager.PROTOCOL_DNS_SD`;
- advertise AFTER the listening status so a failure stays visible;
- `Log.e` the registration exception; `Log.i` on registered success.

On-device validation (adb, live): `mDNS registered:
screencast._screencast._tcp` in logcat; `dumpsys servicediscovery` shows
the active Advertiser (key diagnostic: `mClientRequests` empty == no
request ever issued); desktop `avahi-browse` and `screencast --discover`
list the phone at 192.168.178.29:5005; a 25s `--send --target monitor
--peer 192.168.178.29:5005` session decoded (in-band SPS reconfigured
320x240 → 2496x1040) and rendered (screencap mean brightness 0.51).
Note: with both the Pi and the phone on the LAN, plain `--send` refuses
(two receivers found) — target the phone with `--peer`.

Follow-up from the same on-device session — startup squish eliminated:
`H264Decoder.outputSize()` previously reported the 320x240 configure()
placeholder until the codec parsed the SPS, so `fitVideo` sized the
TextureView 1488x1116 (1.33 aspect) and the first rendered frame(s) of
2.4 content were visibly squished. It now returns null until
`INFO_OUTPUT_FORMAT_CHANGED` fires; the first frame renders into the
fullscreen surface and the correct letterbox (2484x1035) follows within
~40ms. Validated live: no placeholder "video size" line in logcat.

Commits: 9a24933 (PROTOCOL_DNS_SD discovery fix + docs), plus the
outputSize fix (see git log).

Still open (accepted, needs a device or a new test dep):

- Untested on device: `setOutputSurface` mid-session (surface switch) and
  the whole pendingOffer path; API-35 `detachOutputSurface()` could replace
  the render-flag approach.
- No JVM tests for `SignalingMessage`/`SignalingServer` (would need the
  `org.json:json` test dependency; android.jar stubs throw).
- Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200),
  RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a
  future phase).
- Verified NOT a bug: reusing one `DatagramPacket` without resetting its
  length — modern JVMs recv by buffer capacity (JDK-21 probe + the
  successful on-device streaming confirm it).