screen_cast/.agents/MEMORY.md

Project Memory — screen_cast

Last updated: Phase 8 (Android receiver app) complete and validated on a Fairphone 6; all prior phases done.

Project state

• Phase 8 done: Android receiver app (android/, Kotlin, minSdk 30, app id screen_cast.receiver): phone as second receiver (screen → HDMI via USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++ changes. AGP 9 built-in Kotlin (no kotlin plugin, no kotlinOptions; Kotlin targets compileOptions, Java 17); no androidx (framework + org.json + JUnit). 25 JVM tests green. Validated end-to-end on a Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in RUNBOOK): INTERNET permission is REQUIRED for NsdService; DatagramSocket.localPort (.port is -1 unconnected; .localAddress is Inet6Address, not InetSocketAddress); pass the Surface to MediaCodec.configure() + start(); render via releaseOutputBuffer(render=true); C2 AVC needs a concrete size at configure (in-band SPS reconfigures); MediaFormat.format()/ KEY_MIME_TYPE not public in API 36; NSD RegistrationListener replaced ResolutionListener (reflection fallback for older devices); android._video-scaling: C2 scales output to the Surface → letterbox by sizing the TextureView to the video aspect, NOT a transform matrix (double scale). Sender-side: Hyprland + GTK portal's --target monitor picks the FIRST output (eDP-1, wrong display) — use --target window (hyprland-share-picker). Activity FQN for am start: screen_cast.receiver/screen_cast.ReceiverActivity (Kotlin package = namespace screen_cast).

• Phase 6 done: Avahi mDNS discovery (_screencast._tcp — receiver announces its signaling port via a threaded-poll Avahi client; senders browse+resolve) plus JSON session signaling (offer/answer) over TCP with a signaling server on the receiver (port 5005). --send auto-discovers when exactly one receiver is found; --discover lists receivers; --peer targets a receiver directly. meson test 5/5, valgrind clean.

• Signaling is newline-delimited JSON over TCP, not WebSocket: no WS library was installed and the rtp-networking skill permits plain TCP. The wire format (one JSON object per line: offer/answer with session id, codec, rtp port) is transport-agnostic; Phase 7 adds the WS dependency if needed.

• Dual-stack everywhere: this machine resolves its own services over IPv6 (ULA + link-local), so the UDP transport, signaling client, and both listeners now support both families (IPv6 sockets with IPV6_V6ONLY=0 for dual-stack listening; AF_UNSPEC getaddrinfo for peers). Validated over both 127.0.0.1 and ::1.

• Discovery dedupe: one entry per (service_name, signaling_port) — a host with many interfaces otherwise registers dozens of address variants.

• Receiver-only builds (-Dsender=false, meson option): skip the capture backend and sender pipeline (SC_HAS_SENDER guards in main.cpp/pipelines.cpp; --send refuses cleanly in such builds). scripts/install-receiver.sh uses this to install on small ARM boards (user's Raspberry Pi Zero 2 W receiver) without PipeWire/portal deps, building SDL3 from source when the distro lacks it. Requires GCC 13+ (<format>). Validated on x86_64: builds, tests 5/5, --discover works, --send refuses with a message.

• systemd autostart: systemd/screencast-receiver.service is a template (__SC_RECEIVER_BIN__/__SC_RECEIVER_USER__); the install script substitutes and enables it (opt out: SC_RECEIVER_SERVICE=0), adding the run user to video/render/input for headless KMSDRM.

• systemd tty trap (hit on the real Pi, reproduced locally): with StandardInput=tty + TTYPath=tty1, the service NEVER started — systemd blocks PRE-EXEC in acquire_terminal() waiting for a tty that the console session/getty already owns. Symptoms: status shows the main PID as "(screencast)" with Tasks:1 and ~40ms CPU, silent journal, no mDNS. Diagnosed by gdb-attaching the stuck process (acquire_terminal backtrace) after reproducing with a local transient unit. Fix: the service owns a dedicated free VT (tty7) + tolerant ExecStartPre=-/usr/bin/chvt 7; tty1 keeps the console. Also: systemctl status Tasks counts THREADS (a healthy receiver shows ~15), and "Console Autologin" advice was WRONG (it made the hang deterministic) — removed from all docs.

• Fullscreen headless rendering: under the KMSDRM video driver (no window manager) the renderer goes fullscreen automatically; --fullscreen forces it on desktops. Aspect is preserved via SDL_SetRenderLogicalPresentation(LETTERBOX) (verified: 4:3 feed on a 3440x1440 monitor rendered with black pillarbox bars; --fullscreen window covered the full monitor). The service lifecycle was validated with a user unit: SIGTERM stop → clean exit (success), mDNS withdrawn, no restart.

• mDNS operational lesson (hit during validation): kill -9 on a process holding an avahi registration leaves stale daemon records; the service then browses but never resolves (timeout for every peer) until records expire (~75 min). systemctl restart avahi-daemon clears it. Recorded in RUNBOOK; always stop with SIGTERM.

• Low-RAM build validated on hardware: the Zero 2 W receiver build completes with the install script's temporary-swap + single-job handling (the user confirmed; nlohmann/json's signaling TU was the OOM trigger before the fix). Cross-compiling on the dev machine was considered and dropped — the on-Pi build works and the toolchain/container effort was not needed.

• App icons wired up (screencast_icon/): single 256 px PNG used everywhere. meson install ships it to share/icons/hicolor/256x256/apps/screencast.png; the GTK panel sets it via set_icon_name("screencast") (GTK4 removed the pixel-buffer window icon — themed names only; gtk_icon_theme_has_icon verified OK); the SDL receiver window embeds the PNG at build time (icon_png_data.h via scripts/icon_to_header.py custom_target) and SDL_SetWindowIcon with an optional SDL3_image dependency (no pkg-config ships with it → meson.get_compiler('cpp').find_library('SDL3_image', required: false)

  • disabler; -DSC_HAS_WINDOW_ICON gate; Pi builds without it still work). The waybar widget now shows the icon as background-image in ~/.config/waybar/style.css (▶/⏸ text hidden; .idle dimmed, .streaming green tint) instead of text glyphs; backup style.css.bak-20260909-*. Lesson: icons appear in compositor taskbars/switchers, not title bars (GTK4 CSD and Hyprland decorations don't draw them), so validate via theme lookup, not title-bar screenshots. sudo pacman -S sdl3_image done (extra/sdl3_image 3.4.6).

• GTK panel + waybar widget: screencast-gui (gtkmm-4.0, behind -Dgui=true; app internals now live in the sc_app_core static lib so CLI and GUI share pipelines/session/state). screencast waybar [--toggle] prints the waybar module line and toggles streaming via SIGTERM (stop) or a detached re-exec spawn from last-session.json (start). State lives in $XDG_RUNTIME_DIR/screencast/sender.json (written by the pipeline with session id, receiver, bitrate, pid, start time; stale files are detected by pid-liveness). The waybar config was wired into the user's bar (custom/screencast before custom/timetrack, with backup) and the binaries installed to /usr/local/bin.

  • Pi Wi-Fi hotspot (scripts/pi-hotspot.sh on|off|status): NetworkManager AP mode (WPA2, ipv4 shared → built-in DHCP/NAT, Pi at 10.42.0.1). Takes over wlan0 while active; generated PSK stored in /etc/screencast-hotspot.conf. No application changes needed — the receiver already announces on all interfaces. nmcli property syntax validated against NM 1.58 with a disposable profile; AP bring-up itself can only be validated on the Pi.

• Phase 7 resilience shipped (pending Pi-side hw-decode run):

  • PLI over signaling: SessionPli message; the depacketizer now returns DepacketizeResult{access_unit, frame_dropped}; the receiver rate-limits PLIs to 1/500 ms; the sender keeps the signaling channel open and calls SenderPipeline::request_keyframe() (thread-safe atomic → run-loop → encoder). Validated end-to-end with a probe that drops a mid-keyframe packet: receiver logs "frame damaged", PLI arrives with the session id.
  • RtpJitterBuffer (rtp_packet.h): sequence reordering, 16 pkt/60 ms, straggler discard via serial-number arithmetic, overflow flush for genuine loss. Zero added latency on in-order streams.
  • Hardware decode probe in DecoderFactory (DecoderConfig.hardware_accel, default true): h264_v4l2m2m first, software fallback, --swdecode opts out. On the desktop the probe fails cleanly ("Could not find a valid device") and falls back; on the Pi it should pick the VideoCore m2m device — NEEDS THE USER'S PI RUN to confirm.

• REAL-HARDWARE VALIDATION (desktop → Pi Zero 2 W over Wi-Fi): the full chain works on two machines: mDNS discovery → signaling negotiation → RTP over Wi-Fi → software H.264 decode → fullscreen KMSDRM letterboxed rendering on the Pi's display. Two transport lessons from that network: the LAN's 6to4 (2002::) addresses are unreachable between peers — senders now try all discovered addresses in reachability order (private IPv4 first; build/pi_probe.cpp in the build dir is the probe that validated this end-to-end against the real Pi).

• Phase 5 (local UDP sender→receiver loopback) is implemented:

  • UdpRtpTransport (src/network/udp_transport.cpp): raw POSIX sockets, AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop() closes the socket to unblock the receive jthread. ASIO was deliberately deferred to Phase 6 (see decisions).
  • screencast binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline (recv→depacketize→decode→bounded 3-frame queue→render thread).
  • SdlRenderer (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture, RGBA texture upload, texture recreated on resolution change. RendererFactory::create now returns RendererResult (error channel added, mirroring codec/capture patterns).
  • Encoder change: GLOBAL_HEADER removed so libx264 repeats SPS/PPS in-band at every keyframe; a receiver now decodes from the bitstream alone (mid-stream join, PLI recovery-ready). get_extradata() is empty in this mode; codec round-trip test updated to match the streaming path.
  • Burst-control fixes after the first real run (receiver saw non-existing PPS 0 forever): without VBV, a 2256x1504 IDR burst (~100s of KB of back-to-back FU-A packets) overflowed the ~208KB default UDP receive buffer; the sequence gap made drop-on-damage discard whole keyframes including their in-band SPS/PPS, so the receiver never recovered. Fixes: encoder VBV (rc_max_rate = bitrate, rc_buffer_size = bitrate*2/fps — caps any keyframe to ~2 frame periods of bytes), microsecond encoder time_base (kills the 25fps pts quantization that duplicated timestamps), and a best-effort 4MB SO_RCVBUF on the receive socket (kernel clamps to rmem_max).
  • Display fixes after the receiver window never appeared: (1) SDL must own the window from one thread — creating it on main and pumping/presenting from the render thread left the Wayland surface unmapped; the renderer now creates/polls/presents/destroys entirely on the render thread (with a start() init handshake). (2) Wayland windows are invisible until the first render commit, so the renderer presents a blank frame at init (visible black window while waiting). (3) The receiver logs stream started (WxH) on the first decoded frame and any first present failure.
  • Phase 5 validated end-to-end (headless, without the portal): a synthetic RTP feed of solid red/green frames drove the real receiver over localhost UDP; the window mapped, logged stream started (320x240), and a grim screenshot of the window region showed the fed color (V=198, SAT=74 during the red feed) — decoded video visibly rendering. The user's real sender run showed capture→encode→send working (1 IDR + 12 P-frames, IDR capped at ~20KB by the VBV).

• Automated validation: meson test 4/4 — new udp loopback test encodes synthetic frames, packetizes, sends over a real localhost UDP socket, depacketizes, and decodes 10/10 frames with correct dimensions. Valgrind clean (loopback + codec tests).

• Manual validation pending (needs the desktop): run the two commands in docs/RUNBOOK.md — receiver window should show the captured desktop. Tick docs/PHASES.md Phase 5 after this works.

• Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A; 3-byte canonical start codes; drop-on-damage loss handling).

• Phase 3 capture done and validated (PipeWire/portal backend; the impl_ext_end_proxy wrong-context warnings were fixed by holding the thread-loop lock across all pw proxy operations).

Decisions

• Language: C++20 with explicit modern-C++ guidelines in AGENTS.md and cpp-meson-build/SKILL.md.
• Build system: Meson.
• Capture: PipeWire + xdg-desktop-portal.
• Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
• H.264 encoder path: software libx264, low-latency settings, Annex-B output.
• SPS/PPS are sent in-band ahead of every keyframe (no GLOBAL_HEADER); the decoder starts from the bitstream alone. DecoderConfig.extradata remains available if signaling ever negotiates parameters out of band.
• Transport: RTP over UDP via raw POSIX sockets for now; ASIO stays a Phase 6+ option (the ARCHITECTURE.md dependency table places it with signaling/discovery). IPv4 only at the transport level for now.
• Rendering: SDL3 (sdl3 pkg-config, 3.4 installed); plain texture upload, no GPU pipeline yet.
• Pixel-format convention trap: FFmpeg names packed formats in memory byte order (AV_PIX_FMT_RGBA = R,G,B,A in memory), but SDL names 32-bit formats MSB-first (SDL_PIXELFORMAT_RGBA8888 = A,B,G,R in memory). FFmpeg RGBA data therefore needs SDL_PIXELFORMAT_ABGR8888 — using RGBA8888 paints the alpha byte as red (red-tinted image).
• Discovery: mDNS/Avahi.
• Namespace: sc.
• Module error results use per-module std::variant<T, XError> types (CodecResult, CaptureResult, RendererResult) since C++20 has no std::expected.

Active blockers

None.

Open questions

• GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
• Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after software encode works; note the Pi Zero 2 W receiver use case makes hardware decode (V4L2/MMAL) the more urgent half.
• IPv6 at the transport layer — RESOLVED in Phase 6: dual-stack everywhere (AF_UNSPEC resolution, IPV6_V6ONLY=0 listeners); validated over both families.

Forward-looking review notes (for later phases)

• RtpTransport::start/send return plain bools (scaffold API); error messages are lost — consider an error channel when signaling lands.
• Very high bitrates can still exceed even a raised receive buffer if net.core.rmem_max is low on the receiver; the encoder VBV bounds bursts to ~2 frame periods, so this needs --bitrate ≳ 100 Mbps to matter. Document in RUNBOOK.
• DMA-BUF-only portal streams are rejected with a clear message (hardware path is Phase 7).
• No negative-path tests yet (bad config, bad stride, undersized buffer).
• to_annex_b_h264 sniffs AVCC vs Annex-B by content; if an AVCC-emitting encoder is ever added, prefer an explicit config flag over the heuristic.
• Region targets are rejected: the desktop portal has no region capture.
• CaptureSession::next_frame() returns nullopt on stream error without surfacing the reason (logged to stderr).
• Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders we control never emit them, but third-party interop would need support.