screen_cast/.agents/MEMORY.md

Project Memory — screen_cast

Last updated: Phase 8 (Android receiver app) complete and validated on a Fairphone 6; all prior phases done. The Android app's review findings were fixed in a follow-up pass (same day) — see "Android app review" at the bottom of this file.

Project state

• Phase 9 in progress: iOS receiver app (ios/, Swift, min iOS 17): iPhone as a second receiver. Speaks the same signaling+RTP protocol — no C++ changes. Mirrors the Android receiver (Phase 8) source-to-source.

  • Scaffolding: XcodeGen project.yml (Info.plist carries NSLocalNetworkUsageDescription + NSBonjourServices: _screencast._tcp) + bootstrap.sh (downloads XcodeGen from the GitHub release, no Homebrew; generates the project; builds/tests via xcodebuild). ios/README.md has build/install/device steps.
  • App (Receiver/App): SwiftUI + AVSampleBufferDisplayLayer (.resizeAspect = letterbox — the same "size the surface, not a transform" lesson as Android); ReceiverController (ObservableObject) drives start/stop on scenePhase.
  • Protocol core (Receiver/Rtp, Receiver/Signaling): RtpHeader/RtpPacket/ JitterBuffer/H264Depacketizer ported source-to-source; SignalingMessage (JSONSerialization) + LineAssembler + SignalingServer (BSD sockets, dual-stack, most-recent-peer-wins, MSG_NOSIGNAL sends that never throw); AvccConverter (Annex-B ↔ AVCC) + NalExtractor (SPS/PPS) are new for the VideoToolbox path.
  • Decode (Receiver/Decode): H264FormatDescription (Core Foundation H.264 config recipe) + H264VideoToolboxDecoder (in-band SPS/PPS → session; kVTDecompressionPropertyKey_RealTime; session recreated on size change; the output callback may run on a worker thread, so state is lock-guarded) → AVSampleBufferRenderSink (AVSampleBufferDisplayLayerSession).
  • Support (Receiver/Support): UdpTransport (poll-based recv so close() can't strand a blocked recvfrom) + LocalAddress (getifaddrs for the --peer hint).
  • Pipeline (Receiver/Pipeline): ReceiverPipeline — one serial queue for all state + decode; reader thread only polls/receives UDP; pendingOffer for late surface attach (+ PLI); PLI rate-limited 500 ms; first-frame flag; video size tracked (never reports a placeholder before the first real keyframe — the Android startup-squish lesson).
  • Tests (ios/ReceiverTests): the 25 Android JVM tests ported to XCTest plus new coverage for signaling JSON, line framing, AVCC, NAL extraction.
  • .gitignore excludes the generated ios/Receiver.xcodeproj/, ios/tools/, ios/Receiver/Info.plist (the project.yml info block is the source of truth).
  • NOT YET VALIDATED (this box is Linux, no Xcode/iOS SDK): nothing here compiles or runs. 9.5 needs a Mac with Xcode 26 + the iPhone 16. Highest-risk on-device items: (1) local-network + Bonjour consent (NSBonjourServices must be _screencast._tcp; iOS 26 tightened the prompt), (2) the H264FormatDescription CF ownership recipe (a bug crashes on the first keyframe — loud, not silent), (3) VideoToolbox decode → AVSampleBufferDisplayLayer render. Run ios/bootstrap.sh test first on the Mac.

• Phase 8 done: Android receiver app (android/, Kotlin, minSdk 30, app id screen_cast.receiver): phone as second receiver (screen → HDMI via USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++ changes. AGP 9 built-in Kotlin (no kotlin plugin, no kotlinOptions; Kotlin targets compileOptions, Java 17); no androidx (framework + org.json + JUnit). 25 JVM tests green. Validated end-to-end on a Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in RUNBOOK): INTERNET permission is REQUIRED for NsdService; DatagramSocket.localPort (.port is -1 unconnected; .localAddress is Inet6Address, not InetSocketAddress); pass the Surface to MediaCodec.configure() + start(); render via releaseOutputBuffer(render=true); C2 AVC needs a concrete size at configure (in-band SPS reconfigures); MediaFormat.format()/ KEY_MIME_TYPE not public in API 36; NSD: the classic registerService(info, flags, RegistrationListener) API exists from API 16 through 36 (javap-verified on the android-36 SDK) — an old reflection fallback targeting a never-existent "ResolutionListener" was removed as dead code; android._video-scaling: C2 scales output to the Surface → letterbox by sizing the TextureView to the video aspect, NOT a transform matrix (double scale). Sender-side: Hyprland + GTK portal's --target monitor picks the FIRST output (eDP-1, wrong display) — use --target window (hyprland-share-picker). Activity FQN for am start: screen_cast.receiver/screen_cast.ReceiverActivity (Kotlin package = namespace screen_cast).

• Phase 6 done: Avahi mDNS discovery (_screencast._tcp — receiver announces its signaling port via a threaded-poll Avahi client; senders browse+resolve) plus JSON session signaling (offer/answer) over TCP with a signaling server on the receiver (port 5005). --send auto-discovers when exactly one receiver is found; --discover lists receivers; --peer targets a receiver directly. meson test 5/5, valgrind clean.

• Signaling is newline-delimited JSON over TCP, not WebSocket: no WS library was installed and the rtp-networking skill permits plain TCP. The wire format (one JSON object per line: offer/answer with session id, codec, rtp port) is transport-agnostic; Phase 7 adds the WS dependency if needed.

• Dual-stack everywhere: this machine resolves its own services over IPv6 (ULA + link-local), so the UDP transport, signaling client, and both listeners now support both families (IPv6 sockets with IPV6_V6ONLY=0 for dual-stack listening; AF_UNSPEC getaddrinfo for peers). Validated over both 127.0.0.1 and ::1.

• Discovery dedupe: one entry per (service_name, signaling_port) — a host with many interfaces otherwise registers dozens of address variants.

• Receiver-only builds (-Dsender=false, meson option): skip the capture backend and sender pipeline (SC_HAS_SENDER guards in main.cpp/pipelines.cpp; --send refuses cleanly in such builds). scripts/install-receiver.sh uses this to install on small ARM boards (user's Raspberry Pi Zero 2 W receiver) without PipeWire/portal deps, building SDL3 from source when the distro lacks it. Requires GCC 13+ (<format>). Validated on x86_64: builds, tests 5/5, --discover works, --send refuses with a message.

• systemd autostart: systemd/screencast-receiver.service is a template (__SC_RECEIVER_BIN__/__SC_RECEIVER_USER__); the install script substitutes and enables it (opt out: SC_RECEIVER_SERVICE=0), adding the run user to video/render/input for headless KMSDRM.

• systemd tty trap (hit on the real Pi, reproduced locally): with StandardInput=tty + TTYPath=tty1, the service NEVER started — systemd blocks PRE-EXEC in acquire_terminal() waiting for a tty that the console session/getty already owns. Symptoms: status shows the main PID as "(screencast)" with Tasks:1 and ~40ms CPU, silent journal, no mDNS. Diagnosed by gdb-attaching the stuck process (acquire_terminal backtrace) after reproducing with a local transient unit. Fix: the service owns a dedicated free VT (tty7) + tolerant ExecStartPre=-/usr/bin/chvt 7; tty1 keeps the console. Also: systemctl status Tasks counts THREADS (a healthy receiver shows ~15), and "Console Autologin" advice was WRONG (it made the hang deterministic) — removed from all docs.

• Fullscreen headless rendering: under the KMSDRM video driver (no window manager) the renderer goes fullscreen automatically; --fullscreen forces it on desktops. Aspect is preserved via SDL_SetRenderLogicalPresentation(LETTERBOX) (verified: 4:3 feed on a 3440x1440 monitor rendered with black pillarbox bars; --fullscreen window covered the full monitor). The service lifecycle was validated with a user unit: SIGTERM stop → clean exit (success), mDNS withdrawn, no restart.

• mDNS operational lesson (hit during validation): kill -9 on a process holding an avahi registration leaves stale daemon records; the service then browses but never resolves (timeout for every peer) until records expire (~75 min). systemctl restart avahi-daemon clears it. Recorded in RUNBOOK; always stop with SIGTERM.

• Low-RAM build validated on hardware: the Zero 2 W receiver build completes with the install script's temporary-swap + single-job handling (the user confirmed; nlohmann/json's signaling TU was the OOM trigger before the fix). Cross-compiling on the dev machine was considered and dropped — the on-Pi build works and the toolchain/container effort was not needed.

• App icons wired up (screencast_icon/): single 256 px PNG used everywhere. meson install ships it to share/icons/hicolor/256x256/apps/screencast.png; the GTK panel sets it via set_icon_name("screencast") (GTK4 removed the pixel-buffer window icon — themed names only; gtk_icon_theme_has_icon verified OK); the SDL receiver window embeds the PNG at build time (icon_png_data.h via scripts/icon_to_header.py custom_target) and SDL_SetWindowIcon with an optional SDL3_image dependency (no pkg-config ships with it → meson.get_compiler('cpp').find_library('SDL3_image', required: false)

  • disabler; -DSC_HAS_WINDOW_ICON gate; Pi builds without it still work). The waybar widget now shows the icon as background-image in ~/.config/waybar/style.css (▶/⏸ text hidden; .idle dimmed, .streaming green tint) instead of text glyphs; backup style.css.bak-20260909-*. Lesson: icons appear in compositor taskbars/switchers, not title bars (GTK4 CSD and Hyprland decorations don't draw them), so validate via theme lookup, not title-bar screenshots. sudo pacman -S sdl3_image done (extra/sdl3_image 3.4.6).

• GTK panel + waybar widget: screencast-gui (gtkmm-4.0, behind -Dgui=true; app internals now live in the sc_app_core static lib so CLI and GUI share pipelines/session/state). screencast waybar [--toggle] prints the waybar module line and toggles streaming via SIGTERM (stop) or a detached re-exec spawn from last-session.json (start). State lives in $XDG_RUNTIME_DIR/screencast/sender.json (written by the pipeline with session id, receiver, bitrate, pid, start time; stale files are detected by pid-liveness). The waybar config was wired into the user's bar (custom/screencast before custom/timetrack, with backup) and the binaries installed to /usr/local/bin.

  • Pi Wi-Fi hotspot (scripts/pi-hotspot.sh on|off|status): NetworkManager AP mode (WPA2, ipv4 shared → built-in DHCP/NAT, Pi at 10.42.0.1). Takes over wlan0 while active; generated PSK stored in /etc/screencast-hotspot.conf. No application changes needed — the receiver already announces on all interfaces. nmcli property syntax validated against NM 1.58 with a disposable profile; AP bring-up itself can only be validated on the Pi.

• Phase 7 resilience shipped (pending Pi-side hw-decode run):

  • PLI over signaling: SessionPli message; the depacketizer now returns DepacketizeResult{access_unit, frame_dropped}; the receiver rate-limits PLIs to 1/500 ms; the sender keeps the signaling channel open and calls SenderPipeline::request_keyframe() (thread-safe atomic → run-loop → encoder). Validated end-to-end with a probe that drops a mid-keyframe packet: receiver logs "frame damaged", PLI arrives with the session id.
  • RtpJitterBuffer (rtp_packet.h): sequence reordering, 16 pkt/60 ms, straggler discard via serial-number arithmetic, overflow flush for genuine loss. Zero added latency on in-order streams.
  • Hardware decode probe in DecoderFactory (DecoderConfig.hardware_accel, default true): h264_v4l2m2m first, software fallback, --swdecode opts out. On the desktop the probe fails cleanly ("Could not find a valid device") and falls back; on the Pi it should pick the VideoCore m2m device — NEEDS THE USER'S PI RUN to confirm.

• REAL-HARDWARE VALIDATION (desktop → Pi Zero 2 W over Wi-Fi): the full chain works on two machines: mDNS discovery → signaling negotiation → RTP over Wi-Fi → software H.264 decode → fullscreen KMSDRM letterboxed rendering on the Pi's display. Two transport lessons from that network: the LAN's 6to4 (2002::) addresses are unreachable between peers — senders now try all discovered addresses in reachability order (private IPv4 first; build/pi_probe.cpp in the build dir is the probe that validated this end-to-end against the real Pi).

• Phase 5 (local UDP sender→receiver loopback) is implemented:

  • UdpRtpTransport (src/network/udp_transport.cpp): raw POSIX sockets, AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop() closes the socket to unblock the receive jthread. ASIO was deliberately deferred to Phase 6 (see decisions).
  • screencast binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline (recv→depacketize→decode→bounded 3-frame queue→render thread).
  • SdlRenderer (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture, RGBA texture upload, texture recreated on resolution change. RendererFactory::create now returns RendererResult (error channel added, mirroring codec/capture patterns).
  • Encoder change: GLOBAL_HEADER removed so libx264 repeats SPS/PPS in-band at every keyframe; a receiver now decodes from the bitstream alone (mid-stream join, PLI recovery-ready). get_extradata() is empty in this mode; codec round-trip test updated to match the streaming path.
  • Burst-control fixes after the first real run (receiver saw non-existing PPS 0 forever): without VBV, a 2256x1504 IDR burst (~100s of KB of back-to-back FU-A packets) overflowed the ~208KB default UDP receive buffer; the sequence gap made drop-on-damage discard whole keyframes including their in-band SPS/PPS, so the receiver never recovered. Fixes: encoder VBV (rc_max_rate = bitrate, rc_buffer_size = bitrate*2/fps — caps any keyframe to ~2 frame periods of bytes), microsecond encoder time_base (kills the 25fps pts quantization that duplicated timestamps), and a best-effort 4MB SO_RCVBUF on the receive socket (kernel clamps to rmem_max).
  • Display fixes after the receiver window never appeared: (1) SDL must own the window from one thread — creating it on main and pumping/presenting from the render thread left the Wayland surface unmapped; the renderer now creates/polls/presents/destroys entirely on the render thread (with a start() init handshake). (2) Wayland windows are invisible until the first render commit, so the renderer presents a blank frame at init (visible black window while waiting). (3) The receiver logs stream started (WxH) on the first decoded frame and any first present failure.
  • Phase 5 validated end-to-end (headless, without the portal): a synthetic RTP feed of solid red/green frames drove the real receiver over localhost UDP; the window mapped, logged stream started (320x240), and a grim screenshot of the window region showed the fed color (V=198, SAT=74 during the red feed) — decoded video visibly rendering. The user's real sender run showed capture→encode→send working (1 IDR + 12 P-frames, IDR capped at ~20KB by the VBV).

• Automated validation: meson test 4/4 — new udp loopback test encodes synthetic frames, packetizes, sends over a real localhost UDP socket, depacketizes, and decodes 10/10 frames with correct dimensions. Valgrind clean (loopback + codec tests).

• Manual validation pending (needs the desktop): run the two commands in docs/RUNBOOK.md — receiver window should show the captured desktop. Tick docs/PHASES.md Phase 5 after this works.

• Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A; 3-byte canonical start codes; drop-on-damage loss handling).

• Phase 3 capture done and validated (PipeWire/portal backend; the impl_ext_end_proxy wrong-context warnings were fixed by holding the thread-loop lock across all pw proxy operations).

Decisions

• Language: C++20 with explicit modern-C++ guidelines in AGENTS.md and cpp-meson-build/SKILL.md.
• Build system: Meson.
• Capture: PipeWire + xdg-desktop-portal.
• Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
• H.264 encoder path: software libx264, low-latency settings, Annex-B output.
• SPS/PPS are sent in-band ahead of every keyframe (no GLOBAL_HEADER); the decoder starts from the bitstream alone. DecoderConfig.extradata remains available if signaling ever negotiates parameters out of band.
• Transport: RTP over UDP via raw POSIX sockets for now; ASIO stays a Phase 6+ option (the ARCHITECTURE.md dependency table places it with signaling/discovery). IPv4 only at the transport level for now.
• Rendering: SDL3 (sdl3 pkg-config, 3.4 installed); plain texture upload, no GPU pipeline yet.
• Pixel-format convention trap: FFmpeg names packed formats in memory byte order (AV_PIX_FMT_RGBA = R,G,B,A in memory), but SDL names 32-bit formats MSB-first (SDL_PIXELFORMAT_RGBA8888 = A,B,G,R in memory). FFmpeg RGBA data therefore needs SDL_PIXELFORMAT_ABGR8888 — using RGBA8888 paints the alpha byte as red (red-tinted image).
• Discovery: mDNS/Avahi.
• Namespace: sc.
• Module error results use per-module std::variant<T, XError> types (CodecResult, CaptureResult, RendererResult) since C++20 has no std::expected.

Active blockers

None.

Open questions

• GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
• Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after software encode works; note the Pi Zero 2 W receiver use case makes hardware decode (V4L2/MMAL) the more urgent half.
• IPv6 at the transport layer — RESOLVED in Phase 6: dual-stack everywhere (AF_UNSPEC resolution, IPV6_V6ONLY=0 listeners); validated over both families.

Forward-looking review notes (for later phases)

• RtpTransport::start/send return plain bools (scaffold API); error messages are lost — consider an error channel when signaling lands.
• Very high bitrates can still exceed even a raised receive buffer if net.core.rmem_max is low on the receiver; the encoder VBV bounds bursts to ~2 frame periods, so this needs --bitrate ≳ 100 Mbps to matter. Document in RUNBOOK.
• DMA-BUF-only portal streams are rejected with a clear message (hardware path is Phase 7).
• No negative-path tests yet (bad config, bad stride, undersized buffer).
• to_annex_b_h264 sniffs AVCC vs Annex-B by content; if an AVCC-emitting encoder is ever added, prefer an explicit config flag over the heuristic.
• Region targets are rejected: the desktop portal has no region capture.
• CaptureSession::next_frame() returns nullopt on stream error without surfacing the reason (logged to stderr).
• Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders we control never emit them, but third-party interop would need support.

Android app review (2026-09-10) — findings FIXED same day

Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified faithful to the C++ side source-to-source), followed by a fix pass that landed all findings. No commit yet (user has not asked).

Fixed:

• SignalingServer.send() no longer throws (mirrors the C++ server, which ignores write failures): a broken signaling TCP no longer kills the rtp-reader thread via requestPli(); peerOut is dropped (closing the socket) on write failure.
• H264Decoder.configure() assigns the created codec before configuring, so a configure/start failure can no longer orphan the MediaCodec instance (no finalizer; scarce native slots).
• Offer-before-surface no longer configures a ByteBuffer-mode decoder: the offer is parked as pendingOffer and attachSurface() configures later (a surface-less codec can never take one — setOutputSurface refuses it, an IllegalStateException crash on the main thread). The late configure sends a PLI (the sender only emits IDRs when asked).
• ReceiverActivity.onSurfaceTextureDestroyed → pipeline.detachSurface(); the pipeline/decoder forget dead surfaces instead of configuring against them (frames decode unrendered until the next attach).
• The status overlay reappears: onStatus sets visibility = VISIBLE (it used to write into a GONE view after the first frame).
• H264Decoder.feed() returns false on input-queue timeout → the pipeline requests a PLI instead of silently dropping the frame (no flush — the codec is healthy).
• The depacketizer accumulates into ByteArrayOutputStreams instead of boxed ArrayList<Int> (was ~MB/s of Integer allocations on keyframes).
• NSD reflection fallback deleted (dead code — see the quirk note above).
• Thread-safety: all codec calls now run under decoderLock (MediaCodec is not thread-safe; feed/drain previously raced attachSurface); requestPli() is thread-safe via pliLock.

Validation: gradle :app:assembleDebug :app:testDebugUnitTest green (25/25, full --rerun-tasks rebuild, only two pre-existing warnings); meson test 5/5 unchanged (no C++ touched).

Discovery bug found and fixed on-device (2026-09-10, same day)

The user reported the desktop sender never discovered the phone. Root cause (found live on the Fairphone 6): registerService(info, 0, listener) — Android 16's NsdManager.checkProtocol() rejects protocol 0 with IllegalArgumentException: Unsupported protocol. The old code swallowed it into the dead reflection fallback, and start() posted the "Listening…" status AFTER advertiseNsd, overwriting the failure text — so mDNS never advertised and Phase 8's NSD validation was only ever "no crash" (8.5 streamed via --peer, masking it).

Fixes in ReceiverPipeline.advertiseNsd()/start():

• pass NsdManager.PROTOCOL_DNS_SD;
• advertise AFTER the listening status so a failure stays visible;
• Log.e the registration exception; Log.i on registered success.

On-device validation (adb, live): mDNS registered: screencast._screencast._tcp in logcat; dumpsys servicediscovery shows the active Advertiser (key diagnostic: mClientRequests empty == no request ever issued); desktop avahi-browse and screencast --discover list the phone at 192.168.178.29:5005; a 25s --send --target monitor --peer 192.168.178.29:5005 session decoded (in-band SPS reconfigured 320x240 → 2496x1040) and rendered (screencap mean brightness 0.51). Note: with both the Pi and the phone on the LAN, plain --send refuses (two receivers found) — target the phone with --peer.

Follow-up from the same on-device session — startup squish eliminated: H264Decoder.outputSize() previously reported the 320x240 configure() placeholder until the codec parsed the SPS, so fitVideo sized the TextureView 1488x1116 (1.33 aspect) and the first rendered frame(s) of 2.4 content were visibly squished. It now returns null until INFO_OUTPUT_FORMAT_CHANGED fires; the first frame renders into the fullscreen surface and the correct letterbox (2484x1035) follows within ~40ms. Validated live: no placeholder "video size" line in logcat.

Commits: 9a24933 (PROTOCOL_DNS_SD discovery fix + docs), plus the outputSize fix (see git log).

Still open (accepted, needs a device or a new test dep):

• Untested on device: setOutputSurface mid-session (surface switch) and the whole pendingOffer path; API-35 detachOutputSurface() could replace the render-flag approach.
• No JVM tests for SignalingMessage/SignalingServer (would need the org.json:json test dependency; android.jar stubs throw).
• Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200), RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a future phase).
• Verified NOT a bug: reusing one DatagramPacket without resetting its length — modern JVMs recv by buffer capacity (JDK-21 probe + the successful on-device streaming confirm it).