screen_cast/.agents/MEMORY.md

Project Memory — screen_cast

Last updated: Phase 6 (discovery + signaling) complete and validated over loopback; current phase is Phase 7.

Project state

• Phase 6 done: Avahi mDNS discovery (_screencast._tcp — receiver announces its signaling port via a threaded-poll Avahi client; senders browse+resolve) plus JSON session signaling (offer/answer) over TCP with a signaling server on the receiver (port 5005). --send auto-discovers when exactly one receiver is found; --discover lists receivers; --peer targets a receiver directly. meson test 5/5, valgrind clean.

• Signaling is newline-delimited JSON over TCP, not WebSocket: no WS library was installed and the rtp-networking skill permits plain TCP. The wire format (one JSON object per line: offer/answer with session id, codec, rtp port) is transport-agnostic; Phase 7 adds the WS dependency if needed.

• Dual-stack everywhere: this machine resolves its own services over IPv6 (ULA + link-local), so the UDP transport, signaling client, and both listeners now support both families (IPv6 sockets with IPV6_V6ONLY=0 for dual-stack listening; AF_UNSPEC getaddrinfo for peers). Validated over both 127.0.0.1 and ::1.

• Discovery dedupe: one entry per (service_name, signaling_port) — a host with many interfaces otherwise registers dozens of address variants.

• mDNS operational lesson (hit during validation): kill -9 on a process holding an avahi registration leaves stale daemon records; the service then browses but never resolves (timeout for every peer) until records expire (~75 min). systemctl restart avahi-daemon clears it. Recorded in RUNBOOK; always stop with SIGTERM.

• Phase 5 (local UDP sender→receiver loopback) is implemented:

  • UdpRtpTransport (src/network/udp_transport.cpp): raw POSIX sockets, AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop() closes the socket to unblock the receive jthread. ASIO was deliberately deferred to Phase 6 (see decisions).
  • screencast binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline (recv→depacketize→decode→bounded 3-frame queue→render thread).
  • SdlRenderer (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture, RGBA texture upload, texture recreated on resolution change. RendererFactory::create now returns RendererResult (error channel added, mirroring codec/capture patterns).
  • Encoder change: GLOBAL_HEADER removed so libx264 repeats SPS/PPS in-band at every keyframe; a receiver now decodes from the bitstream alone (mid-stream join, PLI recovery-ready). get_extradata() is empty in this mode; codec round-trip test updated to match the streaming path.
  • Burst-control fixes after the first real run (receiver saw non-existing PPS 0 forever): without VBV, a 2256x1504 IDR burst (~100s of KB of back-to-back FU-A packets) overflowed the ~208KB default UDP receive buffer; the sequence gap made drop-on-damage discard whole keyframes including their in-band SPS/PPS, so the receiver never recovered. Fixes: encoder VBV (rc_max_rate = bitrate, rc_buffer_size = bitrate*2/fps — caps any keyframe to ~2 frame periods of bytes), microsecond encoder time_base (kills the 25fps pts quantization that duplicated timestamps), and a best-effort 4MB SO_RCVBUF on the receive socket (kernel clamps to rmem_max).
  • Display fixes after the receiver window never appeared: (1) SDL must own the window from one thread — creating it on main and pumping/presenting from the render thread left the Wayland surface unmapped; the renderer now creates/polls/presents/destroys entirely on the render thread (with a start() init handshake). (2) Wayland windows are invisible until the first render commit, so the renderer presents a blank frame at init (visible black window while waiting). (3) The receiver logs stream started (WxH) on the first decoded frame and any first present failure.
  • Phase 5 validated end-to-end (headless, without the portal): a synthetic RTP feed of solid red/green frames drove the real receiver over localhost UDP; the window mapped, logged stream started (320x240), and a grim screenshot of the window region showed the fed color (V=198, SAT=74 during the red feed) — decoded video visibly rendering. The user's real sender run showed capture→encode→send working (1 IDR + 12 P-frames, IDR capped at ~20KB by the VBV).

• Automated validation: meson test 4/4 — new udp loopback test encodes synthetic frames, packetizes, sends over a real localhost UDP socket, depacketizes, and decodes 10/10 frames with correct dimensions. Valgrind clean (loopback + codec tests).

• Manual validation pending (needs the desktop): run the two commands in docs/RUNBOOK.md — receiver window should show the captured desktop. Tick docs/PHASES.md Phase 5 after this works.

• Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A; 3-byte canonical start codes; drop-on-damage loss handling).

• Phase 3 capture done and validated (PipeWire/portal backend; the impl_ext_end_proxy wrong-context warnings were fixed by holding the thread-loop lock across all pw proxy operations).

Decisions

• Language: C++20 with explicit modern-C++ guidelines in AGENTS.md and cpp-meson-build/SKILL.md.
• Build system: Meson.
• Capture: PipeWire + xdg-desktop-portal.
• Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
• H.264 encoder path: software libx264, low-latency settings, Annex-B output.
• SPS/PPS are sent in-band ahead of every keyframe (no GLOBAL_HEADER); the decoder starts from the bitstream alone. DecoderConfig.extradata remains available if signaling ever negotiates parameters out of band.
• Transport: RTP over UDP via raw POSIX sockets for now; ASIO stays a Phase 6+ option (the ARCHITECTURE.md dependency table places it with signaling/discovery). IPv4 only at the transport level for now.
• Rendering: SDL3 (sdl3 pkg-config, 3.4 installed); plain texture upload, no GPU pipeline yet.
• Pixel-format convention trap: FFmpeg names packed formats in memory byte order (AV_PIX_FMT_RGBA = R,G,B,A in memory), but SDL names 32-bit formats MSB-first (SDL_PIXELFORMAT_RGBA8888 = A,B,G,R in memory). FFmpeg RGBA data therefore needs SDL_PIXELFORMAT_ABGR8888 — using RGBA8888 paints the alpha byte as red (red-tinted image).
• Discovery: mDNS/Avahi.
• Namespace: sc.
• Module error results use per-module std::variant<T, XError> types (CodecResult, CaptureResult, RendererResult) since C++20 has no std::expected.

Active blockers

None.

Open questions

• GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
• Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after software encode path works.
• IPv6 at the transport layer — revisit when LAN streaming lands (Phase 6+).

Forward-looking review notes (for later phases)

• RtpTransport::start/send return plain bools (scaffold API); error messages are lost — consider an error channel when signaling lands.
• Very high bitrates can still exceed even a raised receive buffer if net.core.rmem_max is low on the receiver; the encoder VBV bounds bursts to ~2 frame periods, so this needs --bitrate ≳ 100 Mbps to matter. Document in RUNBOOK.
• DMA-BUF-only portal streams are rejected with a clear message (hardware path is Phase 7).
• No negative-path tests yet (bad config, bad stride, undersized buffer).
• to_annex_b_h264 sniffs AVCC vs Annex-B by content; if an AVCC-emitting encoder is ever added, prefer an explicit config flag over the heuristic.
• Region targets are rejected: the desktop portal has no region capture.
• CaptureSession::next_frame() returns nullopt on stream error without surfacing the reason (logged to stderr).
• Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders we control never emit them, but third-party interop would need support.