screen_cast/.agents/MEMORY.md

Project Memory — screen_cast

Last updated: Phase 4 (RTP framing) complete and tested; current phase is Phase 5.

Project state

• Phase 4 done: sc_network library implements RtpHeader/RtpPacket (RFC 3550 serialize/parse; tolerates CSRC lists, extension headers, and padding on the receive side) plus H264Packetizer/H264Depacketizer (RFC 6184 single-NAL + FU-A; STAP-A never emitted, unsupported types mark the frame damaged on receive).
• Depacketized access units use 3-byte start codes, and the splitter keeps a zero byte preceding a start code with the previous NAL, so both 3- and 4-byte-start-code streams round-trip byte-exactly (tested).
• Loss handling is drop-on-damage: a sequence gap or missing FU fragment marks the frame damaged and it is dropped silently at its marker. Full loss recovery / jitter handling is Phase 7.
• test_rtp covers header/packet round-trips, malformed rejections, NAL splitting (3- and 4-byte codes), FU-A chunking with MTU bounds, full packetize→depacketize round-trip, gap dropping, frame separation by marker alone, sequence wrap, and random default SSRC/sequence. Valgrind-clean; meson test 3/3.
• Phase 3 remains validated; earlier review fixes still in place.
• The first smoke run emitted impl_ext_end_proxy called from wrong context warnings: pw_context_connect_fd and pw_core_disconnect ran outside the thread-loop lock. Fixed by holding the lock across all pw setup/teardown proxy operations. Re-running the smoke tool should now be warning-free (capture worked both ways; the warnings only meant the first two marshaled messages were rejected and retried from the right context).
• src/capture/pipewire_capture.cpp now implements the full backend: libportal 0.10 handshake (create_screencast_session → session_start → open_pipewire_remote), PipeWire 1.6 stream on the first portal node, BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff. Portal handshake blocks on the caller thread; frames arrive on the pw thread. stop() is thread-safe; teardown follows the pw-required order.
• Encoder now accepts padded strides for packed RGB formats and the new PixelFormat::Bgrx (mapped to AV_PIX_FMT_BGRA); planar Yuv420p still requires a packed layout. This was the review note blocking Phase 3.
• tools/capture_smoke (manual, not in meson test) captures N frames → encodes → writes Annex-B including prepended SPS/PPS extradata (verified: libx264 GLOBAL_HEADER extradata is Annex-B).
• Earlier review fixes remain in place; valgrind on the codec round-trip test is still clean after the encoder stride changes.
• Encoder PTS caveat: the encoder time_base is derived from the configured frame rate (default 25fps), but portal frames arrive at monitor refresh (often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless for the smoke test; revisit when RTP timestamps matter (Phase 4/5).

Decisions

• Language: C++20 with explicit modern-C++ guidelines in AGENTS.md and cpp-meson-build/SKILL.md.
• Build system: Meson.
• Capture: PipeWire + xdg-desktop-portal.
• Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
• H.264 encoder path: software libx264, low-latency settings, Annex-B output.
• Decoder receives SPS/PPS through DecoderConfig.extradata.
• Transport: RTP over UDP; signaling via WebSocket/JSON.
• Discovery: mDNS/Avahi.
• Rendering: SDL2 or SDL3 + OpenGL.
• Namespace: sc.
• Module error results use per-module std::variant<T, XError> types (CodecResult, CaptureResult) since C++20 has no std::expected.

Active blockers

None.

Open questions

• GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
• Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after software encode path works.

Forward-looking review notes (for later phases)

• AV_CODEC_FLAG_GLOBAL_HEADER suppresses in-band SPS/PPS; a receiver cannot join mid-stream or recover after PLI without parameter sets. Phase 5 must prepend SPS/PPS to keyframes or negotiate them in signaling.
• Encoder sets no VBV (maxrate/buffer_size) — ABR only; add for smoother UDP streaming in Phase 7.
• DMA-BUF-only portal streams are rejected with a clear message (hardware path is Phase 7).
• No negative-path tests yet (bad config, bad stride, undersized buffer).
• to_annex_b_h264 sniffs AVCC vs Annex-B by content; if an AVCC-emitting encoder is ever added, prefer an explicit config flag over the heuristic.
• Region targets are rejected: the desktop portal has no region capture.
• next_frame() returns nullopt on stream error without surfacing the reason (logged to stderr); consider an error channel when the receiver pipeline lands.