screen_cast/docs/PHASES.md

Development Phases

This file breaks the project into incremental milestones. Each phase produces a working, testable slice of functionality. Do not start a phase until the previous one is validated.

Phase 1 — Project Skeleton

Goal: configure, compile, and run tests with no real dependencies.

• Meson build files (meson.build, meson_options.txt).
• Public module headers with namespace sc.
• Unit test that exercises a trivial utility function.
• README.md with build instructions.

Validation: meson setup build && meson compile -C build && meson test -C build.

Phase 2 — Software H.264 Encode / Decode

Goal: encode raw pixel buffers to H.264 and decode them back, purely with FFmpeg software paths.

• Add libavcodec, libavutil, libswscale dependencies.
• Implement EncoderFactory::create() and Encoder::encode().
• Implement DecoderFactory::create() and Decoder::decode().
• Round-trip test: synthetic RGB frames → H.264 → decoded RGB.

Validation: unit test produces visually/structurally correct round-trip frames.

Phase 3 — PipeWire Screen Capture

Goal: capture the desktop and feed frames into the encoder.

• Add libpipewire-0.3 dependency.
• Implement CaptureFactory::create() using the xdg-desktop-portal.
• Wire CaptureSession::next_frame() → Encoder::encode() in a local smoke test that just writes a few encoded frames to disk.

Validation: manual run on a real Linux desktop session produces a valid H.264 bitstream.

Phase 4 — RTP Framing

Goal: packetize NAL units into RTP and depacketize them.

• Implement RtpHeader and RtpPacket serialize/parse.
• Add H.264 NAL splitting and FU-A fragmentation.
• Unit test for serialization, fragmentation, and reassembly.

Validation: unit tests cover single-NAL and fragmented packet paths.

Phase 5 — Local UDP Sender → Receiver Loopback

Goal: send RTP packets over UDP and render the result locally.

• Implement RtpTransport with ASIO or raw UDP sockets.
• Wire sender pipeline: capture → encode → RTP → localhost UDP.
• Wire receiver pipeline: localhost UDP → RTP → decode → renderer.
• Add SDL2/SDL3 dependency and a minimal Renderer.

Validation: screencast --send and screencast --receive on the same machine show the captured desktop in a window.

Phase 6 — LAN Signaling and Discovery

Goal: two peers on the same LAN can find each other and negotiate a session.

• Implement mDNS/DNS-SD discovery with Avahi.
• Implement JSON signaling (newline-delimited JSON over TCP for now; the WebSocket transport is deferred to Phase 7 with its dependency).
• Extend CLI with --peer and --discover.

Validation: two peers connect without hard-coded IP addresses. Validated over loopback AND on real two-machine hardware: a Raspberry Pi Zero 2 W (Wi-Fi, headless KMSDRM, systemd autostart) was discovered, negotiated with, and streamed to from the desktop; the receiver rendered a synthetic feed fullscreen on its display. On that network the LAN's 6to4 addresses are unreachable between peers, so senders try all discovered addresses in reachability order (private IPv4 first).

Phase 7 — Resilience and Polish

Goal: loss recovery, hardware acceleration, and polish.

• PLI keyframe feedback over signaling (receiver asks, sender re-encodes a keyframe; validated end-to-end with an induced loss).
• Jitter/reorder window on the receiver (16 packets / 60 ms).
• Hardware H.264 decode probe with software fallback (h264_v4l2m2m — the Pi's VideoCore path; --swdecode opts out).
• GTK4 sender GUI behind meson -Dgui=true (gtkmm; receiver list, bitrate, start/stop) plus a waybar widget (screencast waybar, click-to-toggle, right-click opens the panel; state shared via $XDG_RUNTIME_DIR/screencast/sender.json).
• VAAPI hardware encode probe on the sender (deferred: software encode is not the bottleneck).
• Deferred: .desktop file, packaging.

Validation: sustained streaming under packet loss; hardware accel smoke where available. PLI + jitter validated on the desktop (induced-loss probe); hardware decode validated as clean-fallback on the desktop, hardware path pending a run on the Pi.

Phase 8 — Android Receiver App

Goal: a native Android receiver app (Kotlin) so a phone can act as the second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the existing signaling + RTP protocol; no C++ changes.

• 8.1 Gradle project builds (android/, AGP 9 built-in Kotlin, no androidx; gradle :app:assembleDebug)
• 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer, depacketizer — 25 tests green)
• 8.3 Signaling + NSD validated on device (offer → answer over the network; mDNS registration via the API-36 RegistrationListener) — correction 2026-09-10: registration had in fact never succeeded (registerService passed protocol 0, which API 36 rejects with "Unsupported protocol"; the swallowed failure was hidden behind the "Listening…" status). Fixed with PROTOCOL_DNS_SD + real on-device validation: desktop --discover lists the phone and a live --send session decodes and renders.
• 8.4 MediaCodec decode + Surface render validated on device (in-band SPS sizing, letterbox fit)
• 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed fullscreen render, PLI keyframe recovery on Wi-Fi loss
• 8.6 Docs: README receiver section, RUNBOOK build/install/test, PHASES + MEMORY updates

Validation: gradle :app:testDebugUnitTest green; live session --send --target window --peer <phone>:5005 rendered fullscreen on the phone (and HDMI via DP-alt-mode) with loss recovery.

Phase 9 — iOS Receiver App

Goal: a native iOS receiver app (Swift) so an iPhone can act as the second receiver. The app speaks the existing signaling + RTP protocol; no C++ changes. Mirrors the Android receiver (Phase 8) source-to-source.

• 9.1 Xcode project scaffolding (ios/, XcodeGen spec + bootstrap script; Info.plist with local-network + Bonjour privacy keys)
• 9.2 Swift protocol core + XCTest (rtp header/packet, jitter, depacketizer ported source-to-source; added signaling JSON, line framing, AVCC, NAL extraction tests — the Android side had no signaling tests)
• 9.3 Signaling server (BSD sockets, dual-stack, most-recent-peer, never throws) + NSBonjourServices advertisement
• 9.4 Media path: UDP (poll-based) → jitter → depacketize → VideoToolbox (in-band SPS/PPS, real-time decode) → AVSampleBufferDisplayLayer (letterbox) + PLI on damage + pendingOffer for late surface attach
• 9.5 On-device validation (needs a Mac + Xcode 26 + iPhone 16): unit tests green on the simulator; --discover lists the phone; --send --peer <ip>:5005 streams letterboxed; PLI recovery on Wi-Fi loss
• 9.6 Docs: iOS RUNBOOK quirks, README receiver section, PHASES + MEMORY

Validation: ios/bootstrap.sh test (simulator) green; live session --send --peer <iphone>:5005 renders fullscreen letterboxed on the iPhone with loss recovery. Note: this box is Linux — no Xcode/iOS SDK — so 9.2–9.4 are authored but only 9.5 can validate the VideoToolbox + local-network paths.

Current phase

Phase 9 — iOS Receiver App (implementation authored; on-device validation pending a Mac + Xcode 26 + iPhone 16).