Three changes to make the stream survive constrained links:
Adaptive quality: the sender pipeline now tracks the PLI rate from
the receiver. Every 5 seconds it evaluates: >0.5 PLI/s means the
link is saturated (the receiver is dropping frames), so the CRF
increases by 2 (lower quality, fewer bits) and the encoder restarts
with a keyframe. <0.1 PLI/s means the link is stable, so the CRF
decreases by 1 (better quality) and the encoder probes upward.
Clamped to [user CRF, user CRF + 10] so quality never degrades
below what the link can handle, and never exceeds what the user
asked for. The adaptation is logged to stderr for visibility.
Frame rate capping (--fps N): throttles the capture loop to N
frames per second (0 = no cap; monitor rate). At 15fps instead of
60fps, the bandwidth requirement drops 4x at the same quality
level. Desktop content is still smooth at 15-20fps.
Tighter VBV: one frame period of buffer instead of two. A two-frame
buffer lets a keyframe spike to twice the target rate in one burst,
which overflows any constrained hop (Wi-Fi hotspot, slow switch)
and cascades into PLI storms. One frame period keeps bursts
within what the link can absorb in real time.
Four encoder quality improvements, all sender-side:
- CRF rate control (default 22, --crf to override): targets a
constant visual quality level instead of a fixed bitrate. Static
desktop content uses 300-800 kbps (vs. forced 4000+), and the saved
bits go to sharp text and clean motion when they appear. The VBV
max rate (the --bitrate value, now a cap rather than a target)
bounds bursts so the receiver's UDP buffers stay safe. Round-trip
test bitrate dropped from 1390 kb/s to 47 kb/s on synthetic frames
— the encoder uses only what it needs.
- 5-second GOP (was 1 second): 80% fewer keyframe bits freed for
detail frames. Screen content changes incrementally, not
wholesale; PLI feedback recovers from loss in one frame time
regardless of GOP length.
- faster preset (was veryfast): better sub-pixel estimation and
RDO on more decisions. The desktop handles it trivially at 1080p.
- Screen-content x264 tuning: aq-mode=2 (auto-variance AQ moves
bits away from flat areas toward text edges) and psy-rd=1.5
(preserves texture sharpness).
Combined with the earlier veryfast upgrade and sender-side
downscaling, this is roughly 2x the perceived quality at the same
average bandwidth compared to the original ultrafast ABR encoder.
meson test 5/5 in both configurations, valgrind clean.
Two quality improvements:
Preset: ultrafast -> veryfast. Unlocks Main profile with CABAC
entropy coding, hexagonal motion search, 3 reference frames, and
adaptive quantization — typically 30-40% better quality at the same
bitrate. The desktop handles the extra encoding cost trivially
(150+ fps at 1080p).
Downscaling: the receiver now advertises its display resolution in
the signaling answer (display_width/display_height, 0 = unknown).
When the capture exceeds the display (e.g. 2256x1504 source on a
1920x1080 receiver), the sender scales down preserving aspect ratio
before encoding — the same sws_scale pass that already converts the
pixel format also handles the resolution change, so there is no
extra step. This concentrates the entire bitrate into pixels the
display actually shows (~2.7x more bits per visible pixel at
4000 kbps when going from 2256x1504 to 1620x1080).
The renderer caches the display size during window creation
(native monitor resolution in fullscreen/KMSDRM; window size
otherwise). The receiver includes it in every signaling answer; the
sender pipeline computes aspect-preserving, even-rounded scaled
dimensions when the display is smaller than the capture.
meson test 5/5 in both configurations, valgrind clean.
The receiver decoded H.264 to YUV420P, converted it to RGBA via a
CPU-intensive swscale pass, then uploaded 4 bytes/pixel to an SDL
texture — only for the GPU to convert back to RGB during rendering.
This eliminated the swscale pass entirely (40-60% of receiver CPU at
1080p) and cut the texture upload by 62%.
- DecodedFrame now carries three YUV420P planes with their strides
instead of a packed RGBA buffer; the decoder copies the planes
directly from the AVFrame (zero conversion for the common software
path). Non-YUV420P decoder output (e.g. NV12 from v4l2m2m) is
converted once to YUV420P.
- The SDL renderer uploads via SDL_UpdateYUVTexture with
SDL_PIXELFORMAT_IYUV; the GPU does the YUV→RGB conversion during
rendering.
- Decoder threading: slice-level with 4 threads (parallelizes within a
frame, no added latency), not frame-level (which buffers multiple
frames — the initial thread_count=0 broke the loopback test because
the H.264 decoder introduced a multi-frame delay before producing
output).
- The round-trip test converts decoded YUV back to RGBA for pixel
comparison via a test-local swscale call (the pipeline itself never
converts).
meson test 5/5 in both configurations, valgrind clean.
A gtkmm-4.0 control panel (behind -Dgui=true, default off): refresh
shows discovered receivers (grouped and preference-sorted), a bitrate
scale, and start/stop that runs the whole session on a worker thread
so the interactive portal picker never blocks the UI. The CLI and the
GUI now share the new sc_app_core static library holding the
pipelines, session orchestration (negotiation + PLI feedback), and a
state store.
The sender pipeline publishes its state to
$XDG_RUNTIME_DIR/screencast/sender.json (session id, receiver,
bitrate, pid, start time; stale files detected by pid liveness) and
persists the last session for one-click restarts. The new
'screencast waybar' subcommand prints a waybar module line and its
--toggle flag stops a running sender gracefully or spawns a detached
restart of the last receiver.
Waybar on the dev machine is wired: custom/screencast module with
click-to-toggle and right-click panel, plus styles, with a timestamped
backup of both config files. Both binaries are installed to
/usr/local/bin.
Validated: waybar output (idle and streaming states with a synthetic
state file), GUI launches on the desktop (window observed via
hyprctl), meson test 5/5 in both build configurations, formatting
clean.
Loss recovery for the streaming path:
- PLI over signaling: the depacketizer now reports damaged frames
(DepacketizeResult) and the receiver asks the sender for a keyframe
(SessionPli, rate-limited to one per 500 ms). The sender keeps the
signaling channel open during the session and honors PLIs through
the new thread-safe SenderPipeline::request_keyframe(). Recovery
takes one frame time instead of waiting out the GOP.
- RtpJitterBuffer: reorders RTP packets by sequence number (16 packets
/ 60 ms) before the in-order depacketizer, so Wi-Fi reordering is
not misread as loss; in-order streams release immediately, and a
straggler older than the delivered sequence is discarded.
- Hardware H.264 decode probe: DecoderFactory tries h264_v4l2m2m (the
VideoCore path on the Pi) with an automatic software fallback and a
clear journal line for the chosen path; --swdecode opts out.
Validated: PLI end-to-end with a probe that drops a mid-keyframe
packet over real UDP (receiver logged the damaged frame and the PLI
arrived with the session id); hardware probe fails cleanly and falls
back on this desktop; jitter reordering covered by unit tests.
meson test 5/5 in both build configurations, valgrind clean.
A headless receiver has no window manager, so a windowed window is
meaningless there — the screencast should simply fill the screen. The
renderer now goes borderless fullscreen automatically when SDL runs
the KMSDRM backend, and --fullscreen forces the same behavior in
desktop sessions. The video keeps its aspect ratio via
SDL_SetRenderLogicalPresentation(LETTERBOX): a 4:3 desktop on a 16:9
TV renders with black bars instead of a stretched picture, the cursor
is hidden, and the clear-before-draw keeps the bars black.
Verified live on a 3440x1440 monitor: the --fullscreen receiver window
covered the entire display while a 4:3 synthetic feed rendered with
pure-black pillarbox bars (Y=0/SAT=0) and a saturated-red center
(V=254). meson test 5/5 in both build configurations.
Add mDNS/DNS-SD discovery and JSON session negotiation so two peers on
a LAN connect without hard-coded addresses.
- Discovery (Avahi threaded-poll client): the receiver announces
_screencast._tcp with its signaling port; senders browse and resolve
peers. Strict lock ordering (poll lock before state mutex) keeps the
callbacks deadlock-free; name collisions rename via
avahi_alternative_service_name.
- Signaling: one JSON object per newline-terminated TCP line. The
receiver hosts a server (port 5005) and answers session offers with
its RTP port; senders connect, offer, and stream to the negotiated
endpoint. WebSocket was deferred: no WS library is installed, the
skill permits plain TCP, and the wire format is transport-agnostic.
- Dual-stack transports: this machine resolves its own services over
IPv6, so getaddrinfo now runs AF_UNSPEC and listeners bind IPv6 with
IPV6_V6ONLY=0 (IPv4 fallback), covering UDP and TCP alike. The
signaling client shutdown now uses shutdown() so a reader blocked in
recv() cannot hang the join (a plain close() does not wake it).
- CLI: --discover lists receivers (deduped to one entry per host);
--send auto-disovers when exactly one receiver is found; --peer
targets a receiver directly; --signaling-port overrides the default.
Validation: meson test 5/5 (new signaling round-trip test), valgrind
clean. End-to-end over loopback: --discover finds the announced
receiver, a probe negotiated a session and streamed 60 frames over
both IPv4 and IPv6, and the receiver reported stream started. mDNS
resolution was verified against avahi-browse as an independent
reference.
Wire the first end-to-end pipeline: capture -> encode -> packetize ->
UDP -> depacketize -> decode -> render.
- UdpRtpTransport: raw POSIX UDP sockets (IPv4 via getaddrinfo), a
receive jthread woken by socket close on stop; port 0 skips binding
so the sender uses an OS-assigned source port. ASIO stays deferred
to the signaling phase per ARCHITECTURE.md.
- SdlRenderer: SDL3 window/renderer with RGBA texture upload; the
texture is recreated on resolution change. RendererFactory now
returns RendererResult so SDL init failures carry a message,
mirroring the codec/capture error patterns.
- screencast binary: parse_cli plus SenderPipeline/ReceiverPipeline
per the app scaffolds; the sender creates its encoder once capture
reports real dimensions, the receiver keeps a bounded 3-frame queue
to hold latency down and renders on its own thread until the window
closes. cli argv signature fixed to 'const char* const*' so main's
argv converts implicitly.
- Encoder: drop AV_CODEC_FLAG_GLOBAL_HEADER so libx264 repeats SPS/PPS
in-band at each keyframe -- the receiver decodes from the bitstream
alone, which also makes mid-stream joins and later PLI recovery
work without out-of-band parameter negotiation. The round-trip test
now exercises exactly that path.
- tests: new udp-loopback integration test pushes synthetic frames
through a real localhost socket and decodes 10/10 frames with the
right dimensions; valgrind clean (loopback + codec). meson test 4/4.
Manual validation on the desktop (receiver window shows the captured
desktop) is documented in docs/RUNBOOK.md.
Add the sc_network library: RFC 3550 RtpHeader/RtpPacket serialize and
parse (the receiver tolerates CSRC lists, extension headers, and
padding by skipping/stripping them) and RFC 6184 H.264 payloading via
H264Packetizer/H264Depacketizer.
The packetizer splits Annex-B frames into NAL units (3- and 4-byte
start codes), emitting single-NAL packets or FU-A fragments within the
configured MTU, with the marker bit closing each frame and randomized
SSRC/sequence by default. The depacketizer reassembles access units
with 3-byte start codes, so both start-code widths round-trip
byte-exactly; frames damaged by sequence gaps or missing fragments
are dropped until the Phase 7 loss-recovery work.
test_rtp covers header and packet round-trips, malformed-input
rejections, splitter behavior, FU-A chunk bounds, full packetize ->
depacketize round-trip, gap dropping, marker-only frame separation,
sequence wrap, and empty inputs. meson test 3/3, valgrind clean.
Implement the xdg-desktop-portal ScreenCast backend via libportal: a
blocking portal handshake (interactive source picker), a PipeWire stream
on the portal's node enumerating BGRx/BGRA/RGBx/RGBA, and a latest-frame
slot handing frames to next_frame(). stop() is thread-safe; teardown
follows the order PipeWire requires. All proxy operations run under the
thread-loop lock to satisfy the protocol extension context checks
('impl_ext_end_proxy called from wrong context' otherwise).
The encoder now accepts padded strides for packed RGB inputs (real
PipeWire row pitches) and maps the new PixelFormat::Bgrx to
AV_PIX_FMT_BGRA.
Add tools/capture_smoke: a manual smoke tool (interactive, not in
meson test) that captures N frames, encodes them, and writes a
self-contained Annex-B elementary stream with prepended SPS/PPS.
Validated manually on Wayland/Hyprland: 2256x1504 H.264 elementary
stream, ffprobe clean. Phase 3 marked complete in docs/PHASES.md.
Replace the nullable unique_ptr returned by CaptureFactory::create()
with CaptureResult<std::unique_ptr<CaptureSession>> using the new
CaptureError/CaptureResult pattern, mirroring codec/error.h. The stub
now reports 'not implemented yet' as an error instead of returning
nullptr. Update handoff memory with the review outcome and
forward-looking notes for phases 3, 5, and 7.
Add a linkable CaptureFactory stub so the capture API can be consumed
without unresolved symbols. Change remaining config string_view fields to
std::string to prevent dangling references. Update handoff memory.
Add FFmpeg-based encoder/decoder with SPS/PPS extradata, Annex-B output
normalization, low-latency libx264 settings, and a round-trip unit test.
Includes review hardening: cached SwsContext, bitrate-only rate control,
std::byte/uin8_t cast helpers, and richer test assertions.
fegger