feat(capture): implement Phase 3 PipeWire/portal desktop capture
Implement the xdg-desktop-portal ScreenCast backend via libportal: a
blocking portal handshake (interactive source picker), a PipeWire stream
on the portal's node enumerating BGRx/BGRA/RGBx/RGBA, and a latest-frame
slot handing frames to next_frame(). stop() is thread-safe; teardown
follows the order PipeWire requires. All proxy operations run under the
thread-loop lock to satisfy the protocol extension context checks
('impl_ext_end_proxy called from wrong context' otherwise).
The encoder now accepts padded strides for packed RGB inputs (real
PipeWire row pitches) and maps the new PixelFormat::Bgrx to
AV_PIX_FMT_BGRA.
Add tools/capture_smoke: a manual smoke tool (interactive, not in
meson test) that captures N frames, encodes them, and writes a
self-contained Annex-B elementary stream with prepended SPS/PPS.
Validated manually on Wayland/Hyprland: 2256x1504 H.264 elementary
stream, ffprobe clean. Phase 3 marked complete in docs/PHASES.md.
This commit is contained in:
+36
-27
@@ -1,31 +1,37 @@
|
||||
# Project Memory — screen_cast
|
||||
|
||||
Last updated: Phase 2 codec review fixes applied.
|
||||
Last updated: Phase 3 validated and complete; current phase is Phase 4.
|
||||
|
||||
## Project state
|
||||
|
||||
- Phase 2 codec implementation reviewed with valgrind; all confirmed defects
|
||||
fixed on top of the capture-stub commit:
|
||||
- decoder leaked every packet payload (`av_malloc` + direct `packet->data`
|
||||
assignment bypassed the packet's owning `AVBufferRef`); payloads are now
|
||||
allocated with `av_new_packet`.
|
||||
- decoder extradata lacked `AV_INPUT_BUFFER_PADDING_SIZE`; FFmpeg's
|
||||
extradata parser over-read the buffer (valgrind invalid reads). Now
|
||||
allocated padded and zeroed.
|
||||
- oversized encoded frames are rejected before the int cast.
|
||||
- EAGAIN-retry loops in encoder/decoder now handle unexpected EOF instead
|
||||
of retrying forever.
|
||||
- fixed "RTP packet" → "AVPacket" error message in the encoder.
|
||||
- `CaptureFactory::create()` now returns
|
||||
`CaptureResult<std::unique_ptr<CaptureSession>>` (variant with
|
||||
`CaptureError`) instead of a nullable unique_ptr; the stub reports
|
||||
"PipeWire capture is not implemented yet" as an error. New pattern lives in
|
||||
`include/screencast/capture/error.h`, mirroring `codec/error.h`.
|
||||
- Validation: `meson test` 2/2 OK; valgrind on `test_codec_roundtrip` is now
|
||||
clean (0 definite losses, 0 invalid reads); clang-format clean. See
|
||||
`docs/RUNBOOK.md` for the reusable checks.
|
||||
- Phase 3 capture stub remains in place; remaining implementation: capture,
|
||||
transport, rendering, and CLI/pipeline glue.
|
||||
- **Phase 3 is done and validated on the desktop**: a manual
|
||||
`./build/tools/capture_smoke 10 out.h264` run on Wayland/Hyprland produced
|
||||
a valid 2256x1504 H.264 elementary stream (ffprobe clean). `docs/PHASES.md`
|
||||
is ticked; current phase is Phase 4 — RTP framing.
|
||||
- The first smoke run emitted `impl_ext_end_proxy called from wrong context`
|
||||
warnings: `pw_context_connect_fd` and `pw_core_disconnect` ran outside the
|
||||
thread-loop lock. Fixed by holding the lock across all pw setup/teardown
|
||||
proxy operations. Re-running the smoke tool should now be warning-free
|
||||
(capture worked both ways; the warnings only meant the first two
|
||||
marshaled messages were rejected and retried from the right context).
|
||||
- `src/capture/pipewire_capture.cpp` now implements the full backend:
|
||||
libportal 0.10 handshake (`create_screencast_session` → `session_start` →
|
||||
`open_pipewire_remote`), PipeWire 1.6 stream on the first portal node,
|
||||
BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff.
|
||||
Portal handshake blocks on the caller thread; frames arrive on the pw
|
||||
thread. `stop()` is thread-safe; teardown follows the pw-required order.
|
||||
- Encoder now accepts padded strides for packed RGB formats and the new
|
||||
`PixelFormat::Bgrx` (mapped to `AV_PIX_FMT_BGRA`); planar Yuv420p still
|
||||
requires a packed layout. This was the review note blocking Phase 3.
|
||||
- `tools/capture_smoke` (manual, not in `meson test`) captures N frames →
|
||||
encodes → writes Annex-B including prepended SPS/PPS extradata (verified:
|
||||
libx264 GLOBAL_HEADER extradata is Annex-B).
|
||||
- Earlier review fixes remain in place; valgrind on the codec round-trip
|
||||
test is still clean after the encoder stride changes.
|
||||
- Encoder PTS caveat: the encoder time_base is derived from the configured
|
||||
frame rate (default 25fps), but portal frames arrive at monitor refresh
|
||||
(often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless
|
||||
for the smoke test; revisit when RTP timestamps matter (Phase 4/5).
|
||||
|
||||
## Decisions
|
||||
|
||||
@@ -55,14 +61,17 @@ None.
|
||||
|
||||
## Forward-looking review notes (for later phases)
|
||||
|
||||
- Encoder rejects non-packed strides in `make_input_frame`; PipeWire/portal
|
||||
frames usually have alignment-padded strides — Phase 3 must pass the real
|
||||
stride through to swscale instead of rejecting it.
|
||||
- `AV_CODEC_FLAG_GLOBAL_HEADER` suppresses in-band SPS/PPS; a receiver cannot
|
||||
join mid-stream or recover after PLI without parameter sets. Phase 5 must
|
||||
prepend SPS/PPS to keyframes or negotiate them in signaling.
|
||||
- Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother
|
||||
UDP streaming in Phase 7.
|
||||
- DMA-BUF-only portal streams are rejected with a clear message (hardware
|
||||
path is Phase 7).
|
||||
- No negative-path tests yet (bad config, bad stride, undersized buffer).
|
||||
- `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting
|
||||
encoder is ever added, prefer an explicit config flag over the heuristic.
|
||||
encoder is ever added, prefer an explicit config flag over the heuristic.
|
||||
- Region targets are rejected: the desktop portal has no region capture.
|
||||
- `next_frame()` returns `nullopt` on stream error without surfacing the
|
||||
reason (logged to stderr); consider an error channel when the receiver
|
||||
pipeline lands.
|
||||
Reference in New Issue
Block a user