feat(capture): implement Phase 3 PipeWire/portal desktop capture

Implement the xdg-desktop-portal ScreenCast backend via libportal: a
blocking portal handshake (interactive source picker), a PipeWire stream
on the portal's node enumerating BGRx/BGRA/RGBx/RGBA, and a latest-frame
slot handing frames to next_frame(). stop() is thread-safe; teardown
follows the order PipeWire requires. All proxy operations run under the
thread-loop lock to satisfy the protocol extension context checks
('impl_ext_end_proxy called from wrong context' otherwise).

The encoder now accepts padded strides for packed RGB inputs (real
PipeWire row pitches) and maps the new PixelFormat::Bgrx to
AV_PIX_FMT_BGRA.

Add tools/capture_smoke: a manual smoke tool (interactive, not in
meson test) that captures N frames, encodes them, and writes a
self-contained Annex-B elementary stream with prepended SPS/PPS.

Validated manually on Wayland/Hyprland: 2256x1504 H.264 elementary
stream, ffprobe clean. Phase 3 marked complete in docs/PHASES.md.
This commit is contained in:
2026-09-07 10:37:14 +02:00
parent ac9dc02b51
commit ce52f64e52
11 changed files with 883 additions and 55 deletions
+35 -26
View File
@@ -1,31 +1,37 @@
# Project Memory — screen_cast
Last updated: Phase 2 codec review fixes applied.
Last updated: Phase 3 validated and complete; current phase is Phase 4.
## Project state
- Phase 2 codec implementation reviewed with valgrind; all confirmed defects
fixed on top of the capture-stub commit:
- decoder leaked every packet payload (`av_malloc` + direct `packet->data`
assignment bypassed the packet's owning `AVBufferRef`); payloads are now
allocated with `av_new_packet`.
- decoder extradata lacked `AV_INPUT_BUFFER_PADDING_SIZE`; FFmpeg's
extradata parser over-read the buffer (valgrind invalid reads). Now
allocated padded and zeroed.
- oversized encoded frames are rejected before the int cast.
- EAGAIN-retry loops in encoder/decoder now handle unexpected EOF instead
of retrying forever.
- fixed "RTP packet" → "AVPacket" error message in the encoder.
- `CaptureFactory::create()` now returns
`CaptureResult<std::unique_ptr<CaptureSession>>` (variant with
`CaptureError`) instead of a nullable unique_ptr; the stub reports
"PipeWire capture is not implemented yet" as an error. New pattern lives in
`include/screencast/capture/error.h`, mirroring `codec/error.h`.
- Validation: `meson test` 2/2 OK; valgrind on `test_codec_roundtrip` is now
clean (0 definite losses, 0 invalid reads); clang-format clean. See
`docs/RUNBOOK.md` for the reusable checks.
- Phase 3 capture stub remains in place; remaining implementation: capture,
transport, rendering, and CLI/pipeline glue.
- **Phase 3 is done and validated on the desktop**: a manual
`./build/tools/capture_smoke 10 out.h264` run on Wayland/Hyprland produced
a valid 2256x1504 H.264 elementary stream (ffprobe clean). `docs/PHASES.md`
is ticked; current phase is Phase 4 — RTP framing.
- The first smoke run emitted `impl_ext_end_proxy called from wrong context`
warnings: `pw_context_connect_fd` and `pw_core_disconnect` ran outside the
thread-loop lock. Fixed by holding the lock across all pw setup/teardown
proxy operations. Re-running the smoke tool should now be warning-free
(capture worked both ways; the warnings only meant the first two
marshaled messages were rejected and retried from the right context).
- `src/capture/pipewire_capture.cpp` now implements the full backend:
libportal 0.10 handshake (`create_screencast_session``session_start`
`open_pipewire_remote`), PipeWire 1.6 stream on the first portal node,
BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff.
Portal handshake blocks on the caller thread; frames arrive on the pw
thread. `stop()` is thread-safe; teardown follows the pw-required order.
- Encoder now accepts padded strides for packed RGB formats and the new
`PixelFormat::Bgrx` (mapped to `AV_PIX_FMT_BGRA`); planar Yuv420p still
requires a packed layout. This was the review note blocking Phase 3.
- `tools/capture_smoke` (manual, not in `meson test`) captures N frames →
encodes → writes Annex-B including prepended SPS/PPS extradata (verified:
libx264 GLOBAL_HEADER extradata is Annex-B).
- Earlier review fixes remain in place; valgrind on the codec round-trip
test is still clean after the encoder stride changes.
- Encoder PTS caveat: the encoder time_base is derived from the configured
frame rate (default 25fps), but portal frames arrive at monitor refresh
(often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless
for the smoke test; revisit when RTP timestamps matter (Phase 4/5).
## Decisions
@@ -55,14 +61,17 @@ None.
## Forward-looking review notes (for later phases)
- Encoder rejects non-packed strides in `make_input_frame`; PipeWire/portal
frames usually have alignment-padded strides — Phase 3 must pass the real
stride through to swscale instead of rejecting it.
- `AV_CODEC_FLAG_GLOBAL_HEADER` suppresses in-band SPS/PPS; a receiver cannot
join mid-stream or recover after PLI without parameter sets. Phase 5 must
prepend SPS/PPS to keyframes or negotiate them in signaling.
- Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother
UDP streaming in Phase 7.
- DMA-BUF-only portal streams are rejected with a clear message (hardware
path is Phase 7).
- No negative-path tests yet (bad config, bad stride, undersized buffer).
- `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting
encoder is ever added, prefer an explicit config flag over the heuristic.
- Region targets are rejected: the desktop portal has no region capture.
- `next_frame()` returns `nullopt` on stream error without surfacing the
reason (logged to stderr); consider an error channel when the receiver
pipeline lands.
+3
View File
@@ -16,6 +16,9 @@ meson-private/
*.exe
compile_commands.json
# Smoke test output
*.h264
# IDE
.vscode/
.idea/
+4 -4
View File
@@ -32,9 +32,9 @@ frames.
**Goal**: capture the desktop and feed frames into the encoder.
- Add `libpipewire-0.3` dependency.
- Implement `CaptureFactory::create()` using the xdg-desktop-portal.
- Wire `CaptureSession::next_frame()``Encoder::encode()` in a local smoke
- [x] Add `libpipewire-0.3` dependency.
- [x] Implement `CaptureFactory::create()` using the xdg-desktop-portal.
- [x] Wire `CaptureSession::next_frame()``Encoder::encode()` in a local smoke
test that just writes a few encoded frames to disk.
**Validation**: manual run on a real Linux desktop session produces a valid
@@ -89,4 +89,4 @@ where available.
## Current phase
Phase 3PipeWire screen capture.
Phase 4RTP Framing.
+22
View File
@@ -27,6 +27,28 @@ FFmpeg may keep some "still reachable" allocations at exit; that is normal.
This check caught two real defects in the Phase 2 decoder: a per-packet
payload leak and an unpadded extradata buffer over-read. Keep using it.
## Capture smoke test (Phase 3, manual)
Requires a running desktop session with `xdg-desktop-portal` and a backend
that implements the ScreenCast portal (e.g. Hyprland, GNOME, KDE). The
portal shows an interactive source picker, so this cannot run unattended.
```sh
meson compile -C build # builds tools/capture_smoke
./build/tools/capture_smoke 10 out.h264 # captures 10 frames
ffprobe -v error -show_entries stream=codec_name,width,height out.h264
```
Expected: the tool prints the negotiated resolution/format/stride and
writes a non-empty file; `ffprobe` reports `codec_name=h264` and the correct
width/height. The encoder prepends its Annex-B SPS/PPS extradata so the
file is a self-contained elementary stream.
If you see `impl_ext_end_proxy called from wrong context` warnings, a
PipeWire proxy operation ran without the thread-loop lock — all pw
calls that send messages (connect, stream, core) must happen under
`pw_thread_loop_lock`.
## Formatting
```sh
+1
View File
@@ -13,6 +13,7 @@ namespace sc {
enum class PixelFormat {
Rgba,
Bgrx, // 4 bytes/pixel, memory order B, G, R, unused
Yuv420p,
};
+3
View File
@@ -10,6 +10,9 @@ project('screen_cast', 'cpp',
# Public and private include directories are declared in `src/meson.build`.
subdir('src')
# Manual smoke tools
subdir('tools')
# Tests
enable_tests = get_option('tests')
if enable_tests
+12 -6
View File
@@ -1,11 +1,17 @@
# Phase 3 capture placeholder. libpipewire-0.3 will be added when the portal
# implementation is wired in.
# Phase 3 capture backend: PipeWire via the xdg-desktop-portal ScreenCast
# interface, with libportal handling the portal D-Bus handshake.
dep_pipewire = dependency('libpipewire-0.3')
dep_portal = dependency('libportal')
sc_capture_sources = files('pipewire_capture.cpp')
# Built even though nothing links it yet, so the stub stays compile-clean.
static_library('sc_capture',
sc_capture = static_library('sc_capture',
sc_capture_sources,
include_directories : sc_core_inc)
include_directories : sc_core_inc,
dependencies : [dep_pipewire, dep_portal])
# sc_capture_dep will be declared once the app/pipeline code links against it.
sc_capture_dep = declare_dependency(
link_with : sc_capture,
include_directories : sc_core_inc,
dependencies : [dep_pipewire, dep_portal])
+573 -4
View File
@@ -1,13 +1,582 @@
#include "screencast/capture/capture.h"
#include <glib.h>
#include <libportal/portal-helpers.h>
#include <libportal/portal.h>
#include <libportal/remote.h>
#include <libportal/session.h>
#include <pipewire/pipewire.h>
#include <pipewire/thread-loop.h>
#include <spa/param/param.h>
#include <spa/param/video/format-utils.h>
#include <spa/pod/builder.h>
#include <spa/utils/result.h>
#include <unistd.h>
#include <array>
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cstdint>
#include <cstring>
#include <format>
#include <iostream>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <utility>
namespace sc {
namespace {
CaptureResult<std::unique_ptr<CaptureSession>> CaptureFactory::create(CaptureTarget /*target*/) {
// Phase 3 placeholder. The PipeWire / xdg-desktop-portal capture backend
// will be implemented here once the portal integration lands.
return CaptureError{"PipeWire capture is not implemented yet"};
struct GObjectDeleter {
void operator()(gpointer object) const noexcept {
if (object != nullptr) {
g_object_unref(object);
}
}
};
using PortalPtr = std::unique_ptr<XdpPortal, GObjectDeleter>;
using XdpSessionPtr = std::unique_ptr<XdpSession, GObjectDeleter>;
class ScopedMainContext {
public:
ScopedMainContext() : context_(g_main_context_new()) {
if (context_ != nullptr) {
g_main_context_push_thread_default(context_);
}
}
~ScopedMainContext() {
if (context_ != nullptr) {
g_main_context_pop_thread_default(context_);
g_main_context_unref(context_);
}
}
ScopedMainContext(const ScopedMainContext&) = delete;
ScopedMainContext& operator=(const ScopedMainContext&) = delete;
GMainContext* get() const noexcept {
return context_;
}
private:
GMainContext* context_;
};
class ScopedMainLoop {
public:
explicit ScopedMainLoop(GMainContext* context) : loop_(g_main_loop_new(context, FALSE)) {}
~ScopedMainLoop() {
if (loop_ != nullptr) {
g_main_loop_unref(loop_);
}
}
ScopedMainLoop(const ScopedMainLoop&) = delete;
ScopedMainLoop& operator=(const ScopedMainLoop&) = delete;
void run() {
g_main_loop_run(loop_);
}
void quit() {
g_main_loop_quit(loop_);
}
private:
GMainLoop* loop_;
};
// Owns a raw fd and closes it unless release()d.
class OwnedFd {
public:
explicit OwnedFd(int fd) noexcept : fd_(fd) {}
~OwnedFd() {
if (fd_ >= 0) {
close(fd_);
}
}
OwnedFd(const OwnedFd&) = delete;
OwnedFd& operator=(const OwnedFd&) = delete;
OwnedFd(OwnedFd&& other) noexcept : fd_(other.fd_) {
other.fd_ = -1;
}
OwnedFd& operator=(OwnedFd&& other) noexcept {
if (this != &other) {
if (fd_ >= 0) {
close(fd_);
}
fd_ = other.fd_;
other.fd_ = -1;
}
return *this;
}
int get() const noexcept {
return fd_;
}
int release() noexcept {
const int fd = fd_;
fd_ = -1;
return fd;
}
private:
int fd_;
};
std::string gerror_message(const GError* error) {
if (error == nullptr || error->message == nullptr) {
return "unknown error";
}
return std::string{error->message};
}
uint64_t monotonic_now_ns() {
const auto now = std::chrono::steady_clock::now().time_since_epoch();
return static_cast<uint64_t>(std::chrono::duration_cast<std::chrono::nanoseconds>(now).count());
}
std::optional<PixelFormat> pixel_format_from_spa(uint32_t spa_format) noexcept {
switch (spa_format) {
case SPA_VIDEO_FORMAT_BGRx:
case SPA_VIDEO_FORMAT_BGRA:
return PixelFormat::Bgrx;
case SPA_VIDEO_FORMAT_RGBx:
case SPA_VIDEO_FORMAT_RGBA:
return PixelFormat::Rgba;
default:
return std::nullopt;
}
}
// The portal Start response exposes one entry per stream as (u a{sv}); the
// first child of the first stream holds the PipeWire node id.
std::optional<uint32_t> first_stream_node_id(XdpSession* session) {
GVariant* streams = xdp_session_get_streams(session); // transfer full
if (streams == nullptr) {
return std::nullopt;
}
std::optional<uint32_t> node_id;
if (g_variant_n_children(streams) > 0) {
GVariant* stream = g_variant_get_child_value(streams, 0);
if (stream != nullptr) {
if (g_variant_is_of_type(stream, G_VARIANT_TYPE("(ua{sv})"))) {
guint32 id = 0;
GVariant* properties = nullptr;
g_variant_get(stream, "(u@a{sv})", &id, &properties);
if (properties != nullptr) {
g_variant_unref(properties);
}
node_id = static_cast<uint32_t>(id);
}
g_variant_unref(stream);
}
}
g_variant_unref(streams);
return node_id;
}
struct SessionCreatedOp {
ScopedMainLoop* loop = nullptr;
XdpSession* session = nullptr; // owned, transferred by the caller
GError* error = nullptr; // owned, cleared by the caller
};
void on_screencast_session_created(GObject* source, GAsyncResult* result, gpointer user_data) {
auto* op = static_cast<SessionCreatedOp*>(user_data);
op->session = xdp_portal_create_screencast_session_finish(XDP_PORTAL(source), result, &op->error);
op->loop->quit();
}
struct SessionStartedOp {
ScopedMainLoop* loop = nullptr;
bool started = false;
GError* error = nullptr; // owned, cleared by the caller
};
void on_screencast_session_started(GObject* source, GAsyncResult* result, gpointer user_data) {
auto* op = static_cast<SessionStartedOp*>(user_data);
op->started = xdp_session_start_finish(XDP_SESSION(source), result, &op->error) == TRUE;
op->loop->quit();
}
} // namespace
// PipeWire screencast session driven by the xdg-desktop-portal ScreenCast
// interface (via libportal). The portal handshake runs synchronously on the
// thread that calls CaptureFactory::create(); frames are produced on a
// PipeWire thread and handed to next_frame() through a latest-frame slot.
class PipeWireCaptureSession final : public CaptureSession {
public:
PipeWireCaptureSession(uint32_t node_id, XdpSessionPtr portal_session, PortalPtr portal)
: node_id_(node_id), portal_session_(std::move(portal_session)), portal_(std::move(portal)) {
stream_events_.version = PW_VERSION_STREAM_EVENTS;
stream_events_.state_changed = &PipeWireCaptureSession::on_state_changed;
stream_events_.param_changed = &PipeWireCaptureSession::on_param_changed;
stream_events_.process = &PipeWireCaptureSession::on_process;
}
~PipeWireCaptureSession() override {
stop();
teardown_pipewire();
}
PipeWireCaptureSession(const PipeWireCaptureSession&) = delete;
PipeWireCaptureSession& operator=(const PipeWireCaptureSession&) = delete;
std::optional<CapturedFrame> next_frame() override {
std::unique_lock lock(frame_mutex_);
frame_cv_.wait(lock, [this] { return stopped_ || has_new_frame_; });
if (!has_new_frame_) {
return std::nullopt; // stopped (or the stream ended with an error)
}
has_new_frame_ = false;
return std::move(latest_frame_);
}
void stop() override {
{
std::lock_guard lock(frame_mutex_);
stopped_ = true;
}
frame_cv_.notify_all();
pw_thread_loop* loop = loop_;
if (loop != nullptr) {
pw_thread_loop_lock(loop);
if (stream_ != nullptr) {
pw_stream_disconnect(stream_);
}
pw_thread_loop_unlock(loop);
}
}
// Connects the capture stream to the portal's PipeWire node. On success
// the fd has been handed to PipeWire; on failure it is closed here.
//
// Every PipeWire operation that marshals a proxy message must run under
// the thread-loop lock; the protocol extension rejects calls from other
// contexts ("impl_ext_end_proxy called from wrong context").
std::optional<std::string> connect_to_pipewire(OwnedFd fd) {
loop_ = pw_thread_loop_new("screencast-capture", nullptr);
if (loop_ == nullptr) {
return std::string{"failed to create the PipeWire thread loop"};
}
if (pw_thread_loop_start(loop_) < 0) {
return std::string{"failed to start the PipeWire thread loop"};
}
pw_thread_loop_lock(loop_);
context_ = pw_context_new(pw_thread_loop_get_loop(loop_), nullptr, 0);
if (context_ == nullptr) {
pw_thread_loop_unlock(loop_);
return std::string{"failed to create the PipeWire context"};
}
// pw_context_connect_fd takes ownership of the socket (see core.h).
core_ = pw_context_connect_fd(context_, fd.release(), nullptr, 0);
if (core_ == nullptr) {
pw_thread_loop_unlock(loop_);
return std::string{"failed to connect to the PipeWire remote"};
}
pw_properties* props = pw_properties_new(PW_KEY_MEDIA_TYPE, "Video", PW_KEY_MEDIA_CATEGORY, "Capture", nullptr);
stream_ = pw_stream_new(core_, "screencast", props);
if (stream_ == nullptr) {
pw_thread_loop_unlock(loop_);
return std::string{"failed to create the capture stream"};
}
pw_stream_add_listener(stream_, &stream_listener_, &stream_events_, this);
// Enumerate the formats the portal stream can provide. Properties that
// are omitted (size, framerate) mean "any".
std::array<uint8_t, 1024> pod_buffer{};
struct spa_pod_builder pod_builder = SPA_POD_BUILDER_INIT(pod_buffer.data(), pod_buffer.size());
const struct spa_pod* connect_params[1] = {nullptr};
{
struct spa_pod_frame outer[1];
struct spa_pod_frame choice[1];
spa_pod_builder_push_object(&pod_builder, &outer[0], SPA_TYPE_OBJECT_Format, SPA_PARAM_EnumFormat);
spa_pod_builder_add(&pod_builder,
SPA_FORMAT_mediaType,
SPA_POD_Id(SPA_MEDIA_TYPE_video),
SPA_FORMAT_mediaSubtype,
SPA_POD_Id(SPA_MEDIA_SUBTYPE_raw),
0);
spa_pod_builder_prop(&pod_builder, SPA_FORMAT_VIDEO_format, 0);
spa_pod_builder_push_choice(&pod_builder, &choice[0], SPA_CHOICE_Enum, 0);
spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_BGRx); // default
spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_BGRA);
spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_RGBx);
spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_RGBA);
spa_pod_builder_pop(&pod_builder, &choice[0]);
connect_params[0] = static_cast<const struct spa_pod*>(spa_pod_builder_pop(&pod_builder, &outer[0]));
}
const int connect_result = pw_stream_connect(stream_,
PW_DIRECTION_INPUT,
node_id_,
static_cast<enum pw_stream_flags>(PW_STREAM_FLAG_AUTOCONNECT |
PW_STREAM_FLAG_MAP_BUFFERS |
PW_STREAM_FLAG_DONT_RECONNECT),
connect_params,
1);
pw_thread_loop_unlock(loop_);
if (connect_result < 0) {
return std::string{"failed to connect the capture stream: "} + spa_strerror(connect_result);
}
return std::nullopt;
}
private:
static void on_state_changed(void* data, enum pw_stream_state, enum pw_stream_state state, const char* error) {
if (state != PW_STREAM_STATE_ERROR) {
return;
}
auto* self = static_cast<PipeWireCaptureSession*>(data);
std::cerr << std::format("screencast: capture stream error: {}\n", error != nullptr ? error : "unknown error");
self->mark_stopped();
}
static void on_param_changed(void* data, uint32_t id, const struct spa_pod* param) {
if (param == nullptr || id != SPA_PARAM_Format) {
return;
}
auto* self = static_cast<PipeWireCaptureSession*>(data);
struct spa_video_info info = {};
if (spa_format_parse(param, &info.media_type, &info.media_subtype) < 0) {
return;
}
if (info.media_type != SPA_MEDIA_TYPE_video || info.media_subtype != SPA_MEDIA_SUBTYPE_raw) {
return;
}
if (spa_format_video_raw_parse(param, &info.info.raw) < 0) {
return;
}
self->video_width_.store(info.info.raw.size.width, std::memory_order_relaxed);
self->video_height_.store(info.info.raw.size.height, std::memory_order_relaxed);
self->video_spa_format_.store(info.info.raw.format, std::memory_order_relaxed);
}
static void on_process(void* data) {
auto* self = static_cast<PipeWireCaptureSession*>(data);
struct pw_buffer* buffer = nullptr;
while ((buffer = pw_stream_dequeue_buffer(self->stream_)) != nullptr) {
self->store_frame(*buffer);
pw_stream_queue_buffer(self->stream_, buffer);
}
}
void store_frame(const struct pw_buffer& pw_buffer) {
const struct spa_buffer* buffer = pw_buffer.buffer;
if (buffer == nullptr || buffer->n_datas < 1) {
return;
}
const struct spa_data& data = buffer->datas[0];
if (data.type == SPA_DATA_DmaBuf && !dma_buf_logged_.exchange(true)) {
std::cerr << "screencast: the portal offers DMA-BUF buffers only; software capture "
"requires mapped memory (deferred to the hardware-acceleration phase)\n";
}
if (data.data == nullptr || data.chunk == nullptr) {
return; // unmapped (e.g. DMA-BUF) or empty buffer
}
const int width = video_width_.load(std::memory_order_relaxed);
const int height = video_height_.load(std::memory_order_relaxed);
const uint32_t spa_format = video_spa_format_.load(std::memory_order_relaxed);
if (width <= 0 || height <= 0) {
return;
}
const auto pixel_format = pixel_format_from_spa(spa_format);
if (!pixel_format.has_value()) {
if (!unsupported_format_logged_.exchange(true)) {
std::cerr << std::format("screencast: unsupported negotiated pixel format {}\n", spa_format);
}
return;
}
const int stride = data.chunk->stride;
const int row_bytes = width * 4; // every enumerated format is 4 bytes/pixel
if (stride < row_bytes) {
return;
}
const std::size_t required_size = static_cast<std::size_t>(stride) * (static_cast<std::size_t>(height) - 1) +
static_cast<std::size_t>(row_bytes);
if (data.chunk->size < required_size && data.maxsize < required_size) {
return; // buffer smaller than the negotiated frame
}
CapturedFrame frame;
frame.width = width;
frame.height = height;
frame.pixel_format = *pixel_format;
frame.stride = stride;
frame.timestamp_ns = monotonic_now_ns();
frame.pixels.resize(required_size);
std::memcpy(frame.pixels.data(), data.data, required_size);
{
std::lock_guard lock(frame_mutex_);
latest_frame_ = std::move(frame);
has_new_frame_ = true;
}
frame_cv_.notify_all();
}
void mark_stopped() {
{
std::lock_guard lock(frame_mutex_);
stopped_ = true;
}
frame_cv_.notify_all();
}
// PipeWire teardown in the order required by the API: destroy the stream
// and disconnect the core under the thread-loop lock, then drop the
// context, then stop the loop.
void teardown_pipewire() noexcept {
if (loop_ != nullptr) {
pw_thread_loop_lock(loop_);
if (stream_ != nullptr) {
spa_hook_remove(&stream_listener_);
pw_stream_destroy(stream_);
stream_ = nullptr;
}
if (core_ != nullptr) {
pw_core_disconnect(core_);
core_ = nullptr;
}
pw_thread_loop_unlock(loop_);
}
if (context_ != nullptr) {
pw_context_destroy(context_);
context_ = nullptr;
}
if (loop_ != nullptr) {
pw_thread_loop_stop(loop_);
pw_thread_loop_destroy(loop_);
loop_ = nullptr;
}
pw_deinit();
}
uint32_t node_id_ = 0;
XdpSessionPtr portal_session_;
PortalPtr portal_;
pw_thread_loop* loop_ = nullptr;
pw_context* context_ = nullptr;
pw_core* core_ = nullptr;
pw_stream* stream_ = nullptr;
struct spa_hook stream_listener_ = {};
struct pw_stream_events stream_events_ = {};
std::atomic<int> video_width_{0};
std::atomic<int> video_height_{0};
std::atomic<uint32_t> video_spa_format_{SPA_VIDEO_FORMAT_UNKNOWN};
std::atomic<bool> dma_buf_logged_{false};
std::atomic<bool> unsupported_format_logged_{false};
std::mutex frame_mutex_;
std::condition_variable frame_cv_;
std::optional<CapturedFrame> latest_frame_;
bool has_new_frame_ = false;
bool stopped_ = false;
};
CaptureResult<std::unique_ptr<CaptureSession>> CaptureFactory::create(CaptureTarget target) {
XdpOutputType outputs = XDP_OUTPUT_NONE;
if (std::holds_alternative<CaptureTargetWholeScreen>(target)) {
outputs = XDP_OUTPUT_MONITOR;
} else if (std::holds_alternative<CaptureTargetWindow>(target)) {
outputs = XDP_OUTPUT_WINDOW;
} else {
return CaptureError{"region capture is not supported by the desktop portal"};
}
pw_init(nullptr, nullptr);
ScopedMainContext main_context;
if (main_context.get() == nullptr) {
return CaptureError{"failed to create a GLib main context"};
}
ScopedMainLoop main_loop(main_context.get());
PortalPtr portal(xdp_portal_new());
if (portal == nullptr) {
return CaptureError{"failed to connect to the desktop portal"};
}
// Step 1: create the screencast session. The user picks a source in the
// portal dialog while this call blocks.
SessionCreatedOp created{&main_loop};
xdp_portal_create_screencast_session(portal.get(),
outputs,
XDP_SCREENCAST_FLAG_NONE,
XDP_CURSOR_MODE_EMBEDDED,
XDP_PERSIST_MODE_NONE,
nullptr,
nullptr,
&on_screencast_session_created,
&created);
main_loop.run();
XdpSessionPtr portal_session(created.session);
if (portal_session == nullptr) {
const std::string message = gerror_message(created.error);
g_clear_error(&created.error);
return CaptureError{"failed to create the screencast session: " + message};
}
g_clear_error(&created.error);
// Step 2: start the session.
SessionStartedOp started{&main_loop};
xdp_session_start(portal_session.get(), nullptr, nullptr, &on_screencast_session_started, &started);
main_loop.run();
if (!started.started) {
const std::string message = gerror_message(started.error);
g_clear_error(&started.error);
return CaptureError{"failed to start the screencast session: " + message};
}
g_clear_error(&started.error);
// Step 3: find the PipeWire node to capture.
const std::optional<uint32_t> node_id = first_stream_node_id(portal_session.get());
if (!node_id.has_value()) {
return CaptureError{"the screencast session exposes no streams"};
}
OwnedFd remote_fd(xdp_session_open_pipewire_remote(portal_session.get()));
if (remote_fd.get() < 0) {
return CaptureError{"failed to open the PipeWire remote of the screencast session"};
}
auto session = std::make_unique<PipeWireCaptureSession>(*node_id, std::move(portal_session), std::move(portal));
if (auto error = session->connect_to_pipewire(std::move(remote_fd))) {
return CaptureError{std::move(*error)};
}
return std::unique_ptr<CaptureSession>(std::move(session));
}
} // namespace sc
+56 -14
View File
@@ -3,6 +3,7 @@
#include "ffmpeg_raii.h"
#include <algorithm>
#include <array>
#include <atomic>
#include <cstdint>
@@ -18,15 +19,24 @@ namespace {
using namespace sc::detail;
AVPixelFormat to_ffmpeg_format(PixelFormat format) noexcept {
return format == PixelFormat::Rgba ? AV_PIX_FMT_RGBA : AV_PIX_FMT_YUV420P;
switch (format) {
case PixelFormat::Rgba:
return AV_PIX_FMT_RGBA;
case PixelFormat::Bgrx:
// BGRx is BGR with an unused fourth byte; FFmpeg models that layout as BGRA.
return AV_PIX_FMT_BGRA;
case PixelFormat::Yuv420p:
return AV_PIX_FMT_YUV420P;
}
return AV_PIX_FMT_NONE;
}
int expected_first_plane_stride(PixelFormat format, int width) noexcept {
return format == PixelFormat::Rgba ? width * 4 : width;
bool is_packed_rgb(PixelFormat format) noexcept {
return format == PixelFormat::Rgba || format == PixelFormat::Bgrx;
}
std::size_t expected_buffer_size(PixelFormat format, int width, int height) {
return static_cast<std::size_t>(av_image_get_buffer_size(to_ffmpeg_format(format), width, height, 1));
int minimum_row_bytes(PixelFormat format, int width) noexcept {
return is_packed_rgb(format) ? width * 4 : width;
}
enum class ReceiveStatus {
@@ -244,12 +254,29 @@ class FfmpegEncoder final : public Encoder {
}
const AVPixelFormat input_format = to_ffmpeg_format(frame.pixel_format);
const int expected_stride = expected_first_plane_stride(frame.pixel_format, frame.width);
if (frame.stride != 0 && frame.stride != expected_stride) {
return CodecError{"unsupported input stride"};
if (input_format == AV_PIX_FMT_NONE) {
return CodecError{"unsupported pixel format"};
}
const std::size_t required_size = expected_buffer_size(frame.pixel_format, frame.width, frame.height);
const int row_bytes = minimum_row_bytes(frame.pixel_format, frame.width);
std::size_t stride = static_cast<std::size_t>(row_bytes);
if (frame.stride != 0) {
if (frame.stride < row_bytes) {
return CodecError{"input stride is smaller than one pixel row"};
}
if (!is_packed_rgb(frame.pixel_format) && frame.stride != row_bytes) {
return CodecError{"unsupported input stride for planar format"};
}
stride = static_cast<std::size_t>(frame.stride);
}
std::size_t required_size = 0;
if (is_packed_rgb(frame.pixel_format)) {
required_size = stride * (static_cast<std::size_t>(frame.height) - 1) + static_cast<std::size_t>(row_bytes);
} else {
required_size =
static_cast<std::size_t>(av_image_get_buffer_size(input_format, frame.width, frame.height, 1));
}
if (frame.pixels.size() < required_size) {
return CodecError{"input pixel buffer is too small"};
}
@@ -274,12 +301,27 @@ class FfmpegEncoder final : public Encoder {
return CodecError{"failed to create swscale context"};
}
std::array<uint8_t*, 4> src{nullptr, nullptr, nullptr, nullptr};
std::array<const uint8_t*, 4> src{nullptr, nullptr, nullptr, nullptr};
std::array<int, 4> src_lines{0, 0, 0, 0};
if (av_image_fill_arrays(
src.data(), src_lines.data(), as_u8(frame.pixels.data()), input_format, frame.width, frame.height, 1) <
0) {
return CodecError{"failed to fill input pixel arrays"};
if (is_packed_rgb(frame.pixel_format)) {
// Packed RGB is a single plane whose row pitch may be padded beyond
// the packed row size, so build the source arrays by hand.
src[0] = as_u8(frame.pixels.data());
src_lines[0] = static_cast<int>(stride);
} else {
std::array<uint8_t*, 4> planar_src{nullptr, nullptr, nullptr, nullptr};
std::array<int, 4> planar_lines{0, 0, 0, 0};
if (av_image_fill_arrays(planar_src.data(),
planar_lines.data(),
as_u8(frame.pixels.data()),
input_format,
frame.width,
frame.height,
1) < 0) {
return CodecError{"failed to fill input pixel arrays"};
}
std::copy(planar_src.begin(), planar_src.end(), src.begin());
std::copy(planar_lines.begin(), planar_lines.end(), src_lines.begin());
}
if (sws_scale(scaler_.get(), src.data(), src_lines.data(), 0, frame.height, output->data, output->linesize) <=
+167
View File
@@ -0,0 +1,167 @@
// Phase 3 smoke test: capture a few real desktop frames through the portal,
// encode them to H.264, and write an Annex-B elementary stream to disk.
//
// This tool is intentionally manual: it needs a running desktop session and
// the user must confirm the source picker dialog, so it is not registered
// with `meson test`. Validation steps are documented in docs/RUNBOOK.md.
#include "screencast/capture/capture.h"
#include "screencast/codec/encoder.h"
#include <charconv>
#include <cstdint>
#include <format>
#include <fstream>
#include <iostream>
#include <memory>
#include <string>
#include <string_view>
#include <vector>
namespace {
int positive_int_arg(std::string_view text) {
int value = 0;
const auto [ptr, ec] = std::from_chars(text.data(), text.data() + text.size(), value);
if (ec != std::errc{} || ptr != text.data() + text.size() || value <= 0) {
return -1;
}
return value;
}
std::string_view pixel_format_name(sc::PixelFormat format) {
switch (format) {
case sc::PixelFormat::Rgba:
return "rgba";
case sc::PixelFormat::Bgrx:
return "bgrx";
case sc::PixelFormat::Yuv420p:
return "yuv420p";
}
return "unknown";
}
bool write_bytes(std::ofstream& output, const std::vector<std::byte>& bytes) {
output.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
return output.good();
}
} // namespace
int main(int argc, char** argv) {
const int max_frames = argc > 1 ? positive_int_arg(argv[1]) : 10;
if (max_frames < 0) {
std::cerr << "usage: capture_smoke [frames] [output.h264]\n";
return 1;
}
const std::string output_path = argc > 2 ? argv[2] : "screencast_smoke.h264";
std::cout << "creating capture session (choose a source in the portal dialog)...\n";
auto capture_result = sc::CaptureFactory::create(sc::CaptureTargetWholeScreen{});
if (sc::is_capture_error(capture_result)) {
std::cerr << std::format("capture failed: {}\n", sc::capture_error(capture_result).message);
return 1;
}
auto capture = std::move(sc::capture_value(capture_result));
std::ofstream output(output_path, std::ios::binary | std::ios::trunc);
if (!output.is_open()) {
std::cerr << std::format("failed to open {} for writing\n", output_path);
return 1;
}
std::unique_ptr<sc::Encoder> encoder;
std::uint64_t total_bytes = 0;
std::uint64_t encoded_frames = 0;
std::uint64_t keyframes = 0;
for (int captured = 0; captured < max_frames; ++captured) {
auto frame = capture->next_frame();
if (!frame.has_value()) {
std::cout << std::format("capture session ended after {} frames\n", captured);
break;
}
if (encoder == nullptr) {
sc::EncoderConfig config;
config.width = frame->width;
config.height = frame->height;
config.frame_rate_num = 25;
config.frame_rate_den = 1;
config.bitrate_kbps = 8000;
auto encoder_result = sc::EncoderFactory::create(config);
if (sc::is_codec_error(encoder_result)) {
std::cerr << std::format("encoder creation failed: {}\n", sc::codec_error(encoder_result).message);
return 1;
}
encoder = std::move(sc::codec_value(encoder_result));
// The encoder is configured with AV_CODEC_FLAG_GLOBAL_HEADER, so
// parameter sets live in extradata; prepend them so the output
// file is a self-contained Annex-B stream.
const auto extradata = encoder->get_extradata();
if (!extradata.empty()) {
if (!write_bytes(output, extradata)) {
std::cerr << std::format("failed to write parameter sets to {}\n", output_path);
return 1;
}
total_bytes += extradata.size();
}
std::cout << std::format("capturing {}x{} ({}, stride {})\n",
frame->width,
frame->height,
pixel_format_name(frame->pixel_format),
frame->stride);
}
auto encoded_result = encoder->encode(*frame);
if (sc::is_codec_error(encoded_result)) {
std::cerr << std::format("encode failed: {}\n", sc::codec_error(encoded_result).message);
return 1;
}
for (const auto& encoded : sc::codec_value(encoded_result)) {
if (!write_bytes(output, encoded.data)) {
std::cerr << std::format("failed to write encoded data to {}\n", output_path);
return 1;
}
total_bytes += encoded.data.size();
++encoded_frames;
keyframes += encoded.is_keyframe ? 1 : 0;
}
}
if (encoder != nullptr) {
auto flushed = encoder->flush();
if (sc::is_codec_error(flushed)) {
std::cerr << std::format("flush failed: {}\n", sc::codec_error(flushed).message);
return 1;
}
for (const auto& encoded : sc::codec_value(flushed)) {
if (!write_bytes(output, encoded.data)) {
std::cerr << std::format("failed to write flushed data to {}\n", output_path);
return 1;
}
total_bytes += encoded.data.size();
++encoded_frames;
}
}
capture->stop();
output.close();
if (total_bytes == 0 || encoded_frames == 0) {
std::cerr << "no frames were captured\n";
return 1;
}
std::cout << std::format("wrote {} bytes to {} ({} encoded frames, {} keyframes)\n",
total_bytes,
output_path,
encoded_frames,
keyframes);
std::cout << std::format("validate with: ffprobe -v error -show_entries stream=codec_name,width,height {}\n",
output_path);
return 0;
}
+6
View File
@@ -0,0 +1,6 @@
# Manual smoke tools. These are not registered with `meson test` because they
# need an interactive desktop session (portal consent dialog).
executable('capture_smoke',
'capture_smoke.cpp',
dependencies : [sc_capture_dep, sc_codec_dep])