diff --git a/.agents/MEMORY.md b/.agents/MEMORY.md index bb9a025..7b420f3 100644 --- a/.agents/MEMORY.md +++ b/.agents/MEMORY.md @@ -1,31 +1,37 @@ # Project Memory — screen_cast -Last updated: Phase 2 codec review fixes applied. +Last updated: Phase 3 validated and complete; current phase is Phase 4. ## Project state -- Phase 2 codec implementation reviewed with valgrind; all confirmed defects - fixed on top of the capture-stub commit: - - decoder leaked every packet payload (`av_malloc` + direct `packet->data` - assignment bypassed the packet's owning `AVBufferRef`); payloads are now - allocated with `av_new_packet`. - - decoder extradata lacked `AV_INPUT_BUFFER_PADDING_SIZE`; FFmpeg's - extradata parser over-read the buffer (valgrind invalid reads). Now - allocated padded and zeroed. - - oversized encoded frames are rejected before the int cast. - - EAGAIN-retry loops in encoder/decoder now handle unexpected EOF instead - of retrying forever. - - fixed "RTP packet" → "AVPacket" error message in the encoder. -- `CaptureFactory::create()` now returns - `CaptureResult>` (variant with - `CaptureError`) instead of a nullable unique_ptr; the stub reports - "PipeWire capture is not implemented yet" as an error. New pattern lives in - `include/screencast/capture/error.h`, mirroring `codec/error.h`. -- Validation: `meson test` 2/2 OK; valgrind on `test_codec_roundtrip` is now - clean (0 definite losses, 0 invalid reads); clang-format clean. See - `docs/RUNBOOK.md` for the reusable checks. -- Phase 3 capture stub remains in place; remaining implementation: capture, - transport, rendering, and CLI/pipeline glue. +- **Phase 3 is done and validated on the desktop**: a manual + `./build/tools/capture_smoke 10 out.h264` run on Wayland/Hyprland produced + a valid 2256x1504 H.264 elementary stream (ffprobe clean). `docs/PHASES.md` + is ticked; current phase is Phase 4 — RTP framing. +- The first smoke run emitted `impl_ext_end_proxy called from wrong context` + warnings: `pw_context_connect_fd` and `pw_core_disconnect` ran outside the + thread-loop lock. Fixed by holding the lock across all pw setup/teardown + proxy operations. Re-running the smoke tool should now be warning-free + (capture worked both ways; the warnings only meant the first two + marshaled messages were rejected and retried from the right context). +- `src/capture/pipewire_capture.cpp` now implements the full backend: + libportal 0.10 handshake (`create_screencast_session` → `session_start` → + `open_pipewire_remote`), PipeWire 1.6 stream on the first portal node, + BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff. + Portal handshake blocks on the caller thread; frames arrive on the pw + thread. `stop()` is thread-safe; teardown follows the pw-required order. +- Encoder now accepts padded strides for packed RGB formats and the new + `PixelFormat::Bgrx` (mapped to `AV_PIX_FMT_BGRA`); planar Yuv420p still + requires a packed layout. This was the review note blocking Phase 3. +- `tools/capture_smoke` (manual, not in `meson test`) captures N frames → + encodes → writes Annex-B including prepended SPS/PPS extradata (verified: + libx264 GLOBAL_HEADER extradata is Annex-B). +- Earlier review fixes remain in place; valgrind on the codec round-trip + test is still clean after the encoder stride changes. +- Encoder PTS caveat: the encoder time_base is derived from the configured + frame rate (default 25fps), but portal frames arrive at monitor refresh + (often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless + for the smoke test; revisit when RTP timestamps matter (Phase 4/5). ## Decisions @@ -55,14 +61,17 @@ None. ## Forward-looking review notes (for later phases) -- Encoder rejects non-packed strides in `make_input_frame`; PipeWire/portal - frames usually have alignment-padded strides — Phase 3 must pass the real - stride through to swscale instead of rejecting it. - `AV_CODEC_FLAG_GLOBAL_HEADER` suppresses in-band SPS/PPS; a receiver cannot join mid-stream or recover after PLI without parameter sets. Phase 5 must prepend SPS/PPS to keyframes or negotiate them in signaling. - Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother UDP streaming in Phase 7. +- DMA-BUF-only portal streams are rejected with a clear message (hardware + path is Phase 7). - No negative-path tests yet (bad config, bad stride, undersized buffer). - `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting - encoder is ever added, prefer an explicit config flag over the heuristic. \ No newline at end of file + encoder is ever added, prefer an explicit config flag over the heuristic. +- Region targets are rejected: the desktop portal has no region capture. +- `next_frame()` returns `nullopt` on stream error without surfacing the + reason (logged to stderr); consider an error channel when the receiver + pipeline lands. \ No newline at end of file diff --git a/.gitignore b/.gitignore index 84ff330..75c6bc0 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,9 @@ meson-private/ *.exe compile_commands.json +# Smoke test output +*.h264 + # IDE .vscode/ .idea/ diff --git a/docs/PHASES.md b/docs/PHASES.md index 29c5d8b..b95fed0 100644 --- a/docs/PHASES.md +++ b/docs/PHASES.md @@ -32,9 +32,9 @@ frames. **Goal**: capture the desktop and feed frames into the encoder. -- Add `libpipewire-0.3` dependency. -- Implement `CaptureFactory::create()` using the xdg-desktop-portal. -- Wire `CaptureSession::next_frame()` → `Encoder::encode()` in a local smoke +- [x] Add `libpipewire-0.3` dependency. +- [x] Implement `CaptureFactory::create()` using the xdg-desktop-portal. +- [x] Wire `CaptureSession::next_frame()` → `Encoder::encode()` in a local smoke test that just writes a few encoded frames to disk. **Validation**: manual run on a real Linux desktop session produces a valid @@ -89,4 +89,4 @@ where available. ## Current phase -Phase 3 — PipeWire screen capture. +Phase 4 — RTP Framing. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index a66c710..2f74288 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -27,6 +27,28 @@ FFmpeg may keep some "still reachable" allocations at exit; that is normal. This check caught two real defects in the Phase 2 decoder: a per-packet payload leak and an unpadded extradata buffer over-read. Keep using it. +## Capture smoke test (Phase 3, manual) + +Requires a running desktop session with `xdg-desktop-portal` and a backend +that implements the ScreenCast portal (e.g. Hyprland, GNOME, KDE). The +portal shows an interactive source picker, so this cannot run unattended. + +```sh +meson compile -C build # builds tools/capture_smoke +./build/tools/capture_smoke 10 out.h264 # captures 10 frames +ffprobe -v error -show_entries stream=codec_name,width,height out.h264 +``` + +Expected: the tool prints the negotiated resolution/format/stride and +writes a non-empty file; `ffprobe` reports `codec_name=h264` and the correct +width/height. The encoder prepends its Annex-B SPS/PPS extradata so the +file is a self-contained elementary stream. + +If you see `impl_ext_end_proxy called from wrong context` warnings, a +PipeWire proxy operation ran without the thread-loop lock — all pw +calls that send messages (connect, stream, core) must happen under +`pw_thread_loop_lock`. + ## Formatting ```sh diff --git a/include/screencast/capture/capture.h b/include/screencast/capture/capture.h index c5f3660..484bb33 100644 --- a/include/screencast/capture/capture.h +++ b/include/screencast/capture/capture.h @@ -13,6 +13,7 @@ namespace sc { enum class PixelFormat { Rgba, + Bgrx, // 4 bytes/pixel, memory order B, G, R, unused Yuv420p, }; diff --git a/meson.build b/meson.build index c80950f..159b4c6 100644 --- a/meson.build +++ b/meson.build @@ -10,6 +10,9 @@ project('screen_cast', 'cpp', # Public and private include directories are declared in `src/meson.build`. subdir('src') +# Manual smoke tools +subdir('tools') + # Tests enable_tests = get_option('tests') if enable_tests diff --git a/src/capture/meson.build b/src/capture/meson.build index 9a26209..f5a05b5 100644 --- a/src/capture/meson.build +++ b/src/capture/meson.build @@ -1,11 +1,17 @@ -# Phase 3 capture placeholder. libpipewire-0.3 will be added when the portal -# implementation is wired in. +# Phase 3 capture backend: PipeWire via the xdg-desktop-portal ScreenCast +# interface, with libportal handling the portal D-Bus handshake. + +dep_pipewire = dependency('libpipewire-0.3') +dep_portal = dependency('libportal') sc_capture_sources = files('pipewire_capture.cpp') -# Built even though nothing links it yet, so the stub stays compile-clean. -static_library('sc_capture', +sc_capture = static_library('sc_capture', sc_capture_sources, - include_directories : sc_core_inc) + include_directories : sc_core_inc, + dependencies : [dep_pipewire, dep_portal]) -# sc_capture_dep will be declared once the app/pipeline code links against it. +sc_capture_dep = declare_dependency( + link_with : sc_capture, + include_directories : sc_core_inc, + dependencies : [dep_pipewire, dep_portal]) \ No newline at end of file diff --git a/src/capture/pipewire_capture.cpp b/src/capture/pipewire_capture.cpp index 4e489d3..99d8259 100644 --- a/src/capture/pipewire_capture.cpp +++ b/src/capture/pipewire_capture.cpp @@ -1,13 +1,582 @@ #include "screencast/capture/capture.h" +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include #include +#include +#include +#include +#include namespace sc { +namespace { -CaptureResult> CaptureFactory::create(CaptureTarget /*target*/) { - // Phase 3 placeholder. The PipeWire / xdg-desktop-portal capture backend - // will be implemented here once the portal integration lands. - return CaptureError{"PipeWire capture is not implemented yet"}; +struct GObjectDeleter { + void operator()(gpointer object) const noexcept { + if (object != nullptr) { + g_object_unref(object); + } + } +}; + +using PortalPtr = std::unique_ptr; +using XdpSessionPtr = std::unique_ptr; + +class ScopedMainContext { + public: + ScopedMainContext() : context_(g_main_context_new()) { + if (context_ != nullptr) { + g_main_context_push_thread_default(context_); + } + } + + ~ScopedMainContext() { + if (context_ != nullptr) { + g_main_context_pop_thread_default(context_); + g_main_context_unref(context_); + } + } + + ScopedMainContext(const ScopedMainContext&) = delete; + ScopedMainContext& operator=(const ScopedMainContext&) = delete; + + GMainContext* get() const noexcept { + return context_; + } + + private: + GMainContext* context_; +}; + +class ScopedMainLoop { + public: + explicit ScopedMainLoop(GMainContext* context) : loop_(g_main_loop_new(context, FALSE)) {} + + ~ScopedMainLoop() { + if (loop_ != nullptr) { + g_main_loop_unref(loop_); + } + } + + ScopedMainLoop(const ScopedMainLoop&) = delete; + ScopedMainLoop& operator=(const ScopedMainLoop&) = delete; + + void run() { + g_main_loop_run(loop_); + } + + void quit() { + g_main_loop_quit(loop_); + } + + private: + GMainLoop* loop_; +}; + +// Owns a raw fd and closes it unless release()d. +class OwnedFd { + public: + explicit OwnedFd(int fd) noexcept : fd_(fd) {} + + ~OwnedFd() { + if (fd_ >= 0) { + close(fd_); + } + } + + OwnedFd(const OwnedFd&) = delete; + OwnedFd& operator=(const OwnedFd&) = delete; + + OwnedFd(OwnedFd&& other) noexcept : fd_(other.fd_) { + other.fd_ = -1; + } + + OwnedFd& operator=(OwnedFd&& other) noexcept { + if (this != &other) { + if (fd_ >= 0) { + close(fd_); + } + fd_ = other.fd_; + other.fd_ = -1; + } + return *this; + } + + int get() const noexcept { + return fd_; + } + + int release() noexcept { + const int fd = fd_; + fd_ = -1; + return fd; + } + + private: + int fd_; +}; + +std::string gerror_message(const GError* error) { + if (error == nullptr || error->message == nullptr) { + return "unknown error"; + } + return std::string{error->message}; +} + +uint64_t monotonic_now_ns() { + const auto now = std::chrono::steady_clock::now().time_since_epoch(); + return static_cast(std::chrono::duration_cast(now).count()); +} + +std::optional pixel_format_from_spa(uint32_t spa_format) noexcept { + switch (spa_format) { + case SPA_VIDEO_FORMAT_BGRx: + case SPA_VIDEO_FORMAT_BGRA: + return PixelFormat::Bgrx; + case SPA_VIDEO_FORMAT_RGBx: + case SPA_VIDEO_FORMAT_RGBA: + return PixelFormat::Rgba; + default: + return std::nullopt; + } +} + +// The portal Start response exposes one entry per stream as (u a{sv}); the +// first child of the first stream holds the PipeWire node id. +std::optional first_stream_node_id(XdpSession* session) { + GVariant* streams = xdp_session_get_streams(session); // transfer full + if (streams == nullptr) { + return std::nullopt; + } + + std::optional node_id; + if (g_variant_n_children(streams) > 0) { + GVariant* stream = g_variant_get_child_value(streams, 0); + if (stream != nullptr) { + if (g_variant_is_of_type(stream, G_VARIANT_TYPE("(ua{sv})"))) { + guint32 id = 0; + GVariant* properties = nullptr; + g_variant_get(stream, "(u@a{sv})", &id, &properties); + if (properties != nullptr) { + g_variant_unref(properties); + } + node_id = static_cast(id); + } + g_variant_unref(stream); + } + } + g_variant_unref(streams); + return node_id; +} + +struct SessionCreatedOp { + ScopedMainLoop* loop = nullptr; + XdpSession* session = nullptr; // owned, transferred by the caller + GError* error = nullptr; // owned, cleared by the caller +}; + +void on_screencast_session_created(GObject* source, GAsyncResult* result, gpointer user_data) { + auto* op = static_cast(user_data); + op->session = xdp_portal_create_screencast_session_finish(XDP_PORTAL(source), result, &op->error); + op->loop->quit(); +} + +struct SessionStartedOp { + ScopedMainLoop* loop = nullptr; + bool started = false; + GError* error = nullptr; // owned, cleared by the caller +}; + +void on_screencast_session_started(GObject* source, GAsyncResult* result, gpointer user_data) { + auto* op = static_cast(user_data); + op->started = xdp_session_start_finish(XDP_SESSION(source), result, &op->error) == TRUE; + op->loop->quit(); +} + +} // namespace + +// PipeWire screencast session driven by the xdg-desktop-portal ScreenCast +// interface (via libportal). The portal handshake runs synchronously on the +// thread that calls CaptureFactory::create(); frames are produced on a +// PipeWire thread and handed to next_frame() through a latest-frame slot. +class PipeWireCaptureSession final : public CaptureSession { + public: + PipeWireCaptureSession(uint32_t node_id, XdpSessionPtr portal_session, PortalPtr portal) + : node_id_(node_id), portal_session_(std::move(portal_session)), portal_(std::move(portal)) { + stream_events_.version = PW_VERSION_STREAM_EVENTS; + stream_events_.state_changed = &PipeWireCaptureSession::on_state_changed; + stream_events_.param_changed = &PipeWireCaptureSession::on_param_changed; + stream_events_.process = &PipeWireCaptureSession::on_process; + } + + ~PipeWireCaptureSession() override { + stop(); + teardown_pipewire(); + } + + PipeWireCaptureSession(const PipeWireCaptureSession&) = delete; + PipeWireCaptureSession& operator=(const PipeWireCaptureSession&) = delete; + + std::optional next_frame() override { + std::unique_lock lock(frame_mutex_); + frame_cv_.wait(lock, [this] { return stopped_ || has_new_frame_; }); + if (!has_new_frame_) { + return std::nullopt; // stopped (or the stream ended with an error) + } + has_new_frame_ = false; + return std::move(latest_frame_); + } + + void stop() override { + { + std::lock_guard lock(frame_mutex_); + stopped_ = true; + } + frame_cv_.notify_all(); + + pw_thread_loop* loop = loop_; + if (loop != nullptr) { + pw_thread_loop_lock(loop); + if (stream_ != nullptr) { + pw_stream_disconnect(stream_); + } + pw_thread_loop_unlock(loop); + } + } + + // Connects the capture stream to the portal's PipeWire node. On success + // the fd has been handed to PipeWire; on failure it is closed here. + // + // Every PipeWire operation that marshals a proxy message must run under + // the thread-loop lock; the protocol extension rejects calls from other + // contexts ("impl_ext_end_proxy called from wrong context"). + std::optional connect_to_pipewire(OwnedFd fd) { + loop_ = pw_thread_loop_new("screencast-capture", nullptr); + if (loop_ == nullptr) { + return std::string{"failed to create the PipeWire thread loop"}; + } + if (pw_thread_loop_start(loop_) < 0) { + return std::string{"failed to start the PipeWire thread loop"}; + } + + pw_thread_loop_lock(loop_); + + context_ = pw_context_new(pw_thread_loop_get_loop(loop_), nullptr, 0); + if (context_ == nullptr) { + pw_thread_loop_unlock(loop_); + return std::string{"failed to create the PipeWire context"}; + } + + // pw_context_connect_fd takes ownership of the socket (see core.h). + core_ = pw_context_connect_fd(context_, fd.release(), nullptr, 0); + if (core_ == nullptr) { + pw_thread_loop_unlock(loop_); + return std::string{"failed to connect to the PipeWire remote"}; + } + + pw_properties* props = pw_properties_new(PW_KEY_MEDIA_TYPE, "Video", PW_KEY_MEDIA_CATEGORY, "Capture", nullptr); + stream_ = pw_stream_new(core_, "screencast", props); + if (stream_ == nullptr) { + pw_thread_loop_unlock(loop_); + return std::string{"failed to create the capture stream"}; + } + pw_stream_add_listener(stream_, &stream_listener_, &stream_events_, this); + + // Enumerate the formats the portal stream can provide. Properties that + // are omitted (size, framerate) mean "any". + std::array pod_buffer{}; + struct spa_pod_builder pod_builder = SPA_POD_BUILDER_INIT(pod_buffer.data(), pod_buffer.size()); + const struct spa_pod* connect_params[1] = {nullptr}; + { + struct spa_pod_frame outer[1]; + struct spa_pod_frame choice[1]; + + spa_pod_builder_push_object(&pod_builder, &outer[0], SPA_TYPE_OBJECT_Format, SPA_PARAM_EnumFormat); + spa_pod_builder_add(&pod_builder, + SPA_FORMAT_mediaType, + SPA_POD_Id(SPA_MEDIA_TYPE_video), + SPA_FORMAT_mediaSubtype, + SPA_POD_Id(SPA_MEDIA_SUBTYPE_raw), + 0); + spa_pod_builder_prop(&pod_builder, SPA_FORMAT_VIDEO_format, 0); + spa_pod_builder_push_choice(&pod_builder, &choice[0], SPA_CHOICE_Enum, 0); + spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_BGRx); // default + spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_BGRA); + spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_RGBx); + spa_pod_builder_id(&pod_builder, SPA_VIDEO_FORMAT_RGBA); + spa_pod_builder_pop(&pod_builder, &choice[0]); + connect_params[0] = static_cast(spa_pod_builder_pop(&pod_builder, &outer[0])); + } + + const int connect_result = pw_stream_connect(stream_, + PW_DIRECTION_INPUT, + node_id_, + static_cast(PW_STREAM_FLAG_AUTOCONNECT | + PW_STREAM_FLAG_MAP_BUFFERS | + PW_STREAM_FLAG_DONT_RECONNECT), + connect_params, + 1); + + pw_thread_loop_unlock(loop_); + + if (connect_result < 0) { + return std::string{"failed to connect the capture stream: "} + spa_strerror(connect_result); + } + return std::nullopt; + } + + private: + static void on_state_changed(void* data, enum pw_stream_state, enum pw_stream_state state, const char* error) { + if (state != PW_STREAM_STATE_ERROR) { + return; + } + auto* self = static_cast(data); + std::cerr << std::format("screencast: capture stream error: {}\n", error != nullptr ? error : "unknown error"); + self->mark_stopped(); + } + + static void on_param_changed(void* data, uint32_t id, const struct spa_pod* param) { + if (param == nullptr || id != SPA_PARAM_Format) { + return; + } + auto* self = static_cast(data); + + struct spa_video_info info = {}; + if (spa_format_parse(param, &info.media_type, &info.media_subtype) < 0) { + return; + } + if (info.media_type != SPA_MEDIA_TYPE_video || info.media_subtype != SPA_MEDIA_SUBTYPE_raw) { + return; + } + if (spa_format_video_raw_parse(param, &info.info.raw) < 0) { + return; + } + + self->video_width_.store(info.info.raw.size.width, std::memory_order_relaxed); + self->video_height_.store(info.info.raw.size.height, std::memory_order_relaxed); + self->video_spa_format_.store(info.info.raw.format, std::memory_order_relaxed); + } + + static void on_process(void* data) { + auto* self = static_cast(data); + struct pw_buffer* buffer = nullptr; + while ((buffer = pw_stream_dequeue_buffer(self->stream_)) != nullptr) { + self->store_frame(*buffer); + pw_stream_queue_buffer(self->stream_, buffer); + } + } + + void store_frame(const struct pw_buffer& pw_buffer) { + const struct spa_buffer* buffer = pw_buffer.buffer; + if (buffer == nullptr || buffer->n_datas < 1) { + return; + } + const struct spa_data& data = buffer->datas[0]; + if (data.type == SPA_DATA_DmaBuf && !dma_buf_logged_.exchange(true)) { + std::cerr << "screencast: the portal offers DMA-BUF buffers only; software capture " + "requires mapped memory (deferred to the hardware-acceleration phase)\n"; + } + if (data.data == nullptr || data.chunk == nullptr) { + return; // unmapped (e.g. DMA-BUF) or empty buffer + } + + const int width = video_width_.load(std::memory_order_relaxed); + const int height = video_height_.load(std::memory_order_relaxed); + const uint32_t spa_format = video_spa_format_.load(std::memory_order_relaxed); + if (width <= 0 || height <= 0) { + return; + } + const auto pixel_format = pixel_format_from_spa(spa_format); + if (!pixel_format.has_value()) { + if (!unsupported_format_logged_.exchange(true)) { + std::cerr << std::format("screencast: unsupported negotiated pixel format {}\n", spa_format); + } + return; + } + + const int stride = data.chunk->stride; + const int row_bytes = width * 4; // every enumerated format is 4 bytes/pixel + if (stride < row_bytes) { + return; + } + + const std::size_t required_size = static_cast(stride) * (static_cast(height) - 1) + + static_cast(row_bytes); + if (data.chunk->size < required_size && data.maxsize < required_size) { + return; // buffer smaller than the negotiated frame + } + + CapturedFrame frame; + frame.width = width; + frame.height = height; + frame.pixel_format = *pixel_format; + frame.stride = stride; + frame.timestamp_ns = monotonic_now_ns(); + frame.pixels.resize(required_size); + std::memcpy(frame.pixels.data(), data.data, required_size); + + { + std::lock_guard lock(frame_mutex_); + latest_frame_ = std::move(frame); + has_new_frame_ = true; + } + frame_cv_.notify_all(); + } + + void mark_stopped() { + { + std::lock_guard lock(frame_mutex_); + stopped_ = true; + } + frame_cv_.notify_all(); + } + + // PipeWire teardown in the order required by the API: destroy the stream + // and disconnect the core under the thread-loop lock, then drop the + // context, then stop the loop. + void teardown_pipewire() noexcept { + if (loop_ != nullptr) { + pw_thread_loop_lock(loop_); + if (stream_ != nullptr) { + spa_hook_remove(&stream_listener_); + pw_stream_destroy(stream_); + stream_ = nullptr; + } + if (core_ != nullptr) { + pw_core_disconnect(core_); + core_ = nullptr; + } + pw_thread_loop_unlock(loop_); + } + if (context_ != nullptr) { + pw_context_destroy(context_); + context_ = nullptr; + } + if (loop_ != nullptr) { + pw_thread_loop_stop(loop_); + pw_thread_loop_destroy(loop_); + loop_ = nullptr; + } + pw_deinit(); + } + + uint32_t node_id_ = 0; + XdpSessionPtr portal_session_; + PortalPtr portal_; + + pw_thread_loop* loop_ = nullptr; + pw_context* context_ = nullptr; + pw_core* core_ = nullptr; + pw_stream* stream_ = nullptr; + struct spa_hook stream_listener_ = {}; + struct pw_stream_events stream_events_ = {}; + + std::atomic video_width_{0}; + std::atomic video_height_{0}; + std::atomic video_spa_format_{SPA_VIDEO_FORMAT_UNKNOWN}; + std::atomic dma_buf_logged_{false}; + std::atomic unsupported_format_logged_{false}; + + std::mutex frame_mutex_; + std::condition_variable frame_cv_; + std::optional latest_frame_; + bool has_new_frame_ = false; + bool stopped_ = false; +}; + +CaptureResult> CaptureFactory::create(CaptureTarget target) { + XdpOutputType outputs = XDP_OUTPUT_NONE; + if (std::holds_alternative(target)) { + outputs = XDP_OUTPUT_MONITOR; + } else if (std::holds_alternative(target)) { + outputs = XDP_OUTPUT_WINDOW; + } else { + return CaptureError{"region capture is not supported by the desktop portal"}; + } + + pw_init(nullptr, nullptr); + + ScopedMainContext main_context; + if (main_context.get() == nullptr) { + return CaptureError{"failed to create a GLib main context"}; + } + ScopedMainLoop main_loop(main_context.get()); + + PortalPtr portal(xdp_portal_new()); + if (portal == nullptr) { + return CaptureError{"failed to connect to the desktop portal"}; + } + + // Step 1: create the screencast session. The user picks a source in the + // portal dialog while this call blocks. + SessionCreatedOp created{&main_loop}; + xdp_portal_create_screencast_session(portal.get(), + outputs, + XDP_SCREENCAST_FLAG_NONE, + XDP_CURSOR_MODE_EMBEDDED, + XDP_PERSIST_MODE_NONE, + nullptr, + nullptr, + &on_screencast_session_created, + &created); + main_loop.run(); + + XdpSessionPtr portal_session(created.session); + if (portal_session == nullptr) { + const std::string message = gerror_message(created.error); + g_clear_error(&created.error); + return CaptureError{"failed to create the screencast session: " + message}; + } + g_clear_error(&created.error); + + // Step 2: start the session. + SessionStartedOp started{&main_loop}; + xdp_session_start(portal_session.get(), nullptr, nullptr, &on_screencast_session_started, &started); + main_loop.run(); + if (!started.started) { + const std::string message = gerror_message(started.error); + g_clear_error(&started.error); + return CaptureError{"failed to start the screencast session: " + message}; + } + g_clear_error(&started.error); + + // Step 3: find the PipeWire node to capture. + const std::optional node_id = first_stream_node_id(portal_session.get()); + if (!node_id.has_value()) { + return CaptureError{"the screencast session exposes no streams"}; + } + + OwnedFd remote_fd(xdp_session_open_pipewire_remote(portal_session.get())); + if (remote_fd.get() < 0) { + return CaptureError{"failed to open the PipeWire remote of the screencast session"}; + } + + auto session = std::make_unique(*node_id, std::move(portal_session), std::move(portal)); + if (auto error = session->connect_to_pipewire(std::move(remote_fd))) { + return CaptureError{std::move(*error)}; + } + return std::unique_ptr(std::move(session)); } } // namespace sc diff --git a/src/codec/ffmpeg_encoder.cpp b/src/codec/ffmpeg_encoder.cpp index cdf4d26..38a5751 100644 --- a/src/codec/ffmpeg_encoder.cpp +++ b/src/codec/ffmpeg_encoder.cpp @@ -3,6 +3,7 @@ #include "ffmpeg_raii.h" +#include #include #include #include @@ -18,15 +19,24 @@ namespace { using namespace sc::detail; AVPixelFormat to_ffmpeg_format(PixelFormat format) noexcept { - return format == PixelFormat::Rgba ? AV_PIX_FMT_RGBA : AV_PIX_FMT_YUV420P; + switch (format) { + case PixelFormat::Rgba: + return AV_PIX_FMT_RGBA; + case PixelFormat::Bgrx: + // BGRx is BGR with an unused fourth byte; FFmpeg models that layout as BGRA. + return AV_PIX_FMT_BGRA; + case PixelFormat::Yuv420p: + return AV_PIX_FMT_YUV420P; + } + return AV_PIX_FMT_NONE; } -int expected_first_plane_stride(PixelFormat format, int width) noexcept { - return format == PixelFormat::Rgba ? width * 4 : width; +bool is_packed_rgb(PixelFormat format) noexcept { + return format == PixelFormat::Rgba || format == PixelFormat::Bgrx; } -std::size_t expected_buffer_size(PixelFormat format, int width, int height) { - return static_cast(av_image_get_buffer_size(to_ffmpeg_format(format), width, height, 1)); +int minimum_row_bytes(PixelFormat format, int width) noexcept { + return is_packed_rgb(format) ? width * 4 : width; } enum class ReceiveStatus { @@ -244,12 +254,29 @@ class FfmpegEncoder final : public Encoder { } const AVPixelFormat input_format = to_ffmpeg_format(frame.pixel_format); - const int expected_stride = expected_first_plane_stride(frame.pixel_format, frame.width); - if (frame.stride != 0 && frame.stride != expected_stride) { - return CodecError{"unsupported input stride"}; + if (input_format == AV_PIX_FMT_NONE) { + return CodecError{"unsupported pixel format"}; } - const std::size_t required_size = expected_buffer_size(frame.pixel_format, frame.width, frame.height); + const int row_bytes = minimum_row_bytes(frame.pixel_format, frame.width); + std::size_t stride = static_cast(row_bytes); + if (frame.stride != 0) { + if (frame.stride < row_bytes) { + return CodecError{"input stride is smaller than one pixel row"}; + } + if (!is_packed_rgb(frame.pixel_format) && frame.stride != row_bytes) { + return CodecError{"unsupported input stride for planar format"}; + } + stride = static_cast(frame.stride); + } + + std::size_t required_size = 0; + if (is_packed_rgb(frame.pixel_format)) { + required_size = stride * (static_cast(frame.height) - 1) + static_cast(row_bytes); + } else { + required_size = + static_cast(av_image_get_buffer_size(input_format, frame.width, frame.height, 1)); + } if (frame.pixels.size() < required_size) { return CodecError{"input pixel buffer is too small"}; } @@ -274,12 +301,27 @@ class FfmpegEncoder final : public Encoder { return CodecError{"failed to create swscale context"}; } - std::array src{nullptr, nullptr, nullptr, nullptr}; + std::array src{nullptr, nullptr, nullptr, nullptr}; std::array src_lines{0, 0, 0, 0}; - if (av_image_fill_arrays( - src.data(), src_lines.data(), as_u8(frame.pixels.data()), input_format, frame.width, frame.height, 1) < - 0) { - return CodecError{"failed to fill input pixel arrays"}; + if (is_packed_rgb(frame.pixel_format)) { + // Packed RGB is a single plane whose row pitch may be padded beyond + // the packed row size, so build the source arrays by hand. + src[0] = as_u8(frame.pixels.data()); + src_lines[0] = static_cast(stride); + } else { + std::array planar_src{nullptr, nullptr, nullptr, nullptr}; + std::array planar_lines{0, 0, 0, 0}; + if (av_image_fill_arrays(planar_src.data(), + planar_lines.data(), + as_u8(frame.pixels.data()), + input_format, + frame.width, + frame.height, + 1) < 0) { + return CodecError{"failed to fill input pixel arrays"}; + } + std::copy(planar_src.begin(), planar_src.end(), src.begin()); + std::copy(planar_lines.begin(), planar_lines.end(), src_lines.begin()); } if (sws_scale(scaler_.get(), src.data(), src_lines.data(), 0, frame.height, output->data, output->linesize) <= diff --git a/tools/capture_smoke.cpp b/tools/capture_smoke.cpp new file mode 100644 index 0000000..7ab8450 --- /dev/null +++ b/tools/capture_smoke.cpp @@ -0,0 +1,167 @@ +// Phase 3 smoke test: capture a few real desktop frames through the portal, +// encode them to H.264, and write an Annex-B elementary stream to disk. +// +// This tool is intentionally manual: it needs a running desktop session and +// the user must confirm the source picker dialog, so it is not registered +// with `meson test`. Validation steps are documented in docs/RUNBOOK.md. + +#include "screencast/capture/capture.h" +#include "screencast/codec/encoder.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +int positive_int_arg(std::string_view text) { + int value = 0; + const auto [ptr, ec] = std::from_chars(text.data(), text.data() + text.size(), value); + if (ec != std::errc{} || ptr != text.data() + text.size() || value <= 0) { + return -1; + } + return value; +} + +std::string_view pixel_format_name(sc::PixelFormat format) { + switch (format) { + case sc::PixelFormat::Rgba: + return "rgba"; + case sc::PixelFormat::Bgrx: + return "bgrx"; + case sc::PixelFormat::Yuv420p: + return "yuv420p"; + } + return "unknown"; +} + +bool write_bytes(std::ofstream& output, const std::vector& bytes) { + output.write(reinterpret_cast(bytes.data()), static_cast(bytes.size())); + return output.good(); +} + +} // namespace + +int main(int argc, char** argv) { + const int max_frames = argc > 1 ? positive_int_arg(argv[1]) : 10; + if (max_frames < 0) { + std::cerr << "usage: capture_smoke [frames] [output.h264]\n"; + return 1; + } + const std::string output_path = argc > 2 ? argv[2] : "screencast_smoke.h264"; + + std::cout << "creating capture session (choose a source in the portal dialog)...\n"; + auto capture_result = sc::CaptureFactory::create(sc::CaptureTargetWholeScreen{}); + if (sc::is_capture_error(capture_result)) { + std::cerr << std::format("capture failed: {}\n", sc::capture_error(capture_result).message); + return 1; + } + auto capture = std::move(sc::capture_value(capture_result)); + + std::ofstream output(output_path, std::ios::binary | std::ios::trunc); + if (!output.is_open()) { + std::cerr << std::format("failed to open {} for writing\n", output_path); + return 1; + } + + std::unique_ptr encoder; + std::uint64_t total_bytes = 0; + std::uint64_t encoded_frames = 0; + std::uint64_t keyframes = 0; + + for (int captured = 0; captured < max_frames; ++captured) { + auto frame = capture->next_frame(); + if (!frame.has_value()) { + std::cout << std::format("capture session ended after {} frames\n", captured); + break; + } + + if (encoder == nullptr) { + sc::EncoderConfig config; + config.width = frame->width; + config.height = frame->height; + config.frame_rate_num = 25; + config.frame_rate_den = 1; + config.bitrate_kbps = 8000; + + auto encoder_result = sc::EncoderFactory::create(config); + if (sc::is_codec_error(encoder_result)) { + std::cerr << std::format("encoder creation failed: {}\n", sc::codec_error(encoder_result).message); + return 1; + } + encoder = std::move(sc::codec_value(encoder_result)); + + // The encoder is configured with AV_CODEC_FLAG_GLOBAL_HEADER, so + // parameter sets live in extradata; prepend them so the output + // file is a self-contained Annex-B stream. + const auto extradata = encoder->get_extradata(); + if (!extradata.empty()) { + if (!write_bytes(output, extradata)) { + std::cerr << std::format("failed to write parameter sets to {}\n", output_path); + return 1; + } + total_bytes += extradata.size(); + } + + std::cout << std::format("capturing {}x{} ({}, stride {})\n", + frame->width, + frame->height, + pixel_format_name(frame->pixel_format), + frame->stride); + } + + auto encoded_result = encoder->encode(*frame); + if (sc::is_codec_error(encoded_result)) { + std::cerr << std::format("encode failed: {}\n", sc::codec_error(encoded_result).message); + return 1; + } + for (const auto& encoded : sc::codec_value(encoded_result)) { + if (!write_bytes(output, encoded.data)) { + std::cerr << std::format("failed to write encoded data to {}\n", output_path); + return 1; + } + total_bytes += encoded.data.size(); + ++encoded_frames; + keyframes += encoded.is_keyframe ? 1 : 0; + } + } + + if (encoder != nullptr) { + auto flushed = encoder->flush(); + if (sc::is_codec_error(flushed)) { + std::cerr << std::format("flush failed: {}\n", sc::codec_error(flushed).message); + return 1; + } + for (const auto& encoded : sc::codec_value(flushed)) { + if (!write_bytes(output, encoded.data)) { + std::cerr << std::format("failed to write flushed data to {}\n", output_path); + return 1; + } + total_bytes += encoded.data.size(); + ++encoded_frames; + } + } + + capture->stop(); + output.close(); + + if (total_bytes == 0 || encoded_frames == 0) { + std::cerr << "no frames were captured\n"; + return 1; + } + + std::cout << std::format("wrote {} bytes to {} ({} encoded frames, {} keyframes)\n", + total_bytes, + output_path, + encoded_frames, + keyframes); + std::cout << std::format("validate with: ffprobe -v error -show_entries stream=codec_name,width,height {}\n", + output_path); + return 0; +} diff --git a/tools/meson.build b/tools/meson.build new file mode 100644 index 0000000..236b2fe --- /dev/null +++ b/tools/meson.build @@ -0,0 +1,6 @@ +# Manual smoke tools. These are not registered with `meson test` because they +# need an interactive desktop session (portal consent dialog). + +executable('capture_smoke', + 'capture_smoke.cpp', + dependencies : [sc_capture_dep, sc_codec_dep]) \ No newline at end of file