diff --git a/.agents/MEMORY.md b/.agents/MEMORY.md index 98180b2..b7f384f 100644 --- a/.agents/MEMORY.md +++ b/.agents/MEMORY.md @@ -1,51 +1,38 @@ # Project Memory — screen_cast -Last updated: Phase 4 (RTP framing) complete and tested; current phase is -Phase 5. +Last updated: Phase 5 implemented and automated tests green; manual windowed +loopback validation pending. See bottom for the run commands. ## Project state -- Phase 4 done: `sc_network` library implements `RtpHeader`/`RtpPacket` - (RFC 3550 serialize/parse; tolerates CSRC lists, extension headers, and - padding on the receive side) plus `H264Packetizer`/`H264Depacketizer` - (RFC 6184 single-NAL + FU-A; STAP-A never emitted, unsupported types mark - the frame damaged on receive). -- Depacketized access units use **3-byte start codes**, and the splitter - keeps a zero byte preceding a start code with the previous NAL, so both - 3- and 4-byte-start-code streams round-trip byte-exactly (tested). -- Loss handling is drop-on-damage: a sequence gap or missing FU fragment - marks the frame damaged and it is dropped silently at its marker. Full - loss recovery / jitter handling is Phase 7. -- `test_rtp` covers header/packet round-trips, malformed rejections, NAL - splitting (3- and 4-byte codes), FU-A chunking with MTU bounds, full - packetize→depacketize round-trip, gap dropping, frame separation by - marker alone, sequence wrap, and random default SSRC/sequence. - Valgrind-clean; `meson test` 3/3. -- Phase 3 remains validated; earlier review fixes still in place. -- The first smoke run emitted `impl_ext_end_proxy called from wrong context` - warnings: `pw_context_connect_fd` and `pw_core_disconnect` ran outside the - thread-loop lock. Fixed by holding the lock across all pw setup/teardown - proxy operations. Re-running the smoke tool should now be warning-free - (capture worked both ways; the warnings only meant the first two - marshaled messages were rejected and retried from the right context). -- `src/capture/pipewire_capture.cpp` now implements the full backend: - libportal 0.10 handshake (`create_screencast_session` → `session_start` → - `open_pipewire_remote`), PipeWire 1.6 stream on the first portal node, - BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff. - Portal handshake blocks on the caller thread; frames arrive on the pw - thread. `stop()` is thread-safe; teardown follows the pw-required order. -- Encoder now accepts padded strides for packed RGB formats and the new - `PixelFormat::Bgrx` (mapped to `AV_PIX_FMT_BGRA`); planar Yuv420p still - requires a packed layout. This was the review note blocking Phase 3. -- `tools/capture_smoke` (manual, not in `meson test`) captures N frames → - encodes → writes Annex-B including prepended SPS/PPS extradata (verified: - libx264 GLOBAL_HEADER extradata is Annex-B). -- Earlier review fixes remain in place; valgrind on the codec round-trip - test is still clean after the encoder stride changes. -- Encoder PTS caveat: the encoder time_base is derived from the configured - frame rate (default 25fps), but portal frames arrive at monitor refresh - (often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless - for the smoke test; revisit when RTP timestamps matter (Phase 4/5). +- **Phase 5 (local UDP sender→receiver loopback) is implemented**: + - `UdpRtpTransport` (src/network/udp_transport.cpp): raw POSIX sockets, + AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop() + closes the socket to unblock the receive jthread. ASIO was deliberately + deferred to Phase 6 (see decisions). + - `screencast` binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring + SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline + (recv→depacketize→decode→bounded 3-frame queue→render thread). + - `SdlRenderer` (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture, + RGBA texture upload, texture recreated on resolution change. + `RendererFactory::create` now returns `RendererResult` (error channel + added, mirroring codec/capture patterns). + - Encoder change: **GLOBAL_HEADER removed** so libx264 repeats SPS/PPS + in-band at every keyframe; a receiver now decodes from the bitstream + alone (mid-stream join, PLI recovery-ready). `get_extradata()` is empty + in this mode; codec round-trip test updated to match the streaming path. +- **Automated validation**: `meson test` 4/4 — new `udp loopback` test + encodes synthetic frames, packetizes, sends over a real localhost UDP + socket, depacketizes, and decodes 10/10 frames with correct dimensions. + Valgrind clean (loopback + codec tests). +- **Manual validation pending (needs the desktop)**: run the two commands in + `docs/RUNBOOK.md` — receiver window should show the captured desktop. Tick + `docs/PHASES.md` Phase 5 after this works. +- Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A; + 3-byte canonical start codes; drop-on-damage loss handling). +- Phase 3 capture done and validated (PipeWire/portal backend; the + `impl_ext_end_proxy` wrong-context warnings were fixed by holding the + thread-loop lock across all pw proxy operations). ## Decisions @@ -55,13 +42,19 @@ Phase 5. - Capture: PipeWire + xdg-desktop-portal. - Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale). - H.264 encoder path: software `libx264`, low-latency settings, Annex-B output. -- Decoder receives SPS/PPS through `DecoderConfig.extradata`. -- Transport: RTP over UDP; signaling via WebSocket/JSON. +- **SPS/PPS are sent in-band ahead of every keyframe** (no GLOBAL_HEADER); + the decoder starts from the bitstream alone. `DecoderConfig.extradata` + remains available if signaling ever negotiates parameters out of band. +- Transport: RTP over UDP via raw POSIX sockets for now; **ASIO stays a + Phase 6+ option** (the ARCHITECTURE.md dependency table places it with + signaling/discovery). IPv4 only at the transport level for now. +- Rendering: **SDL3** (`sdl3` pkg-config, 3.4 installed); plain texture + upload, no GPU pipeline yet. - Discovery: mDNS/Avahi. -- Rendering: SDL2 or SDL3 + OpenGL. - Namespace: `sc`. - Module error results use per-module `std::variant` types - (`CodecResult`, `CaptureResult`) since C++20 has no `std::expected`. + (`CodecResult`, `CaptureResult`, `RendererResult`) since C++20 has no + `std::expected`. ## Active blockers @@ -72,20 +65,26 @@ None. - GUI framework (Qt6 vs. none / CLI only) — deferred to later phase. - Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after software encode path works. +- IPv6 at the transport layer — revisit when LAN streaming lands (Phase 6+). ## Forward-looking review notes (for later phases) -- `AV_CODEC_FLAG_GLOBAL_HEADER` suppresses in-band SPS/PPS; a receiver cannot - join mid-stream or recover after PLI without parameter sets. Phase 5 must - prepend SPS/PPS to keyframes or negotiate them in signaling. - Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother UDP streaming in Phase 7. +- Encoder PTS caveat: time_base derives from the configured frame rate + (default 25fps) while portal frames arrive at monitor refresh (often 60Hz), + so pts values quantize and can repeat. RTP timestamps come from the capture + clock instead, so streaming is unaffected; revisit if decoder-side + presentation timing ever matters. +- `RtpTransport::start/send` return plain bools (scaffold API); error + messages are lost — consider an error channel when signaling lands. - DMA-BUF-only portal streams are rejected with a clear message (hardware path is Phase 7). - No negative-path tests yet (bad config, bad stride, undersized buffer). - `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting encoder is ever added, prefer an explicit config flag over the heuristic. - Region targets are rejected: the desktop portal has no region capture. -- `next_frame()` returns `nullopt` on stream error without surfacing the - reason (logged to stderr); consider an error channel when the receiver - pipeline lands. \ No newline at end of file +- `CaptureSession::next_frame()` returns `nullopt` on stream error without + surfacing the reason (logged to stderr). +- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders + we control never emit them, but third-party interop would need support. \ No newline at end of file diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 2f74288..1b1853f 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -49,6 +49,23 @@ PipeWire proxy operation ran without the thread-loop lock — all pw calls that send messages (connect, stream, core) must happen under `pw_thread_loop_lock`. +## Sender / receiver loopback (Phase 5, manual) + +Two terminals on the same desktop session: + +```sh +./build/src/app/screencast --receive # terminal 1: window appears +./build/src/app/screencast --send # terminal 2: portal source picker +``` + +Expected: the receiver window shows the captured desktop in near real time. +Stop either side with Ctrl-C. Optional flags: `--port`, `--peer HOST[:PORT]`, +`--bitrate KBPS`, `--target window`. + +The headless equivalent runs as part of `meson test` (`udp loopback` test): +synthetic frames → encode → packetize → localhost UDP → depacketize → +decode, no portal or window involved. + ## Formatting ```sh diff --git a/include/screencast/app/cli.h b/include/screencast/app/cli.h index 128a156..35bca1b 100644 --- a/include/screencast/app/cli.h +++ b/include/screencast/app/cli.h @@ -20,6 +20,8 @@ struct ReceiveCommand { using Command = std::variant; // Parse command line arguments. Prints usage and returns std::nullopt on error. -std::optional parse_cli(int argc, const char* argv[]); +// `argv` is `char const* const*` so both `main`'s `char**` and const arrays +// convert implicitly. +std::optional parse_cli(int argc, const char* const argv[]); } // namespace sc diff --git a/include/screencast/app/pipeline.h b/include/screencast/app/pipeline.h index e2cc19b..7b44d53 100644 --- a/include/screencast/app/pipeline.h +++ b/include/screencast/app/pipeline.h @@ -16,6 +16,9 @@ struct SenderPipelineConfig { EncoderConfig encoder; Endpoint local_rtp_endpoint; std::optional signaling_server; + // Where encoded RTP packets are sent. Defaults to the local loopback so + // a sender and receiver on one machine work without any configuration. + Endpoint peer_rtp_endpoint{"127.0.0.1", 5004}; }; struct ReceiverPipelineConfig { diff --git a/include/screencast/render/error.h b/include/screencast/render/error.h new file mode 100644 index 0000000..871c0c0 --- /dev/null +++ b/include/screencast/render/error.h @@ -0,0 +1,36 @@ +#pragma once + +#include +#include + +namespace sc { + +struct RendererError { + std::string message; +}; + +// C++20 does not provide std::expected. Use a variant-based result type so +// fallible renderer operations do not rely on exceptions. +template using RendererResult = std::variant; + +template constexpr bool is_renderer_error(const RendererResult& result) noexcept { + return std::holds_alternative(result); +} + +template T& renderer_value(RendererResult& result) { + return std::get(result); +} + +template const T& renderer_value(const RendererResult& result) { + return std::get(result); +} + +template RendererError& renderer_error(RendererResult& result) { + return std::get(result); +} + +template const RendererError& renderer_error(const RendererResult& result) { + return std::get(result); +} + +} // namespace sc diff --git a/include/screencast/render/renderer.h b/include/screencast/render/renderer.h index 61bad66..e78607b 100644 --- a/include/screencast/render/renderer.h +++ b/include/screencast/render/renderer.h @@ -1,6 +1,7 @@ #pragma once #include "screencast/codec/decoder.h" +#include "screencast/render/error.h" #include #include @@ -29,7 +30,7 @@ class Renderer { class RendererFactory { public: - static std::unique_ptr create(const RendererConfig& config); + static RendererResult> create(const RendererConfig& config); }; } // namespace sc diff --git a/src/app/cli.cpp b/src/app/cli.cpp new file mode 100644 index 0000000..4313d89 --- /dev/null +++ b/src/app/cli.cpp @@ -0,0 +1,109 @@ +#include "screencast/app/cli.h" + +#include +#include +#include + +namespace sc { +namespace { + +void print_usage() { + std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate KBPS]\n" + " screencast --receive [--port PORT]\n", + stderr); +} + +bool parse_int(std::string_view text, int& value) { + const auto [pointer, error] = std::from_chars(text.data(), text.data() + text.size(), value); + return error == std::errc{} && pointer == text.data() + text.size(); +} + +bool next_argument(int argc, const char* const argv[], int& index, std::string_view& value) { + if (index + 1 >= argc) { + print_usage(); + return false; + } + value = argv[++index]; + return true; +} + +} // namespace + +std::optional parse_cli(int argc, const char* const argv[]) { + enum class Mode { + None, + Send, + Receive, + }; + + Mode mode = Mode::None; + SendCommand send; + ReceiveCommand receive; + + for (int index = 1; index < argc; ++index) { + const std::string_view argument = argv[index]; + + if (argument == "--send") { + if (mode != Mode::None) { + print_usage(); + return std::nullopt; + } + mode = Mode::Send; + } else if (argument == "--receive") { + if (mode != Mode::None) { + print_usage(); + return std::nullopt; + } + mode = Mode::Receive; + } else if (argument == "--target") { + std::string_view value; + if (!next_argument(argc, argv, index, value)) { + return std::nullopt; + } + if (value != "monitor" && value != "window") { + print_usage(); + return std::nullopt; + } + send.target = value; + } else if (argument == "--peer") { + std::string_view value; + if (!next_argument(argc, argv, index, value)) { + return std::nullopt; + } + if (mode == Mode::Receive) { + receive.peer_address = value; + } else { + send.peer_address = value; + } + } else if (argument == "--bitrate") { + std::string_view value; + if (!next_argument(argc, argv, index, value) || !parse_int(value, send.bitrate_kbps) || + send.bitrate_kbps <= 0) { + print_usage(); + return std::nullopt; + } + } else if (argument == "--port") { + std::string_view value; + int port = 0; + if (!next_argument(argc, argv, index, value) || !parse_int(value, port) || port <= 0 || port > 65535) { + print_usage(); + return std::nullopt; + } + receive.local_rtp_port = port; + } else { + print_usage(); + return std::nullopt; + } + } + + if (mode == Mode::Send) { + return Command{std::move(send)}; + } + if (mode == Mode::Receive) { + return Command{std::move(receive)}; + } + print_usage(); + return std::nullopt; +} + +} // namespace sc diff --git a/src/app/main.cpp b/src/app/main.cpp new file mode 100644 index 0000000..6d0ea83 --- /dev/null +++ b/src/app/main.cpp @@ -0,0 +1,87 @@ +#include "screencast/app/cli.h" +#include "screencast/app/pipeline.h" + +#include +#include +#include +#include +#include +#include + +namespace { + +std::atomic g_interrupted{false}; + +void handle_interrupt(int) { + g_interrupted.store(true); +} + +sc::Endpoint parse_endpoint(std::string_view address, uint16_t default_port) { + sc::Endpoint endpoint; + const std::size_t separator = address.rfind(':'); + if (separator != std::string_view::npos) { + endpoint.address = std::string{address.substr(0, separator)}; + int port = 0; + const auto [pointer, error] = + std::from_chars(address.data() + separator + 1, address.data() + address.size(), port); + endpoint.port = error == std::errc{} ? static_cast(port) : default_port; + } else { + endpoint.address = std::string{address}; + endpoint.port = default_port; + } + return endpoint; +} + +int run_sender(const sc::SendCommand& command) { + sc::SenderPipelineConfig config; + if (command.target == "window") { + config.capture_target = sc::CaptureTargetWindow{}; + } else { + config.capture_target = sc::CaptureTargetWholeScreen{}; + } + config.peer_rtp_endpoint = + command.peer_address.empty() ? sc::Endpoint{"127.0.0.1", 5004} : parse_endpoint(command.peer_address, 5004); + config.encoder.bitrate_kbps = command.bitrate_kbps; + + sc::SenderPipeline pipeline{std::move(config)}; + if (!pipeline.start()) { + return 1; + } + while (!g_interrupted.load()) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + pipeline.stop(); + return 0; +} + +int run_receiver(const sc::ReceiveCommand& command) { + sc::ReceiverPipelineConfig config; + config.local_rtp_endpoint = sc::Endpoint{"0.0.0.0", static_cast(command.local_rtp_port)}; + + sc::ReceiverPipeline pipeline{std::move(config)}; + if (!pipeline.start()) { + return 1; + } + while (!g_interrupted.load()) { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + pipeline.stop(); + return 0; +} + +} // namespace + +int main(int argc, char* argv[]) { + const std::optional command = sc::parse_cli(argc, argv); + if (!command.has_value()) { + return 1; + } + + std::signal(SIGINT, handle_interrupt); + std::signal(SIGTERM, handle_interrupt); + + if (const sc::SendCommand* send = std::get_if(&*command)) { + return run_sender(*send); + } + return run_receiver(std::get(*command)); +} diff --git a/src/app/meson.build b/src/app/meson.build new file mode 100644 index 0000000..029016e --- /dev/null +++ b/src/app/meson.build @@ -0,0 +1,9 @@ +# The screencast application binary wiring every module together. + +executable('screencast', + [ + 'cli.cpp', + 'main.cpp', + 'pipelines.cpp', + ], + dependencies : [sc_capture_dep, sc_codec_dep, sc_network_dep, sc_render_dep]) \ No newline at end of file diff --git a/src/app/pipelines.cpp b/src/app/pipelines.cpp new file mode 100644 index 0000000..8c92bec --- /dev/null +++ b/src/app/pipelines.cpp @@ -0,0 +1,224 @@ +#include "screencast/app/pipeline.h" + +#include "screencast/network/h264_packetizer.h" + +#include +#include +#include +#include +#include +#include + +namespace sc { + +class SenderPipeline::Impl { + public: + explicit Impl(SenderPipelineConfig config) : config_(std::move(config)) {} + + ~Impl() { + stop(); + } + + bool start() { + auto capture_result = CaptureFactory::create(config_.capture_target); + if (is_capture_error(capture_result)) { + std::cerr << std::format("screencast: capture failed: {}\n", capture_error(capture_result).message); + return false; + } + capture_ = std::move(capture_value(capture_result)); + + if (!transport_->start(config_.local_rtp_endpoint, [](RtpPacket) {})) { + std::cerr << "screencast: failed to start the RTP transport\n"; + return false; + } + transport_->set_peer(config_.peer_rtp_endpoint); + + run_thread_ = std::jthread([this](std::stop_token stop_token) { run(std::move(stop_token)); }); + return true; + } + + void stop() { + if (capture_ != nullptr) { + capture_->stop(); + } + run_thread_ = std::jthread{}; + transport_->stop(); + } + + private: + void run(std::stop_token stop_token) { + while (!stop_token.stop_requested()) { + const std::optional frame = capture_->next_frame(); + if (!frame.has_value()) { + break; + } + + if (encoder_ == nullptr && !create_encoder(*frame)) { + break; + } + + auto encoded_result = encoder_->encode(*frame); + if (is_codec_error(encoded_result)) { + std::cerr << std::format("screencast: encode failed: {}\n", codec_error(encoded_result).message); + break; + } + for (const EncodedFrame& encoded : codec_value(encoded_result)) { + for (RtpPacket& packet : packetizer_.packetize(encoded)) { + (void)transport_->send(packet); + } + } + } + } + + bool create_encoder(const CapturedFrame& frame) { + EncoderConfig config = config_.encoder; + config.width = frame.width; + config.height = frame.height; + + auto encoder_result = EncoderFactory::create(config); + if (is_codec_error(encoder_result)) { + std::cerr << std::format("screencast: encoder creation failed: {}\n", codec_error(encoder_result).message); + return false; + } + encoder_ = std::move(codec_value(encoder_result)); + return true; + } + + SenderPipelineConfig config_; + std::unique_ptr capture_; + std::unique_ptr encoder_; + H264Packetizer packetizer_; + std::unique_ptr transport_ = RtpTransportFactory::create(); + std::jthread run_thread_; +}; + +SenderPipeline::SenderPipeline(SenderPipelineConfig config) : impl_(std::make_unique(std::move(config))) {} + +SenderPipeline::~SenderPipeline() = default; + +bool SenderPipeline::start() { + return impl_->start(); +} + +void SenderPipeline::stop() { + impl_->stop(); +} + +class ReceiverPipeline::Impl { + public: + explicit Impl(ReceiverPipelineConfig config) : config_(std::move(config)) {} + + ~Impl() { + stop(); + } + + bool start() { + auto renderer_result = RendererFactory::create(config_.renderer); + if (is_renderer_error(renderer_result)) { + std::cerr << std::format("screencast: renderer creation failed: {}\n", + renderer_error(renderer_result).message); + return false; + } + renderer_ = std::move(renderer_value(renderer_result)); + + auto decoder_result = DecoderFactory::create(config_.decoder); + if (is_codec_error(decoder_result)) { + std::cerr << std::format("screencast: decoder creation failed: {}\n", codec_error(decoder_result).message); + return false; + } + decoder_ = std::move(codec_value(decoder_result)); + + if (!transport_->start(config_.local_rtp_endpoint, + [this](RtpPacket packet) { on_packet(std::move(packet)); })) { + std::cerr << "screencast: failed to start the RTP transport\n"; + return false; + } + + render_thread_ = std::jthread([this](std::stop_token stop_token) { render_loop(std::move(stop_token)); }); + return true; + } + + void stop() { + render_thread_ = std::jthread{}; + transport_->stop(); + renderer_ = nullptr; + decoder_ = nullptr; + { + std::lock_guard lock(queue_mutex_); + queue_.clear(); + } + } + + private: + void on_packet(RtpPacket packet) { + std::optional> access_unit = depacketizer_.depacketize(packet); + if (!access_unit.has_value()) { + return; + } + + EncodedFrame encoded; + encoded.data = std::move(*access_unit); + encoded.rtp_timestamp = packet.header.timestamp; + encoded.is_keyframe = false; + + auto decoded_result = decoder_->decode(encoded); + if (is_codec_error(decoded_result)) { + std::cerr << std::format("screencast: decode failed: {}\n", codec_error(decoded_result).message); + return; + } + for (DecodedFrame& decoded : codec_value(decoded_result)) { + std::lock_guard lock(queue_mutex_); + // Keep latency low: drop the oldest frame when the queue is full. + if (queue_.size() >= kMaxQueuedFrames) { + queue_.pop_front(); + } + queue_.push_back(std::move(decoded)); + } + } + + void render_loop(std::stop_token stop_token) { + while (!stop_token.stop_requested()) { + if (!renderer_->poll_events()) { + break; // window closed + } + std::optional frame; + { + std::unique_lock lock(queue_mutex_); + if (!queue_.empty()) { + frame = std::move(queue_.front()); + queue_.pop_front(); + } + } + if (frame.has_value()) { + (void)renderer_->present(*frame); + } else { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + } + } + + static constexpr std::size_t kMaxQueuedFrames = 3; + + ReceiverPipelineConfig config_; + std::unique_ptr renderer_; + std::unique_ptr decoder_; + H264Depacketizer depacketizer_; + std::unique_ptr transport_ = RtpTransportFactory::create(); + std::jthread render_thread_; + std::mutex queue_mutex_; + std::deque queue_; +}; + +ReceiverPipeline::ReceiverPipeline(ReceiverPipelineConfig config) : impl_(std::make_unique(std::move(config))) {} + +ReceiverPipeline::~ReceiverPipeline() = default; + +bool ReceiverPipeline::start() { + return impl_->start(); +} + +void ReceiverPipeline::stop() { + impl_->stop(); +} + +} // namespace sc diff --git a/src/codec/ffmpeg_encoder.cpp b/src/codec/ffmpeg_encoder.cpp index 38a5751..92f9e5d 100644 --- a/src/codec/ffmpeg_encoder.cpp +++ b/src/codec/ffmpeg_encoder.cpp @@ -397,7 +397,10 @@ CodecResult> EncoderFactory::create(const EncoderConfig ctx->max_b_frames = 0; ctx->thread_count = 1; ctx->profile = AV_PROFILE_H264_MAIN; - ctx->flags |= AV_CODEC_FLAG_LOW_DELAY | AV_CODEC_FLAG_GLOBAL_HEADER; + // No GLOBAL_HEADER: SPS/PPS are repeated in-band at every keyframe so a + // receiver that joins mid-stream (or recovers after loss) can decode + // without out-of-band parameter negotiation. + ctx->flags |= AV_CODEC_FLAG_LOW_DELAY; if (av_opt_set(ctx->priv_data, "preset", "ultrafast", 0) < 0) { return CodecError{"failed to set libx264 preset"}; diff --git a/src/meson.build b/src/meson.build index 7e510b8..50cf8a1 100644 --- a/src/meson.build +++ b/src/meson.build @@ -24,3 +24,6 @@ sc_codec_dep = declare_dependency( dependencies : [dep_avcodec, dep_avutil, dep_swscale]) subdir('network') +subdir('render') + +subdir('app') diff --git a/src/network/meson.build b/src/network/meson.build index 8633e5e..a5d3290 100644 --- a/src/network/meson.build +++ b/src/network/meson.build @@ -3,6 +3,7 @@ sc_network_sources = files( 'rtp_packet.cpp', 'h264_packetizer.cpp', + 'udp_transport.cpp', ) sc_network = static_library('sc_network', diff --git a/src/network/udp_transport.cpp b/src/network/udp_transport.cpp new file mode 100644 index 0000000..a1903f1 --- /dev/null +++ b/src/network/udp_transport.cpp @@ -0,0 +1,163 @@ +#include "screencast/network/transport.h" + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +namespace sc { +namespace { + +constexpr int kInvalidSocket = -1; + +// A UDP datagram cannot exceed 64 KiB on IPv4; one buffer fits any RTP +// packet the receiver will ever see. +constexpr std::size_t kReceiveBufferSize = 65536; + +std::optional resolve_ipv4(const Endpoint& endpoint) { + addrinfo hints{}; + hints.ai_family = AF_INET; + hints.ai_socktype = SOCK_DGRAM; + + addrinfo* result = nullptr; + const std::string port = std::to_string(endpoint.port); + const char* node = endpoint.address.empty() ? nullptr : endpoint.address.c_str(); + if (getaddrinfo(node, port.c_str(), &hints, &result) != 0) { + return std::nullopt; + } + + std::optional address; + if (result != nullptr && result->ai_family == AF_INET && result->ai_addrlen >= sizeof(sockaddr_in)) { + sockaddr_in resolved{}; + std::memcpy(&resolved, result->ai_addr, sizeof(sockaddr_in)); + address = resolved; + } + freeaddrinfo(result); + return address; +} + +} // namespace + +class UdpRtpTransport final : public RtpTransport { + public: + UdpRtpTransport() = default; + + ~UdpRtpTransport() override { + stop(); + } + + UdpRtpTransport(const UdpRtpTransport&) = delete; + UdpRtpTransport& operator=(const UdpRtpTransport&) = delete; + + bool start(const Endpoint& local_endpoint, ReceiveCallback on_receive) override { + if (running_.load()) { + return false; + } + socket_ = ::socket(AF_INET, SOCK_DGRAM, 0); + if (socket_ < 0) { + return false; + } + int reuse = 1; + (void)::setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse)); + + // A zero port skips binding: the OS picks the source port on send. + if (local_endpoint.port != 0) { + const std::optional address = resolve_ipv4(local_endpoint); + if (!address.has_value()) { + (void)::close(socket_); + socket_ = kInvalidSocket; + return false; + } + if (::bind(socket_, reinterpret_cast(&*address), sizeof(*address)) < 0) { + (void)::close(socket_); + socket_ = kInvalidSocket; + return false; + } + } + + running_.store(true); + receive_thread_ = std::jthread([this, callback = std::move(on_receive)]() mutable { receive_loop(callback); }); + return true; + } + + bool send(const RtpPacket& packet) override { + if (!running_.load() || !has_peer_.load()) { + return false; + } + const std::vector bytes = packet.serialize(); + if (bytes.empty()) { + return false; + } + sockaddr_in peer{}; + { + // Snapshot the peer so set_peer() can be called concurrently. + std::lock_guard lock(peer_mutex_); + peer = peer_; + } + const ssize_t sent = + ::sendto(socket_, bytes.data(), bytes.size(), 0, reinterpret_cast(&peer), sizeof(peer)); + return sent == static_cast(bytes.size()); + } + + void set_peer(const Endpoint& peer) override { + const std::optional address = resolve_ipv4(peer); + if (!address.has_value()) { + return; + } + std::lock_guard lock(peer_mutex_); + peer_ = *address; + has_peer_.store(true); + } + + void stop() override { + if (!running_.exchange(false)) { + return; + } + if (socket_ >= 0) { + (void)::shutdown(socket_, SHUT_RDWR); + (void)::close(socket_); + socket_ = kInvalidSocket; + } + // Closing the socket unblocks recvfrom; joining happens implicitly + // when the jthread assignment destroys the previous thread. + receive_thread_ = std::jthread{}; + } + + private: + void receive_loop(ReceiveCallback& callback) { + std::array buffer{}; + while (running_.load()) { + const ssize_t received = ::recvfrom(socket_, buffer.data(), buffer.size(), 0, nullptr, nullptr); + if (received <= 0) { + continue; // closed socket while running_ still true, or error + } + const std::optional packet = + RtpPacket::parse(std::span{buffer.data(), static_cast(received)}); + if (packet.has_value()) { + callback(*packet); + } + } + } + + int socket_ = kInvalidSocket; + std::atomic running_{false}; + std::atomic has_peer_{false}; + std::mutex peer_mutex_; + sockaddr_in peer_{}; + std::jthread receive_thread_; +}; + +std::unique_ptr RtpTransportFactory::create() { + return std::make_unique(); +} + +} // namespace sc diff --git a/src/render/meson.build b/src/render/meson.build new file mode 100644 index 0000000..2ad6a1b --- /dev/null +++ b/src/render/meson.build @@ -0,0 +1,15 @@ +# Phase 5 renderer: SDL3 software texture upload, no GPU pipeline yet. + +dep_sdl3 = dependency('sdl3') + +sc_render_sources = files('sdl_renderer.cpp') + +sc_render = static_library('sc_render', + sc_render_sources, + include_directories : sc_core_inc, + dependencies : [dep_sdl3]) + +sc_render_dep = declare_dependency( + link_with : sc_render, + include_directories : sc_core_inc, + dependencies : [dep_sdl3]) \ No newline at end of file diff --git a/src/render/sdl_renderer.cpp b/src/render/sdl_renderer.cpp new file mode 100644 index 0000000..68f8fcc --- /dev/null +++ b/src/render/sdl_renderer.cpp @@ -0,0 +1,147 @@ +#include "screencast/render/renderer.h" + +#include + +#include + +namespace sc { +namespace { + +// DecodedFrame pixels are AV_PIX_FMT_RGBA: memory order R, G, B, A, which +// matches SDL_PIXELFORMAT_RGBA8888. +constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_RGBA8888; + +class SdlRenderer final : public Renderer { + public: + explicit SdlRenderer(const RendererConfig& config) : config_(config) {} + + ~SdlRenderer() override { + shutdown(); + } + + SdlRenderer(const SdlRenderer&) = delete; + SdlRenderer& operator=(const SdlRenderer&) = delete; + + const std::string& last_error() const { + return last_error_; + } + + bool initialize() { + if (!SDL_Init(SDL_INIT_VIDEO)) { + last_error_ = std::string{"SDL_Init failed: "} + SDL_GetError(); + return false; + } + sdl_inited_ = true; + + window_ = SDL_CreateWindow(config_.window_title.c_str(), config_.initial_width, config_.initial_height, 0); + if (window_ == nullptr) { + last_error_ = std::string{"SDL_CreateWindow failed: "} + SDL_GetError(); + shutdown(); + return false; + } + + renderer_ = SDL_CreateRenderer(window_, nullptr); + if (renderer_ == nullptr) { + last_error_ = std::string{"SDL_CreateRenderer failed: "} + SDL_GetError(); + shutdown(); + return false; + } + + return true; + } + + bool present(const DecodedFrame& frame) override { + if (frame.width <= 0 || frame.height <= 0) { + return false; + } + if (frame.width != texture_width_ || frame.height != texture_height_) { + if (!recreate_texture(frame.width, frame.height)) { + return false; + } + } + + const int pitch = frame.width * 4; + if (!SDL_UpdateTexture(texture_, nullptr, frame.rgba_pixels.data(), pitch)) { + return false; + } + if (!SDL_RenderTexture(renderer_, texture_, nullptr, nullptr)) { + return false; + } + SDL_RenderPresent(renderer_); + return true; + } + + bool poll_events() override { + if (closed_) { + return false; + } + SDL_Event event{}; + while (SDL_PollEvent(&event)) { + if (event.type == SDL_EVENT_QUIT || + (event.type == SDL_EVENT_WINDOW_CLOSE_REQUESTED && event.window.windowID == SDL_GetWindowID(window_))) { + closed_ = true; + return false; + } + } + return true; + } + + void shutdown() override { + if (texture_ != nullptr) { + SDL_DestroyTexture(texture_); + texture_ = nullptr; + } + if (renderer_ != nullptr) { + SDL_DestroyRenderer(renderer_); + renderer_ = nullptr; + } + if (window_ != nullptr) { + SDL_DestroyWindow(window_); + window_ = nullptr; + } + if (sdl_inited_) { + SDL_Quit(); + sdl_inited_ = false; + } + } + + private: + bool recreate_texture(int width, int height) { + if (texture_ != nullptr) { + SDL_DestroyTexture(texture_); + texture_ = nullptr; + } + texture_ = SDL_CreateTexture(renderer_, kSdlPixelFormat, SDL_TEXTUREACCESS_STREAMING, width, height); + if (texture_ == nullptr) { + last_error_ = std::string{"SDL_CreateTexture failed: "} + SDL_GetError(); + texture_width_ = 0; + texture_height_ = 0; + return false; + } + texture_width_ = width; + texture_height_ = height; + return true; + } + + RendererConfig config_; + std::string last_error_; + bool sdl_inited_ = false; + bool closed_ = false; + SDL_Window* window_ = nullptr; + SDL_Renderer* renderer_ = nullptr; + SDL_Texture* texture_ = nullptr; + int texture_width_ = 0; + int texture_height_ = 0; +}; + +} // namespace + +RendererResult> RendererFactory::create(const RendererConfig& config) { + auto renderer = std::make_unique(config); + if (!renderer->initialize()) { + return RendererError{renderer->last_error()}; + } + return std::unique_ptr(std::move(renderer)); +} + +} // namespace sc diff --git a/tests/app/test_loopback.cpp b/tests/app/test_loopback.cpp new file mode 100644 index 0000000..d83f890 --- /dev/null +++ b/tests/app/test_loopback.cpp @@ -0,0 +1,170 @@ +// Phase 5 integration test: the local UDP sender -> receiver loopback without +// the interactive parts. Synthetic frames go through the real chain — +// encode, RTP packetization, a localhost UDP socket, depacketization, and +// decode (with in-band SPS/PPS, since there is no signaling channel). + +#include "screencast/codec/decoder.h" +#include "screencast/codec/encoder.h" +#include "screencast/network/h264_packetizer.h" +#include "screencast/network/transport.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +[[noreturn]] void fail(const char* what) { + std::fprintf(stderr, "test_loopback: FAIL: %s\n", what); + std::abort(); +} + +void check(bool condition, const char* what) { + if (!condition) { + fail(what); + } +} + +template void expect_result(const char* what, const sc::CodecResult& result) { + if (sc::is_codec_error(result)) { + std::fprintf(stderr, "test_loopback: FAIL: %s: %s\n", what, sc::codec_error(result).message.c_str()); + std::abort(); + } +} + +constexpr int kWidth = 64; +constexpr int kHeight = 64; +constexpr int kFrameCount = 10; +constexpr uint64_t kNsPerFrame = 40'000'000; // 25 fps + +sc::CapturedFrame make_frame(uint32_t index) { + sc::CapturedFrame frame; + frame.width = kWidth; + frame.height = kHeight; + frame.timestamp_ns = static_cast(index) * kNsPerFrame; + frame.pixel_format = sc::PixelFormat::Rgba; + frame.stride = kWidth * 4; + frame.pixels.resize(static_cast(kWidth) * kHeight * 4); + + auto* pixels = reinterpret_cast(frame.pixels.data()); + for (int y = 0; y < kHeight; ++y) { + for (int x = 0; x < kWidth; ++x) { + const std::size_t offset = (static_cast(y) * kWidth + x) * 4; + pixels[offset + 0] = static_cast((x + static_cast(index) * 4) & 0xFF); + pixels[offset + 1] = static_cast((y + static_cast(index) * 2) & 0xFF); + pixels[offset + 2] = static_cast(((x ^ y) + static_cast(index)) & 0xFF); + pixels[offset + 3] = 0xFF; + } + } + return frame; +} + +// Receiver state driven entirely by the transport's receive thread. +struct ReceiverSink { + sc::H264Depacketizer depacketizer; + std::unique_ptr decoder; + std::atomic decoded_frames{0}; + bool saw_frame = false; + int last_width = 0; + int last_height = 0; + + void on_packet(sc::RtpPacket packet) { + std::optional> access_unit = depacketizer.depacketize(packet); + if (!access_unit.has_value()) { + return; + } + + sc::EncodedFrame encoded; + encoded.data = std::move(*access_unit); + encoded.rtp_timestamp = packet.header.timestamp; + + auto decoded_result = decoder->decode(encoded); + if (sc::is_codec_error(decoded_result)) { + return; + } + for (const sc::DecodedFrame& decoded : sc::codec_value(decoded_result)) { + last_width = decoded.width; + last_height = decoded.height; + saw_frame = decoded.width > 0 && !decoded.rgba_pixels.empty(); + decoded_frames.fetch_add(1); + } + } +}; + +} // namespace + +int main() { + // Encoder on the sender side. + sc::EncoderConfig encoder_config; + encoder_config.codec_name = "h264"; + encoder_config.width = kWidth; + encoder_config.height = kHeight; + encoder_config.frame_rate_num = 25; + encoder_config.frame_rate_den = 1; + encoder_config.bitrate_kbps = 2000; + + auto encoder_result = sc::EncoderFactory::create(encoder_config); + expect_result("encoder create", encoder_result); + auto encoder = std::move(sc::codec_value(encoder_result)); + + // The receiver decodes purely from the bitstream (in-band SPS/PPS). + sc::DecoderConfig decoder_config; + decoder_config.codec_name = "h264"; + auto decoder_result = sc::DecoderFactory::create(decoder_config); + expect_result("decoder create", decoder_result); + + ReceiverSink sink; + sink.decoder = std::move(sc::codec_value(decoder_result)); + + // Bind the receiver to the first free port in a small range. + auto receiver_transport = sc::RtpTransportFactory::create(); + uint16_t bound_port = 0; + for (uint16_t port = 45904; port < 45934; ++port) { + if (receiver_transport->start(sc::Endpoint{"0.0.0.0", port}, + [&sink](sc::RtpPacket packet) { sink.on_packet(std::move(packet)); })) { + bound_port = port; + break; + } + } + check(bound_port != 0, "receiver transport bound"); + + // Sender transport: no bind (OS picks the source port), peer is loopback. + auto sender_transport = sc::RtpTransportFactory::create(); + check(sender_transport->start(sc::Endpoint{"", 0}, [](sc::RtpPacket) {}), "sender transport started"); + sender_transport->set_peer(sc::Endpoint{"127.0.0.1", bound_port}); + + sc::H264Packetizer packetizer; + for (uint32_t index = 0; index < kFrameCount; ++index) { + const sc::CapturedFrame frame = make_frame(index); + + auto encoded_result = encoder->encode(frame); + expect_result("encode", encoded_result); + + for (const sc::EncodedFrame& encoded : sc::codec_value(encoded_result)) { + for (const sc::RtpPacket& packet : packetizer.packetize(encoded)) { + check(sender_transport->send(packet), "send packet"); + } + } + } + + // Wait (bounded) for the receiver to decode the frames. + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (sink.decoded_frames.load() < kFrameCount && std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + + check(sink.decoded_frames.load() >= kFrameCount / 2, "most frames decoded"); + check(sink.saw_frame, "decoded frame has pixels"); + check(sink.last_width == kWidth && sink.last_height == kHeight, "decoded dimensions"); + + sender_transport->stop(); + receiver_transport->stop(); + + std::printf("test_loopback: %d frames decoded over UDP loopback\n", sink.decoded_frames.load()); + return 0; +} diff --git a/tests/codec/test_roundtrip.cpp b/tests/codec/test_roundtrip.cpp index 15c2f98..5b193a2 100644 --- a/tests/codec/test_roundtrip.cpp +++ b/tests/codec/test_roundtrip.cpp @@ -85,14 +85,16 @@ int main() { expect_codec_result("encoder create", encoder_result); auto encoder = std::move(sc::codec_value(encoder_result)); + // Without GLOBAL_HEADER the encoder carries no extradata; SPS/PPS are + // emitted in-band ahead of every keyframe. This mirrors the streaming + // path, where a receiver starts decoding from the bitstream alone. const auto extradata = encoder->get_extradata(); - assert(!extradata.empty()); + assert(extradata.empty()); sc::DecoderConfig decoder_config; decoder_config.codec_name = "h264"; decoder_config.width = kWidth; decoder_config.height = kHeight; - decoder_config.extradata = extradata; auto decoder_result = sc::DecoderFactory::create(decoder_config); expect_codec_result("decoder create", decoder_result); diff --git a/tests/meson.build b/tests/meson.build index 72c0ce9..7f427cc 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -15,3 +15,9 @@ test_rtp = executable('test_rtp', dependencies : sc_network_dep) test('rtp framing', test_rtp) + +test_loopback = executable('test_loopback', + 'app/test_loopback.cpp', + dependencies : [sc_codec_dep, sc_network_dep]) + +test('udp loopback', test_loopback) diff --git a/tools/capture_smoke.cpp b/tools/capture_smoke.cpp index 7ab8450..41a05b0 100644 --- a/tools/capture_smoke.cpp +++ b/tools/capture_smoke.cpp @@ -97,9 +97,8 @@ int main(int argc, char** argv) { } encoder = std::move(sc::codec_value(encoder_result)); - // The encoder is configured with AV_CODEC_FLAG_GLOBAL_HEADER, so - // parameter sets live in extradata; prepend them so the output - // file is a self-contained Annex-B stream. + // The encoder repeats SPS/PPS in-band at each keyframe, so the + // file is a self-contained Annex-B stream without a prefix. const auto extradata = encoder->get_extradata(); if (!extradata.empty()) { if (!write_bytes(output, extradata)) {