feat(app): implement Phase 5 local UDP sender->receiver loopback

Wire the first end-to-end pipeline: capture -> encode -> packetize ->
UDP -> depacketize -> decode -> render.

- UdpRtpTransport: raw POSIX UDP sockets (IPv4 via getaddrinfo), a
  receive jthread woken by socket close on stop; port 0 skips binding
  so the sender uses an OS-assigned source port. ASIO stays deferred
  to the signaling phase per ARCHITECTURE.md.
- SdlRenderer: SDL3 window/renderer with RGBA texture upload; the
  texture is recreated on resolution change. RendererFactory now
  returns RendererResult so SDL init failures carry a message,
  mirroring the codec/capture error patterns.
- screencast binary: parse_cli plus SenderPipeline/ReceiverPipeline
  per the app scaffolds; the sender creates its encoder once capture
  reports real dimensions, the receiver keeps a bounded 3-frame queue
  to hold latency down and renders on its own thread until the window
  closes. cli argv signature fixed to 'const char* const*' so main's
  argv converts implicitly.
- Encoder: drop AV_CODEC_FLAG_GLOBAL_HEADER so libx264 repeats SPS/PPS
  in-band at each keyframe -- the receiver decodes from the bitstream
  alone, which also makes mid-stream joins and later PLI recovery
  work without out-of-band parameter negotiation. The round-trip test
  now exercises exactly that path.
- tests: new udp-loopback integration test pushes synthetic frames
  through a real localhost socket and decodes 10/10 frames with the
  right dimensions; valgrind clean (loopback + codec). meson test 4/4.

Manual validation on the desktop (receiver window shows the captured
desktop) is documented in docs/RUNBOOK.md.
This commit is contained in:
2026-09-07 11:02:40 +02:00
parent b5e8d7174c
commit 10870bc6c9
20 changed files with 1057 additions and 61 deletions
+52 -53
View File
@@ -1,51 +1,38 @@
# Project Memory — screen_cast # Project Memory — screen_cast
Last updated: Phase 4 (RTP framing) complete and tested; current phase is Last updated: Phase 5 implemented and automated tests green; manual windowed
Phase 5. loopback validation pending. See bottom for the run commands.
## Project state ## Project state
- Phase 4 done: `sc_network` library implements `RtpHeader`/`RtpPacket` - **Phase 5 (local UDP sender→receiver loopback) is implemented**:
(RFC 3550 serialize/parse; tolerates CSRC lists, extension headers, and - `UdpRtpTransport` (src/network/udp_transport.cpp): raw POSIX sockets,
padding on the receive side) plus `H264Packetizer`/`H264Depacketizer` AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop()
(RFC 6184 single-NAL + FU-A; STAP-A never emitted, unsupported types mark closes the socket to unblock the receive jthread. ASIO was deliberately
the frame damaged on receive). deferred to Phase 6 (see decisions).
- Depacketized access units use **3-byte start codes**, and the splitter - `screencast` binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring
keeps a zero byte preceding a start code with the previous NAL, so both SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline
3- and 4-byte-start-code streams round-trip byte-exactly (tested). (recv→depacketize→decode→bounded 3-frame queue→render thread).
- Loss handling is drop-on-damage: a sequence gap or missing FU fragment - `SdlRenderer` (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture,
marks the frame damaged and it is dropped silently at its marker. Full RGBA texture upload, texture recreated on resolution change.
loss recovery / jitter handling is Phase 7. `RendererFactory::create` now returns `RendererResult` (error channel
- `test_rtp` covers header/packet round-trips, malformed rejections, NAL added, mirroring codec/capture patterns).
splitting (3- and 4-byte codes), FU-A chunking with MTU bounds, full - Encoder change: **GLOBAL_HEADER removed** so libx264 repeats SPS/PPS
packetize→depacketize round-trip, gap dropping, frame separation by in-band at every keyframe; a receiver now decodes from the bitstream
marker alone, sequence wrap, and random default SSRC/sequence. alone (mid-stream join, PLI recovery-ready). `get_extradata()` is empty
Valgrind-clean; `meson test` 3/3. in this mode; codec round-trip test updated to match the streaming path.
- Phase 3 remains validated; earlier review fixes still in place. - **Automated validation**: `meson test` 4/4 — new `udp loopback` test
- The first smoke run emitted `impl_ext_end_proxy called from wrong context` encodes synthetic frames, packetizes, sends over a real localhost UDP
warnings: `pw_context_connect_fd` and `pw_core_disconnect` ran outside the socket, depacketizes, and decodes 10/10 frames with correct dimensions.
thread-loop lock. Fixed by holding the lock across all pw setup/teardown Valgrind clean (loopback + codec tests).
proxy operations. Re-running the smoke tool should now be warning-free - **Manual validation pending (needs the desktop)**: run the two commands in
(capture worked both ways; the warnings only meant the first two `docs/RUNBOOK.md` — receiver window should show the captured desktop. Tick
marshaled messages were rejected and retried from the right context). `docs/PHASES.md` Phase 5 after this works.
- `src/capture/pipewire_capture.cpp` now implements the full backend: - Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A;
libportal 0.10 handshake (`create_screencast_session``session_start` 3-byte canonical start codes; drop-on-damage loss handling).
`open_pipewire_remote`), PipeWire 1.6 stream on the first portal node, - Phase 3 capture done and validated (PipeWire/portal backend; the
BGRx/BGRA/RGBx/RGBA enumeration, latest-frame slot with condvar handoff. `impl_ext_end_proxy` wrong-context warnings were fixed by holding the
Portal handshake blocks on the caller thread; frames arrive on the pw thread-loop lock across all pw proxy operations).
thread. `stop()` is thread-safe; teardown follows the pw-required order.
- Encoder now accepts padded strides for packed RGB formats and the new
`PixelFormat::Bgrx` (mapped to `AV_PIX_FMT_BGRA`); planar Yuv420p still
requires a packed layout. This was the review note blocking Phase 3.
- `tools/capture_smoke` (manual, not in `meson test`) captures N frames →
encodes → writes Annex-B including prepended SPS/PPS extradata (verified:
libx264 GLOBAL_HEADER extradata is Annex-B).
- Earlier review fixes remain in place; valgrind on the codec round-trip
test is still clean after the encoder stride changes.
- Encoder PTS caveat: the encoder time_base is derived from the configured
frame rate (default 25fps), but portal frames arrive at monitor refresh
(often 60Hz), so pts values quantize to 40ms units and can repeat. Harmless
for the smoke test; revisit when RTP timestamps matter (Phase 4/5).
## Decisions ## Decisions
@@ -55,13 +42,19 @@ Phase 5.
- Capture: PipeWire + xdg-desktop-portal. - Capture: PipeWire + xdg-desktop-portal.
- Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale). - Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
- H.264 encoder path: software `libx264`, low-latency settings, Annex-B output. - H.264 encoder path: software `libx264`, low-latency settings, Annex-B output.
- Decoder receives SPS/PPS through `DecoderConfig.extradata`. - **SPS/PPS are sent in-band ahead of every keyframe** (no GLOBAL_HEADER);
- Transport: RTP over UDP; signaling via WebSocket/JSON. the decoder starts from the bitstream alone. `DecoderConfig.extradata`
remains available if signaling ever negotiates parameters out of band.
- Transport: RTP over UDP via raw POSIX sockets for now; **ASIO stays a
Phase 6+ option** (the ARCHITECTURE.md dependency table places it with
signaling/discovery). IPv4 only at the transport level for now.
- Rendering: **SDL3** (`sdl3` pkg-config, 3.4 installed); plain texture
upload, no GPU pipeline yet.
- Discovery: mDNS/Avahi. - Discovery: mDNS/Avahi.
- Rendering: SDL2 or SDL3 + OpenGL.
- Namespace: `sc`. - Namespace: `sc`.
- Module error results use per-module `std::variant<T, XError>` types - Module error results use per-module `std::variant<T, XError>` types
(`CodecResult`, `CaptureResult`) since C++20 has no `std::expected`. (`CodecResult`, `CaptureResult`, `RendererResult`) since C++20 has no
`std::expected`.
## Active blockers ## Active blockers
@@ -72,20 +65,26 @@ None.
- GUI framework (Qt6 vs. none / CLI only) — deferred to later phase. - GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
- Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after - Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after
software encode path works. software encode path works.
- IPv6 at the transport layer — revisit when LAN streaming lands (Phase 6+).
## Forward-looking review notes (for later phases) ## Forward-looking review notes (for later phases)
- `AV_CODEC_FLAG_GLOBAL_HEADER` suppresses in-band SPS/PPS; a receiver cannot
join mid-stream or recover after PLI without parameter sets. Phase 5 must
prepend SPS/PPS to keyframes or negotiate them in signaling.
- Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother - Encoder sets no VBV (`maxrate`/`buffer_size`) — ABR only; add for smoother
UDP streaming in Phase 7. UDP streaming in Phase 7.
- Encoder PTS caveat: time_base derives from the configured frame rate
(default 25fps) while portal frames arrive at monitor refresh (often 60Hz),
so pts values quantize and can repeat. RTP timestamps come from the capture
clock instead, so streaming is unaffected; revisit if decoder-side
presentation timing ever matters.
- `RtpTransport::start/send` return plain bools (scaffold API); error
messages are lost — consider an error channel when signaling lands.
- DMA-BUF-only portal streams are rejected with a clear message (hardware - DMA-BUF-only portal streams are rejected with a clear message (hardware
path is Phase 7). path is Phase 7).
- No negative-path tests yet (bad config, bad stride, undersized buffer). - No negative-path tests yet (bad config, bad stride, undersized buffer).
- `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting - `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting
encoder is ever added, prefer an explicit config flag over the heuristic. encoder is ever added, prefer an explicit config flag over the heuristic.
- Region targets are rejected: the desktop portal has no region capture. - Region targets are rejected: the desktop portal has no region capture.
- `next_frame()` returns `nullopt` on stream error without surfacing the - `CaptureSession::next_frame()` returns `nullopt` on stream error without
reason (logged to stderr); consider an error channel when the receiver surfacing the reason (logged to stderr).
pipeline lands. - Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
we control never emit them, but third-party interop would need support.
+17
View File
@@ -49,6 +49,23 @@ PipeWire proxy operation ran without the thread-loop lock — all pw
calls that send messages (connect, stream, core) must happen under calls that send messages (connect, stream, core) must happen under
`pw_thread_loop_lock`. `pw_thread_loop_lock`.
## Sender / receiver loopback (Phase 5, manual)
Two terminals on the same desktop session:
```sh
./build/src/app/screencast --receive # terminal 1: window appears
./build/src/app/screencast --send # terminal 2: portal source picker
```
Expected: the receiver window shows the captured desktop in near real time.
Stop either side with Ctrl-C. Optional flags: `--port`, `--peer HOST[:PORT]`,
`--bitrate KBPS`, `--target window`.
The headless equivalent runs as part of `meson test` (`udp loopback` test):
synthetic frames → encode → packetize → localhost UDP → depacketize →
decode, no portal or window involved.
## Formatting ## Formatting
```sh ```sh
+3 -1
View File
@@ -20,6 +20,8 @@ struct ReceiveCommand {
using Command = std::variant<SendCommand, ReceiveCommand>; using Command = std::variant<SendCommand, ReceiveCommand>;
// Parse command line arguments. Prints usage and returns std::nullopt on error. // Parse command line arguments. Prints usage and returns std::nullopt on error.
std::optional<Command> parse_cli(int argc, const char* argv[]); // `argv` is `char const* const*` so both `main`'s `char**` and const arrays
// convert implicitly.
std::optional<Command> parse_cli(int argc, const char* const argv[]);
} // namespace sc } // namespace sc
+3
View File
@@ -16,6 +16,9 @@ struct SenderPipelineConfig {
EncoderConfig encoder; EncoderConfig encoder;
Endpoint local_rtp_endpoint; Endpoint local_rtp_endpoint;
std::optional<Endpoint> signaling_server; std::optional<Endpoint> signaling_server;
// Where encoded RTP packets are sent. Defaults to the local loopback so
// a sender and receiver on one machine work without any configuration.
Endpoint peer_rtp_endpoint{"127.0.0.1", 5004};
}; };
struct ReceiverPipelineConfig { struct ReceiverPipelineConfig {
+36
View File
@@ -0,0 +1,36 @@
#pragma once
#include <string>
#include <variant>
namespace sc {
struct RendererError {
std::string message;
};
// C++20 does not provide std::expected. Use a variant-based result type so
// fallible renderer operations do not rely on exceptions.
template <typename T> using RendererResult = std::variant<T, RendererError>;
template <typename T> constexpr bool is_renderer_error(const RendererResult<T>& result) noexcept {
return std::holds_alternative<RendererError>(result);
}
template <typename T> T& renderer_value(RendererResult<T>& result) {
return std::get<T>(result);
}
template <typename T> const T& renderer_value(const RendererResult<T>& result) {
return std::get<T>(result);
}
template <typename T> RendererError& renderer_error(RendererResult<T>& result) {
return std::get<RendererError>(result);
}
template <typename T> const RendererError& renderer_error(const RendererResult<T>& result) {
return std::get<RendererError>(result);
}
} // namespace sc
+2 -1
View File
@@ -1,6 +1,7 @@
#pragma once #pragma once
#include "screencast/codec/decoder.h" #include "screencast/codec/decoder.h"
#include "screencast/render/error.h"
#include <memory> #include <memory>
#include <string> #include <string>
@@ -29,7 +30,7 @@ class Renderer {
class RendererFactory { class RendererFactory {
public: public:
static std::unique_ptr<Renderer> create(const RendererConfig& config); static RendererResult<std::unique_ptr<Renderer>> create(const RendererConfig& config);
}; };
} // namespace sc } // namespace sc
+109
View File
@@ -0,0 +1,109 @@
#include "screencast/app/cli.h"
#include <charconv>
#include <cstdio>
#include <string_view>
namespace sc {
namespace {
void print_usage() {
std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate KBPS]\n"
" screencast --receive [--port PORT]\n",
stderr);
}
bool parse_int(std::string_view text, int& value) {
const auto [pointer, error] = std::from_chars(text.data(), text.data() + text.size(), value);
return error == std::errc{} && pointer == text.data() + text.size();
}
bool next_argument(int argc, const char* const argv[], int& index, std::string_view& value) {
if (index + 1 >= argc) {
print_usage();
return false;
}
value = argv[++index];
return true;
}
} // namespace
std::optional<Command> parse_cli(int argc, const char* const argv[]) {
enum class Mode {
None,
Send,
Receive,
};
Mode mode = Mode::None;
SendCommand send;
ReceiveCommand receive;
for (int index = 1; index < argc; ++index) {
const std::string_view argument = argv[index];
if (argument == "--send") {
if (mode != Mode::None) {
print_usage();
return std::nullopt;
}
mode = Mode::Send;
} else if (argument == "--receive") {
if (mode != Mode::None) {
print_usage();
return std::nullopt;
}
mode = Mode::Receive;
} else if (argument == "--target") {
std::string_view value;
if (!next_argument(argc, argv, index, value)) {
return std::nullopt;
}
if (value != "monitor" && value != "window") {
print_usage();
return std::nullopt;
}
send.target = value;
} else if (argument == "--peer") {
std::string_view value;
if (!next_argument(argc, argv, index, value)) {
return std::nullopt;
}
if (mode == Mode::Receive) {
receive.peer_address = value;
} else {
send.peer_address = value;
}
} else if (argument == "--bitrate") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, send.bitrate_kbps) ||
send.bitrate_kbps <= 0) {
print_usage();
return std::nullopt;
}
} else if (argument == "--port") {
std::string_view value;
int port = 0;
if (!next_argument(argc, argv, index, value) || !parse_int(value, port) || port <= 0 || port > 65535) {
print_usage();
return std::nullopt;
}
receive.local_rtp_port = port;
} else {
print_usage();
return std::nullopt;
}
}
if (mode == Mode::Send) {
return Command{std::move(send)};
}
if (mode == Mode::Receive) {
return Command{std::move(receive)};
}
print_usage();
return std::nullopt;
}
} // namespace sc
+87
View File
@@ -0,0 +1,87 @@
#include "screencast/app/cli.h"
#include "screencast/app/pipeline.h"
#include <atomic>
#include <charconv>
#include <chrono>
#include <csignal>
#include <string_view>
#include <thread>
namespace {
std::atomic<bool> g_interrupted{false};
void handle_interrupt(int) {
g_interrupted.store(true);
}
sc::Endpoint parse_endpoint(std::string_view address, uint16_t default_port) {
sc::Endpoint endpoint;
const std::size_t separator = address.rfind(':');
if (separator != std::string_view::npos) {
endpoint.address = std::string{address.substr(0, separator)};
int port = 0;
const auto [pointer, error] =
std::from_chars(address.data() + separator + 1, address.data() + address.size(), port);
endpoint.port = error == std::errc{} ? static_cast<uint16_t>(port) : default_port;
} else {
endpoint.address = std::string{address};
endpoint.port = default_port;
}
return endpoint;
}
int run_sender(const sc::SendCommand& command) {
sc::SenderPipelineConfig config;
if (command.target == "window") {
config.capture_target = sc::CaptureTargetWindow{};
} else {
config.capture_target = sc::CaptureTargetWholeScreen{};
}
config.peer_rtp_endpoint =
command.peer_address.empty() ? sc::Endpoint{"127.0.0.1", 5004} : parse_endpoint(command.peer_address, 5004);
config.encoder.bitrate_kbps = command.bitrate_kbps;
sc::SenderPipeline pipeline{std::move(config)};
if (!pipeline.start()) {
return 1;
}
while (!g_interrupted.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
pipeline.stop();
return 0;
}
int run_receiver(const sc::ReceiveCommand& command) {
sc::ReceiverPipelineConfig config;
config.local_rtp_endpoint = sc::Endpoint{"0.0.0.0", static_cast<uint16_t>(command.local_rtp_port)};
sc::ReceiverPipeline pipeline{std::move(config)};
if (!pipeline.start()) {
return 1;
}
while (!g_interrupted.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
pipeline.stop();
return 0;
}
} // namespace
int main(int argc, char* argv[]) {
const std::optional<sc::Command> command = sc::parse_cli(argc, argv);
if (!command.has_value()) {
return 1;
}
std::signal(SIGINT, handle_interrupt);
std::signal(SIGTERM, handle_interrupt);
if (const sc::SendCommand* send = std::get_if<sc::SendCommand>(&*command)) {
return run_sender(*send);
}
return run_receiver(std::get<sc::ReceiveCommand>(*command));
}
+9
View File
@@ -0,0 +1,9 @@
# The screencast application binary wiring every module together.
executable('screencast',
[
'cli.cpp',
'main.cpp',
'pipelines.cpp',
],
dependencies : [sc_capture_dep, sc_codec_dep, sc_network_dep, sc_render_dep])
+224
View File
@@ -0,0 +1,224 @@
#include "screencast/app/pipeline.h"
#include "screencast/network/h264_packetizer.h"
#include <chrono>
#include <deque>
#include <iostream>
#include <memory>
#include <mutex>
#include <thread>
namespace sc {
class SenderPipeline::Impl {
public:
explicit Impl(SenderPipelineConfig config) : config_(std::move(config)) {}
~Impl() {
stop();
}
bool start() {
auto capture_result = CaptureFactory::create(config_.capture_target);
if (is_capture_error(capture_result)) {
std::cerr << std::format("screencast: capture failed: {}\n", capture_error(capture_result).message);
return false;
}
capture_ = std::move(capture_value(capture_result));
if (!transport_->start(config_.local_rtp_endpoint, [](RtpPacket) {})) {
std::cerr << "screencast: failed to start the RTP transport\n";
return false;
}
transport_->set_peer(config_.peer_rtp_endpoint);
run_thread_ = std::jthread([this](std::stop_token stop_token) { run(std::move(stop_token)); });
return true;
}
void stop() {
if (capture_ != nullptr) {
capture_->stop();
}
run_thread_ = std::jthread{};
transport_->stop();
}
private:
void run(std::stop_token stop_token) {
while (!stop_token.stop_requested()) {
const std::optional<CapturedFrame> frame = capture_->next_frame();
if (!frame.has_value()) {
break;
}
if (encoder_ == nullptr && !create_encoder(*frame)) {
break;
}
auto encoded_result = encoder_->encode(*frame);
if (is_codec_error(encoded_result)) {
std::cerr << std::format("screencast: encode failed: {}\n", codec_error(encoded_result).message);
break;
}
for (const EncodedFrame& encoded : codec_value(encoded_result)) {
for (RtpPacket& packet : packetizer_.packetize(encoded)) {
(void)transport_->send(packet);
}
}
}
}
bool create_encoder(const CapturedFrame& frame) {
EncoderConfig config = config_.encoder;
config.width = frame.width;
config.height = frame.height;
auto encoder_result = EncoderFactory::create(config);
if (is_codec_error(encoder_result)) {
std::cerr << std::format("screencast: encoder creation failed: {}\n", codec_error(encoder_result).message);
return false;
}
encoder_ = std::move(codec_value(encoder_result));
return true;
}
SenderPipelineConfig config_;
std::unique_ptr<CaptureSession> capture_;
std::unique_ptr<Encoder> encoder_;
H264Packetizer packetizer_;
std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create();
std::jthread run_thread_;
};
SenderPipeline::SenderPipeline(SenderPipelineConfig config) : impl_(std::make_unique<Impl>(std::move(config))) {}
SenderPipeline::~SenderPipeline() = default;
bool SenderPipeline::start() {
return impl_->start();
}
void SenderPipeline::stop() {
impl_->stop();
}
class ReceiverPipeline::Impl {
public:
explicit Impl(ReceiverPipelineConfig config) : config_(std::move(config)) {}
~Impl() {
stop();
}
bool start() {
auto renderer_result = RendererFactory::create(config_.renderer);
if (is_renderer_error(renderer_result)) {
std::cerr << std::format("screencast: renderer creation failed: {}\n",
renderer_error(renderer_result).message);
return false;
}
renderer_ = std::move(renderer_value(renderer_result));
auto decoder_result = DecoderFactory::create(config_.decoder);
if (is_codec_error(decoder_result)) {
std::cerr << std::format("screencast: decoder creation failed: {}\n", codec_error(decoder_result).message);
return false;
}
decoder_ = std::move(codec_value(decoder_result));
if (!transport_->start(config_.local_rtp_endpoint,
[this](RtpPacket packet) { on_packet(std::move(packet)); })) {
std::cerr << "screencast: failed to start the RTP transport\n";
return false;
}
render_thread_ = std::jthread([this](std::stop_token stop_token) { render_loop(std::move(stop_token)); });
return true;
}
void stop() {
render_thread_ = std::jthread{};
transport_->stop();
renderer_ = nullptr;
decoder_ = nullptr;
{
std::lock_guard lock(queue_mutex_);
queue_.clear();
}
}
private:
void on_packet(RtpPacket packet) {
std::optional<std::vector<std::byte>> access_unit = depacketizer_.depacketize(packet);
if (!access_unit.has_value()) {
return;
}
EncodedFrame encoded;
encoded.data = std::move(*access_unit);
encoded.rtp_timestamp = packet.header.timestamp;
encoded.is_keyframe = false;
auto decoded_result = decoder_->decode(encoded);
if (is_codec_error(decoded_result)) {
std::cerr << std::format("screencast: decode failed: {}\n", codec_error(decoded_result).message);
return;
}
for (DecodedFrame& decoded : codec_value(decoded_result)) {
std::lock_guard lock(queue_mutex_);
// Keep latency low: drop the oldest frame when the queue is full.
if (queue_.size() >= kMaxQueuedFrames) {
queue_.pop_front();
}
queue_.push_back(std::move(decoded));
}
}
void render_loop(std::stop_token stop_token) {
while (!stop_token.stop_requested()) {
if (!renderer_->poll_events()) {
break; // window closed
}
std::optional<DecodedFrame> frame;
{
std::unique_lock lock(queue_mutex_);
if (!queue_.empty()) {
frame = std::move(queue_.front());
queue_.pop_front();
}
}
if (frame.has_value()) {
(void)renderer_->present(*frame);
} else {
std::this_thread::sleep_for(std::chrono::milliseconds(5));
}
}
}
static constexpr std::size_t kMaxQueuedFrames = 3;
ReceiverPipelineConfig config_;
std::unique_ptr<Renderer> renderer_;
std::unique_ptr<Decoder> decoder_;
H264Depacketizer depacketizer_;
std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create();
std::jthread render_thread_;
std::mutex queue_mutex_;
std::deque<DecodedFrame> queue_;
};
ReceiverPipeline::ReceiverPipeline(ReceiverPipelineConfig config) : impl_(std::make_unique<Impl>(std::move(config))) {}
ReceiverPipeline::~ReceiverPipeline() = default;
bool ReceiverPipeline::start() {
return impl_->start();
}
void ReceiverPipeline::stop() {
impl_->stop();
}
} // namespace sc
+4 -1
View File
@@ -397,7 +397,10 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
ctx->max_b_frames = 0; ctx->max_b_frames = 0;
ctx->thread_count = 1; ctx->thread_count = 1;
ctx->profile = AV_PROFILE_H264_MAIN; ctx->profile = AV_PROFILE_H264_MAIN;
ctx->flags |= AV_CODEC_FLAG_LOW_DELAY | AV_CODEC_FLAG_GLOBAL_HEADER; // No GLOBAL_HEADER: SPS/PPS are repeated in-band at every keyframe so a
// receiver that joins mid-stream (or recovers after loss) can decode
// without out-of-band parameter negotiation.
ctx->flags |= AV_CODEC_FLAG_LOW_DELAY;
if (av_opt_set(ctx->priv_data, "preset", "ultrafast", 0) < 0) { if (av_opt_set(ctx->priv_data, "preset", "ultrafast", 0) < 0) {
return CodecError{"failed to set libx264 preset"}; return CodecError{"failed to set libx264 preset"};
+3
View File
@@ -24,3 +24,6 @@ sc_codec_dep = declare_dependency(
dependencies : [dep_avcodec, dep_avutil, dep_swscale]) dependencies : [dep_avcodec, dep_avutil, dep_swscale])
subdir('network') subdir('network')
subdir('render')
subdir('app')
+1
View File
@@ -3,6 +3,7 @@
sc_network_sources = files( sc_network_sources = files(
'rtp_packet.cpp', 'rtp_packet.cpp',
'h264_packetizer.cpp', 'h264_packetizer.cpp',
'udp_transport.cpp',
) )
sc_network = static_library('sc_network', sc_network = static_library('sc_network',
+163
View File
@@ -0,0 +1,163 @@
#include "screencast/network/transport.h"
#include <arpa/inet.h>
#include <netdb.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <unistd.h>
#include <array>
#include <atomic>
#include <cstring>
#include <mutex>
#include <optional>
#include <string>
#include <thread>
namespace sc {
namespace {
constexpr int kInvalidSocket = -1;
// A UDP datagram cannot exceed 64 KiB on IPv4; one buffer fits any RTP
// packet the receiver will ever see.
constexpr std::size_t kReceiveBufferSize = 65536;
std::optional<sockaddr_in> resolve_ipv4(const Endpoint& endpoint) {
addrinfo hints{};
hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_DGRAM;
addrinfo* result = nullptr;
const std::string port = std::to_string(endpoint.port);
const char* node = endpoint.address.empty() ? nullptr : endpoint.address.c_str();
if (getaddrinfo(node, port.c_str(), &hints, &result) != 0) {
return std::nullopt;
}
std::optional<sockaddr_in> address;
if (result != nullptr && result->ai_family == AF_INET && result->ai_addrlen >= sizeof(sockaddr_in)) {
sockaddr_in resolved{};
std::memcpy(&resolved, result->ai_addr, sizeof(sockaddr_in));
address = resolved;
}
freeaddrinfo(result);
return address;
}
} // namespace
class UdpRtpTransport final : public RtpTransport {
public:
UdpRtpTransport() = default;
~UdpRtpTransport() override {
stop();
}
UdpRtpTransport(const UdpRtpTransport&) = delete;
UdpRtpTransport& operator=(const UdpRtpTransport&) = delete;
bool start(const Endpoint& local_endpoint, ReceiveCallback on_receive) override {
if (running_.load()) {
return false;
}
socket_ = ::socket(AF_INET, SOCK_DGRAM, 0);
if (socket_ < 0) {
return false;
}
int reuse = 1;
(void)::setsockopt(socket_, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse));
// A zero port skips binding: the OS picks the source port on send.
if (local_endpoint.port != 0) {
const std::optional<sockaddr_in> address = resolve_ipv4(local_endpoint);
if (!address.has_value()) {
(void)::close(socket_);
socket_ = kInvalidSocket;
return false;
}
if (::bind(socket_, reinterpret_cast<const sockaddr*>(&*address), sizeof(*address)) < 0) {
(void)::close(socket_);
socket_ = kInvalidSocket;
return false;
}
}
running_.store(true);
receive_thread_ = std::jthread([this, callback = std::move(on_receive)]() mutable { receive_loop(callback); });
return true;
}
bool send(const RtpPacket& packet) override {
if (!running_.load() || !has_peer_.load()) {
return false;
}
const std::vector<std::byte> bytes = packet.serialize();
if (bytes.empty()) {
return false;
}
sockaddr_in peer{};
{
// Snapshot the peer so set_peer() can be called concurrently.
std::lock_guard lock(peer_mutex_);
peer = peer_;
}
const ssize_t sent =
::sendto(socket_, bytes.data(), bytes.size(), 0, reinterpret_cast<const sockaddr*>(&peer), sizeof(peer));
return sent == static_cast<ssize_t>(bytes.size());
}
void set_peer(const Endpoint& peer) override {
const std::optional<sockaddr_in> address = resolve_ipv4(peer);
if (!address.has_value()) {
return;
}
std::lock_guard lock(peer_mutex_);
peer_ = *address;
has_peer_.store(true);
}
void stop() override {
if (!running_.exchange(false)) {
return;
}
if (socket_ >= 0) {
(void)::shutdown(socket_, SHUT_RDWR);
(void)::close(socket_);
socket_ = kInvalidSocket;
}
// Closing the socket unblocks recvfrom; joining happens implicitly
// when the jthread assignment destroys the previous thread.
receive_thread_ = std::jthread{};
}
private:
void receive_loop(ReceiveCallback& callback) {
std::array<std::byte, kReceiveBufferSize> buffer{};
while (running_.load()) {
const ssize_t received = ::recvfrom(socket_, buffer.data(), buffer.size(), 0, nullptr, nullptr);
if (received <= 0) {
continue; // closed socket while running_ still true, or error
}
const std::optional<RtpPacket> packet =
RtpPacket::parse(std::span{buffer.data(), static_cast<std::size_t>(received)});
if (packet.has_value()) {
callback(*packet);
}
}
}
int socket_ = kInvalidSocket;
std::atomic<bool> running_{false};
std::atomic<bool> has_peer_{false};
std::mutex peer_mutex_;
sockaddr_in peer_{};
std::jthread receive_thread_;
};
std::unique_ptr<RtpTransport> RtpTransportFactory::create() {
return std::make_unique<UdpRtpTransport>();
}
} // namespace sc
+15
View File
@@ -0,0 +1,15 @@
# Phase 5 renderer: SDL3 software texture upload, no GPU pipeline yet.
dep_sdl3 = dependency('sdl3')
sc_render_sources = files('sdl_renderer.cpp')
sc_render = static_library('sc_render',
sc_render_sources,
include_directories : sc_core_inc,
dependencies : [dep_sdl3])
sc_render_dep = declare_dependency(
link_with : sc_render,
include_directories : sc_core_inc,
dependencies : [dep_sdl3])
+147
View File
@@ -0,0 +1,147 @@
#include "screencast/render/renderer.h"
#include <SDL3/SDL.h>
#include <string>
namespace sc {
namespace {
// DecodedFrame pixels are AV_PIX_FMT_RGBA: memory order R, G, B, A, which
// matches SDL_PIXELFORMAT_RGBA8888.
constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_RGBA8888;
class SdlRenderer final : public Renderer {
public:
explicit SdlRenderer(const RendererConfig& config) : config_(config) {}
~SdlRenderer() override {
shutdown();
}
SdlRenderer(const SdlRenderer&) = delete;
SdlRenderer& operator=(const SdlRenderer&) = delete;
const std::string& last_error() const {
return last_error_;
}
bool initialize() {
if (!SDL_Init(SDL_INIT_VIDEO)) {
last_error_ = std::string{"SDL_Init failed: "} + SDL_GetError();
return false;
}
sdl_inited_ = true;
window_ = SDL_CreateWindow(config_.window_title.c_str(), config_.initial_width, config_.initial_height, 0);
if (window_ == nullptr) {
last_error_ = std::string{"SDL_CreateWindow failed: "} + SDL_GetError();
shutdown();
return false;
}
renderer_ = SDL_CreateRenderer(window_, nullptr);
if (renderer_ == nullptr) {
last_error_ = std::string{"SDL_CreateRenderer failed: "} + SDL_GetError();
shutdown();
return false;
}
return true;
}
bool present(const DecodedFrame& frame) override {
if (frame.width <= 0 || frame.height <= 0) {
return false;
}
if (frame.width != texture_width_ || frame.height != texture_height_) {
if (!recreate_texture(frame.width, frame.height)) {
return false;
}
}
const int pitch = frame.width * 4;
if (!SDL_UpdateTexture(texture_, nullptr, frame.rgba_pixels.data(), pitch)) {
return false;
}
if (!SDL_RenderTexture(renderer_, texture_, nullptr, nullptr)) {
return false;
}
SDL_RenderPresent(renderer_);
return true;
}
bool poll_events() override {
if (closed_) {
return false;
}
SDL_Event event{};
while (SDL_PollEvent(&event)) {
if (event.type == SDL_EVENT_QUIT ||
(event.type == SDL_EVENT_WINDOW_CLOSE_REQUESTED && event.window.windowID == SDL_GetWindowID(window_))) {
closed_ = true;
return false;
}
}
return true;
}
void shutdown() override {
if (texture_ != nullptr) {
SDL_DestroyTexture(texture_);
texture_ = nullptr;
}
if (renderer_ != nullptr) {
SDL_DestroyRenderer(renderer_);
renderer_ = nullptr;
}
if (window_ != nullptr) {
SDL_DestroyWindow(window_);
window_ = nullptr;
}
if (sdl_inited_) {
SDL_Quit();
sdl_inited_ = false;
}
}
private:
bool recreate_texture(int width, int height) {
if (texture_ != nullptr) {
SDL_DestroyTexture(texture_);
texture_ = nullptr;
}
texture_ = SDL_CreateTexture(renderer_, kSdlPixelFormat, SDL_TEXTUREACCESS_STREAMING, width, height);
if (texture_ == nullptr) {
last_error_ = std::string{"SDL_CreateTexture failed: "} + SDL_GetError();
texture_width_ = 0;
texture_height_ = 0;
return false;
}
texture_width_ = width;
texture_height_ = height;
return true;
}
RendererConfig config_;
std::string last_error_;
bool sdl_inited_ = false;
bool closed_ = false;
SDL_Window* window_ = nullptr;
SDL_Renderer* renderer_ = nullptr;
SDL_Texture* texture_ = nullptr;
int texture_width_ = 0;
int texture_height_ = 0;
};
} // namespace
RendererResult<std::unique_ptr<Renderer>> RendererFactory::create(const RendererConfig& config) {
auto renderer = std::make_unique<SdlRenderer>(config);
if (!renderer->initialize()) {
return RendererError{renderer->last_error()};
}
return std::unique_ptr<Renderer>(std::move(renderer));
}
} // namespace sc
+170
View File
@@ -0,0 +1,170 @@
// Phase 5 integration test: the local UDP sender -> receiver loopback without
// the interactive parts. Synthetic frames go through the real chain —
// encode, RTP packetization, a localhost UDP socket, depacketization, and
// decode (with in-band SPS/PPS, since there is no signaling channel).
#include "screencast/codec/decoder.h"
#include "screencast/codec/encoder.h"
#include "screencast/network/h264_packetizer.h"
#include "screencast/network/transport.h"
#include <atomic>
#include <chrono>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <memory>
#include <thread>
#include <vector>
namespace {
[[noreturn]] void fail(const char* what) {
std::fprintf(stderr, "test_loopback: FAIL: %s\n", what);
std::abort();
}
void check(bool condition, const char* what) {
if (!condition) {
fail(what);
}
}
template <typename T> void expect_result(const char* what, const sc::CodecResult<T>& result) {
if (sc::is_codec_error(result)) {
std::fprintf(stderr, "test_loopback: FAIL: %s: %s\n", what, sc::codec_error(result).message.c_str());
std::abort();
}
}
constexpr int kWidth = 64;
constexpr int kHeight = 64;
constexpr int kFrameCount = 10;
constexpr uint64_t kNsPerFrame = 40'000'000; // 25 fps
sc::CapturedFrame make_frame(uint32_t index) {
sc::CapturedFrame frame;
frame.width = kWidth;
frame.height = kHeight;
frame.timestamp_ns = static_cast<uint64_t>(index) * kNsPerFrame;
frame.pixel_format = sc::PixelFormat::Rgba;
frame.stride = kWidth * 4;
frame.pixels.resize(static_cast<std::size_t>(kWidth) * kHeight * 4);
auto* pixels = reinterpret_cast<std::uint8_t*>(frame.pixels.data());
for (int y = 0; y < kHeight; ++y) {
for (int x = 0; x < kWidth; ++x) {
const std::size_t offset = (static_cast<std::size_t>(y) * kWidth + x) * 4;
pixels[offset + 0] = static_cast<std::uint8_t>((x + static_cast<int>(index) * 4) & 0xFF);
pixels[offset + 1] = static_cast<std::uint8_t>((y + static_cast<int>(index) * 2) & 0xFF);
pixels[offset + 2] = static_cast<std::uint8_t>(((x ^ y) + static_cast<int>(index)) & 0xFF);
pixels[offset + 3] = 0xFF;
}
}
return frame;
}
// Receiver state driven entirely by the transport's receive thread.
struct ReceiverSink {
sc::H264Depacketizer depacketizer;
std::unique_ptr<sc::Decoder> decoder;
std::atomic<int> decoded_frames{0};
bool saw_frame = false;
int last_width = 0;
int last_height = 0;
void on_packet(sc::RtpPacket packet) {
std::optional<std::vector<std::byte>> access_unit = depacketizer.depacketize(packet);
if (!access_unit.has_value()) {
return;
}
sc::EncodedFrame encoded;
encoded.data = std::move(*access_unit);
encoded.rtp_timestamp = packet.header.timestamp;
auto decoded_result = decoder->decode(encoded);
if (sc::is_codec_error(decoded_result)) {
return;
}
for (const sc::DecodedFrame& decoded : sc::codec_value(decoded_result)) {
last_width = decoded.width;
last_height = decoded.height;
saw_frame = decoded.width > 0 && !decoded.rgba_pixels.empty();
decoded_frames.fetch_add(1);
}
}
};
} // namespace
int main() {
// Encoder on the sender side.
sc::EncoderConfig encoder_config;
encoder_config.codec_name = "h264";
encoder_config.width = kWidth;
encoder_config.height = kHeight;
encoder_config.frame_rate_num = 25;
encoder_config.frame_rate_den = 1;
encoder_config.bitrate_kbps = 2000;
auto encoder_result = sc::EncoderFactory::create(encoder_config);
expect_result("encoder create", encoder_result);
auto encoder = std::move(sc::codec_value(encoder_result));
// The receiver decodes purely from the bitstream (in-band SPS/PPS).
sc::DecoderConfig decoder_config;
decoder_config.codec_name = "h264";
auto decoder_result = sc::DecoderFactory::create(decoder_config);
expect_result("decoder create", decoder_result);
ReceiverSink sink;
sink.decoder = std::move(sc::codec_value(decoder_result));
// Bind the receiver to the first free port in a small range.
auto receiver_transport = sc::RtpTransportFactory::create();
uint16_t bound_port = 0;
for (uint16_t port = 45904; port < 45934; ++port) {
if (receiver_transport->start(sc::Endpoint{"0.0.0.0", port},
[&sink](sc::RtpPacket packet) { sink.on_packet(std::move(packet)); })) {
bound_port = port;
break;
}
}
check(bound_port != 0, "receiver transport bound");
// Sender transport: no bind (OS picks the source port), peer is loopback.
auto sender_transport = sc::RtpTransportFactory::create();
check(sender_transport->start(sc::Endpoint{"", 0}, [](sc::RtpPacket) {}), "sender transport started");
sender_transport->set_peer(sc::Endpoint{"127.0.0.1", bound_port});
sc::H264Packetizer packetizer;
for (uint32_t index = 0; index < kFrameCount; ++index) {
const sc::CapturedFrame frame = make_frame(index);
auto encoded_result = encoder->encode(frame);
expect_result("encode", encoded_result);
for (const sc::EncodedFrame& encoded : sc::codec_value(encoded_result)) {
for (const sc::RtpPacket& packet : packetizer.packetize(encoded)) {
check(sender_transport->send(packet), "send packet");
}
}
}
// Wait (bounded) for the receiver to decode the frames.
const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5);
while (sink.decoded_frames.load() < kFrameCount && std::chrono::steady_clock::now() < deadline) {
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
check(sink.decoded_frames.load() >= kFrameCount / 2, "most frames decoded");
check(sink.saw_frame, "decoded frame has pixels");
check(sink.last_width == kWidth && sink.last_height == kHeight, "decoded dimensions");
sender_transport->stop();
receiver_transport->stop();
std::printf("test_loopback: %d frames decoded over UDP loopback\n", sink.decoded_frames.load());
return 0;
}
+4 -2
View File
@@ -85,14 +85,16 @@ int main() {
expect_codec_result("encoder create", encoder_result); expect_codec_result("encoder create", encoder_result);
auto encoder = std::move(sc::codec_value(encoder_result)); auto encoder = std::move(sc::codec_value(encoder_result));
// Without GLOBAL_HEADER the encoder carries no extradata; SPS/PPS are
// emitted in-band ahead of every keyframe. This mirrors the streaming
// path, where a receiver starts decoding from the bitstream alone.
const auto extradata = encoder->get_extradata(); const auto extradata = encoder->get_extradata();
assert(!extradata.empty()); assert(extradata.empty());
sc::DecoderConfig decoder_config; sc::DecoderConfig decoder_config;
decoder_config.codec_name = "h264"; decoder_config.codec_name = "h264";
decoder_config.width = kWidth; decoder_config.width = kWidth;
decoder_config.height = kHeight; decoder_config.height = kHeight;
decoder_config.extradata = extradata;
auto decoder_result = sc::DecoderFactory::create(decoder_config); auto decoder_result = sc::DecoderFactory::create(decoder_config);
expect_codec_result("decoder create", decoder_result); expect_codec_result("decoder create", decoder_result);
+6
View File
@@ -15,3 +15,9 @@ test_rtp = executable('test_rtp',
dependencies : sc_network_dep) dependencies : sc_network_dep)
test('rtp framing', test_rtp) test('rtp framing', test_rtp)
test_loopback = executable('test_loopback',
'app/test_loopback.cpp',
dependencies : [sc_codec_dep, sc_network_dep])
test('udp loopback', test_loopback)
+2 -3
View File
@@ -97,9 +97,8 @@ int main(int argc, char** argv) {
} }
encoder = std::move(sc::codec_value(encoder_result)); encoder = std::move(sc::codec_value(encoder_result));
// The encoder is configured with AV_CODEC_FLAG_GLOBAL_HEADER, so // The encoder repeats SPS/PPS in-band at each keyframe, so the
// parameter sets live in extradata; prepend them so the output // file is a self-contained Annex-B stream without a prefix.
// file is a self-contained Annex-B stream.
const auto extradata = encoder->get_extradata(); const auto extradata = encoder->get_extradata();
if (!extradata.empty()) { if (!extradata.empty()) {
if (!write_bytes(output, extradata)) { if (!write_bytes(output, extradata)) {