117 lines
6.6 KiB
Markdown
117 lines
6.6 KiB
Markdown
# Project Memory — screen_cast
|
|
|
|
Last updated: Phase 5 implemented and automated tests green; manual windowed
|
|
loopback validation pending. See bottom for the run commands.
|
|
|
|
## Project state
|
|
|
|
- **Phase 5 (local UDP sender→receiver loopback) is implemented**:
|
|
- `UdpRtpTransport` (src/network/udp_transport.cpp): raw POSIX sockets,
|
|
AF_INET, IPv4 via getaddrinfo; port 0 skips binding (sender side); stop()
|
|
closes the socket to unblock the receive jthread. ASIO was deliberately
|
|
deferred to Phase 6 (see decisions).
|
|
- `screencast` binary (src/app/): cli.cpp + main.cpp + pipelines.cpp wiring
|
|
SenderPipeline (capture→encode→packetize→send) and ReceiverPipeline
|
|
(recv→depacketize→decode→bounded 3-frame queue→render thread).
|
|
- `SdlRenderer` (src/render/sdl_renderer.cpp): SDL3 window/renderer/texture,
|
|
RGBA texture upload, texture recreated on resolution change.
|
|
`RendererFactory::create` now returns `RendererResult` (error channel
|
|
added, mirroring codec/capture patterns).
|
|
- Encoder change: **GLOBAL_HEADER removed** so libx264 repeats SPS/PPS
|
|
in-band at every keyframe; a receiver now decodes from the bitstream
|
|
alone (mid-stream join, PLI recovery-ready). `get_extradata()` is empty
|
|
in this mode; codec round-trip test updated to match the streaming path.
|
|
- **Burst-control fixes after the first real run** (receiver saw
|
|
`non-existing PPS 0` forever): without VBV, a 2256x1504 IDR burst (~100s
|
|
of KB of back-to-back FU-A packets) overflowed the ~208KB default UDP
|
|
receive buffer; the sequence gap made drop-on-damage discard whole
|
|
keyframes *including their in-band SPS/PPS*, so the receiver never
|
|
recovered. Fixes: encoder VBV (`rc_max_rate = bitrate`,
|
|
`rc_buffer_size = bitrate*2/fps` — caps any keyframe to ~2 frame periods
|
|
of bytes), microsecond encoder time_base (kills the 25fps pts
|
|
quantization that duplicated timestamps), and a best-effort 4MB
|
|
SO_RCVBUF on the receive socket (kernel clamps to rmem_max).
|
|
- **Display fixes after the receiver window never appeared**: (1) SDL must
|
|
own the window from one thread — creating it on main and pumping/presenting
|
|
from the render thread left the Wayland surface unmapped; the renderer now
|
|
creates/polls/presents/destroys entirely on the render thread (with a
|
|
start() init handshake). (2) Wayland windows are invisible until the first
|
|
render commit, so the renderer presents a blank frame at init (visible
|
|
black window while waiting). (3) The receiver logs `stream started (WxH)`
|
|
on the first decoded frame and any first present failure.
|
|
- **Phase 5 validated end-to-end** (headless, without the portal): a
|
|
synthetic RTP feed of solid red/green frames drove the real receiver over
|
|
localhost UDP; the window mapped, logged `stream started (320x240)`, and a
|
|
`grim` screenshot of the window region showed the fed color (V=198,
|
|
SAT=74 during the red feed) — decoded video visibly rendering. The user's
|
|
real sender run showed capture→encode→send working (1 IDR + 12 P-frames,
|
|
IDR capped at ~20KB by the VBV).
|
|
- **Automated validation**: `meson test` 4/4 — new `udp loopback` test
|
|
encodes synthetic frames, packetizes, sends over a real localhost UDP
|
|
socket, depacketizes, and decodes 10/10 frames with correct dimensions.
|
|
Valgrind clean (loopback + codec tests).
|
|
- **Manual validation pending (needs the desktop)**: run the two commands in
|
|
`docs/RUNBOOK.md` — receiver window should show the captured desktop. Tick
|
|
`docs/PHASES.md` Phase 5 after this works.
|
|
- Phase 4 network framing done (RFC 3550 + RFC 6184 single-NAL/FU-A;
|
|
3-byte canonical start codes; drop-on-damage loss handling).
|
|
- Phase 3 capture done and validated (PipeWire/portal backend; the
|
|
`impl_ext_end_proxy` wrong-context warnings were fixed by holding the
|
|
thread-loop lock across all pw proxy operations).
|
|
|
|
## Decisions
|
|
|
|
- Language: C++20 with explicit modern-C++ guidelines in `AGENTS.md` and
|
|
`cpp-meson-build/SKILL.md`.
|
|
- Build system: Meson.
|
|
- Capture: PipeWire + xdg-desktop-portal.
|
|
- Encode/Decode: FFmpeg (libavcodec, libavutil, libswscale).
|
|
- H.264 encoder path: software `libx264`, low-latency settings, Annex-B output.
|
|
- **SPS/PPS are sent in-band ahead of every keyframe** (no GLOBAL_HEADER);
|
|
the decoder starts from the bitstream alone. `DecoderConfig.extradata`
|
|
remains available if signaling ever negotiates parameters out of band.
|
|
- Transport: RTP over UDP via raw POSIX sockets for now; **ASIO stays a
|
|
Phase 6+ option** (the ARCHITECTURE.md dependency table places it with
|
|
signaling/discovery). IPv4 only at the transport level for now.
|
|
- Rendering: **SDL3** (`sdl3` pkg-config, 3.4 installed); plain texture
|
|
upload, no GPU pipeline yet.
|
|
- **Pixel-format convention trap**: FFmpeg names packed formats in memory
|
|
byte order (AV_PIX_FMT_RGBA = R,G,B,A in memory), but SDL names 32-bit
|
|
formats MSB-first (SDL_PIXELFORMAT_RGBA8888 = A,B,G,R in memory).
|
|
FFmpeg RGBA data therefore needs SDL_PIXELFORMAT_ABGR8888 — using
|
|
RGBA8888 paints the alpha byte as red (red-tinted image).
|
|
- Discovery: mDNS/Avahi.
|
|
- Namespace: `sc`.
|
|
- Module error results use per-module `std::variant<T, XError>` types
|
|
(`CodecResult`, `CaptureResult`, `RendererResult`) since C++20 has no
|
|
`std::expected`.
|
|
|
|
## Active blockers
|
|
|
|
None.
|
|
|
|
## Open questions
|
|
|
|
- GUI framework (Qt6 vs. none / CLI only) — deferred to later phase.
|
|
- Hardware acceleration strategy (VAAPI / Vulkan Video / NVENC) — evaluate after
|
|
software encode path works.
|
|
- IPv6 at the transport layer — revisit when LAN streaming lands (Phase 6+).
|
|
|
|
## Forward-looking review notes (for later phases)
|
|
|
|
- `RtpTransport::start/send` return plain bools (scaffold API); error
|
|
messages are lost — consider an error channel when signaling lands.
|
|
- Very high bitrates can still exceed even a raised receive buffer if
|
|
`net.core.rmem_max` is low on the receiver; the encoder VBV bounds
|
|
bursts to ~2 frame periods, so this needs `--bitrate` ≳ 100 Mbps to
|
|
matter. Document in RUNBOOK.
|
|
- DMA-BUF-only portal streams are rejected with a clear message (hardware
|
|
path is Phase 7).
|
|
- No negative-path tests yet (bad config, bad stride, undersized buffer).
|
|
- `to_annex_b_h264` sniffs AVCC vs Annex-B by content; if an AVCC-emitting
|
|
encoder is ever added, prefer an explicit config flag over the heuristic.
|
|
- Region targets are rejected: the desktop portal has no region capture.
|
|
- `CaptureSession::next_frame()` returns `nullopt` on stream error without
|
|
surfacing the reason (logged to stderr).
|
|
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
|
|
we control never emit them, but third-party interop would need support. |