Compare commits

...

21 Commits

Author SHA1 Message Date
fegger c43dd3ca4a feat(ios): native receiver app (Swift, min iOS 17)
A native iOS receiver so an iPhone can act as the second receiver, speaking the existing signaling + RTP protocol (no C++ changes) and mirroring the Android receiver (Phase 8) source-to-source.

- RTP core (header/packet, jitter buffer, H.264 depacketizer) ported from the Android receiver
- BSD-socket signaling server (dual-stack, most-recent-peer, never-throwing sends) + NSBonjourServices
- VideoToolbox H.264 decode (in-band SPS/PPS, real-time, rebuilds on size change) -> AVSampleBufferDisplayLayer
- PLI keyframe recovery (500 ms) + pendingOffer for late surface attach
- XcodeGen project + bootstrap.sh; XCTest port of the Android suite + new coverage
- .gitignore for generated artifacts; CHANGELOG; PHASES + MEMORY updated

Status: authored; on-device validation pending a Mac + Xcode 26 + iPhone 16.
2026-09-10 21:31:53 +02:00
fegger 4f93c7cd20 fix(android): don't letterbox against the configure placeholder
H264Decoder.outputSize() reported the 320x240 configure() placeholder
until the codec parsed the in-band SPS, so fitVideo sized the view to
1488x1116 (1.33 aspect) and the first rendered frame(s) of 2.4 content
were visibly squished before the real size corrected it ~40ms later.
Report no size until INFO_OUTPUT_FORMAT_CHANGED delivers the real one:
the first frame renders into the fullscreen surface and the correct
letterbox follows immediately. Validated live on the Fairphone 6 (the
placeholder 'video size' line is gone from logcat; 25/25 JVM tests).
2026-09-10 12:36:15 +02:00
fegger 9a24933203 fix(android): pass PROTOCOL_DNS_SD so mDNS registration works
Android 16 added NsdManager.checkProtocol(): registerService with protocol
0 threw IllegalArgumentException: Unsupported protocol — silently, since
the old code swallowed the exception into a dead reflection fallback and
start() overwrote the failure status with the Listening line. mDNS never
actually advertised (the Phase 8 session streamed via --peer, masking it).

Also advertise after the listening status so a registration failure stays
visible, and log the exception and the registration success.

Validated live on the Fairphone 6: the desktop --discover lists the phone
and a full --send session decodes (in-band SPS 320x240 -> 2496x1040) and
renders.
2026-09-10 12:35:00 +02:00
fegger c6722d164b fix(android): harden the receiver after the review pass
- SignalingServer.send() never throws: a broken signaling connection
  drops the peer instead of killing the RTP reader thread via requestPli()
- H264Decoder: assign the codec before configure/start (no orphaned
  instance on failure); feed() reports input-queue timeouts so the
  pipeline requests a keyframe instead of dropping the frame silently
- Park offers that arrive without a surface as pendingOffer and configure
  on attachSurface(): a codec configured without a surface can never take
  one (setOutputSurface refuses it); the late configure sends a PLI
- Forget destroyed surfaces (onSurfaceTextureDestroyed -> detachSurface)
- Show the status overlay again on later messages
- Accumulate NAL bytes in ByteArrayOutputStreams, not boxed ArrayList<Int>
- Delete the dead NSD reflection fallback (ResolutionListener never
  existed; the classic API is present from API 16 through 36)
- Hold decoderLock around all MediaCodec calls (not thread-safe) and make
  requestPli thread-safe
- Update RUNBOOK quirks (NSD, setOutputSurface) and MEMORY notes
2026-09-10 12:19:21 +02:00
fegger bbe4f21a3b docs: document the Android receiver app (Phase 8)
README: phone-as-second-screen section (build, install, sender usage,
multi-monitor caveat). RUNBOOK: Android build/test procedure, device-test
commands, and the API-36 platform quirks found during validation.
PHASES: Phase 8 milestones, all validated. MEMORY: current phase state
and the hard-won Android/portal lessons.
2026-09-10 11:38:16 +02:00
fegger 455ba6055d feat(android): native receiver app (Kotlin, minSdk 30)
Phone-as-receiver for the USB-C -> HDMI mirroring use case. Speaks the
existing signaling + RTP wire protocol, so the C++ sender needs no
changes. AGP 9 built-in Kotlin (no separate plugin), no androidx.

- rtp/: RtpHeader, RtpPacket, JitterBuffer (16 pkt / 60 ms, straggler
  discard), H264Depacketizer (single-NAL + FU-A, Annex-B out) — JVM tests
- signaling/: newline-JSON server (offer -> answer, PLI, 500 ms rate cap)
- decode/H264Decoder: MediaCodec -> Surface; size from in-band SPS (offer
  is 0x0); surface passed to configure(); releaseOutputBuffer(render)
- pipeline/ReceiverPipeline: NSD advertise (API-36 RegistrationListener
  + reflection fallback), UDP RTP, jitter, decode, PLI recovery
- ReceiverActivity: fullscreen TextureView letterboxed by sizing the
  view to the video aspect (C2 scales output to the surface)

Validated end-to-end on a Fairphone 6 (API 36): offer/answer, first
frame, letterbox fit, PLI on loss.
2026-09-10 11:31:13 +02:00
fegger 69a1cdac1d feat(app): wire the app icon into the GUI, waybar, and SDL window
screencast_icon/ holds the single source icon set; meson install ships
the 256 px PNG into the hicolor theme for the GTK panel (themed icon
name) and the waybar CSS, and a custom target embeds it as
icon_png_data.h so the SDL receiver window sets it via SDL_SetWindowIcon
with an optional SDL3_image dependency (built-in find_library, since no
pkg-config file ships with it).
2026-09-10 11:30:39 +02:00
fegger 9ff8e94679 docs(readme): document all modes, GUI, resilience, and hotspot 2026-09-09 22:25:22 +02:00
fegger 9024930d91 chore(repo): ignore local hotspot.txt notes file 2026-09-09 22:25:16 +02:00
fegger 144aab6e5b feat(gui): recalibrate presets and add a frame rate spinner
The presets now include a frame rate cap alongside CRF and max
bitrate, reflecting three lessons learned since the CRF fix:

- Frame rate is the biggest bandwidth lever for desktop content:
  15-25 fps is perfectly smooth for screencasting and cuts bandwidth
  2-4x vs. uncapped monitor rate.
- CRF values are now actually applied (the global_quality bug is
  fixed), so the ranges are meaningful.
- The 'Low bandwidth' preset targets 15 fps + CRF 26 + 3000 kbps,
  which fits even the Pi's Wi-Fi hotspot; 'Maximum' uncaps the frame
  rate for the smoothest possible output on a good link.

  Low bandwidth   CRF 26,  3000 kbps, 15 fps
  Balanced        CRF 22,  5000 kbps, 20 fps
  Sharp           CRF 18,  8000 kbps, 25 fps
  Very sharp      CRF 16, 12000 kbps, 30 fps
  Maximum         CRF 14, 20000 kbps, uncapped

A frame rate spinner (5-60, 0 = uncapped) sits alongside the
bitrate slider for independent fine-tuning.
2026-09-09 21:45:25 +02:00
fegger 8f1c2aa868 fix(codec): set CRF via x264's private option, not global_quality
The CRF value was set through AVCodecContext.global_quality with
AV_CODEC_FLAG_QSCALE, which FFmpeg's libx264 wrapper divides by
FF_QP2LAMBDA (118) before passing to x264 — turning CRF 16 into
CRF 0.135 (essentially lossless) while x264 logged '-qscale is
ignored, -crf is recommended' and fell back to its own defaults.
The CRF was never actually applied.

Now the CRF is set as x264's private "crf" option via av_opt_set,
which passes the exact value directly to the encoder. The VBV max
rate still caps bursts as before.
2026-09-09 13:28:09 +02:00
fegger e82e7853d1 feat(app): adaptive quality, frame rate capping, and tighter VBV
Three changes to make the stream survive constrained links:

Adaptive quality: the sender pipeline now tracks the PLI rate from
the receiver. Every 5 seconds it evaluates: >0.5 PLI/s means the
link is saturated (the receiver is dropping frames), so the CRF
increases by 2 (lower quality, fewer bits) and the encoder restarts
with a keyframe. <0.1 PLI/s means the link is stable, so the CRF
decreases by 1 (better quality) and the encoder probes upward.
Clamped to [user CRF, user CRF + 10] so quality never degrades
below what the link can handle, and never exceeds what the user
asked for. The adaptation is logged to stderr for visibility.

Frame rate capping (--fps N): throttles the capture loop to N
frames per second (0 = no cap; monitor rate). At 15fps instead of
60fps, the bandwidth requirement drops 4x at the same quality
level. Desktop content is still smooth at 15-20fps.

Tighter VBV: one frame period of buffer instead of two. A two-frame
buffer lets a keyframe spike to twice the target rate in one burst,
which overflows any constrained hop (Wi-Fi hotspot, slow switch)
and cascades into PLI storms. One frame period keeps bursts
within what the link can absorb in real time.
2026-09-09 13:00:03 +02:00
fegger c78dec139b feat(gui): add a quality preset dropdown to the sender panel
Replaces the raw bitrate slider with a preset dropdown that maps
directly to common CLI invocations:

  Low bandwidth  CRF 28, max  2000 kbps   (screencast --send)
  Standard        CRF 22, max  4000 kbps   (screencast --send)
  Sharp           CRF 18, max  8000 kbps   (--crf 18 --bitrate 8000)
  Very sharp      CRF 16, max 12000 kbps   (--crf 16 --bitrate 12000)
  Maximum         CRF 14, max 20000 kbps   (--crf 14 --bitrate 20000)

The bitrate slider remains for fine-tuning the cap independently of
the preset's quality target (CRF). Selecting a preset sets both; the
label shows the current values.
2026-09-09 12:45:14 +02:00
fegger 36d086af4e perf(codec): CRF rate control, longer GOP, faster preset, screen tuning
Four encoder quality improvements, all sender-side:

- CRF rate control (default 22, --crf to override): targets a
  constant visual quality level instead of a fixed bitrate. Static
  desktop content uses 300-800 kbps (vs. forced 4000+), and the saved
  bits go to sharp text and clean motion when they appear. The VBV
  max rate (the --bitrate value, now a cap rather than a target)
  bounds bursts so the receiver's UDP buffers stay safe. Round-trip
  test bitrate dropped from 1390 kb/s to 47 kb/s on synthetic frames
  — the encoder uses only what it needs.

- 5-second GOP (was 1 second): 80% fewer keyframe bits freed for
  detail frames. Screen content changes incrementally, not
  wholesale; PLI feedback recovers from loss in one frame time
  regardless of GOP length.

- faster preset (was veryfast): better sub-pixel estimation and
  RDO on more decisions. The desktop handles it trivially at 1080p.

- Screen-content x264 tuning: aq-mode=2 (auto-variance AQ moves
  bits away from flat areas toward text edges) and psy-rd=1.5
  (preserves texture sharpness).

Combined with the earlier veryfast upgrade and sender-side
downscaling, this is roughly 2x the perceived quality at the same
average bandwidth compared to the original ultrafast ABR encoder.

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 12:12:36 +02:00
fegger 30538fba73 perf(codec): upgrade x264 preset and downscale to the receiver display
Two quality improvements:

Preset: ultrafast -> veryfast. Unlocks Main profile with CABAC
entropy coding, hexagonal motion search, 3 reference frames, and
adaptive quantization — typically 30-40% better quality at the same
bitrate. The desktop handles the extra encoding cost trivially
(150+ fps at 1080p).

Downscaling: the receiver now advertises its display resolution in
the signaling answer (display_width/display_height, 0 = unknown).
When the capture exceeds the display (e.g. 2256x1504 source on a
1920x1080 receiver), the sender scales down preserving aspect ratio
before encoding — the same sws_scale pass that already converts the
pixel format also handles the resolution change, so there is no
extra step. This concentrates the entire bitrate into pixels the
display actually shows (~2.7x more bits per visible pixel at
4000 kbps when going from 2256x1504 to 1620x1080).

The renderer caches the display size during window creation
(native monitor resolution in fullscreen/KMSDRM; window size
otherwise). The receiver includes it in every signaling answer; the
sender pipeline computes aspect-preserving, even-rounded scaled
dimensions when the display is smaller than the capture.

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 11:02:42 +02:00
fegger 6516b45b02 perf(codec): pass YUV through to the renderer and use slice threading
The receiver decoded H.264 to YUV420P, converted it to RGBA via a
CPU-intensive swscale pass, then uploaded 4 bytes/pixel to an SDL
texture — only for the GPU to convert back to RGB during rendering.
This eliminated the swscale pass entirely (40-60% of receiver CPU at
1080p) and cut the texture upload by 62%.

- DecodedFrame now carries three YUV420P planes with their strides
  instead of a packed RGBA buffer; the decoder copies the planes
  directly from the AVFrame (zero conversion for the common software
  path). Non-YUV420P decoder output (e.g. NV12 from v4l2m2m) is
  converted once to YUV420P.
- The SDL renderer uploads via SDL_UpdateYUVTexture with
  SDL_PIXELFORMAT_IYUV; the GPU does the YUV→RGB conversion during
  rendering.
- Decoder threading: slice-level with 4 threads (parallelizes within a
  frame, no added latency), not frame-level (which buffers multiple
  frames — the initial thread_count=0 broke the loopback test because
  the H.264 decoder introduced a multi-frame delay before producing
  output).
- The round-trip test converts decoded YUV back to RGBA for pixel
  comparison via a test-local swscale call (the pipeline itself never
  converts).

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 09:45:22 +02:00
fegger b4d1411fd9 fix(build): make receiver-only builds link again after the GUI refactor
The sc_app_core static library was always compiled with SC_HAS_SENDER=1,
pulling CaptureFactory into receiver-only builds where the capture
backend does not exist. Three fixes:

- sc_app_core's cpp_args and capture dependency now follow the sender
  meson option, so pipelines.cpp and sender_session.cpp compile their
  sender code out on receiver-only targets.
- sender_session.cpp is fully guarded by SC_HAS_SENDER; its body is
  sender pipeline orchestration and has no business in a receiver.
- address_preference moved from a SenderSession static to a free inline
  function in sender_session.h — run_discover (available in every
  build) uses it for address sorting.

Also: the root meson.build now errors early if gui=true without
sender=true (the GUI is a sender panel).

Verified: meson test 5/5 in both configurations; the receiver-only
binary refuses --send with a clear message; the full build's sender,
GUI, and tests are unchanged.
2026-09-08 22:17:21 +02:00
fegger eb86905e67 fix(app): build waybar output with nlohmann/json, not format strings
The hand-rolled std::format strings embedded literal newline and
Unicode characters via C++ universal character name escapes (\u23f8,
\u2014, \u25b6, \n), which the compiler converts to actual control
characters in the output. Raw newlines inside JSON strings are
invalid, so waybar's parser failed and displayed the raw JSON text
instead of the widget. Build the status line with nlohmann::json,
which escapes everything correctly.
2026-09-08 20:51:24 +02:00
fegger 5c39662cc2 feat(gui): add a GTK sender panel and a waybar widget
A gtkmm-4.0 control panel (behind -Dgui=true, default off): refresh
shows discovered receivers (grouped and preference-sorted), a bitrate
scale, and start/stop that runs the whole session on a worker thread
so the interactive portal picker never blocks the UI. The CLI and the
GUI now share the new sc_app_core static library holding the
pipelines, session orchestration (negotiation + PLI feedback), and a
state store.

The sender pipeline publishes its state to
$XDG_RUNTIME_DIR/screencast/sender.json (session id, receiver,
bitrate, pid, start time; stale files detected by pid liveness) and
persists the last session for one-click restarts. The new
'screencast waybar' subcommand prints a waybar module line and its
--toggle flag stops a running sender gracefully or spawns a detached
restart of the last receiver.

Waybar on the dev machine is wired: custom/screencast module with
click-to-toggle and right-click panel, plus styles, with a timestamped
backup of both config files. Both binaries are installed to
/usr/local/bin.

Validated: waybar output (idle and streaming states with a synthetic
state file), GUI launches on the desktop (window observed via
hyprctl), meson test 5/5 in both build configurations, formatting
clean.
2026-09-08 17:14:44 +02:00
fegger 74b3f04082 docs: record Phase 7 resilience state and re-scope 2026-09-08 16:54:09 +02:00
fegger 943596da6d feat(app): add PLI feedback, jitter reordering, and hardware decode
Loss recovery for the streaming path:

- PLI over signaling: the depacketizer now reports damaged frames
  (DepacketizeResult) and the receiver asks the sender for a keyframe
  (SessionPli, rate-limited to one per 500 ms). The sender keeps the
  signaling channel open during the session and honors PLIs through
  the new thread-safe SenderPipeline::request_keyframe(). Recovery
  takes one frame time instead of waiting out the GOP.
- RtpJitterBuffer: reorders RTP packets by sequence number (16 packets
  / 60 ms) before the in-order depacketizer, so Wi-Fi reordering is
  not misread as loss; in-order streams release immediately, and a
  straggler older than the delivered sequence is discarded.
- Hardware H.264 decode probe: DecoderFactory tries h264_v4l2m2m (the
  VideoCore path on the Pi) with an automatic software fallback and a
  clear journal line for the chosen path; --swdecode opts out.

Validated: PLI end-to-end with a probe that drops a mid-keyframe
packet over real UDP (receiver logged the damaged frame and the PLI
arrived with the session id); hardware probe fails cleanly and falls
back on this desktop; jitter reordering covered by unit tests.
meson test 5/5 in both build configurations, valgrind clean.
2026-09-08 16:54:09 +02:00
102 changed files with 6341 additions and 291 deletions
+210 -4
View File
@@ -1,10 +1,82 @@
# Project Memory — screen_cast
Last updated: Phase 6 (discovery + signaling) complete and validated over
loopback; current phase is Phase 7.
Last updated: Phase 8 (Android receiver app) complete and validated on a
Fairphone 6; all prior phases done. The Android app's review findings were
fixed in a follow-up pass (same day) — see "Android app review" at the
bottom of this file.
## Project state
- **Phase 9 in progress: iOS receiver app** (`ios/`, Swift, min iOS 17):
iPhone as a second receiver. Speaks the same signaling+RTP protocol — no C++
changes. Mirrors the Android receiver (Phase 8) source-to-source.
- Scaffolding: XcodeGen `project.yml` (Info.plist carries
`NSLocalNetworkUsageDescription` + `NSBonjourServices: _screencast._tcp`) +
`bootstrap.sh` (downloads XcodeGen from the GitHub release, no Homebrew;
generates the project; builds/tests via xcodebuild). `ios/README.md` has
build/install/device steps.
- App (`Receiver/App`): SwiftUI + `AVSampleBufferDisplayLayer` (`.resizeAspect`
= letterbox — the same "size the surface, not a transform" lesson as
Android); `ReceiverController` (ObservableObject) drives start/stop on
scenePhase.
- Protocol core (`Receiver/Rtp`, `Receiver/Signaling`): RtpHeader/RtpPacket/
JitterBuffer/H264Depacketizer ported source-to-source; SignalingMessage
(JSONSerialization) + LineAssembler + SignalingServer (BSD sockets,
dual-stack, most-recent-peer-wins, MSG_NOSIGNAL sends that never throw);
`AvccConverter` (Annex-B ↔ AVCC) + `NalExtractor` (SPS/PPS) are new for the
VideoToolbox path.
- Decode (`Receiver/Decode`): `H264FormatDescription` (Core Foundation H.264
config recipe) + `H264VideoToolboxDecoder` (in-band SPS/PPS → session;
`kVTDecompressionPropertyKey_RealTime`; session recreated on size change;
the output callback may run on a worker thread, so state is lock-guarded)
`AVSampleBufferRenderSink` (AVSampleBufferDisplayLayerSession).
- Support (`Receiver/Support`): `UdpTransport` (poll-based recv so close() can't
strand a blocked recvfrom) + `LocalAddress` (getifaddrs for the --peer hint).
- Pipeline (`Receiver/Pipeline`): ReceiverPipeline — one serial queue for all
state + decode; reader thread only polls/receives UDP; pendingOffer for late
surface attach (+ PLI); PLI rate-limited 500 ms; first-frame flag; video
size tracked (never reports a placeholder before the first real keyframe —
the Android startup-squish lesson).
- Tests (`ios/ReceiverTests`): the 25 Android JVM tests ported to XCTest plus
new coverage for signaling JSON, line framing, AVCC, NAL extraction.
- `.gitignore` excludes the generated `ios/Receiver.xcodeproj/`, `ios/tools/`,
`ios/Receiver/Info.plist` (the project.yml `info` block is the source of
truth).
- **NOT YET VALIDATED** (this box is Linux, no Xcode/iOS SDK): nothing here
compiles or runs. 9.5 needs a Mac with Xcode 26 + the iPhone 16.
Highest-risk on-device items: (1) local-network + Bonjour consent
(NSBonjourServices must be `_screencast._tcp`; iOS 26 tightened the prompt),
(2) the H264FormatDescription CF ownership recipe (a bug crashes on the
first keyframe — loud, not silent), (3) VideoToolbox decode →
AVSampleBufferDisplayLayer render. Run `ios/bootstrap.sh test` first on the
Mac.
- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30,
app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via
USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++
changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`;
Kotlin targets `compileOptions`, Java 17); no androidx (framework +
org.json + JUnit). 25 JVM tests green. Validated end-to-end on a
Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed
render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in
RUNBOOK): `INTERNET` permission is REQUIRED for NsdService;
`DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is
Inet6Address, not InetSocketAddress); pass the Surface to
`MediaCodec.configure()` + `start()`; render via
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
configure (in-band SPS reconfigures); `MediaFormat.format()`/
`KEY_MIME_TYPE` not public in API 36; NSD: the classic
`registerService(info, flags, RegistrationListener)` API exists from API
16 through 36 (javap-verified on the android-36 SDK) — an old reflection
fallback targeting a never-existent "ResolutionListener" was removed as
dead code;
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
by sizing the TextureView to the video aspect, NOT a transform matrix
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
picks the FIRST output (eDP-1, wrong display) — use `--target window`
(hyprland-share-picker). Activity FQN for `am start`:
`screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package =
namespace `screen_cast`).
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
announces its signaling port via a threaded-poll Avahi client; senders
browse+resolve) plus JSON session signaling (offer/answer) over TCP with
@@ -65,12 +137,55 @@ loopback; current phase is Phase 7.
before the fix). Cross-compiling on the dev machine was considered and
dropped — the on-Pi build works and the toolchain/container effort was
not needed.
- **Pi Wi-Fi hotspot** (`scripts/pi-hotspot.sh on|off|status`): NetworkManager
- **App icons wired up** (`screencast_icon/`): single 256 px PNG used
everywhere. `meson install` ships it to
`share/icons/hicolor/256x256/apps/screencast.png`; the GTK panel sets it
via `set_icon_name("screencast")` (GTK4 removed the pixel-buffer window
icon — themed names only; `gtk_icon_theme_has_icon` verified OK); the
SDL receiver window embeds the PNG at build time (`icon_png_data.h` via
`scripts/icon_to_header.py` custom_target) and `SDL_SetWindowIcon` with
an optional SDL3_image dependency (no pkg-config ships with it →
`meson.get_compiler('cpp').find_library('SDL3_image', required: false)`
+ disabler; `-DSC_HAS_WINDOW_ICON` gate; Pi builds without it still
work). The waybar widget now shows the icon as `background-image` in
`~/.config/waybar/style.css` (▶/⏸ text hidden; `.idle` dimmed,
`.streaming` green tint) instead of text glyphs; backup
`style.css.bak-20260909-*`. Lesson: icons appear in compositor
taskbars/switchers, not title bars (GTK4 CSD and Hyprland decorations
don't draw them), so validate via theme lookup, not title-bar
screenshots. `sudo pacman -S sdl3_image` done (extra/sdl3_image 3.4.6).
- **GTK panel + waybar widget**: `screencast-gui` (gtkmm-4.0, behind
`-Dgui=true`; app internals now live in the `sc_app_core` static lib so
CLI and GUI share pipelines/session/state). `screencast waybar [--toggle]`
prints the waybar module line and toggles streaming via SIGTERM (stop) or
a detached re-exec spawn from `last-session.json` (start). State lives in
`$XDG_RUNTIME_DIR/screencast/sender.json` (written by the pipeline with
session id, receiver, bitrate, pid, start time; stale files are detected
by pid-liveness). The waybar config was wired into the user's bar
(`custom/screencast` before `custom/timetrack`, with backup) and the
binaries installed to /usr/local/bin.
- **Pi Wi-Fi hotspot** (`scripts/pi-hotspot.sh on|off|status`): NetworkManager
AP mode (WPA2, ipv4 shared → built-in DHCP/NAT, Pi at 10.42.0.1). Takes
over wlan0 while active; generated PSK stored in /etc/screencast-hotspot.conf.
No application changes needed — the receiver already announces on all
interfaces. nmcli property syntax validated against NM 1.58 with a
disposable profile; AP bring-up itself can only be validated on the Pi.
- **Phase 7 resilience shipped** (pending Pi-side hw-decode run):
- PLI over signaling: `SessionPli` message; the depacketizer now returns
`DepacketizeResult{access_unit, frame_dropped}`; the receiver
rate-limits PLIs to 1/500 ms; the sender keeps the signaling channel
open and calls `SenderPipeline::request_keyframe()` (thread-safe
atomic → run-loop → encoder). Validated end-to-end with a probe that
drops a mid-keyframe packet: receiver logs "frame damaged",
PLI arrives with the session id.
- `RtpJitterBuffer` (rtp_packet.h): sequence reordering, 16 pkt/60 ms,
straggler discard via serial-number arithmetic, overflow flush for
genuine loss. Zero added latency on in-order streams.
- Hardware decode probe in DecoderFactory (`DecoderConfig.hardware_accel`,
default true): h264_v4l2m2m first, software fallback, `--swdecode` opts
out. On the desktop the probe fails cleanly ("Could not find a valid
device") and falls back; on the Pi it should pick the VideoCore m2m
device — NEEDS THE USER'S PI RUN to confirm.
- **REAL-HARDWARE VALIDATION (desktop → Pi Zero 2 W over Wi-Fi)**: the full
chain works on two machines: mDNS discovery → signaling negotiation →
RTP over Wi-Fi → software H.264 decode → fullscreen KMSDRM letterboxed
@@ -192,4 +307,95 @@ None.
- `CaptureSession::next_frame()` returns `nullopt` on stream error without
surfacing the reason (logged to stderr).
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
we control never emit them, but third-party interop would need support.
we control never emit them, but third-party interop would need support.
### Android app review (2026-09-10) — findings FIXED same day
Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified
faithful to the C++ side source-to-source), followed by a fix pass that
landed all findings. No commit yet (user has not asked).
Fixed:
- `SignalingServer.send()` no longer throws (mirrors the C++ server, which
ignores write failures): a broken signaling TCP no longer kills the
`rtp-reader` thread via `requestPli()`; `peerOut` is dropped (closing the
socket) on write failure.
- `H264Decoder.configure()` assigns the created codec before configuring,
so a configure/start failure can no longer orphan the MediaCodec instance
(no finalizer; scarce native slots).
- Offer-before-surface no longer configures a ByteBuffer-mode decoder: the
offer is parked as `pendingOffer` and `attachSurface()` configures later
(a surface-less codec can never take one — `setOutputSurface` refuses
it, an IllegalStateException crash on the main thread). The late
configure sends a PLI (the sender only emits IDRs when asked).
- `ReceiverActivity.onSurfaceTextureDestroyed``pipeline.detachSurface()`;
the pipeline/decoder forget dead surfaces instead of configuring against
them (frames decode unrendered until the next attach).
- The status overlay reappears: `onStatus` sets `visibility = VISIBLE`
(it used to write into a GONE view after the first frame).
- `H264Decoder.feed()` returns false on input-queue timeout → the pipeline
requests a PLI instead of silently dropping the frame (no flush — the
codec is healthy).
- The depacketizer accumulates into `ByteArrayOutputStream`s instead of
boxed `ArrayList<Int>` (was ~MB/s of Integer allocations on keyframes).
- NSD reflection fallback deleted (dead code — see the quirk note above).
- Thread-safety: all codec calls now run under `decoderLock` (MediaCodec is
not thread-safe; feed/drain previously raced attachSurface);
`requestPli()` is thread-safe via `pliLock`.
Validation: `gradle :app:assembleDebug :app:testDebugUnitTest` green
(25/25, full --rerun-tasks rebuild, only two pre-existing warnings);
`meson test` 5/5 unchanged (no C++ touched).
### Discovery bug found and fixed on-device (2026-09-10, same day)
The user reported the desktop sender never discovered the phone.
Root cause (found live on the Fairphone 6): `registerService(info, 0,
listener)` — Android 16's `NsdManager.checkProtocol()` rejects protocol
`0` with `IllegalArgumentException: Unsupported protocol`. The old code
swallowed it into the dead reflection fallback, and `start()` posted the
"Listening…" status AFTER advertiseNsd, overwriting the failure text — so
mDNS never advertised and Phase 8's NSD validation was only ever "no
crash" (8.5 streamed via --peer, masking it).
Fixes in `ReceiverPipeline.advertiseNsd()`/`start()`:
- pass `NsdManager.PROTOCOL_DNS_SD`;
- advertise AFTER the listening status so a failure stays visible;
- `Log.e` the registration exception; `Log.i` on registered success.
On-device validation (adb, live): `mDNS registered:
screencast._screencast._tcp` in logcat; `dumpsys servicediscovery` shows
the active Advertiser (key diagnostic: `mClientRequests` empty == no
request ever issued); desktop `avahi-browse` and `screencast --discover`
list the phone at 192.168.178.29:5005; a 25s `--send --target monitor
--peer 192.168.178.29:5005` session decoded (in-band SPS reconfigured
320x240 → 2496x1040) and rendered (screencap mean brightness 0.51).
Note: with both the Pi and the phone on the LAN, plain `--send` refuses
(two receivers found) — target the phone with `--peer`.
Follow-up from the same on-device session — startup squish eliminated:
`H264Decoder.outputSize()` previously reported the 320x240 configure()
placeholder until the codec parsed the SPS, so `fitVideo` sized the
TextureView 1488x1116 (1.33 aspect) and the first rendered frame(s) of
2.4 content were visibly squished. It now returns null until
`INFO_OUTPUT_FORMAT_CHANGED` fires; the first frame renders into the
fullscreen surface and the correct letterbox (2484x1035) follows within
~40ms. Validated live: no placeholder "video size" line in logcat.
Commits: 9a24933 (PROTOCOL_DNS_SD discovery fix + docs), plus the
outputSize fix (see git log).
Still open (accepted, needs a device or a new test dep):
- Untested on device: `setOutputSurface` mid-session (surface switch) and
the whole pendingOffer path; API-35 `detachOutputSurface()` could replace
the render-flag approach.
- No JVM tests for `SignalingMessage`/`SignalingServer` (would need the
`org.json:json` test dependency; android.jar stubs throw).
- Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200),
RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a
future phase).
- Verified NOT a bug: reusing one `DatagramPacket` without resetting its
length — modern JVMs recv by buffer capacity (JDK-21 probe + the
successful on-device streaming confirm it).
+6
View File
@@ -19,6 +19,11 @@ compile_commands.json
# Smoke test output
*.h264
# iOS (generated by bootstrap.sh / XcodeGen; project.yml is the source of truth)
ios/Receiver.xcodeproj/
ios/tools/
ios/Receiver/Info.plist
# IDE
.vscode/
.idea/
@@ -29,3 +34,4 @@ compile_commands.json
# OS
.DS_Store
Thumbs.db
hotspot.txt
+27
View File
@@ -0,0 +1,27 @@
# Changelog
Notable changes to `screen_cast`. Loosely follows [Keep a Changelog].
## [Unreleased]
### Added
- **Phase 9 — iOS receiver app** (`ios/`): a native Swift receiver so an iPhone
can act as the second receiver. Speaks the existing signaling + RTP protocol
(no C++ changes), mirroring the Android receiver (Phase 8) source-to-source.
- RTP core (header/packet, jitter buffer, H.264 depacketizer) ported
source-to-source from the Android receiver.
- BSD-socket signaling server (dual-stack, most-recent-peer, never-throwing
sends) + `NSBonjourServices` advertisement.
- VideoToolbox H.264 decode (in-band SPS/PPS, real-time, session rebuilt on
resolution change) rendered via `AVSampleBufferDisplayLayer` (letterbox).
- PLI keyframe recovery (rate-limited 500 ms) and `pendingOffer` for late
surface attach; no placeholder sizing before the first real keyframe.
- XcodeGen project + `bootstrap.sh`; XCTest port of the Android suite plus new
signaling / line-framing / AVCC / NAL-extraction coverage.
> **Status:** authored; on-device validation pending a Mac + Xcode 26 + iPhone 16
> (local-network/Bonjour consent, the H.264 format-description recipe, and the
> VideoToolbox render path are the on-device verification items).
[Keep a Changelog]: https://keepachangelog.com/en/1.1.0/
+124 -10
View File
@@ -3,9 +3,10 @@
A native Linux peer-to-peer screencast application.
- **Send** your desktop or a window to another Linux machine.
- **Receive** a stream and render it in a window.
- **Receive** a stream and render it in a window — or fullscreen headless.
- Discover receivers on the LAN via **mDNS/Avahi**; negotiate sessions with
**JSON signaling**; stream **H.264 over RTP/UDP**.
- Control it from the **CLI**, a **GTK4 panel**, or a **waybar widget**.
Built with **C++20**, **Meson**, **PipeWire**, **FFmpeg**, and **SDL3**.
@@ -19,6 +20,7 @@ Requirements:
- SDL3 development package (`sdl3`)
- nlohmann JSON (`nlohmann_json`) and Avahi client (`avahi-client`)
- For the sender only: PipeWire dev (`libpipewire-0.3`) and libportal
- Optional, for the GUI: gtkmm-4.0
Build and run tests:
@@ -32,11 +34,72 @@ Stream between two machines:
```sh
screencast --receive # machine A: announces itself, opens a window
screencast --send # machine B: discovers A, negotiates, streams
screencast --send # machine B: discovers A, negotiates, streams
```
See `docs/RUNBOOK.md` for all modes, flags, and validation procedures.
## Modes and flags
```text
screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate MAX_KBPS] [--crf 0-51] [--fps 1-60]
screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen] [--swdecode]
screencast --discover [--timeout SECONDS]
screencast waybar [--toggle] # for waybar widgets
```
| Mode | Flag | Default | Meaning |
|-------------|------------------|---------------|------------------------------------------------|
| `--send` | `--target` | `monitor` | `monitor` or `window` (portal source picker) |
| | `--peer` | auto-discover | target a receiver directly, `HOST[:PORT]` |
| | `--bitrate` | `4000` | VBV max bitrate (kbps) |
| | `--crf` | `22` | encoder quality (lower = better) |
| | `--fps` | capture rate | frame-rate cap, 1-60 |
| `--receive` | `--port` | `5004` | local RTP UDP port |
| | `--signaling-port` | `5005` | TCP signaling port announced via mDNS |
| | `--fullscreen` | off | force fullscreen (automatic under KMSDRM) |
| | `--swdecode` | off | skip the hardware decode probe |
| `--discover`| `--timeout` | `3` | seconds to wait for mDNS responses |
| `waybar` | `--toggle` | print status | toggle streaming (stop / restart last session) |
`--send` without `--peer` discovers receivers on the LAN and requires that
exactly one is found; use `--discover` to list them.
## GUI and waybar widget
Build the GUI alongside the CLI (`meson configure build -Dgui=true`, then
recompile; installs as `screencast-gui`):
```sh
screencast-gui # receiver list → pick one → bitrate → Start
screencast waybar # one JSON line for a waybar custom module
screencast waybar --toggle
```
The waybar module shows the app icon (dimmed while idle, highlighted while
streaming; tooltip: receiver, bitrate, elapsed); left-click toggles
streaming to the last receiver, right-click opens the panel. All front-ends
agree on state because the sender publishes it to
`$XDG_RUNTIME_DIR/screencast/sender.json`. The icon (from `screencast_icon/`)
is installed into the hicolor theme by `meson install`, used by the GTK
panel and the waybar CSS, and embedded in the SDL receiver window.
## Under the hood: resilience
The receiver absorbs loss in two stages and recovers actively:
1. **Jitter window**: RTP packets are re-ordered by sequence number in a
small buffer (16 packets / 60 ms), so Wi-Fi reordering is not misread as
loss. In-order streams release immediately (zero added latency).
2. **PLI feedback**: when a frame arrives genuinely damaged, the receiver
drops it and asks the sender for a keyframe over the signaling channel
(rate-limited to one request per 500 ms); the sender re-encodes a
keyframe immediately, so recovery takes one frame time.
**Hardware decode**: the receiver probes `h264_v4l2m2m` (the VideoCore path
on Raspberry Pi) and falls back to software automatically; `--swdecode`
forces software.
## Receiver on a small ARM board (e.g. Raspberry Pi Zero 2 W)
The receiver does not need the sender's PipeWire/portal capture stack. On
@@ -53,13 +116,60 @@ The script installs the dependencies, builds a receiver-only binary
or newer (GCC 13+ for C++20 `<format>`), builds SDL3 from source when the
distribution does not package it, and enables `avahi-daemon`. It also
installs and enables a systemd service so the receiver starts at boot —
`systemctl status screencast-receiver` to check on it.
`systemctl status screencast-receiver` to check on it (stop with
`sudo systemctl stop screencast-receiver`; always SIGTERM, never `kill -9`,
or stale mDNS records linger).
Performance note: decoding is software H.264; on very small boards expect
smooth playback for modest resolutions and reduced frame rates at high
resolutions. Hardware decode is planned for Phase 7. On a headless console
the receiver runs fullscreen automatically with aspect-preserving
letterboxing.
On a headless console the receiver runs fullscreen automatically with
aspect-preserving letterboxing — no window manager needed (SDL3 KMSDRM; it
renders on its own VT, tty7, so `Ctrl+Alt+F1` returns to the console).
See `docs/RUNBOOK.md` for the headless setup details.
## Direct link: receiver as a Wi-Fi hotspot
The receiver can act as a Wi-Fi access point, so a sender connects to the
board directly with no router in between:
```sh
sudo scripts/pi-hotspot.sh on # prints SSID + generated password
sudo scripts/pi-hotspot.sh status # shows the saved credentials
sudo scripts/pi-hotspot.sh off # back to normal router Wi-Fi
```
While active the Pi is reachable at `10.42.0.1`; on the sender, join the
hotspot's Wi-Fi and run `screencast --send`.
## Android receiver: a phone as the second screen
A native receiver app (Kotlin, minSdk 30) lets a phone act as the second
receiver — and with a USB-C DisplayPort-alt-mode cable, the phone screen
mirrors straight to HDMI. The app speaks the same signaling + RTP protocol,
so the sender needs no changes.
```sh
cd android
gradle :app:assembleDebug # or :app:installDebug with a device attached
adb install app/build/outputs/apk/debug/app-debug.apk
```
Launch the app (it advertises `_screencast._tcp` and shows its IP + ports),
then on the sender:
```sh
screencast --send --peer <phone-ip>:5005
```
Notes:
- The sender downscales to the phone's display resolution automatically
(from the signaling answer).
- The phone renders letterboxed (fit-within), fullscreen, screen kept on.
- The app needs only the `INTERNET` permission; no camera/location.
- mDNS discovery works on the same L2 segment; across subnets use `--peer`
(the app prints its IP and the exact fallback command).
- `--target monitor` captures the portal's default output — on multi-monitor
Hyprland/GTK-portal setups that may not be the one you want; use
`--target window` and pick a window on the target display.
## Architecture
@@ -69,8 +179,12 @@ See `docs/ARCHITECTURE.md` for module boundaries and design rules.
Development is split into phases in `docs/PHASES.md`.
Current phase: **Phase 7Resilience and polish**.
Current phase: **Phase 8Android receiver app** (complete): the Kotlin
receiver app streams to a phone (validated on a Fairphone 6, phone → HDMI
via USB-C DP-alt-mode). Phase 7 items (PLI, jitter, hardware decode,
GUI/waybar) are complete and validated; deferred items remain the VAAPI
hardware encode probe and packaging.
## License
MIT — see `LICENSE` (to be added).
MIT — see `LICENSE` (to be added).
+7
View File
@@ -0,0 +1,7 @@
.gradle/
.kotlin/
build/
local.properties
.idea/
captures/
.cxx/
+32
View File
@@ -0,0 +1,32 @@
plugins {
id("com.android.application")
}
android {
namespace = "screen_cast"
compileSdk = 36
defaultConfig {
applicationId = "screen_cast.receiver"
minSdk = 30
targetSdk = 36
versionCode = 1
versionName = "0.1.0"
}
// No release signing configured: the app is sideloaded as a debug build.
buildTypes {
release {
isMinifyEnabled = false
}
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
}
dependencies {
testImplementation("junit:junit:4.13.2")
}
+28
View File
@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<!-- NsdService and all socket I/O require INTERNET. No other permissions:
no camera, no location, no network state. The screen is kept on, which
keeps Wi-Fi up. -->
<uses-permission android:name="android.permission.INTERNET" />
<application
android:label="screencast"
android:icon="@mipmap/ic_launcher"
android:allowBackup="false"
android:theme="@style/Theme.Screencast">
<activity
android:name=".ReceiverActivity"
android:exported="true"
android:launchMode="singleTask"
android:screenOrientation="sensorLandscape"
android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|keyboard|keyboardHidden|navigation|uiMode|density"
android:stateNotNeeded="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
@@ -0,0 +1,164 @@
package screen_cast
import android.app.Activity
import android.graphics.SurfaceTexture
import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.view.Surface
import android.view.TextureView
import android.view.View
import android.view.WindowManager
import android.widget.TextView
import java.net.Inet4Address
import java.net.NetworkInterface
import kotlin.math.min
import screen_cast.pipeline.ReceiverPipeline
/**
* Fullscreen receiver: a letterboxed TextureView plus a status overlay.
* The pipeline lives in the activity lifetime — created on demand, stopped
* when the app leaves the foreground, released on destroy.
*
* For the USB-C → HDMI use case the app only guarantees the screen is on,
* undimmed, landscape, and immersive; the display mirroring itself is the
* OS behavior of a DisplayPort-alt-mode port.
*/
class ReceiverActivity : Activity() {
companion object {
private const val TAG = "ReceiverActivity"
}
private val ui = Handler(Looper.getMainLooper())
private var pipeline: ReceiverPipeline? = null
private lateinit var videoView: TextureView
private lateinit var statusView: TextView
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_receiver)
videoView = findViewById(R.id.video)
statusView = findViewById(R.id.status)
videoView.isOpaque = true
videoView.surfaceTextureListener = object : TextureView.SurfaceTextureListener {
override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) {
pipeline?.attachSurface(Surface(surface))
fitVideo()
}
override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) {
fitVideo()
}
override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
// The TextureView is about to release this SurfaceTexture; the
// pipeline must forget it or a later offer would configure the
// decoder against a dead surface.
pipeline?.detachSurface()
return true
}
override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
}
}
override fun onResume() {
super.onResume()
applyFullscreen()
ensurePipeline()
pipeline?.start()
}
override fun onPause() {
super.onPause()
pipeline?.stop()
}
override fun onDestroy() {
super.onDestroy()
pipeline?.stop()
pipeline = null
}
private fun ensurePipeline() {
if (pipeline != null) return
val pipeline = ReceiverPipeline(
context = applicationContext,
localIp = localIpv4(),
displaySize = {
val metrics = resources.displayMetrics
metrics.widthPixels to metrics.heightPixels
},
onStatus = { text ->
ui.post {
statusView.text = text
statusView.visibility = View.VISIBLE
}
},
onFirstFrame = { ui.post { statusView.visibility = View.GONE } },
onVideoSize = { _, _ -> ui.post { fitVideo() } },
)
// The surface may already exist by the time the pipeline starts.
val surfaceTexture = videoView.surfaceTexture
if (surfaceTexture != null) {
pipeline.attachSurface(Surface(surfaceTexture))
}
this.pipeline = pipeline
}
/**
* Fit-within (letterbox) by sizing the TextureView to the video's aspect
* ratio, centered on the black window background. The C2 decoder scales
* its output to the Surface (android._video-scaling), so a surface with
* the video's aspect ratio renders 1:1 without distortion; a transform
* matrix on top would double-scale the already-stretched buffer.
*/
private fun fitVideo() {
val (videoWidth, videoHeight) = pipeline?.videoSize() ?: return
val parent = videoView.parent as? View ?: return
val parentWidth = parent.width
val parentHeight = parent.height
if (videoWidth <= 0 || videoHeight <= 0 || parentWidth <= 0 || parentHeight <= 0) return
val scale = min(parentWidth.toFloat() / videoWidth, parentHeight.toFloat() / videoHeight)
val width = (videoWidth * scale).toInt()
val height = (videoHeight * scale).toInt()
val params = videoView.layoutParams
if (params.width == width && params.height == height) return
params.width = width
params.height = height
if (params is android.widget.FrameLayout.LayoutParams) {
params.gravity = android.view.Gravity.CENTER
}
videoView.layoutParams = params
android.util.Log.i(TAG, "fitVideo: ${width}x$height in ${parentWidth}x$parentHeight")
}
private fun applyFullscreen() {
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
val controller = window.insetsController ?: return
controller.systemBarsBehavior = android.view.WindowInsetsController.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE
controller.hide(
android.view.WindowInsets.Type.statusBars() or android.view.WindowInsets.Type.navigationBars(),
)
}
private fun localIpv4(): String {
return try {
val interfaces = NetworkInterface.getNetworkInterfaces() ?: return "unknown"
for (iface in interfaces) {
if (!iface.isUp || iface.isLoopback) continue
for (address in iface.inetAddresses) {
if (address is Inet4Address && !address.isLoopbackAddress) {
return address.hostAddress ?: "unknown"
}
}
}
"unknown"
} catch (e: Exception) {
"unknown"
}
}
}
@@ -0,0 +1,179 @@
package screen_cast.decode
import android.media.MediaCodec
import android.media.MediaFormat
import android.view.Surface
/**
* MediaCodec H.264 decoder that renders directly onto a Surface (no CPU
* pixels). The stream is self-describing: the sender repeats SPS/PPS
* in-band at every keyframe, so no out-of-band codec data is needed.
*/
class H264Decoder {
companion object {
// The C++ sender's VBV bounds keyframes to ~2 frame periods of bytes;
// 8 MiB is far beyond anything the negotiated rates can produce.
private const val MAX_INPUT_SIZE = 8 * 1024 * 1024
}
private var codec: MediaCodec? = null
private var configured = false
// True once the codec parsed the in-band SPS (INFO_OUTPUT_FORMAT_CHANGED).
// Before that, outputFormat still carries the configure() placeholder and
// must not drive layout — sizing the view to it squishes the first frame(s).
@Volatile
private var realFormatSeen = false
@Volatile
private var renderSurface: Surface? = null
/** Creates and configures the decoder. Width/height of 0 = unknown (the bitstream decides). */
@Synchronized
fun configure(width: Int, height: Int) {
if (configured) return
// 0x0 (the sender's offer) means the size is unknown: the Qualcomm
// C2 AVC decoder requires a concrete size at configure() and
// reconfigures from the in-band SPS of the first keyframe (the
// standard adaptive-resolution pattern).
val format = if (width > 0 && height > 0) {
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height)
} else {
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240)
}
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE)
// Assign before configuring so a configure()/start() failure cannot
// orphan the created instance: MediaCodec has no finalizer and each
// unreleased instance holds a scarce native codec slot. The caller
// follows an exception with release(), which is a no-op on null.
val created = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
codec = created
try {
// The surface must go into configure(): the codec must start in
// surface mode. setOutputSurface() afterwards only switches an
// already-surface-mode codec; a codec configured without a
// surface can never take one.
created.configure(format, renderSurface, null, 0)
created.start()
} catch (e: Exception) {
codec = null
try {
created.release()
} catch (ignored: Exception) {
}
throw e
}
configured = true
realFormatSeen = false
}
/** Points the decoder at a (possibly new) render surface. */
@Synchronized
fun attachSurface(surface: Surface) {
renderSurface = surface
// Legal only in surface mode: setOutputSurface() dynamically switches
// an output surface on a codec configured WITH one (per the platform
// docs); on a ByteBuffer-mode codec it throws IllegalStateException.
codec?.setOutputSurface(surface)
}
/**
* Forgets a destroyed render surface: frames still decode (returning
* their buffers and keeping the queue moving) but are not rendered
* until the next [attachSurface]. API 35's detachOutputSurface() is the
* codec-side equivalent.
*/
@Synchronized
fun detachSurface() {
renderSurface = null
}
/**
* Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp).
* @return false when the codec's input queue was full and the frame was
* dropped; the caller should request a keyframe, not flush (the codec
* is healthy, merely busy).
*/
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean): Boolean {
val c = codec ?: throw IllegalStateException("decoder not configured")
val index = c.dequeueInputBuffer(10_000)
if (index < 0) return false
val buffer = c.getInputBuffer(index) ?: return false
buffer.clear()
buffer.put(data)
val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_KEY_FRAME else 0
c.queueInputBuffer(index, 0, data.size, presentationUs, flags)
return true
}
/**
* Non-blocking drain of available output frames; in surface mode each
* released buffer is rendered to the output surface by the codec.
* @throws IllegalStateException when the decoder signals a fatal error;
* the caller should flush() and request a keyframe.
*/
fun drain() {
val c = codec ?: return
val info = MediaCodec.BufferInfo()
while (true) {
val index = c.dequeueOutputBuffer(info, 0)
when {
index >= 0 -> {
// releaseOutputBuffer(render=true) is what puts the
// frame on the surface (and frees the pool slot).
c.releaseOutputBuffer(index, renderSurface != null)
}
index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
// The codec parsed the SPS size; outputFormat is ready.
realFormatSeen = true
android.util.Log.i(
"H264Decoder", "output format: " +
c.outputFormat.getInteger(MediaFormat.KEY_WIDTH) +
"x" +
c.outputFormat.getInteger(MediaFormat.KEY_HEIGHT)
)
}
index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED -> Unit
else -> break // no more frames right now
}
}
}
/** Resets decoder state; the next keyframe re-primes it (in-band SPS/PPS). */
fun flush() {
val c = codec ?: return
try {
c.flush()
} catch (e: Exception) {
// A codec in an error state may refuse the flush; the caller
// follows up with a keyframe request either way.
}
}
/** The decoded resolution, once the first keyframe configured the codec. */
@Synchronized
fun outputSize(): Pair<Int, Int>? {
val c = codec ?: return null
if (!realFormatSeen) return null // still the configure() placeholder
return try {
val format = c.outputFormat
val width = format.getInteger(MediaFormat.KEY_WIDTH)
val height = format.getInteger(MediaFormat.KEY_HEIGHT)
if (width > 0 && height > 0) width to height else null
} catch (e: Exception) {
null
}
}
@Synchronized
fun release() {
val c = codec ?: return
try {
c.stop()
} catch (e: Exception) {
// ignore
}
c.release()
codec = null
configured = false
renderSurface = null
}
}
@@ -0,0 +1,383 @@
package screen_cast.pipeline
import android.content.Context
import android.net.nsd.NsdManager
import android.net.nsd.NsdServiceInfo
import android.view.Surface
import screen_cast.decode.H264Decoder
import screen_cast.rtp.H264Depacketizer
import screen_cast.rtp.JitterBuffer
import screen_cast.rtp.RtpPacket
import screen_cast.signaling.SessionAnswer
import screen_cast.signaling.SessionOffer
import screen_cast.signaling.SessionPli
import screen_cast.signaling.SignalingServer
import java.net.DatagramPacket
import java.net.DatagramSocket
import java.net.InetSocketAddress
import java.net.SocketException
import java.util.concurrent.atomic.AtomicBoolean
/**
* The receiver pipeline, mirroring the C++ ReceiverPipeline:
*
* NSD advertise + signaling server (offer → answer)
* UDP RTP → jitter buffer → depacketize → MediaCodec → Surface
*
* Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
* (rate-limited to one per 500 ms) asks the sender for a keyframe.
*/
class ReceiverPipeline(
private val context: Context,
private val localIp: String,
private val displaySize: () -> Pair<Int, Int>,
private val onStatus: (String) -> Unit,
private val onFirstFrame: () -> Unit,
private val onVideoSize: (Int, Int) -> Unit,
) {
companion object {
private const val TAG = "ReceiverPipeline"
private const val SERVICE_NAME = "screencast"
private const val SERVICE_TYPE = "_screencast._tcp"
private const val DESIRED_UDP_PORT = 5004
private const val DESIRED_SIGNALING_PORT = 5005
private const val PLI_MIN_INTERVAL_MS = 500L
}
private val nsdManager: NsdManager = context.getSystemService(Context.NSD_SERVICE) as NsdManager
private val decoderLock = Any()
@Volatile private var running = false
private var udpSocket: DatagramSocket? = null
private var udpPort = 0
private var signalingPort = 0
private var signaling: SignalingServer? = null
private var readerThread: Thread? = null
@Volatile private var registrationListener: NsdManager.RegistrationListener? = null
// Guarded by decoderLock: currentSurface, pendingOffer, decoder.
@Volatile private var decoder: H264Decoder? = null
@Volatile private var depacketizer = H264Depacketizer()
private val jitter = JitterBuffer()
private var currentSurface: Surface? = null
private var pendingOffer: SessionOffer? = null
@Volatile private var activeSession = ""
private val firstFrameSeen = AtomicBoolean(false)
private val pliLock = Any()
private var lastPliAtMs = 0L
@Volatile private var videoWidth = 0
@Volatile private var videoHeight = 0
/** The current decoded resolution (0, 0 until the first keyframe). */
fun videoSize(): Pair<Int, Int> = videoWidth to videoHeight
/** Binds the ports, advertises the service, and starts reading RTP. */
@Synchronized
fun start() {
if (running) return
try {
// Plain DatagramSocket: consistent semantics across platforms
// (Android's DatagramChannel.receive() returns a SocketAddress,
// not the byte count). Port 5004 when free; otherwise ephemeral.
val socket = try {
DatagramSocket(InetSocketAddress(DESIRED_UDP_PORT))
} catch (e: Exception) {
DatagramSocket()
}
udpPort = socket.localPort
udpSocket = socket
} catch (e: Exception) {
onStatus("Failed to bind the media port: ${e.message}")
return
}
val server = SignalingServer(
onOffer = { offer -> onOffer(offer) },
onPli = { /* the receiver never receives PLIs */ },
)
try {
signalingPort = server.start(DESIRED_SIGNALING_PORT)
signaling = server
} catch (e: Exception) {
onStatus("Failed to start signaling: ${e.message}")
try {
udpSocket?.close()
} catch (ignored: Exception) {
}
return
}
running = true
readerThread = Thread({ readLoop() }, "rtp-reader").also { it.start() }
// Advertise AFTER the listening status: a registration failure must
// not be overwritten by it (both post to the same overlay).
onStatus(
"Listening on $localIp (media :$udpPort, signaling :$signalingPort)\n" +
"Waiting for a sender… (fall back to: screencast --send --peer $localIp:$signalingPort)",
)
advertiseNsd(signalingPort)
}
/**
* Points the decoder (current or future) at a render surface. A pending
* offer (accepted while no surface existed) configures its decoder now.
*/
fun attachSurface(surface: Surface) {
var configuredFromPending = false
synchronized(decoderLock) {
currentSurface = surface
decoder?.attachSurface(surface)
val offer = pendingOffer
if (offer != null) {
val created = configureDecoderLocked(offer.width, offer.height)
if (created != null) {
pendingOffer = null
decoder = created
configuredFromPending = true
} else {
// Keep the offer: the next attachSurface retries the
// configure instead of abandoning the session.
onStatus("Could not start the decoder")
}
}
}
if (configuredFromPending) {
// The sender is already streaming P-frames; the late-configured
// decoder needs a keyframe (SPS/PPS + IDR) to start producing
// output — the sender only emits one when asked.
requestPli()
}
}
/** Forgets a destroyed render surface so a later offer cannot configure against it. */
fun detachSurface() {
synchronized(decoderLock) {
currentSurface = null
decoder?.detachSurface()
}
}
/**
* Creates and configures a decoder for the stream size; caller holds
* [decoderLock]. Returns null on failure with the codec released.
*/
private fun configureDecoderLocked(width: Int, height: Int): H264Decoder? {
val fresh = H264Decoder()
try {
// Surface first: configure() needs it before the codec starts.
currentSurface?.let { fresh.attachSurface(it) }
fresh.configure(width, height)
} catch (e: Exception) {
fresh.release()
return null
}
return fresh
}
/** Stops listening; the pipeline can be started again. */
@Synchronized
fun stop() {
if (!running) return
running = false
activeSession = ""
try {
udpSocket?.close() // unblocks the reader's receive()
} catch (ignored: Exception) {
}
udpSocket = null
readerThread?.join(1000)
readerThread = null
signaling?.close()
signaling = null
unregisterNsd()
synchronized(decoderLock) {
decoder?.release()
decoder = null
pendingOffer = null
}
firstFrameSeen.set(false)
onStatus("Stopped")
}
private fun onOffer(offer: SessionOffer) {
if (offer.codec != "h264") {
onStatus("Unsupported codec: ${offer.codec}")
return
}
// The offer's width/height are informational (the C++ sender leaves
// them 0); the bitstream carries SPS/PPS at every keyframe.
var decoderReady = true
synchronized(decoderLock) {
pendingOffer = null
decoder?.release()
decoder = null
if (currentSurface != null) {
decoder = configureDecoderLocked(offer.width, offer.height)
decoderReady = decoder != null
} else {
// No surface yet (the window is between surfaces): remember
// the offer; attachSurface configures the decoder. Configuring
// without a surface is not an option — that codec could never
// take one afterwards (setOutputSurface refuses it).
pendingOffer = offer
}
}
if (!decoderReady) {
onStatus("Could not start the decoder")
return
}
// New session: pristine reassembly state.
depacketizer = H264Depacketizer()
jitter.clear()
firstFrameSeen.set(false)
videoWidth = 0
videoHeight = 0
activeSession = offer.sessionId
android.util.Log.i(TAG, "offer: session=${offer.sessionId} ${offer.width}x${offer.height} @${offer.frameRateNum}/${offer.frameRateDen}")
val (displayWidth, displayHeight) = displaySize()
signaling?.send(
SessionAnswer(
sessionId = offer.sessionId,
rtpAddress = "", // the sender targets the address of its own signaling connection
rtpPort = udpPort,
displayWidth = displayWidth,
displayHeight = displayHeight,
),
)
onStatus(
if (decoder != null) "Session ${offer.sessionId} negotiated — waiting for the first frame…"
else "Session ${offer.sessionId} negotiated — waiting for the display surface…"
)
}
private fun readLoop() {
val socket = udpSocket ?: return
val buffer = ByteArray(2048) // MTU 1200 + headroom
val datagram = DatagramPacket(buffer, buffer.size)
while (running) {
try {
socket.receive(datagram)
} catch (e: SocketException) {
break // socket closed
} catch (e: Exception) {
continue
}
val received = datagram.length
if (received <= 0) continue
val bytes = if (received == buffer.size) buffer.copyOf() else buffer.copyOf(received)
val packet = RtpPacket.parse(bytes) ?: continue
for (released in jitter.push(packet)) {
handleDepacketized(released)
}
}
}
private fun handleDepacketized(packet: RtpPacket) {
val result = depacketizer.depacketize(packet)
val accessUnit = result.accessUnit
if (accessUnit != null) {
val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL
try {
// MediaCodec is not thread-safe: attachSurface() (main
// thread) and onOffer (signaling thread) synchronize on the
// same lock around their codec calls.
synchronized(decoderLock) {
val activeDecoder = decoder ?: return
if (!activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame)) {
// Input queue full: the dropped frame corrupts the
// GOP until the next keyframe — ask for one. No
// flush; the codec is healthy, merely busy.
requestPli()
}
activeDecoder.drain()
if (firstFrameSeen.compareAndSet(false, true)) {
onFirstFrame()
}
updateVideoSize()
}
} catch (e: Exception) {
onStatus("Decoder error (${e.message}) — requesting a keyframe…")
synchronized(decoderLock) { decoder?.flush() }
requestPli()
}
}
if (result.frameDropped) {
requestPli()
}
}
private fun updateVideoSize() {
val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return
if (size.first != videoWidth || size.second != videoHeight) {
videoWidth = size.first
videoHeight = size.second
android.util.Log.i(TAG, "video size: ${size.first}x${size.second}")
onVideoSize(size.first, size.second)
}
}
// Rate-limited keyframe request, callable from any thread (the reader
// thread and the main thread's attachSurface both reach it).
private fun requestPli() {
val session = activeSession
if (session.isEmpty()) return
val now = System.currentTimeMillis()
synchronized(pliLock) {
if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return
lastPliAtMs = now
}
android.util.Log.i(TAG, "PLI requested for session $session")
signaling?.send(SessionPli(session))
}
private fun advertiseNsd(port: Int) {
val info = NsdServiceInfo().apply {
setServiceName(SERVICE_NAME)
setServiceType(SERVICE_TYPE)
setPort(port)
}
// The classic RegistrationListener API exists from API 16 through 36
// (verified against the android-36 SDK with javap), so no fallback is
// needed — the previously reflected "ResolutionListener" never
// existed at any API level and could only ever fail.
val listener = object : NsdManager.RegistrationListener {
override fun onServiceRegistered(serviceInfo: NsdServiceInfo) {
// Registered: the sender's mDNS browser should see it now.
android.util.Log.i(TAG, "mDNS registered: ${serviceInfo.serviceName}.${SERVICE_TYPE}")
}
override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
}
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
}
try {
// PROTOCOL_DNS_SD is mandatory on API 36: NsdManager.checkProtocol()
// rejects anything else (the historical 0 threw
// "IllegalArgumentException: Unsupported protocol", which the
// old code swallowed — mDNS never advertised on this phone).
nsdManager.registerService(info, NsdManager.PROTOCOL_DNS_SD, listener)
registrationListener = listener
} catch (e: Exception) {
android.util.Log.e(TAG, "mDNS registration failed", e)
onStatus("mDNS unavailable (${e.message}) — reach this receiver with --peer $localIp:$port")
}
}
private fun unregisterNsd() {
val listener = registrationListener ?: return
try {
nsdManager.unregisterService(listener)
} catch (ignored: Exception) {
// already unregistered
}
registrationListener = null
}
}
@@ -0,0 +1,179 @@
package screen_cast.rtp
import java.io.ByteArrayOutputStream
/** Result of feeding one packet to the depacketizer. */
data class DepacketizeResult(
/** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */
val accessUnit: ByteArray? = null,
/** True when this call discarded a frame as damaged (packet loss or unsupported packetization). */
val frameDropped: Boolean = false,
/** True when the completed access unit carries SPS/PPS (a keyframe). */
val isKeyFrame: Boolean = false,
)
/**
* Reassembles RFC 6184 packet streams (single NAL unit packets and FU-A)
* into Annex-B access units. Packets must arrive in order; frames damaged
* by sequence gaps or missing fragments are reported via DepacketizeResult.
*
* Mirrors the C++ `H264Depacketizer` (same state machine and start codes).
*/
class H264Depacketizer {
companion object {
private const val FU_A = 28
}
private var lastSequenceNumber: Int? = null
private var frameStarted = false
private var frameDamaged = false
private var frameTimestamp = 0
// Growable byte accumulators: keyframes reach hundreds of KB, and boxing
// each byte (the ArrayList<Int> this replaced) churned the GC hard.
private val accessUnit = ByteArrayOutputStream()
private var fuActive = false
private val fuNal = ByteArrayOutputStream()
/** Feed one packet (in sequence order, from the jitter buffer). */
fun depacketize(packet: RtpPacket): DepacketizeResult {
var result = DepacketizeResult()
// Track sequence continuity: a gap means packets were lost.
lastSequenceNumber?.let { last ->
val expected = (last + 1) and 0xFFFF
if (packet.header.sequenceNumber != expected) {
fuActive = false
fuNal.reset()
if (frameStarted) {
frameDamaged = true
}
}
}
lastSequenceNumber = packet.header.sequenceNumber
// A timestamp change without a closing marker means the previous frame
// lost its tail and can no longer be recovered.
if (frameStarted && packet.header.timestamp != frameTimestamp) {
dropFrame()
result = result.copy(frameDropped = true)
}
if (!frameStarted) {
frameStarted = true
frameDamaged = false
frameTimestamp = packet.header.timestamp
accessUnit.reset()
}
val payload = packet.payload
if (payload.isNotEmpty()) {
val type = payload[0].toInt() and 0x1F
when {
type in 1..23 -> {
// Single NAL unit packet.
if (fuActive) {
// The previous fragmented NAL never received its end packet.
frameDamaged = true
fuActive = false
fuNal.reset()
}
appendStartCode()
accessUnit.write(payload)
}
type == FU_A -> {
if (payload.size < 2) {
frameDamaged = true
} else {
val fuHeader = payload[1].toInt() and 0xFF
val start = fuHeader and 0x80 != 0
val end = fuHeader and 0x40 != 0
val fragment = payload.copyOfRange(2, payload.size)
when {
start -> {
if (fuActive) {
// The previous fragmented NAL lost its end packet.
frameDamaged = true
}
fuActive = true
fuNal.reset()
// The FU indicator keeps the original NAL's F bit (0) and NRI,
// and declares type 28; the FU header carries S/E plus the real type.
fuNal.write((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F))
fuNal.write(fragment)
}
!fuActive -> {
// Continuation without a start: the head of the NAL is lost.
frameDamaged = true
}
else -> {
fuNal.write(fragment)
if (end) {
appendStartCode()
fuNal.writeTo(accessUnit)
fuActive = false
fuNal.reset()
}
}
}
}
}
else -> {
// Unsupported packetization mode (STAP-A, MTAP, FU-B): the frame
// cannot be reconstructed.
frameDamaged = true
}
}
}
if (!packet.header.marker) {
return result
}
if (fuActive) {
// The marker arrived while a NAL was still fragmented.
frameDamaged = true
fuActive = false
fuNal.reset()
}
if (!frameDamaged && accessUnit.size() > 0) {
val unit = accessUnit.toByteArray()
result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit))
} else {
// The frame that just ended is unusable.
result = result.copy(frameDropped = true)
}
dropFrame()
return result
}
private fun appendStartCode() {
accessUnit.write(0)
accessUnit.write(0)
accessUnit.write(1)
}
/** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */
private fun containsParameterSets(unit: ByteArray): Boolean {
for (i in 0..unit.size - 4) {
if (unit[i] == 0.toByte() && unit[i + 1] == 0.toByte() && unit[i + 2] == 1.toByte()) {
val nalType = unit[i + 3].toInt() and 0x1F
if (nalType == 7 || nalType == 8) {
return true
}
}
}
return false
}
private fun dropFrame() {
frameStarted = false
frameDamaged = false
accessUnit.reset()
fuActive = false
fuNal.reset()
}
}
@@ -0,0 +1,71 @@
package screen_cast.rtp
import java.util.TreeMap
import java.util.concurrent.TimeUnit
/**
* Reorders RTP packets by sequence number before depacketization so that a
* reordering link (Wi-Fi) is not read as loss. Delivery stays in order;
* only aged-out or overflowing buffers release out of order, which the
* downstream gap detection still handles for genuine loss.
*
* Mirrors the C++ `RtpJitterBuffer` (same defaults and semantics).
*/
class JitterBuffer(
private val maxDepth: Int = 16,
private val maxDelayMs: Long = 60,
) {
private class BufferEntry(val timeNanos: Long, val packet: RtpPacket)
private val lock = Any()
private val buffer = TreeMap<Int, BufferEntry>()
private var nextExpected: Int? = null
/** Insert one packet and return the packets now ready for in-order delivery. */
fun push(packet: RtpPacket): List<RtpPacket> = synchronized(lock) {
val released = ArrayList<RtpPacket>()
val sequence = packet.header.sequenceNumber
val now = System.nanoTime()
val expected0 = nextExpected ?: sequence.also { nextExpected = it }
// Serial-number comparison: a distance >= 32768 means the packet is
// older than what we already delivered (duplicate or straggler).
val distance = (sequence - expected0 + 65536) % 65536
if (distance < 32768) {
buffer[sequence] = BufferEntry(now, packet)
// Release the consecutive run from the expected sequence.
var expected = expected0
while (true) {
val entry = buffer[expected] ?: break
released.add(entry.packet)
buffer.remove(expected)
expected = (expected + 1) and 0xFFFF
}
nextExpected = expected
// A missing packet stalls the run: age out the backlog (or bound
// the buffer) and release what is there in order, so genuine loss
// reaches the depacketizer's gap detection rather than blocking.
if (buffer.isNotEmpty()) {
val head = buffer.firstEntry().value
val headAgeMs = TimeUnit.NANOSECONDS.toMillis(now - head.timeNanos)
if (headAgeMs > maxDelayMs || buffer.size > maxDepth) {
released.addAll(buffer.values.map { it.packet })
nextExpected = (buffer.lastKey() + 1) and 0xFFFF
buffer.clear()
}
}
}
released
}
/** Discard everything still buffered. */
fun clear() {
synchronized(lock) {
buffer.clear()
nextExpected = null
}
}
}
@@ -0,0 +1,55 @@
package screen_cast.rtp
/** Minimal RTP header (RFC 3550) without extensions. */
data class RtpHeader(
val version: Int = 2,
val padding: Boolean = false,
val extension: Boolean = false,
val csrcCount: Int = 0,
val marker: Boolean = false,
val payloadType: Int = 96,
val sequenceNumber: Int = 0,
val timestamp: Int = 0,
val ssrc: Int = 0,
) {
/** Serializes the bare 12-byte header; requires a version-2, extension-less header. */
fun serialize(): ByteArray {
val out = ByteArray(12)
out[0] = (((version and 0x0F) shl 6) or (if (padding) 0x20 else 0) or (if (extension) 0x10 else 0) or (csrcCount and 0x0F)).toByte()
out[1] = ((if (marker) 0x80 else 0) or (payloadType and 0x7F)).toByte()
out[2] = (sequenceNumber ushr 8).toByte()
out[3] = (sequenceNumber and 0xFF).toByte()
out[4] = (timestamp ushr 24).toByte()
out[5] = (timestamp ushr 16).toByte()
out[6] = (timestamp ushr 8).toByte()
out[7] = (timestamp and 0xFF).toByte()
out[8] = (ssrc ushr 24).toByte()
out[9] = (ssrc ushr 16).toByte()
out[10] = (ssrc ushr 8).toByte()
out[11] = (ssrc and 0xFF).toByte()
return out
}
companion object {
fun parse(input: ByteArray): RtpHeader? {
if (input.size < 12) return null
val b0 = input[0].toInt() and 0xFF
val b1 = input[1].toInt() and 0xFF
val version = b0 ushr 6
if (version != 2) return null
return RtpHeader(
version = version,
padding = b0 and 0x20 != 0,
extension = b0 and 0x10 != 0,
csrcCount = b0 and 0x0F,
marker = b1 and 0x80 != 0,
payloadType = b1 and 0x7F,
sequenceNumber = ((input[2].toInt() and 0xFF) shl 8) or (input[3].toInt() and 0xFF),
timestamp = ((input[4].toInt() and 0xFF) shl 24) or ((input[5].toInt() and 0xFF) shl 16) or
((input[6].toInt() and 0xFF) shl 8) or (input[7].toInt() and 0xFF),
ssrc = ((input[8].toInt() and 0xFF) shl 24) or ((input[9].toInt() and 0xFF) shl 16) or
((input[10].toInt() and 0xFF) shl 8) or (input[11].toInt() and 0xFF),
)
}
}
}
@@ -0,0 +1,36 @@
package screen_cast.rtp
/** An RTP packet: 12-byte base header (plus optional CSRC/extension) and payload. */
class RtpPacket(val header: RtpHeader, val payload: ByteArray) {
companion object {
/**
* Parses a full RTP datagram. Honors CSRC lists, one-level extension
* headers, and RFC 3550 padding, mirroring the C++ receiver.
*/
fun parse(input: ByteArray): RtpPacket? {
if (input.size < 12) return null
val header = RtpHeader.parse(input) ?: return null
var offset = 12 + header.csrcCount * 4
if (input.size < offset) return null
if (header.extension) {
if (input.size < offset + 4) return null
val extensionWords =
((input[offset + 2].toInt() and 0xFF) shl 8) or (input[offset + 3].toInt() and 0xFF)
offset += 4 + extensionWords * 4
if (input.size < offset) return null
}
var payloadSize = input.size - offset
if (header.padding) {
// RFC 3550: the last byte holds the padding size, including itself.
if (payloadSize == 0) return null
val paddingSize = input[input.size - 1].toInt() and 0xFF
if (paddingSize == 0 || paddingSize > payloadSize) return null
payloadSize -= paddingSize
}
return RtpPacket(header, input.copyOfRange(offset, offset + payloadSize))
}
}
}
@@ -0,0 +1,100 @@
package screen_cast.signaling
import org.json.JSONObject
// JSON wire format shared with the C++ implementation: one JSON object per
// newline-terminated TCP line (offer / answer / pli).
data class SessionOffer(
val sessionId: String,
val codec: String,
val width: Int,
val height: Int,
val frameRateNum: Int,
val frameRateDen: Int,
val rtpAddress: String,
val rtpPort: Int,
) : SignalingMessage
data class SessionAnswer(
val sessionId: String,
val rtpAddress: String,
val rtpPort: Int,
val displayWidth: Int,
val displayHeight: Int,
) : SignalingMessage
data class SessionPli(val sessionId: String) : SignalingMessage
sealed interface SignalingMessage {
companion object {
private const val MAX_MESSAGE_BYTES = 64 * 1024
fun parse(line: String): SignalingMessage? {
if (line.length > MAX_MESSAGE_BYTES) return null
val json = try {
JSONObject(line)
} catch (e: Exception) {
return null
}
when (val type = json.optString("type")) {
"offer" ->
return SessionOffer(
sessionId = json.optString("session_id"),
codec = json.optString("codec"),
width = json.optInt("width"),
height = json.optInt("height"),
frameRateNum = json.optInt("frame_rate_num", 30),
frameRateDen = json.optInt("frame_rate_den", 1),
rtpAddress = json.optString("rtp_address"),
rtpPort = json.optInt("rtp_port"),
)
"answer" ->
return SessionAnswer(
sessionId = json.optString("session_id"),
rtpAddress = json.optString("rtp_address"),
rtpPort = json.optInt("rtp_port"),
displayWidth = json.optInt("display_width"),
displayHeight = json.optInt("display_height"),
)
"pli" -> return SessionPli(sessionId = json.optString("session_id"))
else -> return null
}
}
fun serialize(message: SignalingMessage): String {
val json = JSONObject()
when (message) {
is SessionOffer -> {
json.put("type", "offer")
json.put("session_id", message.sessionId)
json.put("codec", message.codec)
json.put("width", message.width)
json.put("height", message.height)
json.put("frame_rate_num", message.frameRateNum)
json.put("frame_rate_den", message.frameRateDen)
json.put("rtp_address", message.rtpAddress)
json.put("rtp_port", message.rtpPort)
}
is SessionAnswer -> {
json.put("type", "answer")
json.put("session_id", message.sessionId)
json.put("rtp_address", message.rtpAddress)
json.put("rtp_port", message.rtpPort)
json.put("display_width", message.displayWidth)
json.put("display_height", message.displayHeight)
}
is SessionPli -> {
json.put("type", "pli")
json.put("session_id", message.sessionId)
}
}
return json.toString() + "\n"
}
}
}
@@ -0,0 +1,144 @@
package screen_cast.signaling
import java.io.BufferedInputStream
import java.io.ByteArrayOutputStream
import java.io.OutputStream
import java.net.InetSocketAddress
import java.net.ServerSocket
import java.net.Socket
/**
* Newline-delimited JSON signaling server (the receiver side). Keeps the
* most recent connection as its active peer, mirroring the C++ server:
* `onOffer` may answer synchronously (the sender blocks on the answer).
*/
class SignalingServer(
private val onOffer: (SessionOffer) -> Unit,
private val onPli: (SessionPli) -> Unit,
) {
private val server = ServerSocket()
private val peerLock = Any()
private var peerOut: OutputStream? = null
private var acceptThread: Thread? = null
private var readerThread: Thread? = null
@Volatile
private var running = false
/** Binds the port (SO_REUSEADDR) and starts accepting. Returns the bound port. */
fun start(port: Int): Int {
server.reuseAddress = true
server.bind(InetSocketAddress(port))
running = true
acceptThread = Thread({ acceptLoop() }, "signaling-accept")
acceptThread!!.start()
return server.localPort
}
/**
* Sends to the current peer; never throws. A lost control message (a
* PLI, an answer) is recoverable — the session re-negotiates or the next
* keyframe arrives — but an exception here would kill the RTP reader
* thread, which reaches send() from requestPli(). Mirrors the C++ server,
* which ignores write failures.
*/
fun send(message: SignalingMessage) {
val bytes = try {
SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8)
} catch (e: Exception) {
return // unreachable for our message types; serialize is total
}
synchronized(peerLock) {
val out = peerOut ?: return
try {
out.write(bytes)
out.flush()
} catch (e: Exception) {
// The peer is gone (e.g. the sender died while media still
// flows). Dropping peerOut also closes the socket; the next
// accepted offer installs a fresh one.
peerOut = null
}
}
}
fun close() {
running = false
try {
server.close()
} catch (e: Exception) {
// already closed
}
synchronized(peerLock) {
try {
peerOut?.close()
} catch (e: Exception) {
// peer already gone
}
peerOut = null
}
acceptThread?.join(500)
readerThread?.join(500)
}
private fun acceptLoop() {
while (running) {
val socket = try {
server.accept()
} catch (e: Exception) {
break // listening socket closed
}
synchronized(peerLock) {
try {
peerOut?.close()
} catch (e: Exception) {
// previous sender already gone
}
peerOut = socket.getOutputStream()
}
readerThread = Thread({ readLoop(socket) }, "signaling-reader").also { it.start() }
}
}
private fun readLoop(socket: Socket) {
val input = BufferedInputStream(socket.getInputStream())
val line = ByteArrayOutputStream()
val chunk = ByteArray(4096)
while (running) {
val read = try {
input.read(chunk)
} catch (e: Exception) {
break
}
if (read < 0) break
for (i in 0 until read) {
val b = chunk[i].toInt() and 0xFF
if (b == '\n'.code) {
val text = line.toString(Charsets.UTF_8.name())
line.reset()
if (text.isNotEmpty()) {
dispatch(text)
}
} else if (b != '\r'.code) {
if (line.size() < 64 * 1024) {
line.write(b)
} else {
line.reset() // hostile or broken peer: drop the oversized line
}
}
}
}
}
private fun dispatch(line: String) {
val message = SignalingMessage.parse(line) ?: return
try {
when (message) {
is SessionOffer -> onOffer(message)
is SessionPli -> onPli(message)
is SessionAnswer -> Unit // the receiver never receives answers
}
} catch (e: Exception) {
// A broken callback must not kill the reader thread.
}
}
}
@@ -0,0 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@android:color/black">
<TextureView
android:id="@+id/video"
android:layout_width="match_parent"
android:layout_height="match_parent" />
<TextView
android:id="@+id/status"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="top|start"
android:layout_margin="16dp"
android:background="#66000000"
android:maxLines="3"
android:ellipsize="end"
android:paddingHorizontal="12dp"
android:paddingVertical="8dp"
android:textColor="@android:color/white"
android:textSize="13sp"
tools:text="Listening… waiting for a sender"
xmlns:tools="http://schemas.android.com/tools" />
</FrameLayout>
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<style name="Theme.Screencast" parent="@android:style/Theme.Material.NoActionBar">
<item name="android:windowFullscreen">true</item>
<item name="android:windowBackground">@android:color/black</item>
<item name="android:statusBarColor">@android:color/black</item>
<item name="android:navigationBarColor">@android:color/black</item>
</style>
</resources>
@@ -0,0 +1,138 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
class H264DepacketizerTest {
companion object {
private val START = byteArrayOf(0x00, 0x00, 0x01)
private fun packet(seq: Int, ts: Int, payload: ByteArray, marker: Boolean = false) =
RtpPacket(
RtpHeader(sequenceNumber = seq, timestamp = ts, marker = marker),
payload,
)
}
@Test
fun single_nal_two_packets() {
val depacketizer = H264Depacketizer()
// SPS (type 7) then a slice (type 5), closed by the marker.
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x67, 0xAA.toByte(), 0xBB.toByte())))
assertNull(first.accessUnit)
val second = depacketizer.depacketize(packet(2, 100, byteArrayOf(0x41, 0x01, 0x02), marker = true))
assertTrue(
second.accessUnit
?.contentEquals(START + byteArrayOf(0x67, 0xAA.toByte(), 0xBB.toByte()) + START + byteArrayOf(0x41, 0x01, 0x02)) == true,
)
assertTrue(second.isKeyFrame)
assertFalse(second.frameDropped)
}
@Test
fun fu_a_reassembly() {
val depacketizer = H264Depacketizer()
// NAL: header 0x41 (type 1, NRI 2) + payload 0x11 0x22 0x33 0x44.
// FU indicator = (0x41 & 0xE0) | 28 = 0x5C; the depacketizer rebuilds
// the NAL header from indicator-NRI | FU-type, so the FU header must
// carry the original type (1).
val indicator = 0x5C.toByte()
val start = packet(1, 100, byteArrayOf(indicator, 0x81.toByte(), 0x11))
val mid = packet(2, 100, byteArrayOf(indicator, 0x01, 0x22))
val last = packet(3, 100, byteArrayOf(indicator, 0x41, 0x33, 0x44), marker = true)
assertNull(depacketizer.depacketize(start).accessUnit)
assertNull(depacketizer.depacketize(mid).accessUnit)
val done = depacketizer.depacketize(last)
assertTrue(done.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0x11, 0x22, 0x33, 0x44)) == true)
assertFalse(done.frameDropped)
}
@Test
fun drops_gapped_frames() {
val depacketizer = H264Depacketizer()
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01)))
assertNull(first.accessUnit)
// seq 2 is missing; the frame must be reported dropped, not delivered.
val tail = depacketizer.depacketize(packet(3, 100, byteArrayOf(0x41, 0x02), marker = true))
assertNull(tail.accessUnit)
assertTrue(tail.frameDropped)
}
@Test
fun separate_frames_by_marker() {
val depacketizer = H264Depacketizer()
val au1 = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0xAA.toByte()), marker = true))
assertTrue(au1.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0xAA.toByte())) == true)
val au2 = depacketizer.depacketize(packet(2, 200, byteArrayOf(0x41, 0xBB.toByte()), marker = true))
assertTrue(au2.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0xBB.toByte())) == true)
}
@Test
fun drops_fu_without_start() {
val depacketizer = H264Depacketizer()
// Continuation (no S bit) without any start packet.
val result = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte(), 0x41.toByte(), 0x11), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_still_fragmented_at_marker() {
val depacketizer = H264Depacketizer()
val start = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte(), 0x81.toByte(), 0x11)))
assertNull(start.accessUnit)
// Marker arrives while the FU-A NAL is still open.
val result = depacketizer.depacketize(packet(2, 100, byteArrayOf(0x41, 0x01), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_unsupported_packetization() {
val depacketizer = H264Depacketizer()
val stapA = 24.toByte() // STAP-A
val result = depacketizer.depacketize(
packet(1, 100, byteArrayOf(stapA, 0x00, 0x05, 0x41, 0x01), marker = true),
)
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_timestamp_change_without_marker() {
val depacketizer = H264Depacketizer()
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01)))
assertNull(first.accessUnit)
// The stale frame is reported dropped, but this packet starts (and
// closes) the next frame — matching the C++ depacketizer.
val result = depacketizer.depacketize(packet(2, 200, byteArrayOf(0x41, 0x02), marker = true))
assertTrue(result.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0x02)) == true)
assertTrue(result.frameDropped)
}
@Test
fun keyframe_detection_requires_parameter_sets() {
val depacketizer = H264Depacketizer()
val plain = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01), marker = true))
assertFalse(plain.isKeyFrame)
val depacketizer2 = H264Depacketizer()
val withSps = depacketizer2.depacketize(
packet(1, 100, byteArrayOf(0x67, 0xAA.toByte(), 0x88.toByte(), 0x68, 0xBB.toByte(), 0x41, 0x01), marker = true),
)
assertTrue(withSps.isKeyFrame)
}
@Test
fun drops_short_fu_packets() {
val depacketizer = H264Depacketizer()
// FU-A packet without its FU header byte.
val result = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte()), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
}
@@ -0,0 +1,65 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
class JitterBufferTest {
private fun packet(seq: Int, ts: Int = 100) =
RtpPacket(RtpHeader(sequenceNumber = seq, timestamp = ts), byteArrayOf(seq.toByte()))
@Test
fun in_order_releases_immediately() {
val jitter = JitterBuffer()
assertEquals(listOf(1), jitter.push(packet(1)).map { it.header.sequenceNumber })
assertEquals(listOf(2), jitter.push(packet(2)).map { it.header.sequenceNumber })
assertEquals(listOf(3), jitter.push(packet(3)).map { it.header.sequenceNumber })
}
@Test
fun reorders_out_of_order_packets() {
val jitter = JitterBuffer()
assertEquals(listOf(1), jitter.push(packet(1)).map { it.header.sequenceNumber })
assertTrue(jitter.push(packet(3)).isEmpty())
val released = jitter.push(packet(2)).map { it.header.sequenceNumber }
assertEquals(listOf(2, 3), released)
}
@Test
fun overflow_releases_in_order_and_advances() {
val jitter = JitterBuffer(maxDepth = 4)
assertTrue(jitter.push(packet(1)).map { it.header.sequenceNumber } == listOf(1))
// seq 2 is lost; 3..6 stay buffered (within the depth bound).
for (seq in 3..6) {
assertTrue(jitter.push(packet(seq)).isEmpty())
}
// seq 7 overflows the buffer: 3..7 flush in order.
assertEquals(listOf(3, 4, 5, 6, 7), jitter.push(packet(7)).map { it.header.sequenceNumber })
// Delivery continues in order afterwards.
assertEquals(listOf(8), jitter.push(packet(8)).map { it.header.sequenceNumber })
assertEquals(listOf(9), jitter.push(packet(9)).map { it.header.sequenceNumber })
}
@Test
fun discards_stragglers() {
val jitter = JitterBuffer(maxDepth = 4)
jitter.push(packet(1))
for (seq in 3..8) {
jitter.push(packet(seq))
}
assertTrue(jitter.push(packet(9)).isNotEmpty())
// seq 4 is now far behind the expected sequence: discarded, not delivered.
assertTrue(jitter.push(packet(4)).isEmpty())
// In-order delivery continues from 10.
assertEquals(listOf(10), jitter.push(packet(10)).map { it.header.sequenceNumber })
}
@Test
fun clear_resets_state() {
val jitter = JitterBuffer()
jitter.push(packet(5))
jitter.clear()
// A completely different sequence now starts fresh.
assertEquals(listOf(100), jitter.push(packet(100)).map { it.header.sequenceNumber })
}
}
@@ -0,0 +1,112 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
class RtpHeaderTest {
@Test
fun roundtrip() {
val header = RtpHeader(
version = 2,
padding = false,
extension = false,
csrcCount = 0,
marker = true,
payloadType = 96,
sequenceNumber = 0xABCD,
timestamp = 0xDEADBEEF.toInt(),
ssrc = 0x12345678,
)
val wire = header.serialize()
assertEquals(12, wire.size)
val parsed = RtpHeader.parse(wire)
assertEquals(header, parsed)
}
@Test
fun rejects_bad_version() {
val wire = RtpHeader().serialize().copyOf()
wire[0] = (wire[0].toInt() and 0x3F or (1 shl 6)).toByte()
assertNull(RtpHeader.parse(wire))
}
@Test
fun rejects_short_input() {
val header = RtpHeader()
assertNull(RtpHeader.parse(header.serialize().copyOfRange(0, 11)))
assertNull(RtpHeader.parse(ByteArray(0)))
}
@Test
fun preserves_flags() {
val header = RtpHeader(padding = true, csrcCount = 2, marker = true, payloadType = 63)
val parsed = RtpHeader.parse(header.serialize())
assertEquals(true, parsed?.padding)
assertEquals(2, parsed?.csrcCount)
assertEquals(true, parsed?.marker)
assertEquals(63, parsed?.payloadType)
}
}
class RtpPacketTest {
private fun header(seq: Int, marker: Boolean = false) =
RtpHeader(sequenceNumber = seq, payloadType = 96, marker = marker)
@Test
fun roundtrip_with_payload() {
val packet = RtpPacket(header(seq = 7), byteArrayOf(0x11, 0x22, 0x33))
val wire = packet.header.serialize() + packet.payload
val parsed = RtpPacket.parse(wire)
assertEquals(header(seq = 7), parsed?.header)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x11, 0x22, 0x33)) == true)
}
@Test
fun skips_csrc_list() {
val header = RtpHeader(csrcCount = 1, sequenceNumber = 3)
val wire = header.serialize() + byteArrayOf(0x0A, 0x00, 0x00, 0x01) + byteArrayOf(0x99.toByte())
val parsed = RtpPacket.parse(wire)
assertEquals(1, parsed?.header?.csrcCount)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x99.toByte())) == true)
}
@Test
fun skips_extension_header() {
val header = RtpHeader(extension = true, sequenceNumber = 4)
// profile=0x0001, length=1 word, one word of data
val wire = header.serialize() +
byteArrayOf(0x00, 0x01, 0x00, 0x01, 0xDE.toByte(), 0xAD.toByte(), 0xBE.toByte(), 0xEF.toByte()) +
byteArrayOf(0x77)
val parsed = RtpPacket.parse(wire)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x77)) == true)
}
@Test
fun strips_padding() {
val header = RtpHeader(padding = true, sequenceNumber = 5)
// Payload byte, one padding zero, size byte (2 = padding incl. itself).
val wire = header.serialize() + byteArrayOf(0x55, 0x00, 0x02)
val parsed = RtpPacket.parse(wire)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x55)) == true)
}
@Test
fun rejects_truncated_csrc_and_extension() {
val csrc = RtpHeader(csrcCount = 1).serialize()
assertNull(RtpPacket.parse(csrc)) // 12 bytes, needs 16
val ext = RtpHeader(extension = true).serialize() + byteArrayOf(0x00, 0x01)
assertNull(RtpPacket.parse(ext)) // extension length field cut off
}
@Test
fun rejects_bad_padding() {
val zeroPad = RtpHeader(padding = true).serialize() + byteArrayOf(0x00)
assertNull(RtpPacket.parse(zeroPad))
val oversized = RtpHeader(padding = true).serialize() + byteArrayOf(0x00, 0x00, 0x05)
assertNull(RtpPacket.parse(oversized))
assertNull(RtpPacket.parse(ByteArray(11)))
}
}
+3
View File
@@ -0,0 +1,3 @@
plugins {
id("com.android.application") version "9.4.0" apply false
}
+3
View File
@@ -0,0 +1,3 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
org.gradle.caching=true
kotlin.code.style=official
+17
View File
@@ -0,0 +1,17 @@
pluginManagement {
repositories {
google()
mavenCentral()
gradlePluginPortal()
}
}
dependencyResolutionManagement {
repositories {
google()
mavenCentral()
}
}
rootProject.name = "screen_cast-android"
include(":app")
+74 -8
View File
@@ -82,17 +82,83 @@ discovered addresses in reachability order (private IPv4 first).
## Phase 7 — Resilience and Polish
**Goal**: loss recovery, hardware acceleration, and packaging.
**Goal**: loss recovery, hardware acceleration, and polish.
- NACK / PLI feedback loop.
- Jitter buffer on the receiver.
- VAAPI/NVENC hardware encode probes and fallback.
- Optional GUI target behind `meson -Dgui=true`.
- `.desktop` file, icon, packaging notes.
- [x] PLI keyframe feedback over signaling (receiver asks, sender
re-encodes a keyframe; validated end-to-end with an induced loss).
- [x] Jitter/reorder window on the receiver (16 packets / 60 ms).
- [x] Hardware H.264 decode probe with software fallback (h264_v4l2m2m —
the Pi's VideoCore path; --swdecode opts out).
- [x] GTK4 sender GUI behind `meson -Dgui=true` (gtkmm; receiver list,
bitrate, start/stop) plus a waybar widget (`screencast waybar`,
click-to-toggle, right-click opens the panel; state shared via
$XDG_RUNTIME_DIR/screencast/sender.json).
- [ ] VAAPI hardware encode probe on the sender (deferred: software
encode is not the bottleneck).
- Deferred: .desktop file, packaging.
**Validation**: sustained streaming under packet loss; hardware accel smoke
where available.
where available. PLI + jitter validated on the desktop (induced-loss probe);
hardware decode validated as clean-fallback on the desktop, hardware path
pending a run on the Pi.
## Phase 8 — Android Receiver App
**Goal**: a native Android receiver app (Kotlin) so a phone can act as the
second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the
existing signaling + RTP protocol; no C++ changes.
- [x] 8.1 Gradle project builds (`android/`, AGP 9 built-in Kotlin, no
androidx; `gradle :app:assembleDebug`)
- [x] 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer,
depacketizer — 25 tests green)
- [x] 8.3 Signaling + NSD validated on device (offer → answer over the
network; mDNS registration via the API-36 RegistrationListener)
**correction 2026-09-10**: registration had in fact never
succeeded (registerService passed protocol `0`, which API 36
rejects with "Unsupported protocol"; the swallowed failure was
hidden behind the "Listening…" status). Fixed with
`PROTOCOL_DNS_SD` + real on-device validation: desktop
`--discover` lists the phone and a live `--send` session
decodes and renders.
- [x] 8.4 MediaCodec decode + Surface render validated on device (in-band
SPS sizing, letterbox fit)
- [x] 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed
fullscreen render, PLI keyframe recovery on Wi-Fi loss
- [x] 8.6 Docs: README receiver section, RUNBOOK build/install/test,
PHASES + MEMORY updates
**Validation**: `gradle :app:testDebugUnitTest` green; live session
`--send --target window --peer <phone>:5005` rendered fullscreen on the
phone (and HDMI via DP-alt-mode) with loss recovery.
## Phase 9 — iOS Receiver App
**Goal**: a native iOS receiver app (Swift) so an iPhone can act as the second
receiver. The app speaks the existing signaling + RTP protocol; no C++ changes.
Mirrors the Android receiver (Phase 8) source-to-source.
- [x] 9.1 Xcode project scaffolding (`ios/`, XcodeGen spec + bootstrap script;
Info.plist with local-network + Bonjour privacy keys)
- [x] 9.2 Swift protocol core + XCTest (rtp header/packet, jitter, depacketizer
ported source-to-source; added signaling JSON, line framing, AVCC, NAL
extraction tests — the Android side had no signaling tests)
- [x] 9.3 Signaling server (BSD sockets, dual-stack, most-recent-peer, never
throws) + `NSBonjourServices` advertisement
- [x] 9.4 Media path: UDP (poll-based) → jitter → depacketize → VideoToolbox
(in-band SPS/PPS, real-time decode) → AVSampleBufferDisplayLayer
(letterbox) + PLI on damage + pendingOffer for late surface attach
- [ ] 9.5 On-device validation (needs a Mac + Xcode 26 + iPhone 16): unit tests
green on the simulator; `--discover` lists the phone; `--send --peer
<ip>:5005` streams letterboxed; PLI recovery on Wi-Fi loss
- [ ] 9.6 Docs: iOS RUNBOOK quirks, README receiver section, PHASES + MEMORY
**Validation**: `ios/bootstrap.sh test` (simulator) green; live session
`--send --peer <iphone>:5005` renders fullscreen letterboxed on the iPhone with
loss recovery. Note: this box is Linux — no Xcode/iOS SDK — so 9.29.4 are
authored but only 9.5 can validate the VideoToolbox + local-network paths.
## Current phase
Phase 7Resilience and Polish.
Phase 9iOS Receiver App (implementation authored; on-device validation
pending a Mac + Xcode 26 + iPhone 16).
+111
View File
@@ -134,6 +134,54 @@ A window manager only comes with the desktop-session alternative, where
the receiver runs inside it (uncomment the `Environment=` lines in the
service file as described above).
## GTK panel and waybar widget (Phase 7)
Build the GUI alongside the CLI (`meson configure build -Dgui=true`, then
recompile; installs as `screencast-gui`):
```sh
screencast-gui # receiver list → pick one → bitrate → Start
screencast waybar # one JSON line for a waybar custom module
screencast waybar --toggle # stop a running sender / restart the last
```
The waybar module (installed to this machine's `~/.config/waybar/` — see
`custom/screencast` there): the app icon as the module face (dimmed while
idle, green tint while streaming; the `▶`/`⏸` text from the JSON is hidden
in the CSS), tooltip with receiver, bitrate, elapsed; left-click toggles
streaming to the last receiver via the state file, right-click opens the
panel. The sender publishes its state to `$XDG_RUNTIME_DIR/screencast/sender.json`,
so every front-end — CLI, GUI, widget — agrees on what is running.
**Icons**: `screencast_icon/screencast_256.png` is the single app icon.
`meson install` puts it in `share/icons/hicolor/256x256/apps/screencast.png`;
the GTK panel uses it by themed name (`set_icon_name` — GTK4 removed the
pixel-buffer window icon, so themed icons only), and the waybar CSS
references the installed path. The SDL receiver window embeds the PNG at
build time (`icon_png_data.h` generated by `scripts/icon_to_header.py`) and
sets it via `SDL_SetWindowIcon` when SDL3_image is available (optional
dependency; no pkg-config ships with it, so meson finds the library
directly). Icons surface in compositor taskbars/switchers, not in title
bars (neither GTK4's CSD nor Hyprland's decorations draw them).
Restart waybar after changing `style.css` to pick up icon changes.
## Under the hood: resilience (Phase 7)
The receiver absorbs loss in two stages and recovers actively:
1. **Jitter window**: RTP packets are re-ordered by sequence number in a
small buffer (16 packets / 60 ms), so Wi-Fi reordering is not misread as
loss. In-order streams release immediately (zero added latency).
2. **PLI feedback**: when a frame arrives genuinely damaged, the receiver
drops it and asks the sender for a keyframe over the signaling channel
(rate-limited to one request per 500 ms). The sender re-encodes a
keyframe immediately — recovery takes one frame time instead of
waiting out the rest of the GOP.
**Hardware decode**: the receiver probes `h264_v4l2m2m` (the VideoCore
path on Raspberry Pi) and falls back to software automatically; the
journal says which path is active. `--swdecode` forces software.
## Sender / receiver loopback (Phase 5, manual)
Two terminals on the same desktop session:
@@ -161,6 +209,69 @@ The headless equivalent runs as part of `meson test` (`udp loopback` test):
synthetic frames → encode → packetize → localhost UDP → depacketize →
decode, no portal or window involved.
## Android receiver app (Phase 8)
Toolchain: JDK 21, system `gradle` (no wrapper), Android SDK with android-36.
AGP 9 has **built-in Kotlin** — do not apply `org.jetbrains.kotlin.android`
and do not use `kotlinOptions {}`.
```sh
cd android
gradle :app:assembleDebug # -> app/build/outputs/apk/debug/app-debug.apk
gradle :app:testDebugUnitTest # 25 JVM tests (rtp, jitter, depacketizer)
```
Device test (validated on a Fairphone 6, Android 16 / API 36):
```sh
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n screen_cast.receiver/screen_cast.ReceiverActivity
# activity FQN is screen_cast.ReceiverActivity (namespace), not
# screen_cast.receiver.ReceiverActivity
adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml # status text
adb exec-out screencap -p > phone.png # visual check
```
On the sender: `screencast --send --peer <phone-ip>:5005` (add `--target
window` to pick a specific window; the portal's monitor source picks the
first output on multi-monitor Hyprland/GTK-portal setups).
Platform quirks found while validating (Android 16 / API 36):
- `android.permission.INTERNET` is required — NsdService rejects
registration without it.
- `NsdManager.registerService` on Android 16 validates the protocol
argument (`NsdManager.checkProtocol`): the historical `0` throws
`IllegalArgumentException: Unsupported protocol` — pass
`NsdManager.PROTOCOL_DNS_SD`. This was silent for a long time: the app
swallowed the exception and the failure status was overwritten by the
"Listening…" line, so mDNS never advertised even though the UI looked
fine (discovery only worked via `--peer`). Diagnosed via
`adb shell dumpsys servicediscovery` (the client's `mClientRequests`
stays empty when no request was ever issued) plus logging the
exception.
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
(Inet6Address on Android), not an `InetSocketAddress`.
- `MediaCodec`: pass the output Surface to `configure()` — the codec must
start in surface mode (`setOutputSurface` afterwards only switches an
already-surface-mode codec to a new surface; a codec configured without
a surface can never take one, per the API docs); call `start()`; render
with `releaseOutputBuffer(index, render=true)`; the C2 AVC decoder
requires a concrete width/height at configure (use a placeholder — the
in-band SPS reconfigures it).
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
surface.
- NSD: the classic `registerService(info, flags, RegistrationListener)`
API exists from API 16 through 36 (verified with javap on the android-36
SDK jar) — no fallback is needed. A reflection fallback targeting a
pre-36 "ResolutionListener" was removed: that class never existed at
any API level, so the fallback could only ever fail.
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
— letterbox by **sizing the TextureView to the video aspect** (centered
on a black window background), not with a transform matrix (double scale).
- mDNS does not cross subnets; `--peer` is the reliable path.
## Formatting
```sh
+9 -2
View File
@@ -10,7 +10,9 @@ namespace sc {
struct SendCommand {
std::string_view target = "monitor"; // monitor, window
std::string_view peer_address; // optional; empty means auto-discover
int bitrate_kbps = 4000;
int bitrate_kbps = 4000; // VBV max bitrate
int crf = 22; // constant rate factor (quality)
int fps = 0; // 0 = no cap (capture rate)
};
struct ReceiveCommand {
@@ -18,13 +20,18 @@ struct ReceiveCommand {
int local_rtp_port = 5004;
int signaling_port = 5005;
bool fullscreen = false;
bool software_decode = false; // --swdecode disables the hardware probe
};
struct DiscoverCommand {
int timeout_seconds = 3;
};
using Command = std::variant<SendCommand, ReceiveCommand, DiscoverCommand>;
struct WaybarCommand {
bool toggle = false; // toggle streaming instead of printing status
};
using Command = std::variant<SendCommand, ReceiveCommand, DiscoverCommand, WaybarCommand>;
// Parse command line arguments. Prints usage and returns std::nullopt on error.
// `argv` is `char const* const*` so both `main`'s `char**` and const arrays
+16
View File
@@ -19,6 +19,18 @@ struct SenderPipelineConfig {
// Where encoded RTP packets are sent. Defaults to the local loopback so
// a sender and receiver on one machine work without any configuration.
Endpoint peer_rtp_endpoint{"127.0.0.1", 5004};
// Session identity from signaling; written to the sender state file for
// status widgets (waybar) and one-click restarts.
std::string session_id;
// The receiver's display resolution (0 = unknown). The sender downscales
// to this before encoding so bitrate is not spent on pixels the display
// cannot show.
int max_encode_width = 0;
int max_encode_height = 0;
// Cap the encoding frame rate (0 = no cap; use the capture rate).
// Lowering the frame rate halves the bandwidth at the same quality
// level — useful on constrained links.
int max_frame_rate = 0;
};
struct ReceiverPipelineConfig {
@@ -39,6 +51,10 @@ class SenderPipeline {
bool start();
void stop();
// Ask the sender to encode its next frame as a keyframe. Thread-safe;
// used by the PLI feedback path.
void request_keyframe();
private:
class Impl;
std::unique_ptr<Impl> impl_;
+14 -1
View File
@@ -10,11 +10,21 @@
namespace sc {
// Decoded video frame in YUV420P planar format (the native decoder output,
// passed to the renderer without any colorspace conversion).
struct DecodedFrame {
int width = 0;
int height = 0;
uint64_t capture_timestamp_ns = 0;
std::vector<std::byte> rgba_pixels;
// YUV420P planes; each row is `stride` bytes wide and may be padded
// beyond the picture width.
std::vector<std::byte> plane_y;
std::vector<std::byte> plane_u;
std::vector<std::byte> plane_v;
int stride_y = 0;
int stride_u = 0;
int stride_v = 0;
};
struct DecoderConfig {
@@ -22,6 +32,9 @@ struct DecoderConfig {
int width = 0;
int height = 0;
std::vector<std::byte> extradata; // SPS/PPS for H.264
// Probe a hardware decoder (v4l2 mem2mem) first and fall back to the
// software decoder automatically. Disable with --swdecode.
bool hardware_accel = true;
};
class Decoder {
+6
View File
@@ -23,6 +23,12 @@ struct EncoderConfig {
int height = 0;
int frame_rate_num = 30;
int frame_rate_den = 1;
// CRF (constant rate factor, 0-51): the target visual quality level.
// Lower = better quality. 23 is x264's default; screen content looks
// good at 20-24.
int crf = 22;
// Maximum bitrate in kbps (the VBV cap). The encoder uses fewer bits on
// static content and more on motion, but never exceeds this.
int bitrate_kbps = 4000;
bool hardware_accel = false;
};
+13 -4
View File
@@ -42,14 +42,23 @@ class H264Packetizer {
std::uint16_t next_sequence_number_ = 0;
};
// Result of feeding one packet to the depacketizer.
struct DepacketizeResult {
// The completed access unit (Annex-B with 3-byte start codes) when the
// packet closed an undamaged frame.
std::optional<std::vector<std::byte>> access_unit;
// True when this call discarded a frame as damaged (packet loss or an
// unsupported packetization). Pipelines use it to request a keyframe.
bool frame_dropped = false;
};
// Reassembles RFC 6184 packet streams (single NAL unit packets and FU-A)
// into Annex-B access units. Packets must arrive in order; frames damaged by
// sequence gaps or missing fragments are dropped silently.
// sequence gaps or missing fragments are reported via DepacketizeResult.
class H264Depacketizer {
public:
// Feed one packet. Returns the completed access unit (Annex-B with 3-byte
// start codes) when the packet closes a frame, nullopt otherwise.
std::optional<std::vector<std::byte>> depacketize(const RtpPacket& packet);
// Feed one packet.
DepacketizeResult depacketize(const RtpPacket& packet);
private:
void drop_frame();
+29
View File
@@ -1,8 +1,12 @@
#pragma once
#include <chrono>
#include <cstdint>
#include <map>
#include <mutex>
#include <optional>
#include <span>
#include <utility>
#include <vector>
namespace sc {
@@ -31,4 +35,29 @@ struct RtpPacket {
static std::optional<RtpPacket> parse(std::span<const std::byte> in) noexcept;
};
// Reorders RTP packets by sequence number before depacketization so that a
// reordering link (Wi-Fi) does not read as loss. Delivery stays in order;
// only aged-out or overflowing buffers release out of order, which the
// downstream gap detection still handles for genuine loss.
class RtpJitterBuffer {
public:
explicit RtpJitterBuffer(std::size_t max_depth = 16,
std::chrono::milliseconds max_delay = std::chrono::milliseconds{60});
// Insert one packet and return the packets now ready for in-order
// delivery. In-order streams release immediately (zero added latency);
// a straggler older than the next expected sequence is discarded.
std::vector<RtpPacket> push(RtpPacket packet);
// Discard everything still buffered.
void clear();
private:
std::size_t max_depth_;
std::chrono::milliseconds max_delay_;
std::mutex mutex_;
std::map<std::uint16_t, std::pair<std::chrono::steady_clock::time_point, RtpPacket>> buffer_;
std::optional<std::uint16_t> next_expected_;
};
} // namespace sc
+11 -1
View File
@@ -29,9 +29,19 @@ struct SessionAnswer {
// The address may be empty: the sender then targets the address of its
// signaling connection and the port carried here.
Endpoint rtp_endpoint;
// The receiver's display resolution (0 = unknown). The sender may
// downscale to this to avoid encoding pixels the display cannot show.
int display_width = 0;
int display_height = 0;
};
using SignalingMessage = std::variant<SessionOffer, SessionAnswer>;
// Picture Loss Indication: the receiver asks the sender for a keyframe
// after discarding a damaged frame.
struct SessionPli {
std::string session_id;
};
using SignalingMessage = std::variant<SessionOffer, SessionAnswer, SessionPli>;
class SignalingChannel {
public:
+5
View File
@@ -24,6 +24,11 @@ class Renderer {
// Present one decoded frame. Returns false if the window was closed.
virtual bool present(const DecodedFrame& frame) = 0;
// The display resolution (native monitor size in fullscreen mode).
// Used by the receiver to tell the sender what resolution to encode at.
virtual int display_width() const = 0;
virtual int display_height() const = 0;
// Pump events (window close, resize). Non-blocking.
virtual bool poll_events() = 0;
+77
View File
@@ -0,0 +1,77 @@
# screencast iOS receiver
A native iOS receiver app (Swift) so an iPhone can act as a second receiver.
It speaks the **existing signaling + RTP protocol unchanged** — no C++ changes —
mirroring the Android receiver (Phase 8) source-to-source.
- **Min iOS:** 17 (validated on iPhone 16 / iOS 26.6.1)
- **No third-party dependencies** — only Apple system frameworks (SwiftUI,
AVFoundation, VideoToolbox, CoreMedia, BSD sockets).
## Pipeline
```
Bonjour advertise (_screencast._tcp) + TCP signaling server (offer → answer)
UDP RTP → jitter buffer (16 pkt / 60 ms) → depacketize (single-NAL + FU-A)
→ VideoToolbox H.264 (in-band SPS/PPS) → AVSampleBufferDisplayLayer (letterbox)
```
Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
(rate-limited to one per 500 ms) asks the sender for a keyframe.
## Layout
```
Receiver/
App/ SwiftUI app, video surface (AVSampleBufferDisplayLayer), controller
Rtp/ RtpHeader, RtpPacket, JitterBuffer, H264Depacketizer, Avcc, NAL extraction
Signaling/ SignalingMessage (JSON), LineAssembler, SignalingServer (BSD sockets)
Decode/ RenderSink, H.264 format description, VideoToolbox decoder
Support/ UdpTransport (poll-based), LocalAddress
Pipeline/ ReceiverPipeline (coordinates everything)
ReceiverTests/
XCTest port of the Android JVM tests + added coverage (signaling, line
framing, AVCC, NAL extraction)
```
## Build and test (on a Mac with Xcode 26)
```sh
./bootstrap.sh # installs XcodeGen, generates the project, builds for device
./bootstrap.sh test # runs the unit tests on the simulator
```
`bootstrap.sh` downloads XcodeGen from the GitHub release into `tools/` (no
Homebrew). Override the simulator with `IOS_DEST="platform=iOS Simulator,name=…"`.
## Install on a device
Free provisioning (or your team) is set in Xcode — this can't be scripted:
1. Open `Receiver.xcodeproj`.
2. Select the **Receiver** target → *Signing & Capabilities* → set your Apple ID
(a 7-day development certificate is fine for sideloading).
3. Run to the iPhone (USB, trust the computer).
The first run prompts for **Local Network** access — allow it, or the sender
will never discover the phone (silent failure, like the Android
`PROTOCOL_DNS_SD` bug).
## Stream to the phone
On the sender (Linux desktop):
```sh
screencast --send --peer <phone-ip>:5005 # target the phone directly
screencast --discover # or list receivers; the phone shows up
```
Expected: the phone shows the captured desktop, letterboxed to its screen, with
PLI recovery on Wi-Fi loss.
## Testing notes
The pure protocol core (RTP, jitter, depacketizer, signaling JSON, line framing,
AVCC, NAL extraction) is unit-tested in the simulator. The VideoToolbox decode
path and the local-network/Bonjour flow are on-device validation items — the
highest-risk parts to verify on the borrowed Mac + iPhone.
+88
View File
@@ -0,0 +1,88 @@
import SwiftUI
import UIKit
/// Owns the pipeline and mirrors its status to the UI. The pipeline posts its
/// callbacks to the main thread, so the @Published mutations happen there.
final class ReceiverController: ObservableObject {
@Published var status = "Starting…"
@Published var showStatus = true
private let pipeline: ReceiverPipeline
private let pixelSize: CGSize
init() {
let localIP = LocalAddress.primaryIPv4()
let size = ReceiverController.screenPixelSize()
pixelSize = size
pipeline = ReceiverPipeline(
localIP: localIP,
displaySize: { [weak self] in self?.pixelSize ?? .zero },
onStatus: { [weak self] text in self?.apply(status: text) },
onFirstFrame: { [weak self] in self?.apply(showStatus: false) },
onVideoSize: { _ in })
}
func start() { pipeline.start() }
func stop() { pipeline.stop() }
func bindSink(_ sink: RenderSink) { pipeline.attachSink(sink) }
private func apply(status: String? = nil, showStatus: Bool? = nil) {
if let s = status { self.status = s }
if let v = showStatus { self.showStatus = v }
}
static func screenPixelSize() -> CGSize {
let scale = UIScreen.main.scale
let b = UIScreen.main.bounds
return CGSize(width: b.width * scale, height: b.height * scale)
}
}
struct ContentView: View {
@StateObject private var controller = ReceiverController()
@Environment(\.scenePhase) private var scenePhase
var body: some View {
ZStack {
Color.black.ignoresSafeArea()
VideoSurfaceView { sink in controller.bindSink(sink) }
.ignoresSafeArea()
if controller.showStatus {
VStack {
Spacer()
Text(controller.status)
.font(.footnote)
.foregroundStyle(.white)
.multilineTextAlignment(.center)
.padding(.horizontal, 24)
.padding(.vertical, 12)
.background(.black.opacity(0.55), in: RoundedRectangle(cornerRadius: 10))
Spacer()
Spacer()
}
.transition(.opacity)
}
}
.onAppear { controller.start() }
.onChange(of: scenePhase) { _, phase in
switch phase {
case .active: controller.start()
case .inactive, .background: controller.stop()
@unknown default: break
}
}
}
}
@main
struct ReceiverApp: App {
var body: some Scene {
WindowGroup {
ContentView()
.preferredColorScheme(.dark)
.statusBarHidden(true)
.persistentSystemOverlays(.hidden)
}
}
}
+34
View File
@@ -0,0 +1,34 @@
import SwiftUI
import AVFoundation
/// Hosts the `AVSampleBufferDisplayLayer` and exposes it as a `RenderSink`.
/// The layer keeps the full screen and letterboxes via `.resizeAspect`, so the
/// decoded stream (already downscaled to the display size by the sender) is
/// shown 1:1 with no distortion.
struct VideoSurfaceView: UIViewRepresentable {
let onSink: (RenderSink) -> Void
func makeCoordinator() -> Coordinator {
Coordinator()
}
func makeUIView(context: Context) -> UIView {
let view = UIView(frame: .zero)
view.backgroundColor = .black
let layer = AVSampleBufferDisplayLayer()
layer.videoGravity = .resizeAspect
view.layer.addSublayer(layer)
let sink = AVSampleBufferRenderSink(layer: layer)
context.coordinator.sink = sink
onSink(sink)
return view
}
func updateUIView(_ uiView: UIView, context: Context) {}
final class Coordinator {
var sink: AVSampleBufferRenderSink?
}
}
@@ -0,0 +1,57 @@
import CoreMedia
import Foundation
/// Builds a `CMVideoFormatDescription` for H.264 from in-band SPS + PPS.
///
/// VideoToolbox needs the parameter sets up front; the sender repeats them
/// before every keyframe, so any keyframe carries a complete set. This is the
/// Core Foundation recipe for an H.264 "config" format description.
enum H264FormatDescription {
static func create(sps: [UInt8], pps: [UInt8]) -> CMVideoFormatDescription? {
let pointersArray = CFArrayCreateMutable(kCFAllocatorDefault, 0, &kCFTypeArrayCallBacks)
for ps in [sps, pps] {
guard let descriptor = makeParameterSetDescriptor(ps) else {
CFRelease(pointersArray)
return nil
}
CFArrayAppendValue(pointersArray, descriptor.takeUnretainedValue())
CFRelease(descriptor) // the array now owns it
}
let key = kCMFormatDescriptionExtension_SampleDescriptionPointers as CFString
let attrs = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFDictionaryKeyCallBacks, &kCFDictionaryValueCallBacks)
CFDictionarySetValue(attrs, key, pointersArray)
var config: Unmanaged<CMVideoFormatDescription>?
let status = CMVideoFormatDescriptionCreate(
kCFAllocatorDefault,
kCMVideoCodecType_H264,
0, 0, 0,
attrs,
&config)
CFRelease(attrs)
CFRelease(pointersArray)
guard status == noErr, let c = config else { return nil }
return c.takeRetainedValue()
}
private static func makeParameterSetDescriptor(_ ps: [UInt8]) -> Unmanaged<CMVideoFormatDescription>? {
let cfData = Data(ps) as CFData
let oneElement = CFArrayCreateMutable(kCFAllocatorDefault, 0, &kCFTypeArrayCallBacks)
CFArrayAppendValue(oneElement, cfData)
let key = kCMFormatDescriptionExtension_SampleDescriptionPointers as CFString
let attrs = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFDictionaryKeyCallBacks, &kCFDictionaryValueCallBacks)
CFDictionarySetValue(attrs, key, oneElement)
var desc: Unmanaged<CMVideoFormatDescription>?
let status = CMVideoFormatDescriptionCreateForCodecType(
kCFAllocatorDefault,
kCMVideoCodecType_H264,
attrs,
&desc)
CFRelease(oneElement)
CFRelease(attrs)
guard status == noErr, let d = desc else { return nil }
return d
}
}
@@ -0,0 +1,174 @@
import VideoToolbox
import CoreMedia
import CoreGraphics
import Foundation
/// Hardware H.264 decoder using VideoToolbox. The stream is self-describing:
/// the sender repeats SPS/PPS in-band at every keyframe, so no out-of-band
/// codec data is needed. On a keyframe the in-band SPS/PPS (re)establish the
/// stream size; when that size changes, the decode session is recreated the
/// Android C2 "in-band SPS reconfigure" pattern, which also avoids the startup
/// squish (we never report a placeholder size before the first real keyframe).
///
/// The decode output callback may run on a worker thread, so session and
/// format-description access is guarded by a lock.
final class H264VideoToolboxDecoder {
private let renderSink: RenderSink
private let stateLock = NSLock()
private var session: VTDecompressionSession?
private var formatDescription: CMVideoFormatDescription?
private var streamSize = CGSize.zero
private var realSizeSeen = false
init(renderSink: RenderSink) {
self.renderSink = renderSink
}
/// The decoded resolution, once the first keyframe configured the session
/// (nil before that the caller must not drive layout off a placeholder).
func outputSize() -> CGSize? {
guard realSizeSeen, streamSize != .zero else { return nil }
return streamSize
}
/// Decodes one access unit (Annex-B) and renders the output. Returns false
/// when a decode error occurred and the caller should request a keyframe.
func decode(accessUnit annexB: [UInt8], rtpTimestamp: Int, isKeyFrame: Bool) -> Bool {
if isKeyFrame {
guard let sets = NalExtractor.parameterSets(annexB) else { return false }
configureIfNeeded(sps: sets.sps, pps: sets.pps)
}
stateLock.lock()
let session = self.session
let cd = self.formatDescription
stateLock.unlock()
guard let session, let cd else { return false }
guard let avcc = AvccConverter.toAvcc(annexB) else { return false }
let pts = CMTime(value: CMTimeValue(rtpTimestamp), timescale: 90000)
guard let blockBuffer = makeBlockBuffer(avcc) else { return false }
var infoFlags: VTDecodeInfoFlags = []
let status = VTDecompressionSessionDecodeFrame(
session,
blockBuffer,
isKeyFrame ? kVTDecodeFrameFlags_EnableFastPath : 0,
&infoFlags,
pts)
CFRelease(blockBuffer)
if status != noErr {
// Recoverable: the next keyframe (SPS/PPS + IDR) re-primes it.
teardownSession()
return false
}
realSizeSeen = true
return true
}
func release() {
stateLock.lock()
teardownSessionLocked()
if let cd = formatDescription { CFRelease(cd) }
formatDescription = nil
stateLock.unlock()
renderSink.detach()
streamSize = .zero
realSizeSeen = false
}
// MARK: - Internals
private func configureIfNeeded(sps: [UInt8], pps: [UInt8]) {
guard let cd = H264FormatDescription.create(sps: sps, pps: pps) else { return }
var dims = CMVideoDimensions()
guard CMVideoFormatDescriptionGetDimensions(cd, &dims) == noErr else {
CFRelease(cd)
return
}
let size = CGSize(width: CGFloat(dims.width), height: CGFloat(dims.height))
stateLock.lock()
// Same size and a live session: keep it (the sender only changes the
// stream when the receiver's display size changes).
if session != nil && size == streamSize {
CFRelease(cd)
stateLock.unlock()
return
}
teardownSessionLocked()
var outSession: VTDecompressionSession?
let status = VTDecompressionSessionCreate(
kCFAllocatorDefault,
cd,
vtOutputCallback,
Unmanaged.passUnretained(self).toOpaque(),
&outSession)
guard status == noErr, let newSession = outSession else {
CFRelease(cd)
stateLock.unlock()
return
}
// Low-latency decode: emit as soon as the frame is complete.
VTSessionSetProperty(newSession, kVTDecompressionPropertyKey_RealTime, kCFBooleanTrue)
renderSink.setFormatDescription(cd)
formatDescription = cd // we hold the +1 from H264FormatDescription.create
session = newSession
streamSize = size
realSizeSeen = false
stateLock.unlock()
}
// Caller holds stateLock.
private func teardownSessionLocked() {
if let s = session {
VTDecompressionSessionInvalidate(s)
CFRelease(s)
}
session = nil
}
// Caller does not hold the lock.
private func teardownSession() {
stateLock.lock()
teardownSessionLocked()
stateLock.unlock()
}
private func makeBlockBuffer(_ bytes: [UInt8]) -> CMBlockBuffer? {
let cfData = Data(bytes) as CFData
var blockBuffer: CMBlockBuffer?
let status = CMBlockBufferCreateWithData(kCFAllocatorDefault, cfData, &blockBuffer)
guard status == noErr, let bb = blockBuffer else { return nil }
return bb
}
// Runs on whatever thread VideoToolbox uses for the output callback.
private func handleOutput(_ pixelBuffer: CVPixelBuffer?, _ presentationTime: CMTime?) {
stateLock.lock()
let cd = formatDescription
stateLock.unlock()
guard let cd, let pixelBuffer else { return }
let pts = presentationTime ?? CMTime(value: 0, timescale: 600)
var sampleBuffer: CMSampleBuffer?
let status = CMSampleBufferCreate(kCFAllocatorDefault, nil, 0, pts, .invalid, 1, 0, nil, &sampleBuffer)
guard status == noErr, let sb = sampleBuffer else { return }
guard CMSampleBufferSetDataBufferFromPixelBuffer(sb, pixelBuffer) == noErr else {
CFRelease(sb)
return
}
renderSink.enqueue(sb)
}
}
/// C-compatible decode output callback; recovers the decoder from the refCon.
private func vtOutputCallback(_ refCon: UnsafeMutableRawPointer?,
_ pixelBuffer: CVPixelBuffer?,
_ presentationTime: CMTime?,
_ duration: CMTime?,
_ infoFlags: VTDecodeInfoFlags) {
guard let refCon = refCon else { return }
let decoder = Unmanaged<H264VideoToolboxDecoder>.fromOpaque(refCon).takeUnretainedValue()
decoder.handleOutput(pixelBuffer, presentationTime: presentationTime)
}
+61
View File
@@ -0,0 +1,61 @@
import AVFoundation
import CoreMedia
/// A render target for decoded frames. The decoder enqueues a `CMSampleBuffer`
/// (wrapping a CVPixelBuffer) per frame; the sink renders it. Mirrors the role
/// of the Android `Surface` + MediaCodec surface-mode rendering.
protocol RenderSink: AnyObject {
var isAttached: Bool { get }
func attach()
func detach()
/// Updates the layer's video format description (called when the in-band
/// SPS/PPS establish a (new) stream size).
func setFormatDescription(_ formatDescription: CMVideoFormatDescription)
/// Enqueues one decoded frame for display.
func enqueue(_ sampleBuffer: CMSampleBuffer)
}
/// Renders decoded CVPixelBuffers via `AVSampleBufferDisplayLayer`, which does
/// the video-scaling for us. `.resizeAspect` gives letterbox directly, so the
/// host view can stay full-screen without a manual transform (the same lesson
/// as the Android TextureView sizing: size the surface to the aspect, not a
/// transform matrix).
final class AVSampleBufferRenderSink: RenderSink {
private let layer: AVSampleBufferDisplayLayer
private var session: AVSampleBufferDisplayLayerSession?
private let lock = NSLock()
private(set) var isAttached = false
init(layer: AVSampleBufferDisplayLayer) {
self.layer = layer
}
func attach() {
lock.lock()
defer { lock.unlock() }
guard session == nil else { return }
let s = AVSampleBufferDisplayLayerSession(layer)
s.start()
session = s
isAttached = true
}
func detach() {
lock.lock()
defer { lock.unlock() }
session?.stop()
session = nil
isAttached = false
}
func setFormatDescription(_ formatDescription: CMVideoFormatDescription) {
layer.formatDescription = formatDescription
}
func enqueue(_ sampleBuffer: CMSampleBuffer) {
lock.lock()
let s = session
lock.unlock()
s?.enqueue(sampleBuffer)
}
}
@@ -0,0 +1,268 @@
import Foundation
import CoreGraphics
/// The receiver pipeline, mirroring the C++ ReceiverPipeline and the Kotlin
/// receiver:
///
/// signaling server (offer answer)
/// UDP RTP jitter buffer depacketize VideoToolbox render sink
///
/// Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
/// (rate-limited to one per 500 ms) asks the sender for a keyframe.
///
/// Concurrency: all shared state and every decode call run on a single serial
/// queue; the reader thread only polls/receives UDP and forwards datagrams to
/// that queue. UI callbacks are marshalled to the main thread.
final class ReceiverPipeline {
static let desiredUdpPort: UInt16 = 5004
static let desiredSignalingPort: UInt16 = 5005
static let pliMinIntervalMs: Int = 500
static let receiveBufferSize: Int32 = 4 * 1024 * 1024
private let localIP: String
private let displaySize: () -> CGSize
private let onStatus: (String) -> Void
private let onFirstFrame: () -> Void
private let onVideoSize: (CGSize) -> Void
private let queue = DispatchQueue(label: "sc.receiver.pipeline")
private var running = false
private var udp: UdpTransport?
private var signaling: SignalingServer?
private var readerThread: Thread?
private let jitter = JitterBuffer()
private var depacketizer = H264Depacketizer()
private var decoder: H264VideoToolboxDecoder?
private var renderSink: RenderSink?
private var pendingOffer: SignalingMessage?
private var activeSession = ""
private var firstFrameSeen = false
private var currentVideoSize = CGSize.zero
private var lastPliAtMs: Int64 = 0
init(localIP: String,
displaySize: @escaping () -> CGSize,
onStatus: @escaping (String) -> Void,
onFirstFrame: @escaping () -> Void,
onVideoSize: @escaping (CGSize) -> Void) {
self.localIP = localIP
self.displaySize = displaySize
self.onStatus = onStatus
self.onFirstFrame = onFirstFrame
self.onVideoSize = onVideoSize
}
/// The current decoded resolution (zero until the first keyframe).
func videoSize() -> CGSize {
return queue.sync { currentVideoSize }
}
// MARK: - Lifecycle
/// Binds the ports, starts the signaling server, and starts reading RTP.
func start() {
queue.async { [weak self] in
guard let self, !self.running else { return }
let udp = UdpTransport()
guard udp.bind(preferredPort: Self.desiredUdpPort, receiveBufferSize: Self.receiveBufferSize) else {
self.postStatus("Failed to bind the media port")
return
}
let signaling = SignalingServer(
onOffer: { [weak self] offer in self?.queue.async { self?.handleOffer(offer) } },
onPli: { _ in })
guard signaling.start(Self.desiredSignalingPort) else {
udp.close()
self.postStatus("Failed to start signaling")
return
}
self.running = true
self.udp = udp
self.signaling = signaling
let thread = Thread { [weak self] in self?.readLoop(udp: udp) }
thread.name = "rtp-reader"
self.readerThread = thread
thread.start()
let ip = self.localIP
let mediaPort = udp.port
let sigPort = signaling.port
self.postStatus("Listening on \(ip) (media :\(mediaPort), signaling :\(sigPort))\nWaiting for a sender… (fall back to: screencast --send --peer \(ip):\(sigPort))")
}
}
/// Points the (current or future) decoder at a render sink. A pending offer
/// (accepted while no sink existed) configures its decoder now and requests
/// a keyframe, since the sender only emits one when asked.
func attachSink(_ sink: RenderSink) {
queue.async { [weak self] in
guard let self else { return }
self.renderSink = sink
sink.attach()
guard let offer = self.pendingOffer else { return }
self.pendingOffer = nil
// The decoder configures on the first keyframe; the sink is already
// attached, so creation cannot fail. A late-configured decoder needs
// a keyframe (the sender only emits one when asked).
self.decoder = H264VideoToolboxDecoder(renderSink: sink)
self.requestPli()
}
}
/// Forgets a destroyed render sink so a later offer cannot render into it.
func detachSink() {
queue.async { [weak self] in
guard let self else { return }
self.decoder?.release()
self.decoder = nil
self.renderSink?.detach()
self.renderSink = nil
}
}
/// Stops listening; the pipeline can be started again.
func stop() {
queue.async { [weak self] in
guard let self, self.running else { return }
self.running = false
self.activeSession = ""
self.udp?.close()
self.udp = nil
self.readerThread?.join()
self.readerThread = nil
self.signaling?.close()
self.signaling = nil
self.decoder?.release()
self.decoder = nil
self.pendingOffer = nil
self.firstFrameSeen = false
self.currentVideoSize = .zero
self.postStatus("Stopped")
}
}
// MARK: - Media path
private func readLoop(udp: UdpTransport) {
while true {
switch udp.poll(timeoutMs: 100) {
case 1:
if let data = udp.receiveDatagram() {
queue.async { [weak self] in self?.processDatagram(data) }
}
case 0:
continue // timeout: re-check on the next poll
default:
return // closed/errored: the socket was closed by stop()
}
}
}
private func processDatagram(_ data: [UInt8]) {
guard let packet = RtpPacket.parse(data) else { return }
for released in jitter.push(packet) {
handleDepacketized(released)
}
}
private func handleDepacketized(_ packet: RtpPacket) {
let result = depacketizer.depacketize(packet)
if let accessUnit = result.accessUnit {
let presentation = Int(packet.header.timestamp & 0xFFFFFFFF)
guard let decoder = decoder else { return }
if !decoder.decode(accessUnit: accessUnit, rtpTimestamp: presentation, isKeyFrame: result.isKeyFrame) {
// Input/decode trouble: the dropped frame corrupts the GOP
// until the next keyframe ask for one.
requestPli()
}
if !firstFrameSeen {
firstFrameSeen = true
postFirstFrame()
}
postVideoSizeIfChanged()
}
if result.frameDropped {
requestPli()
}
}
private func postVideoSizeIfChanged() {
guard let size = decoder?.outputSize(), size != .zero else { return }
if size != currentVideoSize {
currentVideoSize = size
postVideoSize(size)
}
}
// MARK: - Signaling
private func handleOffer(_ offer: SignalingMessage) {
guard case let .offer(sessionId, codec, _, _, _, _, _, _) = offer else { return }
if codec != "h264" {
postStatus("Unsupported codec: \(codec)")
return
}
// New session: pristine decoder, reassembly state, and session.
decoder?.release()
decoder = nil
pendingOffer = nil
if renderSink != nil {
decoder = H264VideoToolboxDecoder(renderSink: renderSink!)
} else {
// No surface yet: park the offer; attachSink configures later.
pendingOffer = offer
}
depacketizer = H264Depacketizer()
jitter.clear()
firstFrameSeen = false
currentVideoSize = .zero
activeSession = sessionId
let size = displaySize()
let answer = SignalingMessage.answer(
sessionId: sessionId,
rtpAddress: "", // the sender targets the address of its own signaling connection
rtpPort: Int(udp?.port ?? 0),
displayWidth: Int(size.width),
displayHeight: Int(size.height))
signaling?.send(answer)
if decoder != nil {
postStatus("Session \(sessionId) negotiated — waiting for the first frame…")
} else {
postStatus("Session \(sessionId) negotiated — waiting for the display surface…")
}
}
/// Rate-limited keyframe request, callable from any thread (it always runs
/// on the pipeline queue in practice).
private func requestPli() {
let session = activeSession
guard !session.isEmpty else { return }
let now = Int64(Date().timeIntervalSince1970 * 1000)
if now - lastPliAtMs < Int64(Self.pliMinIntervalMs) { return }
lastPliAtMs = now
signaling?.send(.pli(sessionId: session))
}
// MARK: - UI callbacks (main thread)
private func postStatus(_ text: String) {
DispatchQueue.main.async { [onStatus] in onStatus(text) }
}
private func postFirstFrame() {
DispatchQueue.main.async { [onFirstFrame] in onFirstFrame() }
}
private func postVideoSize(_ size: CGSize) {
DispatchQueue.main.async { [onVideoSize] in onVideoSize(size) }
}
}
+73
View File
@@ -0,0 +1,73 @@
import Foundation
/// Converts between Annex-B (start-code delimited) and AVCC (4-byte
/// big-endian length prefixed) H.264 representations.
///
/// VideoToolbox consumes AVCC: each NAL unit is preceded by a 32-bit length.
/// Our depacketizer emits Annex-B (the project's canonical 3-byte start codes),
/// so the decoder feeds AVCC derived here.
enum AvccConverter {
/// Splits an Annex-B access unit into its NAL units (start codes removed).
static func nalUnits(_ annexB: [UInt8]) -> [[UInt8]] {
guard annexB.count >= 4 else { return [] }
let n = annexB.count
// Locate every start code (3-byte `00 00 01` and 4-byte `00 00 00 01`).
var offsets: [Int] = []
var i = 0
while i + 2 < n {
if annexB[i] == 0 && annexB[i + 1] == 0 && annexB[i + 2] == 1 {
offsets.append(i)
i += 3
continue
}
if i + 3 < n, annexB[i + 2] == 0, annexB[i + 3] == 1 {
offsets.append(i)
i += 4
continue
}
i += 1
}
guard !offsets.isEmpty else { return [] }
var units: [[UInt8]] = []
for (idx, offset) in offsets.enumerated() {
let nalStart = offset + 3
let nalEnd = idx + 1 < offsets.count ? offsets[idx + 1] : n
let nal = Array(annexB[nalStart..<nalEnd])
if !nal.isEmpty { units.append(nal) }
}
return units
}
/// Returns the AVCC form of an Annex-B access unit, or nil if it has no NALs.
static func toAvcc(_ annexB: [UInt8]) -> [UInt8]? {
let units = nalUnits(annexB)
guard !units.isEmpty else { return nil }
var out: [UInt8] = []
out.reserveCapacity(annexB.count + units.count * 4)
for nal in units {
let len = nal.count
out.append(UInt8((len >> 24) & 0xFF))
out.append(UInt8((len >> 16) & 0xFF))
out.append(UInt8((len >> 8) & 0xFF))
out.append(UInt8(len & 0xFF))
out.append(contentsOf: nal)
}
return out
}
/// Splits an AVCC byte array (4-byte big-endian length prefixes) into NAL units.
static func fromAvcc(_ avcc: [UInt8]) -> [[UInt8]] {
var units: [[UInt8]] = []
var i = 0
let n = avcc.count
while i + 4 <= n {
let len = (Int(avcc[i]) << 24) | (Int(avcc[i + 1]) << 16) | (Int(avcc[i + 2]) << 8) | Int(avcc[i + 3])
guard len > 0, i + 4 + len <= n else { break }
units.append(Array(avcc[(i + 4)..<(i + 4 + len)]))
i += 4 + len
}
return units
}
}
+160
View File
@@ -0,0 +1,160 @@
import Foundation
/// Result of feeding one packet to the depacketizer.
struct DepacketizeResult {
/// Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged.
var accessUnit: [UInt8]?
/// True when this call discarded a frame as damaged (packet loss or unsupported packetization).
var frameDropped = false
/// True when the completed access unit carries SPS/PPS (a keyframe).
var isKeyFrame = false
}
/// Reassembles RFC 6184 packet streams (single NAL unit packets and FU-A)
/// into Annex-B access units. Packets must arrive in order; frames damaged by
/// sequence gaps or missing fragments are reported via DepacketizeResult.
///
/// Mirrors the C++ `H264Depacketizer` (same state machine and start codes).
final class H264Depacketizer {
static let fuA = 28
private var lastSequenceNumber: Int?
private var frameStarted = false
private var frameDamaged = false
private var frameTimestamp = 0
// Growable byte accumulators: keyframes reach hundreds of KB.
private var accessUnit: [UInt8] = []
private var fuActive = false
private var fuNal: [UInt8] = []
/// Feed one packet (in sequence order, from the jitter buffer).
func depacketize(_ packet: RtpPacket) -> DepacketizeResult {
var result = DepacketizeResult()
// Track sequence continuity: a gap means packets were lost.
if let last = lastSequenceNumber {
let expected = (last + 1) & 0xFFFF
if packet.header.sequenceNumber != expected {
fuActive = false
fuNal.removeAll(keepingCapacity: true)
if frameStarted { frameDamaged = true }
}
}
lastSequenceNumber = packet.header.sequenceNumber
// A timestamp change without a closing marker means the previous frame
// lost its tail and can no longer be recovered.
if frameStarted && packet.header.timestamp != frameTimestamp {
dropFrame()
result.frameDropped = true
}
if !frameStarted {
frameStarted = true
frameDamaged = false
frameTimestamp = packet.header.timestamp
accessUnit.removeAll(keepingCapacity: true)
}
let payload = packet.payload
if !payload.isEmpty {
let type = Int(payload[0]) & 0x1F
if type >= 1 && type <= 23 {
// Single NAL unit packet.
if fuActive {
frameDamaged = true
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
appendStartCode()
accessUnit.append(contentsOf: payload)
} else if type == Self.fuA {
if payload.count < 2 {
frameDamaged = true
} else {
let fuHeader = Int(payload[1])
let start = fuHeader & 0x80 != 0
let end = fuHeader & 0x40 != 0
let fragment = Array(payload[2...])
if start {
if fuActive {
// The previous fragmented NAL lost its end packet.
frameDamaged = true
}
fuActive = true
fuNal.removeAll(keepingCapacity: true)
// The FU indicator keeps the original NAL's F bit (0) and NRI,
// and declares type 28; the FU header carries S/E plus the real type.
fuNal.append(UInt8((Int(payload[0]) & 0xE0) | (fuHeader & 0x1F)))
fuNal.append(contentsOf: fragment)
} else if !fuActive {
// Continuation without a start: the head of the NAL is lost.
frameDamaged = true
} else {
fuNal.append(contentsOf: fragment)
if end {
appendStartCode()
accessUnit.append(contentsOf: fuNal)
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
}
}
} else {
// Unsupported packetization mode (STAP-A, MTAP, FU-B).
frameDamaged = true
}
}
if !packet.header.marker {
return result
}
if fuActive {
// The marker arrived while a NAL was still fragmented.
frameDamaged = true
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
if !frameDamaged && !accessUnit.isEmpty {
let unit = accessUnit
result.accessUnit = unit
result.isKeyFrame = Self.containsParameterSets(unit)
} else {
// The frame that just ended is unusable.
result.frameDropped = true
}
dropFrame()
return result
}
private func appendStartCode() {
accessUnit.append(0)
accessUnit.append(0)
accessUnit.append(1)
}
/// The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8.
private static func containsParameterSets(_ unit: [UInt8]) -> Bool {
guard unit.count >= 4 else { return false }
var i = 0
while i <= unit.count - 4 {
if unit[i] == 0 && unit[i + 1] == 0 && unit[i + 2] == 1 {
let nalType = Int(unit[i + 3]) & 0x1F
if nalType == 7 || nalType == 8 {
return true
}
}
i += 1
}
return false
}
private func dropFrame() {
frameStarted = false
frameDamaged = false
accessUnit.removeAll(keepingCapacity: true)
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
}
+83
View File
@@ -0,0 +1,83 @@
import Foundation
/// Reorders RTP packets by sequence number before depacketization so that a
/// reordering link (Wi-Fi) is not read as loss. Delivery stays in order; only
/// aged-out or overflowing buffers release out of order, which the downstream
/// gap detection still handles for genuine loss.
///
/// Mirrors the C++ `RtpJitterBuffer` (same defaults and semantics) and the
/// Kotlin receiver.
final class JitterBuffer {
private struct Entry {
let timeNs: Int64
let packet: RtpPacket
}
private let maxDepth: Int
private let maxDelayNs: Int64
private var buffer: [Int: Entry] = [:]
private var nextExpected: Int?
init(maxDepth: Int = 16, maxDelayMs: Int = 60) {
self.maxDepth = maxDepth
self.maxDelayNs = Int64(maxDelayMs) * 1_000_000
}
/// Insert one packet and return the packets now ready for in-order delivery.
func push(_ packet: RtpPacket) -> [RtpPacket] {
var released: [RtpPacket] = []
let sequence = packet.header.sequenceNumber
let now = Self.nowNanos()
let expected0: Int
if let e = nextExpected {
expected0 = e
} else {
expected0 = sequence
nextExpected = sequence
}
// Serial-number comparison: a distance >= 32768 means the packet is
// older than what we already delivered (duplicate or straggler).
let distance = (sequence - expected0 + 65536) % 65536
if distance < 32768 {
buffer[sequence] = Entry(timeNs: now, packet: packet)
// Release the consecutive run from the expected sequence.
var expected = expected0
while let entry = buffer[expected] {
released.append(entry.packet)
buffer.removeValue(forKey: expected)
expected = (expected + 1) & 0xFFFF
}
nextExpected = expected
// A missing packet stalls the run: age out the backlog (or bound
// the buffer) and release what is there in order, so genuine loss
// reaches the depacketizer's gap detection rather than blocking.
if !buffer.isEmpty {
let keys = buffer.keys.sorted()
let head = keys.first!
let headAgeNs = now - buffer[head]!.timeNs
if headAgeNs > maxDelayNs || buffer.count > maxDepth {
for key in keys {
released.append(buffer[key]!.packet)
}
nextExpected = (keys.last! + 1) & 0xFFFF
buffer.removeAll(keepingCapacity: true)
}
}
}
return released
}
/// Discard everything still buffered.
func clear() {
buffer.removeAll(keepingCapacity: true)
nextExpected = nil
}
private static func nowNanos() -> Int64 {
return Int64(DispatchTime.now().uptimeNanoseconds)
}
}
+29
View File
@@ -0,0 +1,29 @@
import Foundation
/// The in-band SPS (type 7) and PPS (type 8) NAL units extracted from an
/// access unit. The sender repeats both ahead of every keyframe, so any
/// keyframe carries them; they are the source for the VideoToolbox format
/// description.
struct NalSets {
let sps: [UInt8]
let pps: [UInt8]
}
/// Extracts parameter sets from an Annex-B access unit.
enum NalExtractor {
static func parameterSets(_ annexB: [UInt8]) -> NalSets? {
var sps: [UInt8]?
var pps: [UInt8]?
for nal in AvccConverter.nalUnits(annexB) {
guard !nal.isEmpty else { continue }
let type = Int(nal[0]) & 0x1F
if type == 7 && sps == nil {
sps = nal
} else if type == 8 && pps == nil {
pps = nal
}
}
guard let s = sps, let p = pps else { return nil }
return NalSets(sps: s, pps: p)
}
}
+56
View File
@@ -0,0 +1,56 @@
import Foundation
/// Minimal RTP header (RFC 3550) without extensions, mirroring the C++
/// `RtpHeader` and the Kotlin receiver.
struct RtpHeader: Equatable {
var version = 2
var padding = false
var extensionHeader = false
var csrcCount = 0
var marker = false
var payloadType = 96
var sequenceNumber = 0
var timestamp = 0
var ssrc = 0
/// Serializes the bare 12-byte header; requires a version-2, extension-less header.
func serialize() -> [UInt8] {
var out = [UInt8](repeating: 0, count: 12)
out[0] = UInt8((version & 0x0F) << 6
| (padding ? 0x20 : 0)
| (extensionHeader ? 0x10 : 0)
| (csrcCount & 0x0F))
out[1] = UInt8((marker ? 0x80 : 0) | (payloadType & 0x7F))
out[2] = UInt8((sequenceNumber >> 8) & 0xFF)
out[3] = UInt8(sequenceNumber & 0xFF)
out[4] = UInt8((timestamp >> 24) & 0xFF)
out[5] = UInt8((timestamp >> 16) & 0xFF)
out[6] = UInt8((timestamp >> 8) & 0xFF)
out[7] = UInt8(timestamp & 0xFF)
out[8] = UInt8((ssrc >> 24) & 0xFF)
out[9] = UInt8((ssrc >> 16) & 0xFF)
out[10] = UInt8((ssrc >> 8) & 0xFF)
out[11] = UInt8(ssrc & 0xFF)
return out
}
/// Parses a 12-byte header from the start of a datagram.
static func parse(_ input: [UInt8]) -> RtpHeader? {
guard input.count >= 12 else { return nil }
let b0 = Int(input[0])
let b1 = Int(input[1])
let version = b0 >> 6
guard version == 2 else { return nil }
return RtpHeader(
version: version,
padding: b0 & 0x20 != 0,
extensionHeader: b0 & 0x10 != 0,
csrcCount: b0 & 0x0F,
marker: b1 & 0x80 != 0,
payloadType: b1 & 0x7F,
sequenceNumber: (Int(input[2]) << 8) | Int(input[3]),
timestamp: (Int(input[4]) << 24) | (Int(input[5]) << 16) | (Int(input[6]) << 8) | Int(input[7]),
ssrc: (Int(input[8]) << 24) | (Int(input[9]) << 16) | (Int(input[10]) << 8) | Int(input[11]),
)
}
}
+38
View File
@@ -0,0 +1,38 @@
import Foundation
/// An RTP packet: 12-byte base header (plus optional CSRC/extension) and payload.
struct RtpPacket {
let header: RtpHeader
let payload: [UInt8]
init(header: RtpHeader, payload: [UInt8]) {
self.header = header
self.payload = payload
}
/// Parses a full RTP datagram. Honors CSRC lists, one-level extension
/// headers, and RFC 3550 padding, mirroring the C++ receiver.
static func parse(_ input: [UInt8]) -> RtpPacket? {
guard input.count >= 12, let header = RtpHeader.parse(input) else { return nil }
var offset = 12 + header.csrcCount * 4
guard input.count >= offset else { return nil }
if header.extensionHeader {
guard input.count >= offset + 4 else { return nil }
let extensionWords = (Int(input[offset + 2]) << 8) | Int(input[offset + 3])
offset += 4 + extensionWords * 4
guard input.count >= offset else { return nil }
}
var payloadSize = input.count - offset
if header.padding {
// RFC 3550: the last byte holds the padding size, including itself.
guard payloadSize > 0 else { return nil }
let paddingSize = Int(input[input.count - 1])
guard paddingSize != 0, paddingSize <= payloadSize else { return nil }
payloadSize -= paddingSize
}
return RtpPacket(header: header, payload: Array(input[offset..<(offset + payloadSize)]))
}
}
@@ -0,0 +1,33 @@
import Foundation
/// Splits a byte stream into newline-terminated lines, dropping CR and
/// enforcing the 64 KB line cap the same framing the C++ and Kotlin
/// receivers use. Extracted so the logic is testable in isolation.
final class LineAssembler {
private var pending: [UInt8] = []
private let maxLine = SignalingMessage.maxMessageBytes
/// Feed a chunk of received bytes; returns the complete lines it contained.
func feed(_ chunk: [UInt8]) -> [String] {
var lines: [String] = []
for b in chunk {
if b == 0x0A { // \n
let text = String(bytes: pending, encoding: .utf8) ?? ""
pending.removeAll(keepingCapacity: true)
if !text.isEmpty { lines.append(text) }
} else if b != 0x0D { // \r
if pending.count < maxLine {
pending.append(b)
} else {
// Hostile or broken peer: drop the oversized line.
pending.removeAll(keepingCapacity: true)
}
}
}
return lines
}
func reset() {
pending.removeAll(keepingCapacity: true)
}
}
@@ -0,0 +1,89 @@
import Foundation
/// JSON wire format shared with the C++ implementation: one JSON object per
/// newline-terminated TCP line (offer / answer / pli). Mirrors the C++
/// `SignalingMessage` and the Kotlin receiver.
enum SignalingMessage {
case offer(sessionId: String,
codec: String,
width: Int,
height: Int,
frameRateNum: Int,
frameRateDen: Int,
rtpAddress: String,
rtpPort: Int)
case answer(sessionId: String,
rtpAddress: String,
rtpPort: Int,
displayWidth: Int,
displayHeight: Int)
case pli(sessionId: String)
static let maxMessageBytes = 64 * 1024
static func parse(_ line: String) -> SignalingMessage? {
guard line.count <= maxMessageBytes else { return nil }
guard let data = line.data(using: .utf8),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let type = obj["type"] as? String else {
return nil
}
switch type {
case "offer":
return .offer(
sessionId: obj["session_id"] as? String ?? "",
codec: obj["codec"] as? String ?? "",
width: obj["width"] as? Int ?? 0,
height: obj["height"] as? Int ?? 0,
frameRateNum: obj["frame_rate_num"] as? Int ?? 30,
frameRateDen: obj["frame_rate_den"] as? Int ?? 1,
rtpAddress: obj["rtp_address"] as? String ?? "",
rtpPort: obj["rtp_port"] as? Int ?? 0)
case "answer":
return .answer(
sessionId: obj["session_id"] as? String ?? "",
rtpAddress: obj["rtp_address"] as? String ?? "",
rtpPort: obj["rtp_port"] as? Int ?? 0,
displayWidth: obj["display_width"] as? Int ?? 0,
displayHeight: obj["display_height"] as? Int ?? 0)
case "pli":
return .pli(sessionId: obj["session_id"] as? String ?? "")
default:
return nil
}
}
static func serialize(_ message: SignalingMessage) -> String {
let json: [String: Any]
switch message {
case let .offer(sessionId, codec, width, height, frameRateNum, frameRateDen, rtpAddress, rtpPort):
json = [
"type": "offer",
"session_id": sessionId,
"codec": codec,
"width": width,
"height": height,
"frame_rate_num": frameRateNum,
"frame_rate_den": frameRateDen,
"rtp_address": rtpAddress,
"rtp_port": rtpPort,
]
case let .answer(sessionId, rtpAddress, rtpPort, displayWidth, displayHeight):
json = [
"type": "answer",
"session_id": sessionId,
"rtp_address": rtpAddress,
"rtp_port": rtpPort,
"display_width": displayWidth,
"display_height": displayHeight,
]
case let .pli(sessionId):
json = ["type": "pli", "session_id": sessionId]
}
guard let data = try? JSONSerialization.data(withJSONObject: json),
let s = String(data: data, encoding: .utf8) else {
return "{}\n" // unreachable for our message types; serialize is total
}
return s + "\n"
}
}
@@ -0,0 +1,212 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// Small socket helpers shared by the signaling server and the UDP transport.
enum SocketUtils {
/// Creates a bound, listening TCP socket. Prefers a dual-stack IPv6
/// listener (both families), falls back to IPv4-only. Returns -1 on failure.
static func makeStreamListener(port: UInt16) -> Int32 {
for family in [AF_INET6, AF_INET] {
let fd = socket(family, SOCK_STREAM, 0)
guard fd >= 0 else { continue }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
if family == AF_INET6 {
var no: Int32 = 0
setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &no, socklen_t(MemoryLayout<Int32>.size))
}
let bound: Int32
if family == AF_INET6 {
var a = sockaddr_in6()
a.sin6_family = sa_family_t(AF_INET6)
a.sin6_port = port.bigEndian
a.sin6_addr = in6addr_any
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in6>.size)) }
}
} else {
var a = sockaddr_in()
a.sin_family = sa_family_t(AF_INET)
a.sin_port = port.bigEndian
a.sin_addr = in_addr(s_addr: INADDR_ANY)
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in>.size)) }
}
}
if bound != 0 { close(fd); continue }
if listen(fd, 16) != 0 { close(fd); continue }
return fd
}
return -1
}
/// The local port of a bound socket (the port is at byte offset 2 for both
/// IPv4 and IPv6 sockets).
static func boundPort(_ fd: Int32) -> UInt16 {
var a = sockaddr_storage()
var len = socklen_t(MemoryLayout<sockaddr_storage>.size)
guard getsockname(fd, &a, &len) == 0 else { return 0 }
return withUnsafeBytes(of: &a) { raw in
raw.load(fromByteOffset: 2, as: UInt16.self).bigEndian
}
}
/// A UDP socket bound to [port] (or 0 for ephemeral) for receiving, with a
/// generous receive buffer (the C++ sender's VBV bounds bursts, but a larger
/// buffer absorbs a burst on a lossy link).
static func makeUdpReceiver(port: UInt16, receiveBufferSize: Int32) -> Int32 {
for family in [AF_INET6, AF_INET] {
let fd = socket(family, SOCK_DGRAM, 0)
guard fd >= 0 else { continue }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
if family == AF_INET6 {
var no: Int32 = 0
setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &no, socklen_t(MemoryLayout<Int32>.size))
}
let bound: Int32
if family == AF_INET6 {
var a = sockaddr_in6()
a.sin6_family = sa_family_t(AF_INET6)
a.sin6_port = port.bigEndian
a.sin6_addr = in6addr_any
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in6>.size)) }
}
} else {
var a = sockaddr_in()
a.sin_family = sa_family_t(AF_INET)
a.sin_port = port.bigEndian
a.sin_addr = in_addr(s_addr: INADDR_ANY)
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in>.size)) }
}
}
if bound != 0 { close(fd); continue }
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &receiveBufferSize, socklen_t(MemoryLayout<Int32>.size))
return fd
}
return -1
}
}
/// Newline-delimited JSON signaling server (the receiver side). Keeps the most
/// recent connection as its active peer, mirroring the C++ server: `onOffer`
/// may answer synchronously (the sender blocks on the answer).
final class SignalingServer {
private let onOffer: (SignalingMessage) -> Void
private let onPli: (SignalingMessage) -> Void
private var listenFd: Int32 = -1
private(set) var port: UInt16 = 0
private var peerFd: Int32 = -1
private let peerLock = NSLock()
private let assembler = LineAssembler()
private var acceptThread: Thread?
private var readerThread: Thread?
private var running = false
init(onOffer: @escaping (SignalingMessage) -> Void,
onPli: @escaping (SignalingMessage) -> Void) {
self.onOffer = onOffer
self.onPli = onPli
}
/// Binds the port (SO_REUSEADDR) and starts accepting. Returns true on success.
func start(_ preferredPort: UInt16) -> Bool {
guard let fd = SocketUtils.makeStreamListener(port: preferredPort), fd >= 0 else { return false }
listenFd = fd
port = SocketUtils.boundPort(fd)
running = true
let thread = Thread { [weak self] in self?.acceptLoop() }
thread.name = "signaling-accept"
acceptThread = thread
thread.start()
return true
}
/// Sends to the current peer; never throws. A lost control message is
/// recoverable the session re-negotiates or the next keyframe arrives
/// but an exception here would kill the RTP reader thread that reaches
/// send() from requestPli(). Mirrors the C++ server, which ignores write
/// failures.
func send(_ message: SignalingMessage) {
guard let bytes = SignalingMessage.serialize(message).data(using: .utf8) else { return }
peerLock.lock()
let fd = peerFd
peerLock.unlock()
guard fd >= 0 else { return }
bytes.withUnsafeBytes { raw in
_ = send(fd, raw.baseAddress, raw.count, Int32(MSG_NOSIGNAL))
}
}
func close() {
running = false
peerLock.lock()
let peer = peerFd
peerFd = -1
peerLock.unlock()
if peer >= 0 { close(peer) }
if listenFd >= 0 { close(listenFd) }
listenFd = -1
}
private func acceptLoop() {
while running {
var addr = sockaddr()
var len = socklen_t(MemoryLayout<sockaddr>.size)
let client = accept(listenFd, &addr, &len)
if client < 0 {
if !running { break }
continue
}
// Most-recent-connection-wins: close the previous peer.
peerLock.lock()
let old = peerFd
peerFd = client
peerLock.unlock()
if old >= 0 { close(old) }
let thread = Thread { [weak self] in self?.readLoop(fd: client) }
thread.name = "signaling-reader"
readerThread = thread
thread.start()
}
}
private func readLoop(fd: Int32) {
assembler.reset()
var buffer = [UInt8](repeating: 0, count: 4096)
while running {
let read = buffer.withUnsafeMutableBytes { raw in
recv(fd, raw.baseAddress, raw.count, 0)
}
guard read > 0 else { break }
for line in assembler.feed(Array(buffer.prefix(read))) {
dispatch(line)
}
}
// Peer closed; if it is still our active peer, mark it gone.
peerLock.lock()
if peerFd == fd {
peerFd = -1
}
peerLock.unlock()
}
private func dispatch(_ line: String) {
guard let message = SignalingMessage.parse(line) else { return }
// A broken callback must not kill the reader thread.
switch message {
case .offer:
onOffer(message)
case .pli:
onPli(message)
case .answer:
break // the receiver never receives answers
}
}
}
+41
View File
@@ -0,0 +1,41 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// Finds the primary IPv4 address (e.g. on Wi-Fi) for the status overlay and
/// the `--peer` hint. Best-effort; the sender reaches us by IP over the LAN.
enum LocalAddress {
static func primaryIPv4() -> String {
var result = "unknown"
var fallback = "unknown"
var ptr: UnsafeMutablePointer<ifaddrs>?
guard getifaddrs(&ptr) == 0 else { return result }
defer { freeifaddrs(ptr) }
var current = ptr
while let iface = current {
let next = iface.pointee.ifa_next
current = next
guard let sa = iface.pointee.ifa_addr else { continue }
guard sa.pointee.sa_family == sa_family_t(AF_INET) else { continue }
if Int32(iface.pointee.ifa_flags) & IFF_LOOPBACK == 0 {
let inaddr = sa.assumingMemoryBound(to: sockaddr_in.self).pointee
var host = [CChar](repeating: 0, count: Int(INET_ADDRSTRLEN))
if inet_ntop(AF_INET, &inaddr.sin_addr, &host, socklen_t(INET_ADDRSTRLEN)) != nil {
let ip = String(cString: host)
if ip.hasPrefix("169.254") {
continue // link-local; prefer a routable address
}
fallback = ip
let name = String(cString: iface.pointee.ifa_name)
if name.hasPrefix("en") || name.hasPrefix("wlan") {
return ip // Wi-Fi / Ethernet: good enough for the hint
}
}
}
}
return fallback
}
}
+59
View File
@@ -0,0 +1,59 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// A UDP socket for receiving RTP. Binds to a preferred port (or an ephemeral
/// port when it is busy), then delivers complete datagrams one at a time.
/// Mirrors the C++ `UdpRtpTransport` receive path and the Kotlin `DatagramSocket`
/// reader. Uses poll(2) with a timeout so `close()` from another thread cannot
/// strand a blocked recv (a close does not reliably unblock a POSIX recvfrom).
final class UdpTransport {
private var fd: Int32 = -1
private(set) var port: UInt16 = 0
/// Binds to [preferredPort] (or an ephemeral port when it is busy).
@discardableResult
func bind(preferredPort: UInt16, receiveBufferSize: Int32) -> Bool {
guard fd < 0 else { return true }
for p in [preferredPort, UInt16(0)] {
if let f = SocketUtils.makeUdpReceiver(port: p, receiveBufferSize: receiveBufferSize), f >= 0 {
fd = f
port = SocketUtils.boundPort(fd)
return true
}
}
return false
}
/// Waits up to [timeoutMs] for a datagram. Returns 1 when one is ready to
/// read, 0 on timeout, and -1 when the socket is closed/errored.
func poll(timeoutMs: Int32) -> Int32 {
guard fd >= 0 else { return -1 }
var pfd = pollfd(fd: fd, events: poll_events_t(POLLIN), revents: 0)
let r = withUnsafeMutablePointer(to: &pfd) { poll($0, 1, timeoutMs) }
if r > 0 {
if pfd.revents & poll_events_t(POLLERR) != 0 { return -1 }
return 1
}
return r == 0 ? 0 : -1
}
/// Reads one ready datagram. Returns nil on error or empty read.
func receiveDatagram() -> [UInt8]? {
guard fd >= 0 else { return nil }
var buf = [UInt8](repeating: 0, count: 4096)
let read = buf.withUnsafeMutableBytes { raw in
recvfrom(fd, raw.baseAddress, raw.count, 0, nil, nil)
}
guard read > 0 else { return nil }
return Array(buf.prefix(read))
}
func close() {
if fd >= 0 {
close(fd)
fd = -1
}
}
}
@@ -0,0 +1,27 @@
import XCTest
@testable import Receiver
final class AvccConverterTests: XCTestCase {
func testNalUnits() {
let annexB: [UInt8] = [0, 0, 1, 0x41, 0x11, 0, 0, 1, 0x67, 0xAA]
XCTAssertEqual(AvccConverter.nalUnits(annexB), [[0x41, 0x11], [0x67, 0xAA]])
}
func testToAvcc() {
let annexB: [UInt8] = [0, 0, 1, 0x41, 0x11, 0, 0, 1, 0x67, 0xAA]
let avcc = AvccConverter.toAvcc(annexB)
XCTAssertEqual(avcc, [0, 0, 0, 2, 0x41, 0x11, 0, 0, 0, 2, 0x67, 0xAA])
}
func testFromAvccRoundtrip() {
let units: [[UInt8]] = [[0x41, 0x11], [0x67, 0xAA, 0xBB]]
let annexB = [0, 0, 1] + units[0] + [0, 0, 1] + units[1]
let avcc = AvccConverter.toAvcc(annexB)
XCTAssertEqual(AvccConverter.fromAvcc(avcc ?? []), units)
}
func testEmptyReturnsNil() {
XCTAssertNil(AvccConverter.toAvcc([UInt8]()))
XCTAssertNil(AvccConverter.toAvcc([0, 0, 1]))
}
}
@@ -0,0 +1,103 @@
import XCTest
@testable import Receiver
final class H264DepacketizerTests: XCTestCase {
private let start: [UInt8] = [0x00, 0x00, 0x01]
private func packet(seq: Int, ts: Int, payload: [UInt8], marker: Bool = false) -> RtpPacket {
RtpPacket(header: RtpHeader(sequenceNumber: seq, timestamp: ts, marker: marker), payload: payload)
}
func testSingleNalTwoPackets() {
let d = H264Depacketizer()
// SPS (type 7) then a slice (type 5), closed by the marker.
let first = d.depacketize(packet(seq: 1, ts: 100, payload: [0x67, 0xAA, 0xBB]))
XCTAssertNil(first.accessUnit)
let second = d.depacketize(packet(seq: 2, ts: 100, payload: [0x41, 0x01, 0x02], marker: true))
XCTAssertEqual(second.accessUnit, start + [0x67, 0xAA, 0xBB] + start + [0x41, 0x01, 0x02])
XCTAssertTrue(second.isKeyFrame)
XCTAssertFalse(second.frameDropped)
}
func testFuAReassembly() {
let d = H264Depacketizer()
// NAL: header 0x41 (type 1, NRI 2) + payload 0x11 0x22 0x33 0x44.
// FU indicator = (0x41 & 0xE0) | 28 = 0x5C.
let indicator: UInt8 = 0x5C
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [indicator, 0x81, 0x11]))
_ = d.depacketize(packet(seq: 2, ts: 100, payload: [indicator, 0x01, 0x22]))
let done = d.depacketize(packet(seq: 3, ts: 100, payload: [indicator, 0x41, 0x33, 0x44], marker: true))
XCTAssertEqual(done.accessUnit, start + [0x41, 0x11, 0x22, 0x33, 0x44])
XCTAssertFalse(done.frameDropped)
}
func testDropsGappedFrames() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01]))
// seq 2 is missing; the frame must be reported dropped, not delivered.
let tail = d.depacketize(packet(seq: 3, ts: 100, payload: [0x41, 0x02], marker: true))
XCTAssertNil(tail.accessUnit)
XCTAssertTrue(tail.frameDropped)
}
func testSeparateFramesByMarker() {
let d = H264Depacketizer()
let au1 = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0xAA], marker: true))
XCTAssertEqual(au1.accessUnit, start + [0x41, 0xAA])
let au2 = d.depacketize(packet(seq: 2, ts: 200, payload: [0x41, 0xBB], marker: true))
XCTAssertEqual(au2.accessUnit, start + [0x41, 0xBB])
}
func testDropsFuWithoutStart() {
let d = H264Depacketizer()
// Continuation (no S bit) without any start packet.
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C, 0x41, 0x11], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsStillFragmentedAtMarker() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C, 0x81, 0x11]))
// Marker arrives while the FU-A NAL is still open.
let result = d.depacketize(packet(seq: 2, ts: 100, payload: [0x41, 0x01], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsUnsupportedPacketization() {
let d = H264Depacketizer()
let stapA: UInt8 = 24 // STAP-A
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [stapA, 0x00, 0x05, 0x41, 0x01], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsTimestampChangeWithoutMarker() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01]))
// The stale frame is reported dropped, but this packet starts (and
// closes) the next frame matching the C++ depacketizer.
let result = d.depacketize(packet(seq: 2, ts: 200, payload: [0x41, 0x02], marker: true))
XCTAssertEqual(result.accessUnit, start + [0x41, 0x02])
XCTAssertTrue(result.frameDropped)
}
func testKeyframeDetectionRequiresParameterSets() {
let d = H264Depacketizer()
let plain = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01], marker: true))
XCTAssertFalse(plain.isKeyFrame)
let d2 = H264Depacketizer()
let withSps = d2.depacketize(packet(seq: 1, ts: 100, payload: [0x67, 0xAA, 0x88, 0x68, 0xBB, 0x41, 0x01], marker: true))
XCTAssertTrue(withSps.isKeyFrame)
}
func testDropsShortFuPackets() {
let d = H264Depacketizer()
// FU-A packet without its FU header byte.
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
}
+57
View File
@@ -0,0 +1,57 @@
import XCTest
@testable import Receiver
final class JitterBufferTests: XCTestCase {
private func packet(seq: Int, ts: Int = 100) -> RtpPacket {
RtpPacket(header: RtpHeader(sequenceNumber: seq, timestamp: ts), payload: [UInt8(seq)])
}
func testInOrderReleasesImmediately() {
let jitter = JitterBuffer()
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
XCTAssertEqual(jitter.push(packet(seq: 2)).map { $0.header.sequenceNumber }, [2])
XCTAssertEqual(jitter.push(packet(seq: 3)).map { $0.header.sequenceNumber }, [3])
}
func testReordersOutOfOrderPackets() {
let jitter = JitterBuffer()
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
XCTAssertTrue(jitter.push(packet(seq: 3)).isEmpty)
XCTAssertEqual(jitter.push(packet(seq: 2)).map { $0.header.sequenceNumber }, [2, 3])
}
func testOverflowReleasesInOrderAndAdvances() {
let jitter = JitterBuffer(maxDepth: 4)
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
// seq 2 is lost; 3..6 stay buffered (within the depth bound).
for seq in 3...6 {
XCTAssertTrue(jitter.push(packet(seq: seq)).isEmpty)
}
// seq 7 overflows the buffer: 3..7 flush in order.
XCTAssertEqual(jitter.push(packet(seq: 7)).map { $0.header.sequenceNumber }, [3, 4, 5, 6, 7])
// Delivery continues in order afterwards.
XCTAssertEqual(jitter.push(packet(seq: 8)).map { $0.header.sequenceNumber }, [8])
XCTAssertEqual(jitter.push(packet(seq: 9)).map { $0.header.sequenceNumber }, [9])
}
func testDiscardsStragglers() {
let jitter = JitterBuffer(maxDepth: 4)
_ = jitter.push(packet(seq: 1))
for seq in 3...8 {
_ = jitter.push(packet(seq: seq))
}
XCTAssertTrue(jitter.push(packet(seq: 9)).isNotEmpty)
// seq 4 is now far behind the expected sequence: discarded, not delivered.
XCTAssertTrue(jitter.push(packet(seq: 4)).isEmpty)
// In-order delivery continues from 10.
XCTAssertEqual(jitter.push(packet(seq: 10)).map { $0.header.sequenceNumber }, [10])
}
func testClearResetsState() {
let jitter = JitterBuffer()
_ = jitter.push(packet(seq: 5))
jitter.clear()
// A completely different sequence now starts fresh.
XCTAssertEqual(jitter.push(packet(seq: 100)).map { $0.header.sequenceNumber }, [100])
}
}
@@ -0,0 +1,44 @@
import XCTest
@testable import Receiver
final class LineAssemblerTests: XCTestCase {
private func bytes(_ s: String) -> [UInt8] { Array(s.utf8) }
func testSingleLine() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("hello")), [])
XCTAssertEqual(a.feed(bytes("\n")), ["hello"])
}
func testMultipleLinesInOneChunk() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("one\ntwo\nthree\n")), ["one", "two", "three"])
}
func testDropsCR() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("line\r\n")), ["line"])
}
func testIgnoresEmptyLine() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("\n")), [])
}
func testSplitAcrossChunks() {
let a = LineAssembler()
let payload = "{\"session_id\":\"x\"}"
var lines: [String] = []
lines += a.feed(Array(payload.prefix(5)))
lines += a.feed(Array(payload.suffix(from: 5)))
lines += a.feed(["\n".utf8.first!])
XCTAssertEqual(lines, [payload])
}
func testDropsOversizedLine() {
let a = LineAssembler()
let big = String(repeating: "a", count: SignalingMessage.maxMessageBytes + 1)
_ = a.feed(Array(big.utf8))
XCTAssertEqual(a.feed(bytes("\n")), [])
}
}
+24
View File
@@ -0,0 +1,24 @@
import XCTest
@testable import Receiver
final class NalExtractorTests: XCTestCase {
func testExtractsSpsAndPps() {
// Annex-B: SPS (type 7), PPS (type 8), slice (type 5).
let annexB: [UInt8] = [0, 0, 1, 0x67, 0xAA, 0, 0, 1, 0x68, 0xBB, 0, 0, 1, 0x41, 0x01]
let sets = NalExtractor.parameterSets(annexB)
XCTAssertEqual(sets?.sps, [0x67, 0xAA])
XCTAssertEqual(sets?.pps, [0x68, 0xBB])
}
func testReturnsNilWithoutBoth() {
let annexB: [UInt8] = [0, 0, 1, 0x67, 0xAA, 0, 0, 1, 0x41, 0x01] // SPS but no PPS
XCTAssertNil(NalExtractor.parameterSets(annexB))
}
func testPicksFirstOfEach() {
let annexB: [UInt8] = [0, 0, 1, 0x67, 0x11, 0, 0, 1, 0x68, 0x22, 0, 0, 1, 0x67, 0x33, 0, 0, 1, 0x68, 0x44]
let sets = NalExtractor.parameterSets(annexB)
XCTAssertEqual(sets?.sps, [0x67, 0x11])
XCTAssertEqual(sets?.pps, [0x68, 0x22])
}
}
+87
View File
@@ -0,0 +1,87 @@
import XCTest
@testable import Receiver
final class RtpHeaderTests: XCTestCase {
func testRoundtrip() {
let header = RtpHeader(version: 2, padding: false, extensionHeader: false, csrcCount: 0,
marker: true, payloadType: 96, sequenceNumber: 0xABCD,
timestamp: 0xDEADBEEF, ssrc: 0x12345678)
let wire = header.serialize()
XCTAssertEqual(wire.count, 12)
XCTAssertEqual(RtpHeader.parse(wire), header)
}
func testRejectsBadVersion() {
var wire = RtpHeader().serialize()
wire[0] = (wire[0] & 0x3F) | (1 << 6)
XCTAssertNil(RtpHeader.parse(wire))
}
func testRejectsShortInput() {
let header = RtpHeader()
XCTAssertNil(RtpHeader.parse(Array(header.serialize().prefix(11))))
XCTAssertNil(RtpHeader.parse([]))
}
func testPreservesFlags() {
let header = RtpHeader(padding: true, csrcCount: 2, marker: true, payloadType: 63)
let parsed = RtpHeader.parse(header.serialize())
XCTAssertEqual(parsed?.padding, true)
XCTAssertEqual(parsed?.csrcCount, 2)
XCTAssertEqual(parsed?.marker, true)
XCTAssertEqual(parsed?.payloadType, 63)
}
}
final class RtpPacketTests: XCTestCase {
private func header(seq: Int, marker: Bool = false) -> RtpHeader {
RtpHeader(sequenceNumber: seq, payloadType: 96, marker: marker)
}
func testRoundtripWithPayload() {
let packet = RtpPacket(header: header(seq: 7), payload: [0x11, 0x22, 0x33])
let wire = packet.header.serialize() + packet.payload
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.header, header(seq: 7))
XCTAssertEqual(parsed?.payload, [0x11, 0x22, 0x33])
}
func testSkipsCsrcList() {
let header = RtpHeader(csrcCount: 1, sequenceNumber: 3)
let wire = header.serialize() + [0x0A, 0x00, 0x00, 0x01] + [0x99]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.header.csrcCount, 1)
XCTAssertEqual(parsed?.payload, [0x99])
}
func testSkipsExtensionHeader() {
let header = RtpHeader(extensionHeader: true, sequenceNumber: 4)
// profile=0x0001, length=1 word, one word of data.
let wire = header.serialize() + [0x00, 0x01, 0x00, 0x01, 0xDE, 0xAD, 0xBE, 0xEF] + [0x77]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.payload, [0x77])
}
func testStripsPadding() {
let header = RtpHeader(padding: true, sequenceNumber: 5)
// Payload byte, one padding zero, size byte (2 = padding incl. itself).
let wire = header.serialize() + [0x55, 0x00, 0x02]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.payload, [0x55])
}
func testRejectsTruncatedCsrcAndExtension() {
let csrc = RtpHeader(csrcCount: 1).serialize()
XCTAssertNil(RtpPacket.parse(csrc)) // 12 bytes, needs 16
let ext = RtpHeader(extensionHeader: true).serialize() + [0x00, 0x01]
XCTAssertNil(RtpPacket.parse(ext)) // extension length field cut off
}
func testRejectsBadPadding() {
let zeroPad = RtpHeader(padding: true).serialize() + [0x00]
XCTAssertNil(RtpPacket.parse(zeroPad))
let oversized = RtpHeader(padding: true).serialize() + [0x00, 0x00, 0x05]
XCTAssertNil(RtpPacket.parse(oversized))
XCTAssertNil(RtpPacket.parse([UInt8](repeating: 0, count: 11)))
}
}
@@ -0,0 +1,54 @@
import XCTest
@testable import Receiver
final class SignalingMessageTests: XCTestCase {
func testOfferRoundtrip() {
let offer = SignalingMessage.offer(sessionId: "s-1", codec: "h264", width: 0, height: 0,
frameRateNum: 30, frameRateDen: 1, rtpAddress: "10.0.0.5", rtpPort: 1234)
let line = SignalingMessage.serialize(offer).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .offer(sid, codec, w, h, num, den, addr, port) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-1")
XCTAssertEqual(codec, "h264")
XCTAssertEqual(w, 0)
XCTAssertEqual(h, 0)
XCTAssertEqual(num, 30)
XCTAssertEqual(den, 1)
XCTAssertEqual(addr, "10.0.0.5")
XCTAssertEqual(port, 1234)
}
func testAnswerRoundtrip() {
let answer = SignalingMessage.answer(sessionId: "s-2", rtpAddress: "", rtpPort: 5004,
displayWidth: 1179, displayHeight: 2556)
let line = SignalingMessage.serialize(answer).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .answer(sid, addr, port, dw, dh) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-2")
XCTAssertEqual(addr, "")
XCTAssertEqual(port, 5004)
XCTAssertEqual(dw, 1179)
XCTAssertEqual(dh, 2556)
}
func testPliRoundtrip() {
let pli = SignalingMessage.pli(sessionId: "s-3")
let line = SignalingMessage.serialize(pli).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .pli(sid) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-3")
}
func testRejectsUnknownType() {
XCTAssertNil(SignalingMessage.parse("{\"type\":\"bogus\"}"))
}
func testRejectsInvalidJson() {
XCTAssertNil(SignalingMessage.parse("not json"))
}
func testRejectsOversizedLine() {
let big = String(repeating: "a", count: SignalingMessage.maxMessageBytes + 1)
XCTAssertNil(SignalingMessage.parse(big))
}
}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
#
# Bootstrap for the iOS receiver. Intended to be run on a Mac with Xcode 26.
# It installs XcodeGen (if absent) from the GitHub release (no Homebrew needed),
# generates the Xcode project, and builds/tests.
#
# Usage:
# ./bootstrap.sh # generate + build for device
# ./bootstrap.sh test # generate + run the unit tests (simulator)
# ./bootstrap.sh build-sim # generate + build for the simulator
# ./bootstrap.sh generate # just generate the project
#
# Device install/signing is intentionally left to Xcode: open
# Receiver.xcodeproj, set your Apple ID on the Receiver target, and hit Run.
set -euo pipefail
IOS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TOOLS_DIR="$IOS_DIR/tools"
XCODEGEN_BIN="$TOOLS_DIR/xcodegen"
PROJ="$IOS_DIR/Receiver.xcodeproj"
# 1. Xcode
if ! command -v xcodebuild >/dev/null 2>&1; then
echo "error: xcodebuild not found. Install Xcode (26.x) and select its CLI tools." >&2
echo " xcode-select --install (or: sudo xcode-select -s /Applications/Xcode.app/Contents/Developer)" >&2
exit 1
fi
# 2. XcodeGen: prefer a system copy, then a local install, else download latest.
if ! command -v xcodegen >/dev/null 2>&1 && [ ! -x "$XCODEGEN_BIN" ]; then
echo "Installing XcodeGen into $TOOLS_DIR ..."
mkdir -p "$TOOLS_DIR"
url="https://github.com/yonaskolb/XcodeGen/releases/latest/download/xcodegen.zip"
tmp="$(mktemp -d)"
curl -fL "$url" -o "$tmp/xcodegen.zip"
unzip -o -q "$tmp/xcodegen.zip" -d "$TOOLS_DIR"
# The release zip extracts to a flat binary named "xcodegen".
if [ ! -x "$XCODEGEN_BIN" ] && [ -f "$TOOLS_DIR/xcodegen" ]; then
chmod +x "$XCODEGEN_BIN"
fi
rm -rf "$tmp"
fi
XCODEGEN="$(command -v xcodegen || echo "$XCODEGEN_BIN")"
if [ ! -x "$XCODEGEN" ] && [ ! -x "$XCODEGEN_BIN" ]; then
echo "error: XcodeGen not found and download failed." >&2
exit 1
fi
XCODEGEN="${XCODEGEN_BIN}"
generate() {
"$XCODEGEN_BIN" generate --spec "$IOS_DIR/project.yml" --project "$IOS_DIR"
echo "Generated $PROJ"
}
dest_sim() { echo "${IOS_DEST:-platform=iOS Simulator,name=iPhone 16}"; }
command="${1:-build}"
generate
case "$command" in
generate)
;;
build)
xcodebuild -project "$PROJ" -scheme Receiver -destination "generic/platform=iOS" build
;;
build-sim)
xcodebuild -project "$PROJ" -scheme Receiver -destination "$(dest_sim)" build
;;
test)
xcodebuild test -project "$PROJ" -scheme Receiver -destination "$(dest_sim)" \
-only-testing:ReceiverTests
;;
*)
echo "usage: bootstrap.sh [generate|build|build-sim|test]" >&2
exit 1
;;
esac
echo "Done."
+71
View File
@@ -0,0 +1,71 @@
name: Receiver
options:
bundleIdPrefix: screencast
deploymentTarget:
iOS: "17.0"
createIntermediateGroups: true
generateEmptySchemes: false
targets:
Receiver:
type: application
platform: iOS
deploymentTarget: "17.0"
sources:
- path: Receiver
settings:
base:
PRODUCT_BUNDLE_IDENTIFIER: screencast.receiver
PRODUCT_NAME: Receiver
TARGETED_DEVICE_FAMILY: 1
CODE_SIGN_STYLE: Automatic
DEVELOPMENT_TEAM: ""
IPHONEOS_DEPLOYMENT_TARGET: "17.0"
ENABLE_USER_SCRIPT_SANDBOXING: YES
info:
path: Receiver/Info.plist
properties:
CFBundleDisplayName: screencast
CFBundleName: screencast
CFBundleShortVersionString: "0.9.0"
CFBundleVersion: "1"
UILaunchScreen: {}
UISupportedInterfaceOrientations:
- UIInterfaceOrientationLandscapeLeft
- UIInterfaceOrientationLandscapeRight
UISupportedInterfaceOrientations~ipad:
- UIInterfaceOrientationLandscapeLeft
- UIInterfaceOrientationLandscapeRight
- UIInterfaceOrientationPortrait
- UIInterfaceOrientationPortraitUpsideDown
# Local-network privacy: required for both Bonjour and the TCP/UDP
# sockets. Without NSBonjourServices the sender never resolves us.
NSLocalNetworkUsageDescription: "screencast receives a video stream on your local network."
NSBonjourServices:
- "_screencast._tcp"
ReceiverTests:
type: bundle.unit-test
platform: iOS
deploymentTarget: "17.0"
sources:
- path: ReceiverTests
dependencies:
- target: Receiver
settings:
base:
PRODUCT_BUNDLE_IDENTIFIER: screencast.receiver.tests
BUNDLE_LOADER: "$(TEST_HOST)"
TEST_HOST: "$(BUILT_PRODUCTS_DIR)/Receiver.app/Receiver"
schemes:
Receiver:
build:
targets:
Receiver: all
test:
config: Debug
targets:
- ReceiverTests
archive:
enabled: false
+12 -1
View File
@@ -10,6 +10,11 @@ project('screen_cast', 'cpp',
# Public and private include directories are declared in `src/meson.build`.
subdir('src')
# The GUI is a sender panel and needs the capture backend.
if get_option('gui') and not get_option('sender')
error('The GUI is a sender panel and requires -Dsender=true')
endif
# Manual smoke tools are sender-side.
if get_option('sender')
subdir('tools')
@@ -21,7 +26,13 @@ if enable_tests
subdir('tests')
endif
# Install the application icon into the hicolor theme: the waybar widget CSS
# and any future .desktop file reference it.
install_data('screencast_icon/screencast_256.png',
install_dir : get_option('datadir') / 'icons/hicolor/256x256/apps',
rename : 'screencast.png')
# Summary for the user
summary({
'tests': enable_tests,
}, section: 'Build options')
}, section: 'Build options')
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 398 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.8 KiB

+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""Generate a C++ header that embeds an image file as a byte array.
Usage: icon_to_header.py INPUT_IMAGE OUTPUT_HEADER
"""
import sys
def main() -> int:
if len(sys.argv) != 3:
print(f"usage: {sys.argv[0]} INPUT_IMAGE OUTPUT_HEADER", file=sys.stderr)
return 2
with open(sys.argv[1], "rb") as src:
data = src.read()
words = ", ".join(f"0x{byte:02x}" for byte in data)
header = f"""// GENERATED FILE - do not edit by hand.
// Produced by scripts/icon_to_header.py from screencast_icon/screencast_256.png.
#pragma once
#include <array>
#include <cstdint>
namespace sc {{
inline constexpr std::array<std::uint8_t, {len(data)}> app_icon_png = {{
{words}
}};
}} // namespace sc
"""
with open(sys.argv[2], "w") as dst:
dst.write(header)
return 0
if __name__ == "__main__":
sys.exit(main())
+33 -2
View File
@@ -8,9 +8,11 @@ namespace sc {
namespace {
void print_usage() {
std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate KBPS]\n"
" screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen]\n"
std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate MAX_KBPS] [--crf "
"0-51] [--fps 1-60]\n"
" screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen] [--swdecode]\n"
" screencast --discover [--timeout SECONDS]\n"
" screencast waybar [--toggle] # for waybar widgets\n"
"\n"
"--send without --peer discovers a receiver on the LAN and requires\n"
"that exactly one is found.\n",
@@ -39,12 +41,14 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
Send,
Receive,
Discover,
Waybar,
};
Mode mode = Mode::None;
SendCommand send;
ReceiveCommand receive;
DiscoverCommand discover;
WaybarCommand waybar;
for (int index = 1; index < argc; ++index) {
const std::string_view argument = argv[index];
@@ -67,6 +71,14 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
return std::nullopt;
}
mode = Mode::Discover;
} else if (argument == "waybar") {
if (mode != Mode::None) {
print_usage();
return std::nullopt;
}
mode = Mode::Waybar;
} else if (argument == "--toggle") {
waybar.toggle = true;
} else if (argument == "--target") {
std::string_view value;
if (!next_argument(argc, argv, index, value)) {
@@ -94,6 +106,20 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
print_usage();
return std::nullopt;
}
} else if (argument == "--crf") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, send.crf) || send.crf < 0 ||
send.crf > 51) {
print_usage();
return std::nullopt;
}
} else if (argument == "--fps") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, send.fps) || send.fps < 0 ||
send.fps > 60) {
print_usage();
return std::nullopt;
}
} else if (argument == "--port") {
std::string_view value;
int port = 0;
@@ -112,6 +138,8 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
receive.signaling_port = port;
} else if (argument == "--fullscreen") {
receive.fullscreen = true;
} else if (argument == "--swdecode") {
receive.software_decode = true;
} else if (argument == "--timeout") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, discover.timeout_seconds) ||
@@ -134,6 +162,9 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
if (mode == Mode::Discover) {
return Command{std::move(discover)};
}
if (mode == Mode::Waybar) {
return Command{std::move(waybar)};
}
print_usage();
return std::nullopt;
}
+154 -169
View File
@@ -1,9 +1,17 @@
#include "screencast/app/cli.h"
#include "screencast/app/pipeline.h"
#include "sender_session.h"
#include "state_store.h"
#include "screencast/network/discovery.h"
#include "screencast/network/signaling.h"
#include <nlohmann/json.hpp>
#include <sys/types.h>
#include <unistd.h>
#include <algorithm>
#include <atomic>
#include <charconv>
@@ -53,49 +61,6 @@ sc::Endpoint parse_endpoint(std::string_view address, std::uint16_t default_port
#endif // SC_HAS_SENDER
std::string make_session_id() {
static std::mt19937 engine{std::random_device{}()};
std::string id;
for (int i = 0; i < 16; ++i) {
id += "0123456789abcdef"[static_cast<std::size_t>(engine()) & 0xF];
}
return id;
}
bool is_private_ipv4(std::string_view host) {
if (host.rfind("192.168.", 0) == 0 || host.rfind("10.", 0) == 0) {
return true;
}
if (host.rfind("172.", 0) == 0) {
const std::size_t second = host.find('.', 5);
if (second != std::string_view::npos) {
const int octet = std::stoi(std::string{host.substr(5, second - 5)});
return octet >= 16 && octet <= 31;
}
}
return false;
}
// Ordering for trying a receiver's addresses: private IPv4 first (LANs,
// most reliable), then public IPv4, ULA, and global IPv6. 6to4 (2002::) and
// link-local addresses last: 6to4 is frequently unreachable between LAN
// peers, and link-local needs a scope id to even route.
int address_preference(std::string_view host) {
if (host.find(':') == std::string_view::npos) {
return is_private_ipv4(host) ? 0 : 1;
}
if (host.rfind("fd", 0) == 0 || host.rfind("fc", 0) == 0) {
return 2;
}
if (host.rfind("2002:", 0) == 0) {
return 4;
}
if (host.rfind("fe80:", 0) == 0) {
return 5;
}
return 3;
}
std::vector<sc::DiscoveredPeer> discover_peers(int timeout_seconds, std::string& error) {
std::vector<sc::DiscoveredPeer> peers;
std::mutex mutex;
@@ -140,155 +105,172 @@ std::vector<sc::DiscoveredPeer> discover_peers(int timeout_seconds, std::string&
#ifdef SC_HAS_SENDER
// Offer, wait for the answer, and stream to the negotiated endpoint. The
// channel must already be connected.
int negotiate_and_stream(sc::SignalingChannel& channel, const sc::Endpoint& signaling, const sc::SendCommand& command) {
std::promise<sc::SessionAnswer> answer_promise;
auto answer_future = answer_promise.get_future();
std::atomic<bool> answered{false};
channel.on_message([&](const sc::SignalingMessage& message) {
if (const sc::SessionAnswer* answer = std::get_if<sc::SessionAnswer>(&message)) {
if (!answered.exchange(true)) {
answer_promise.set_value(*answer);
}
}
});
sc::SessionOffer offer;
offer.session_id = make_session_id();
offer.codec_name = "h264";
offer.frame_rate_num = 25;
offer.frame_rate_den = 1;
channel.send(offer);
if (answer_future.wait_for(std::chrono::seconds(5)) != std::future_status::ready) {
std::cerr << "screencast: the receiver did not answer the session offer\n";
channel.disconnect();
return 1;
}
const sc::SessionAnswer answer = answer_future.get();
channel.disconnect();
if (answer.session_id != offer.session_id) {
std::cerr << "screencast: session mismatch in the receiver's answer\n";
return 1;
}
// Stream to the negotiated RTP endpoint. An empty address means
// "the address you reached me on".
const sc::Endpoint rtp_endpoint = answer.rtp_endpoint.address.empty()
? sc::Endpoint{signaling.address, answer.rtp_endpoint.port}
: answer.rtp_endpoint;
sc::SenderPipelineConfig config;
if (command.target == "window") {
config.capture_target = sc::CaptureTargetWindow{};
} else {
config.capture_target = sc::CaptureTargetWholeScreen{};
}
config.peer_rtp_endpoint = rtp_endpoint;
config.encoder.bitrate_kbps = command.bitrate_kbps;
std::cout << std::format("screencast: session {} established; streaming to {}:{}\n",
offer.session_id,
rtp_endpoint.address,
rtp_endpoint.port);
sc::SenderPipeline pipeline{std::move(config)};
if (!pipeline.start()) {
return 1;
}
while (!g_interrupted.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
pipeline.stop();
return 0;
}
int run_sender(const sc::SendCommand& command) {
// Find the receiver's signaling endpoint: explicit --peer, or discover
// exactly one receiver on the LAN. A receiver may resolve to several
// addresses; try them in reachability order until the signaling
// connection succeeds.
auto channel_result = sc::SignalingFactory::create_client();
if (sc::is_network_error(channel_result)) {
std::cerr << std::format("screencast: {}\n", sc::network_error(channel_result).message);
return 1;
}
auto channel = std::move(sc::network_value(channel_result));
sc::Endpoint signaling;
sc::CaptureTarget target = sc::CaptureTargetWholeScreen{};
if (command.target == "window") {
target = sc::CaptureTargetWindow{};
}
if (!command.peer_address.empty()) {
signaling = parse_endpoint(command.peer_address, kDefaultSignalingPort);
if (!channel->connect(signaling)) {
std::cerr << std::format(
"screencast: failed to connect to the receiver at {}:{}\n", signaling.address, signaling.port);
} else {
std::string error;
std::vector<sc::DiscoveredPeer> peers = discover_peers(kSenderDiscoveryTimeoutSeconds, error);
if (!error.empty()) {
std::cerr << std::format("screencast: discovery failed: {}\n", error);
return 1;
}
return negotiate_and_stream(*channel, signaling, command);
}
std::string error;
std::vector<sc::DiscoveredPeer> peers = discover_peers(kSenderDiscoveryTimeoutSeconds, error);
if (!error.empty()) {
std::cerr << std::format("screencast: discovery failed: {}\n", error);
return 1;
}
if (peers.empty()) {
std::cerr << "screencast: no receiver found on the LAN; run 'screencast --receive' on the "
"target machine, or pass --peer\n";
return 1;
}
// Group the addresses by receiver (name + signaling port): one entry
// per address, but they are all the same host.
std::map<std::pair<std::string, std::uint16_t>, std::vector<std::string>> receivers;
for (const sc::DiscoveredPeer& peer : peers) {
receivers[{peer.service_name, peer.signaling_port}].push_back(peer.host);
}
if (receivers.size() > 1) {
for (const auto& [key, hosts] : receivers) {
std::cerr << std::format(
"screencast: {} at {}\n",
key.first,
std::accumulate(hosts.begin(), hosts.end(), std::string{}, [](std::string lhs, const std::string& rhs) {
return lhs.empty() ? rhs : lhs + ", " + rhs;
}));
if (peers.empty()) {
std::cerr << "screencast: no receiver found on the LAN; run 'screencast --receive' on the "
"target machine, or pass --peer\n";
return 1;
}
std::cerr << "screencast: multiple receivers found; pass --peer to choose one\n";
return 1;
}
auto [name, port] = receivers.begin()->first;
std::vector<std::string> hosts = receivers.begin()->second;
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return address_preference(lhs) < address_preference(rhs);
});
std::cout << std::format("screencast: found receiver '{}'\n", name);
bool connected = false;
std::string tried;
for (const std::string& host : hosts) {
if (channel->connect(sc::Endpoint{host, port})) {
signaling = sc::Endpoint{host, port};
connected = true;
break;
// Group the addresses by receiver (name + signaling port): one entry
// per address, but they are all the same host.
std::map<std::pair<std::string, std::uint16_t>, std::vector<std::string>> receivers;
for (const sc::DiscoveredPeer& peer : peers) {
receivers[{peer.service_name, peer.signaling_port}].push_back(peer.host);
}
tried += (tried.empty() ? "" : ", ") + host;
if (receivers.size() > 1) {
for (const auto& [key, hosts] : receivers) {
std::cerr << std::format(
"screencast: {} at {}\n",
key.first,
std::accumulate(
hosts.begin(), hosts.end(), std::string{}, [](std::string lhs, const std::string& rhs) {
return lhs.empty() ? rhs : lhs + ", " + rhs;
}));
}
std::cerr << "screencast: multiple receivers found; pass --peer to choose one\n";
return 1;
}
auto [name, port] = receivers.begin()->first;
std::vector<std::string> hosts = receivers.begin()->second;
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return sc::address_preference(lhs) < sc::address_preference(rhs);
});
std::cout << std::format("screencast: found receiver '{}'\n", name);
signaling = sc::Endpoint{hosts.front(), port};
}
if (!connected) {
std::cerr << std::format("screencast: could not reach the receiver (tried {})\n", tried);
auto session_result = sc::SenderSession::start(signaling, command.bitrate_kbps, command.crf, command.fps, target);
if (auto* error = std::get_if<std::string>(&session_result)) {
std::cerr << std::format("screencast: {}\n", *error);
return 1;
}
return negotiate_and_stream(*channel, signaling, command);
auto session = std::move(std::get<sc::SenderSession>(session_result));
std::cout << std::format("screencast: session {} streaming to {}\n", session.session_id(), session.receiver());
while (!g_interrupted.load()) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
session.stop();
return 0;
}
#endif // SC_HAS_SENDER
#ifdef SC_HAS_SENDER
void spawn_detached_sender(const std::string& peer, int bitrate_kbps) {
char self_path[4096] = {};
const ssize_t length = readlink("/proc/self/exe", self_path, sizeof(self_path) - 1);
if (length <= 0) {
return;
}
self_path[length] = '\0';
const pid_t child = fork();
if (child != 0) {
return; // parent returns immediately; the child streams detached
}
setsid();
(void)freopen("/dev/null", "w", stdout);
const std::string bitrate = std::to_string(bitrate_kbps);
(void)execl(self_path,
"screencast",
"--send",
"--peer",
peer.c_str(),
"--bitrate",
bitrate.c_str(),
static_cast<char*>(nullptr));
_exit(127);
}
#endif
void print_waybar_status() {
const auto state = sc::read_sender_state();
// Built with nlohmann::json so all escaping (newlines in tooltips,
// non-ASCII characters) is handled correctly. Hand-rolled format
// strings produced literal control characters that broke waybar's
// JSON parser.
nlohmann::json json = nlohmann::json::object();
if (!state.has_value()) {
json["text"] = "\u23f8"; // pause symbol
json["alt"] = "idle";
json["class"] = "idle";
json["tooltip"] = "screencast idle \u2014 click to stream to the last receiver\nright-click: open the panel";
std::cout << json.dump() << '\n';
return;
}
const std::int64_t elapsed_ms =
std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
.count() -
state->started_epoch_ms;
const std::int64_t minutes = elapsed_ms / 60000;
const std::int64_t seconds = (elapsed_ms / 1000) % 60;
json["text"] = "\u25b6"; // play symbol
json["alt"] = "streaming";
json["class"] = "streaming";
json["tooltip"] = std::format("screencast \u2192 {}\n{} kbps \u00b7 {}m {:02}s\nsession {}",
state->receiver,
state->bitrate_kbps,
minutes,
seconds,
state->session_id);
std::cout << json.dump() << '\n';
}
int run_waybar(const sc::WaybarCommand& command) {
if (command.toggle) {
if (const auto state = sc::read_sender_state(); state.has_value()) {
(void)kill(static_cast<pid_t>(state->pid), SIGTERM);
// Give the graceful shutdown time to withdraw its state file so
// the immediately following status print reflects reality.
for (int attempt = 0; attempt < 20; ++attempt) {
if (!sc::read_sender_state().has_value()) {
break;
}
std::this_thread::sleep_for(std::chrono::milliseconds(50));
}
} else {
#ifdef SC_HAS_SENDER
if (const auto last = sc::read_last_session(); last.has_value()) {
spawn_detached_sender(last->peer, last->bitrate_kbps);
}
#endif
}
}
print_waybar_status();
return 0;
}
int run_receiver(const sc::ReceiveCommand& command) {
sc::ReceiverPipelineConfig config;
config.local_rtp_endpoint = sc::Endpoint{"0.0.0.0", static_cast<std::uint16_t>(command.local_rtp_port)};
config.signaling_port = static_cast<std::uint16_t>(command.signaling_port);
config.decoder.hardware_accel = !command.software_decode;
// Fullscreen is automatic under KMSDRM (headless); the flag forces it
// on desktop sessions.
config.renderer.fullscreen = command.fullscreen;
@@ -323,7 +305,7 @@ int run_discover(const sc::DiscoverCommand& command) {
}
for (auto& [key, hosts] : receivers) {
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return address_preference(lhs) < address_preference(rhs);
return sc::address_preference(lhs) < sc::address_preference(rhs);
});
std::string joined =
std::accumulate(hosts.begin(), hosts.end(), std::string{}, [](std::string lhs, const std::string& rhs) {
@@ -357,5 +339,8 @@ int main(int argc, char* argv[]) {
if (const sc::ReceiveCommand* receive = std::get_if<sc::ReceiveCommand>(&*command)) {
return run_receiver(*receive);
}
if (const sc::WaybarCommand* waybar = std::get_if<sc::WaybarCommand>(&*command)) {
return run_waybar(*waybar);
}
return run_discover(std::get<sc::DiscoverCommand>(*command));
}
+31 -3
View File
@@ -1,3 +1,32 @@
# Shared application internals used by both the CLI binary and the GUI:
# pipelines, session orchestration, and the state store. The sender
# pipeline is only compiled when the sender is built; receiver-only
# targets must not require the capture backend.
sc_app_core_sources = files(
'pipelines.cpp',
'sender_session.cpp',
'state_store.cpp',
)
sc_app_core_args = []
sc_app_core_deps = [dep_json]
if build_sender
sc_app_core_args += ['-DSC_HAS_SENDER=1']
sc_app_core_deps += [sc_capture_dep]
endif
sc_app_core = static_library('sc_app_core',
sc_app_core_sources,
include_directories : [sc_core_inc, include_directories('.')],
cpp_args : sc_app_core_args,
dependencies : sc_app_core_deps)
sc_app_core_dep = declare_dependency(
link_with : sc_app_core,
include_directories : include_directories('.'),
dependencies : [dep_json])
# The screencast application binary wiring every module together.
# Receiver-only builds (-Dsender=false) exclude the capture backend and the
# sender pipeline.
@@ -5,10 +34,9 @@
screencast_sources = [
'cli.cpp',
'main.cpp',
'pipelines.cpp',
]
screencast_dependencies = [sc_codec_dep, sc_network_dep, sc_render_dep]
screencast_dependencies = [sc_app_core_dep, sc_codec_dep, sc_network_dep, sc_render_dep]
screencast_arguments = []
if build_sender
@@ -16,7 +44,7 @@ if build_sender
screencast_arguments += ['-DSC_HAS_SENDER=1']
endif
screencast = executable('screencast',
executable('screencast',
screencast_sources,
cpp_args : screencast_arguments,
dependencies : screencast_dependencies,
+152 -4
View File
@@ -1,11 +1,16 @@
#include "screencast/app/pipeline.h"
#include "state_store.h"
#include "screencast/network/discovery.h"
#include "screencast/network/h264_packetizer.h"
#include "screencast/network/signaling.h"
#include <unistd.h>
#include <unistd.h>
#include <algorithm>
#include <array>
#include <chrono>
#include <condition_variable>
@@ -32,7 +37,7 @@ std::string receiver_service_name() {
class SenderPipeline::Impl {
public:
explicit Impl(SenderPipelineConfig config) : config_(std::move(config)) {}
explicit Impl(SenderPipelineConfig config) : config_(std::move(config)), current_crf_(config_.encoder.crf) {}
~Impl() {
stop();
@@ -53,10 +58,24 @@ class SenderPipeline::Impl {
transport_->set_peer(config_.peer_rtp_endpoint);
run_thread_ = std::jthread([this](std::stop_token stop_token) { run(std::move(stop_token)); });
// Publish the session for status widgets and one-click restarts.
write_sender_state(SenderState{
.session_id = config_.session_id,
.receiver = std::format("{}:{}", config_.peer_rtp_endpoint.address, config_.peer_rtp_endpoint.port),
.bitrate_kbps = config_.encoder.bitrate_kbps,
.pid = ::getpid(),
.started_epoch_ms = 0});
std::string restart_peer = std::format("{}:5005", config_.peer_rtp_endpoint.address);
if (config_.signaling_server.has_value()) {
restart_peer = std::format("{}:{}", config_.signaling_server->address, config_.signaling_server->port);
}
write_last_session(LastSession{.peer = restart_peer, .bitrate_kbps = config_.encoder.bitrate_kbps});
return true;
}
void stop() {
remove_sender_state();
if (capture_ != nullptr) {
capture_->stop();
}
@@ -64,9 +83,23 @@ class SenderPipeline::Impl {
transport_->stop();
}
void request_keyframe() {
keyframe_requested_.store(true);
pli_count_.fetch_add(1, std::memory_order_relaxed);
}
private:
void run(std::stop_token stop_token) {
auto last_adaptation = std::chrono::steady_clock::now();
auto last_pli_check = last_adaptation;
auto last_pli_count = 0;
auto last_frame_time = last_adaptation;
while (!stop_token.stop_requested()) {
if (keyframe_requested_.exchange(false) && encoder_ != nullptr) {
encoder_->request_keyframe();
}
const std::optional<CapturedFrame> frame = capture_->next_frame();
if (!frame.has_value()) {
break;
@@ -76,6 +109,17 @@ class SenderPipeline::Impl {
break;
}
// Frame rate capping: skip frames that arrive faster than the
// configured target. 0 = no cap (use the monitor rate).
if (config_.max_frame_rate > 0) {
const auto now = std::chrono::steady_clock::now();
const auto min_interval = std::chrono::microseconds(1'000'000 / config_.max_frame_rate);
if (now - last_frame_time < min_interval) {
continue;
}
last_frame_time = now;
}
auto encoded_result = encoder_->encode(*frame);
if (is_codec_error(encoded_result)) {
std::cerr << std::format("screencast: encode failed: {}\n", codec_error(encoded_result).message);
@@ -86,13 +130,70 @@ class SenderPipeline::Impl {
(void)transport_->send(packet);
}
}
// Adaptive quality: evaluate the link every 5 seconds by
// checking how many PLIs the receiver sent. Frequent PLIs
// mean the receiver is dropping frames — the link is
// saturated, so increase the CRF (lower quality, fewer
// bits). When the link is quiet, try lowering the CRF to
// probe for better quality.
const auto now = std::chrono::steady_clock::now();
if (now - last_pli_check >= std::chrono::seconds(5)) {
const auto current_pli = pli_count_.load(std::memory_order_relaxed);
const auto pli_delta = current_pli - last_pli_count;
const auto seconds = std::chrono::duration_cast<std::chrono::seconds>(now - last_pli_check).count();
const auto pli_per_second = static_cast<double>(pli_delta) / static_cast<double>(seconds);
last_pli_count = current_pli;
last_pli_check = now;
if (pli_per_second > 0.5 && current_crf_ < config_.encoder.crf + 10) {
// Saturated: degrade quality (higher CRF = fewer bits)
current_crf_ += 2;
std::cerr << std::format(
"screencast: link saturated ({} PLI/s); adapting CRF to {}\n", pli_per_second, current_crf_);
if (!restart_encoder(*frame)) {
break;
}
} else if (pli_per_second < 0.1 && current_crf_ > config_.encoder.crf) {
// Stable: try better quality (lower CRF = more bits)
current_crf_ -= 1;
std::cerr << std::format("screencast: link stable; probing CRF {}\n", current_crf_);
if (!restart_encoder(*frame)) {
break;
}
}
}
}
}
bool restart_encoder(const CapturedFrame& frame) {
// Create a new encoder with the adjusted CRF; the next encoded
// frame is a keyframe, so the receiver recovers immediately.
encoder_ = nullptr;
return create_encoder(frame);
}
bool create_encoder(const CapturedFrame& frame) {
EncoderConfig config = config_.encoder;
config.width = frame.width;
config.height = frame.height;
config.crf = current_crf_;
// Downscale to the receiver's display when the capture is larger,
// preserving aspect ratio and rounding to even values (YUV420P
// requires even dimensions for the chroma planes).
if (config_.max_encode_width > 0 && config_.max_encode_height > 0 &&
(frame.width > config_.max_encode_width || frame.height > config_.max_encode_height)) {
const double scale = std::min(static_cast<double>(config_.max_encode_width) / frame.width,
static_cast<double>(config_.max_encode_height) / frame.height);
config.width = std::max(2, static_cast<int>(frame.width * scale) & ~1);
config.height = std::max(2, static_cast<int>(frame.height * scale) & ~1);
std::cerr << std::format("screencast: downscaling {}x{} to {}x{} for the receiver's display\n",
frame.width,
frame.height,
config.width,
config.height);
}
auto encoder_result = EncoderFactory::create(config);
if (is_codec_error(encoder_result)) {
@@ -104,13 +205,20 @@ class SenderPipeline::Impl {
}
SenderPipelineConfig config_;
int current_crf_;
std::unique_ptr<CaptureSession> capture_;
std::unique_ptr<Encoder> encoder_;
H264Packetizer packetizer_;
std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create();
std::jthread run_thread_;
std::atomic<bool> keyframe_requested_{false};
std::atomic<int> pli_count_{0};
};
void SenderPipeline::request_keyframe() {
impl_->request_keyframe();
}
SenderPipeline::SenderPipeline(SenderPipelineConfig config) : impl_(std::make_unique<Impl>(std::move(config))) {}
SenderPipeline::~SenderPipeline() = default;
@@ -238,6 +346,7 @@ class ReceiverPipeline::Impl {
signaling_ = nullptr;
discovery_ = nullptr;
transport_->stop();
jitter_.clear();
renderer_ = nullptr;
decoder_ = nullptr;
{
@@ -248,13 +357,24 @@ class ReceiverPipeline::Impl {
private:
void on_packet(RtpPacket packet) {
std::optional<std::vector<std::byte>> access_unit = depacketizer_.depacketize(packet);
if (!access_unit.has_value()) {
// Absorb reordering (Wi-Fi) before the in-order depacketizer, so a
// late packet is not misread as loss.
for (RtpPacket& ordered : jitter_.push(std::move(packet))) {
deliver_packet(ordered);
}
}
void deliver_packet(const RtpPacket& packet) {
const DepacketizeResult result = depacketizer_.depacketize(packet);
if (result.frame_dropped) {
maybe_send_pli();
}
if (!result.access_unit.has_value()) {
return;
}
EncodedFrame encoded;
encoded.data = std::move(*access_unit);
encoded.data = std::move(*result.access_unit);
encoded.rtp_timestamp = packet.header.timestamp;
encoded.is_keyframe = false;
@@ -282,14 +402,38 @@ class ReceiverPipeline::Impl {
if (offer == nullptr) {
return;
}
session_id_ = offer->session_id;
SessionAnswer answer;
answer.session_id = offer->session_id;
// Empty address: the sender targets the address of its signaling
// connection, which reaches this RTP port.
answer.rtp_endpoint = Endpoint{"", config_.local_rtp_endpoint.port};
// Tell the sender what display it is rendering to so it can
// downscale instead of encoding pixels the display cannot show.
if (renderer_ != nullptr) {
answer.display_width = renderer_->display_width();
answer.display_height = renderer_->display_height();
}
signaling_->send(answer);
}
// Ask the sender for a keyframe after a damaged frame, rate-limited so
// sustained loss cannot flood the signaling channel.
void maybe_send_pli() {
if (signaling_ == nullptr || session_id_.empty()) {
return;
}
const auto now = std::chrono::steady_clock::now();
if (now - last_pli_time_ < kPliMinInterval) {
return;
}
last_pli_time_ = now;
SessionPli pli;
pli.session_id = session_id_;
signaling_->send(pli);
std::cerr << std::format("screencast: frame damaged; requesting a keyframe\n");
}
void render_loop(std::stop_token stop_token) {
while (!stop_token.stop_requested()) {
if (!renderer_->poll_events()) {
@@ -315,17 +459,21 @@ class ReceiverPipeline::Impl {
}
static constexpr std::size_t kMaxQueuedFrames = 3;
static constexpr std::chrono::milliseconds kPliMinInterval{500};
ReceiverPipelineConfig config_;
std::unique_ptr<Renderer> renderer_;
std::unique_ptr<Decoder> decoder_;
H264Depacketizer depacketizer_;
RtpJitterBuffer jitter_;
std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create();
std::unique_ptr<SignalingChannel> signaling_;
std::unique_ptr<DiscoveryService> discovery_;
std::jthread render_thread_;
std::mutex queue_mutex_;
std::deque<DecodedFrame> queue_;
std::string session_id_;
std::chrono::steady_clock::time_point last_pli_time_{};
bool stream_started_ = false;
bool present_error_logged_ = false;
};
+174
View File
@@ -0,0 +1,174 @@
#include "sender_session.h"
#include "screencast/network/signaling.h"
#include <atomic>
#include <charconv>
#include <chrono>
#include <format>
#include <future>
#include <memory>
#include <string_view>
#include <utility>
namespace sc {
#ifdef SC_HAS_SENDER
namespace {
bool is_private_ipv4(std::string_view host) {
if (host.rfind("192.168.", 0) == 0 || host.rfind("10.", 0) == 0) {
return true;
}
if (host.rfind("172.", 0) == 0) {
const std::size_t second = host.find('.', 5);
if (second != std::string_view::npos) {
int octet = 0;
const auto [pointer, error] = std::from_chars(host.data() + 5, host.data() + second, octet);
if (error == std::errc{}) {
return octet >= 16 && octet <= 31;
}
}
}
return false;
}
} // namespace
std::variant<SenderSession, std::string>
SenderSession::start(const Endpoint& signaling_endpoint, int bitrate_kbps, int crf, int max_fps, CaptureTarget target) {
auto channel_result = SignalingFactory::create_client();
if (is_network_error(channel_result)) {
return network_error(channel_result).message;
}
auto channel = std::move(network_value(channel_result));
if (!channel->connect(signaling_endpoint)) {
return std::format(
"failed to connect to the receiver at {}:{}", signaling_endpoint.address, signaling_endpoint.port);
}
// Offer + answer.
static std::atomic<std::uint32_t> session_counter{0};
const std::string session_id = std::format(
"sc-{:x}-{:x}",
static_cast<std::uint32_t>(std::chrono::steady_clock::now().time_since_epoch().count()) & 0xffffffffU,
session_counter.fetch_add(1));
std::promise<SessionAnswer> answer_promise;
auto answer_future = answer_promise.get_future();
std::atomic<bool> answered{false};
channel->on_message([&](const SignalingMessage& message) {
if (const SessionAnswer* answer = std::get_if<SessionAnswer>(&message)) {
if (!answered.exchange(true)) {
answer_promise.set_value(*answer);
}
}
});
SessionOffer offer;
offer.session_id = session_id;
offer.codec_name = "h264";
offer.frame_rate_num = 25;
offer.frame_rate_den = 1;
channel->send(offer);
if (answer_future.wait_for(std::chrono::seconds(5)) != std::future_status::ready) {
channel->disconnect();
return std::string{"the receiver did not answer the session offer"};
}
const SessionAnswer answer = answer_future.get();
if (answer.session_id != session_id) {
channel->disconnect();
return std::string{"session mismatch in the receiver's answer"};
}
if (answer.rtp_endpoint.port == 0) {
channel->disconnect();
return std::string{"the receiver did not provide an RTP port"};
}
// Stream to the negotiated endpoint; an empty address means "the
// address you reached me on".
const Endpoint rtp_endpoint = answer.rtp_endpoint.address.empty()
? Endpoint{signaling_endpoint.address, answer.rtp_endpoint.port}
: answer.rtp_endpoint;
SenderPipelineConfig config;
config.capture_target = target;
config.peer_rtp_endpoint = rtp_endpoint;
config.signaling_server = signaling_endpoint;
config.encoder.bitrate_kbps = bitrate_kbps;
config.encoder.crf = crf;
config.max_frame_rate = max_fps;
config.session_id = session_id;
config.max_encode_width = answer.display_width;
config.max_encode_height = answer.display_height;
auto pipeline = std::make_unique<SenderPipeline>(std::move(config));
if (!pipeline->start()) {
channel->disconnect();
return std::string{"failed to start the sender pipeline"};
}
// PLI feedback over the still-open channel.
channel->on_message([pipeline = pipeline.get(), session_id](const SignalingMessage& message) {
const SessionPli* pli = std::get_if<SessionPli>(&message);
if (pli != nullptr && pli->session_id == session_id) {
pipeline->request_keyframe();
}
});
SenderSession session;
session.channel_ = std::move(channel);
session.pipeline_ = std::move(pipeline);
session.session_id_ = session_id;
session.receiver_ = std::format("{}:{}", rtp_endpoint.address, rtp_endpoint.port);
return session;
}
SenderSession::SenderSession(SenderSession&& other) noexcept
: channel_(std::move(other.channel_)),
pipeline_(std::move(other.pipeline_)),
session_id_(std::move(other.session_id_)),
receiver_(std::move(other.receiver_)),
stopped_(other.stopped_) {}
SenderSession& SenderSession::operator=(SenderSession&& other) noexcept {
stop();
channel_ = std::move(other.channel_);
pipeline_ = std::move(other.pipeline_);
session_id_ = std::move(other.session_id_);
receiver_ = std::move(other.receiver_);
stopped_ = other.stopped_;
return *this;
}
SenderSession::~SenderSession() {
stop();
}
void SenderSession::stop() {
if (stopped_) {
return;
}
stopped_ = true;
// The channel first: disconnecting joins its reader threads, so no PLI
// callback can race the pipeline teardown it points at.
if (channel_ != nullptr) {
channel_->disconnect();
channel_ = nullptr;
}
if (pipeline_ != nullptr) {
pipeline_->stop();
pipeline_ = nullptr;
}
}
#else
// Receiver-only builds have no sender pipeline; address_preference is the
// inline header function, and the rest of SenderSession is compiled out.
#endif // SC_HAS_SENDER
} // namespace sc
+72
View File
@@ -0,0 +1,72 @@
#pragma once
#include "screencast/app/pipeline.h"
#include <cstdint>
#include <memory>
#include <string>
#include <string_view>
#include <variant>
namespace sc {
// Ordering for trying a receiver's addresses: private IPv4 first (LANs,
// most reliable), then public IPv4, ULA, and global IPv6. 6to4 (2002::) and
// link-local addresses last: 6to4 is frequently unreachable between LAN
// peers, and link-local needs a scope id to even route. Inline so that
// receiver-only builds (no sender session) can still use it for discovery.
inline int address_preference(std::string_view host) {
if (host.find(':') == std::string_view::npos) {
const bool priv = host.rfind("192.168.", 0) == 0 || host.rfind("10.", 0) == 0 ||
(host.rfind("172.", 0) == 0 && host.find('.', 5) != std::string_view::npos);
return priv ? 0 : 1;
}
if (host.rfind("fd", 0) == 0 || host.rfind("fc", 0) == 0) {
return 2;
}
if (host.rfind("2002:", 0) == 0) {
return 4;
}
if (host.rfind("fe80:", 0) == 0) {
return 5;
}
return 3;
}
// A running sender session: it negotiated over signaling (keeping the
// channel open for PLI feedback) and drives the sender pipeline. Shared by
// the CLI, the GUI, and the waybar widget's one-click restart.
class SenderSession {
public:
// Blocking: connects, offers, waits for the answer, and starts the
// pipeline — which includes the portal's interactive source picker.
// Returns an error message on failure.
static std::variant<SenderSession, std::string>
start(const Endpoint& signaling_endpoint, int bitrate_kbps, int crf, int max_fps, CaptureTarget target);
SenderSession() = default;
~SenderSession();
SenderSession(SenderSession&& other) noexcept;
SenderSession& operator=(SenderSession&& other) noexcept;
// Graceful stop: withdraws state and closes the signaling channel.
void stop();
[[nodiscard]] const std::string& session_id() const {
return session_id_;
}
[[nodiscard]] const std::string& receiver() const {
return receiver_;
}
private:
std::unique_ptr<class SignalingChannel> channel_;
std::unique_ptr<SenderPipeline> pipeline_;
std::string session_id_;
std::string receiver_;
bool stopped_ = false;
};
} // namespace sc
+140
View File
@@ -0,0 +1,140 @@
#include "state_store.h"
#include <nlohmann/json.hpp>
#include <sys/types.h>
#include <unistd.h>
#include <chrono>
#include <csignal>
#include <cstdlib>
#include <filesystem>
#include <fstream>
#include <sstream>
namespace sc {
namespace {
namespace fs = std::filesystem;
fs::path runtime_dir() {
const char* xdg = std::getenv("XDG_RUNTIME_DIR");
if (xdg != nullptr && *xdg != '\0') {
return fs::path{xdg} / "screencast";
}
return fs::temp_directory_path() / ("screencast-" + std::to_string(::getuid()));
}
fs::path config_dir() {
const char* xdg = std::getenv("XDG_CONFIG_HOME");
if (xdg != nullptr && *xdg != '\0') {
return fs::path{xdg} / "screencast";
}
const char* home = std::getenv("HOME");
return fs::path{home != nullptr ? home : "."} / ".config" / "screencast";
}
std::optional<std::string> read_file(const fs::path& path) {
std::ifstream file{path, std::ios::binary};
if (!file.is_open()) {
return std::nullopt;
}
std::ostringstream buffer;
buffer << file.rdbuf();
return buffer.str();
}
bool write_file(const fs::path& path, std::string_view contents) {
std::error_code error;
fs::create_directories(path.parent_path(), error);
std::ofstream file{path, std::ios::binary | std::ios::trunc};
if (!file.is_open()) {
return false;
}
file.write(contents.data(), static_cast<std::streamsize>(contents.size()));
return file.good();
}
std::int64_t epoch_ms() {
const auto now = std::chrono::system_clock::now().time_since_epoch();
return std::chrono::duration_cast<std::chrono::milliseconds>(now).count();
}
bool process_alive(int pid) {
if (pid <= 0) {
return false;
}
return ::kill(static_cast<pid_t>(pid), 0) == 0;
}
} // namespace
bool write_sender_state(const SenderState& state) {
nlohmann::json json;
json["session_id"] = state.session_id;
json["receiver"] = state.receiver;
json["bitrate_kbps"] = state.bitrate_kbps;
json["pid"] = state.pid;
json["started_epoch_ms"] = state.started_epoch_ms != 0 ? state.started_epoch_ms : epoch_ms();
return write_file(runtime_dir() / "sender.json", json.dump() + "\n");
}
bool remove_sender_state() {
std::error_code error;
const bool removed = fs::remove(runtime_dir() / "sender.json", error);
return removed || !fs::exists(runtime_dir() / "sender.json");
}
std::optional<SenderState> read_sender_state() {
const std::optional<std::string> contents = read_file(runtime_dir() / "sender.json");
if (!contents.has_value()) {
return std::nullopt;
}
const nlohmann::json json = nlohmann::json::parse(*contents, nullptr, /*allow_exceptions=*/false);
if (json.is_discarded() || !json.is_object()) {
return std::nullopt;
}
SenderState state;
state.session_id = json.value("session_id", std::string{});
state.receiver = json.value("receiver", std::string{});
state.bitrate_kbps = json.value("bitrate_kbps", 0);
state.pid = json.value("pid", 0);
state.started_epoch_ms = json.value("started_epoch_ms", std::int64_t{0});
// A state file without a live process is a crash remnant: not streaming.
if (!process_alive(state.pid)) {
(void)remove_sender_state();
return std::nullopt;
}
return state;
}
void write_last_session(const LastSession& last) {
nlohmann::json json;
json["peer"] = last.peer;
json["bitrate_kbps"] = last.bitrate_kbps;
(void)write_file(config_dir() / "last-session.json", json.dump() + "\n");
}
std::optional<LastSession> read_last_session() {
const std::optional<std::string> contents = read_file(config_dir() / "last-session.json");
if (!contents.has_value()) {
return std::nullopt;
}
const nlohmann::json json = nlohmann::json::parse(*contents, nullptr, /*allow_exceptions=*/false);
if (json.is_discarded() || !json.is_object()) {
return std::nullopt;
}
LastSession last;
last.peer = json.value("peer", std::string{});
last.bitrate_kbps = json.value("bitrate_kbps", 0);
if (last.peer.empty() || last.bitrate_kbps <= 0) {
return std::nullopt;
}
return last;
}
} // namespace sc
+35
View File
@@ -0,0 +1,35 @@
#pragma once
#include <cstdint>
#include <optional>
#include <string>
namespace sc {
// Live state of a running sender, written by the pipeline so status
// widgets (waybar) and one-click restarts work for every front-end.
struct SenderState {
std::string session_id;
std::string receiver; // "host:port" of the RTP endpoint
int bitrate_kbps = 0;
int pid = 0;
std::int64_t started_epoch_ms = 0;
};
// The last successfully started session, persisted across reboots so a
// single click can restart streaming without picking anything.
struct LastSession {
std::string peer; // "host" or "host:port" for the signaling endpoint
int bitrate_kbps = 0;
};
bool write_sender_state(const SenderState& state);
bool remove_sender_state();
// Returns nullopt when no sender is running (a stale state file is
// treated as not streaming and is cleaned up).
std::optional<SenderState> read_sender_state();
void write_last_session(const LastSession& last);
std::optional<LastSession> read_last_session();
} // namespace sc
+101 -35
View File
@@ -5,6 +5,8 @@
#include <array>
#include <cstdint>
#include <cstring>
#include <format>
#include <iostream>
#include <limits>
#include <optional>
#include <string>
@@ -145,34 +147,61 @@ class FfmpegDecoder final : public Decoder {
? static_cast<uint64_t>(av_rescale_q(input->pts, ctx_->time_base, AVRational{1, 1'000'000'000}))
: fallback_timestamp_ns;
const std::size_t buffer_size =
static_cast<std::size_t>(av_image_get_buffer_size(AV_PIX_FMT_RGBA, frame.width, frame.height, 1));
frame.rgba_pixels.resize(buffer_size);
if (!ensure_scaler(input->width, input->height, static_cast<AVPixelFormat>(input->format))) {
return CodecError{"failed to create swscale context"};
const AVPixelFormat pixel_format = static_cast<AVPixelFormat>(input->format);
if (pixel_format != AV_PIX_FMT_YUV420P) {
// Hardware decoders (e.g. v4l2m2m) may emit NV12 or other planar
// variants; convert to YUV420P once. The common software path
// (YUV420P) skips this entirely.
if (!ensure_scaler(input->width, input->height, pixel_format)) {
return CodecError{"failed to create format conversion context"};
}
AvFramePtr converted(av_frame_alloc(), AvFrameDeleter{});
converted->width = input->width;
converted->height = input->height;
converted->format = AV_PIX_FMT_YUV420P;
if (av_frame_get_buffer(converted.get(), 0) < 0) {
return CodecError{"failed to allocate converted frame"};
}
if (sws_scale(scaler_.get(),
input->data,
input->linesize,
0,
input->height,
converted->data,
converted->linesize) <= 0) {
return CodecError{"failed to convert decoded frame to YUV420P"};
}
return copy_yuv420p_planes(converted.get(), std::move(frame));
}
std::array<uint8_t*, 4> dst{nullptr, nullptr, nullptr, nullptr};
std::array<int, 4> dst_lines{0, 0, 0, 0};
if (av_image_fill_arrays(dst.data(),
dst_lines.data(),
as_u8(frame.rgba_pixels.data()),
AV_PIX_FMT_RGBA,
frame.width,
frame.height,
1) < 0) {
return CodecError{"failed to fill output pixel arrays"};
}
return copy_yuv420p_planes(input, std::move(frame));
}
if (sws_scale(scaler_.get(), input->data, input->linesize, 0, input->height, dst.data(), dst_lines.data()) <=
0) {
return CodecError{"failed to convert decoded frame to RGBA"};
}
// Copies the three YUV420P planes with their strides (which may include
// alignment padding). SDL's UpdateYUVTexture accepts arbitrary pitches.
CodecResult<DecodedFrame> copy_yuv420p_planes(const AVFrame* input, DecodedFrame frame) const {
const int width = input->width;
const int height = input->height;
frame.stride_y = input->linesize[0];
frame.stride_u = input->linesize[1];
frame.stride_v = input->linesize[2];
const std::size_t y_size = static_cast<std::size_t>(frame.stride_y) * height;
const std::size_t uv_size = static_cast<std::size_t>(frame.stride_u) * ((height + 1) / 2);
const std::size_t v_size = static_cast<std::size_t>(frame.stride_v) * ((height + 1) / 2);
frame.plane_y.resize(y_size);
frame.plane_u.resize(uv_size);
frame.plane_v.resize(v_size);
std::memcpy(frame.plane_y.data(), input->data[0], y_size);
std::memcpy(frame.plane_u.data(), input->data[1], uv_size);
std::memcpy(frame.plane_v.data(), input->data[2], v_size);
return frame;
}
// Only used when the decoder emits a non-YUV420P format (hardware paths).
bool ensure_scaler(int width, int height, AVPixelFormat input_format) const {
if (scaler_ != nullptr && scaler_input_width_ == width && scaler_input_height_ == height &&
scaler_input_format_ == input_format) {
@@ -180,7 +209,7 @@ class FfmpegDecoder final : public Decoder {
}
scaler_.reset(sws_getContext(
width, height, input_format, width, height, AV_PIX_FMT_RGBA, SWS_BILINEAR, nullptr, nullptr, nullptr));
width, height, input_format, width, height, AV_PIX_FMT_YUV420P, SWS_BILINEAR, nullptr, nullptr, nullptr));
if (scaler_ == nullptr) {
return false;
}
@@ -199,19 +228,18 @@ class FfmpegDecoder final : public Decoder {
mutable AVPixelFormat scaler_input_format_ = AV_PIX_FMT_NONE;
};
CodecResult<std::unique_ptr<Decoder>> DecoderFactory::create(const DecoderConfig& config) {
if (config.codec_name != "h264") {
return CodecError{"only h264 is supported in phase 2"};
}
namespace {
const AVCodec* codec = avcodec_find_decoder(AV_CODEC_ID_H264);
// Opens the given decoder implementation for the given config. Returns an
// error string on failure (used for the hardware probe + fallback).
std::optional<std::string> open_decoder(const DecoderConfig& config, const AVCodec* codec, AvCodecContextPtr& ctx) {
if (codec == nullptr) {
return CodecError{"h264 decoder not found"};
return std::string{"decoder not found"};
}
AvCodecContextPtr ctx(avcodec_alloc_context3(codec), AvCodecContextDeleter{});
ctx.reset(avcodec_alloc_context3(codec));
if (ctx == nullptr) {
return CodecError{"failed to allocate decoder context"};
return std::string{"failed to allocate decoder context"};
}
ctx->codec_type = AVMEDIA_TYPE_VIDEO;
@@ -221,28 +249,66 @@ CodecResult<std::unique_ptr<Decoder>> DecoderFactory::create(const DecoderConfig
if (config.height > 0) {
ctx->height = config.height;
}
ctx->thread_count = 1;
// Slice-level threading parallelizes within a single frame (no added
// latency), unlike frame-level threading which buffers multiple frames
// before producing output — unacceptable for a live stream.
ctx->thread_count = 4;
ctx->thread_type = FF_THREAD_SLICE;
if (!config.extradata.empty()) {
if (config.extradata.size() > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
return CodecError{"decoder extradata is too large"};
return std::string{"decoder extradata is too large"};
}
// FFmpeg bitstream parsers may read past the end of extradata, so the
// buffer must include the padding they require.
ctx->extradata = static_cast<uint8_t*>(av_malloc(config.extradata.size() + AV_INPUT_BUFFER_PADDING_SIZE));
if (ctx->extradata == nullptr) {
return CodecError{"failed to allocate decoder extradata"};
return std::string{"failed to allocate decoder extradata"};
}
std::memcpy(ctx->extradata, config.extradata.data(), config.extradata.size());
std::memset(ctx->extradata + config.extradata.size(), 0, AV_INPUT_BUFFER_PADDING_SIZE);
ctx->extradata_size = static_cast<int>(config.extradata.size());
}
int open_ret = avcodec_open2(ctx.get(), codec, nullptr);
const int open_ret = avcodec_open2(ctx.get(), codec, nullptr);
if (open_ret < 0) {
return CodecError{std::string{"failed to open h264 decoder: "} + ffmpeg_error(open_ret)};
return std::string{"failed to open decoder: "} + ffmpeg_error(open_ret);
}
return std::nullopt;
}
} // namespace
CodecResult<std::unique_ptr<Decoder>> DecoderFactory::create(const DecoderConfig& config) {
if (config.codec_name != "h264") {
return CodecError{"only h264 is supported"};
}
// Hardware first (v4l2 mem2mem: the Pi's VideoCore H.264 decoder), with
// an automatic software fallback. The kernel's bitstream parser reads
// the stream dimensions from the in-band SPS, so they are not required
// up front.
if (config.hardware_accel) {
if (const AVCodec* hw = avcodec_find_decoder_by_name("h264_v4l2m2m"); hw != nullptr) {
AvCodecContextPtr ctx{nullptr, AvCodecContextDeleter{}};
if (auto error = open_decoder(config, hw, ctx)) {
std::cerr << std::format("screencast: hardware decode unavailable ({}); falling back to "
"software\n",
*error);
} else {
std::cerr << "screencast: using hardware H.264 decode (h264_v4l2m2m)\n";
return std::make_unique<FfmpegDecoder>(std::move(ctx), config);
}
} else {
std::cerr << "screencast: hardware decoder not compiled in; using software decode\n";
}
}
const AVCodec* codec = avcodec_find_decoder(AV_CODEC_ID_H264);
AvCodecContextPtr ctx{nullptr, AvCodecContextDeleter{}};
if (auto error = open_decoder(config, codec, ctx)) {
return CodecError{std::move(*error)};
}
return std::make_unique<FfmpegDecoder>(std::move(ctx), config);
}
+58 -14
View File
@@ -286,8 +286,12 @@ class FfmpegEncoder final : public Encoder {
return CodecError{"failed to allocate AVFrame"};
}
output->width = frame.width;
output->height = frame.height;
// The output frame is at the encoder's configured dimensions (which
// may be smaller than the capture when downscaling to the receiver's
// display); sws_scale handles both the format conversion and the
// resolution change in one pass.
output->width = config_.width;
output->height = config_.height;
output->format = AV_PIX_FMT_YUV420P;
output->time_base = ctx_->time_base;
output->pts =
@@ -338,8 +342,18 @@ class FfmpegEncoder final : public Encoder {
return true;
}
scaler_.reset(sws_getContext(
width, height, input_format, width, height, AV_PIX_FMT_YUV420P, SWS_BILINEAR, nullptr, nullptr, nullptr));
// Scale to the encoder's configured output (which may be smaller
// than the input when downscaling to the receiver's display).
scaler_.reset(sws_getContext(width,
height,
input_format,
config_.width,
config_.height,
AV_PIX_FMT_YUV420P,
SWS_BILINEAR,
nullptr,
nullptr,
nullptr));
if (scaler_ == nullptr) {
return false;
}
@@ -367,7 +381,10 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
return CodecError{"encoder frame rate must be positive"};
}
if (config.bitrate_kbps <= 0) {
return CodecError{"encoder bitrate must be positive"};
return CodecError{"encoder VBV max bitrate must be positive"};
}
if (config.crf < 0 || config.crf > 51) {
return CodecError{"encoder CRF must be between 0 and 51"};
}
if (config.codec_name != "h264" && config.codec_name != "libx264") {
return CodecError{"only h264 is supported in phase 2"};
@@ -395,14 +412,32 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
ctx->time_base = AVRational{1, 1'000'000};
ctx->framerate = AVRational{config.frame_rate_num, config.frame_rate_den};
ctx->pix_fmt = AV_PIX_FMT_YUV420P;
ctx->bit_rate = static_cast<int64_t>(config.bitrate_kbps) * 1000;
// VBV keeps the stream CBR-ish: without it a keyframe may take many
// times the average frame size in one burst, overflowing the receiver's
// UDP socket buffer and dropping the packets that carry SPS/PPS. Two
// frame periods of budget keep latency low while bounding the burst.
ctx->rc_max_rate = ctx->bit_rate;
ctx->rc_buffer_size = static_cast<int>(ctx->bit_rate * 2 / config.frame_rate_num);
ctx->gop_size = config.frame_rate_num;
// CRF rate control: target a constant visual quality level instead of
// a fixed bitrate, and let the encoder use fewer bits on static screen
// content and more on motion or text. The VBV max rate still caps the
// peak so bursts cannot overflow the receiver's UDP buffers.
// CRF rate control: set x264's CRF directly as a private option.
// FFmpeg's global_quality + AV_CODEC_FLAG_QSCALE path divides by
// FF_QP2LAMBDA, giving a wrong CRF value (16/118 ≈ 0.1, essentially
// lossless) — hence the "-qscale is ignored" warning. Setting the
// private "crf" option bypasses that and gives x264 the exact value.
const std::string crf_value = std::to_string(config.crf);
if (av_opt_set(ctx->priv_data, "crf", crf_value.c_str(), 0) < 0) {
return CodecError{"failed to set CRF quality"};
}
ctx->rc_max_rate = static_cast<int64_t>(config.bitrate_kbps) * 1000;
// VBV: one frame period of budget keeps bursts tight — a two-frame
// buffer lets a keyframe spike beyond what a constrained link can
// absorb in real time, causing packet loss that cascades into PLI
// storms. The tighter buffer trades a small quality dip on keyframes
// for much better behavior on slow paths.
ctx->rc_buffer_size = static_cast<int>(ctx->rc_max_rate / config.frame_rate_num);
// A long GOP saves the keyframe overhead for screen content (which
// changes incrementally); PLI feedback recovers from loss within one
// frame time regardless of the GOP length.
ctx->gop_size = config.frame_rate_num * 5;
ctx->max_b_frames = 0;
ctx->thread_count = 1;
ctx->profile = AV_PROFILE_H264_MAIN;
@@ -411,7 +446,7 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
// without out-of-band parameter negotiation.
ctx->flags |= AV_CODEC_FLAG_LOW_DELAY;
if (av_opt_set(ctx->priv_data, "preset", "ultrafast", 0) < 0) {
if (av_opt_set(ctx->priv_data, "preset", "faster", 0) < 0) {
return CodecError{"failed to set libx264 preset"};
}
if (av_opt_set(ctx->priv_data, "tune", "zerolatency", 0) < 0) {
@@ -420,6 +455,15 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
if (av_opt_set(ctx->priv_data, "forced-idr", "1", 0) < 0) {
return CodecError{"failed to enable forced IDR keyframes"};
}
// Screen-content tuning: auto-variance AQ allocates bits away from
// flat areas and toward text edges; higher psy-rd preserves texture
// sharpness at the cost of slight rate efficiency.
if (av_opt_set(ctx->priv_data, "aq-mode", "2", 0) < 0) {
return CodecError{"failed to set adaptive quantization mode"};
}
if (av_opt_set(ctx->priv_data, "psy-rd", "1.5", 0) < 0) {
return CodecError{"failed to set psychovisual rate-distortion strength"};
}
int open_ret = avcodec_open2(ctx.get(), codec, nullptr);
if (open_ret < 0) {
+402
View File
@@ -0,0 +1,402 @@
// GTK4 sender panel: discover receivers on the LAN, pick one, choose a
// quality preset, and start/stop streaming. The blocking parts (portal
// source picker, negotiation) run on worker threads so the UI stays
// responsive.
#include "screencast/network/discovery.h"
#include "sender_session.h"
#include "state_store.h"
#include <gtkmm.h>
#include <algorithm>
#include <chrono>
#include <cstdint>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <thread>
#include <utility>
#include <vector>
namespace {
using sc::DiscoveredPeer;
using sc::Endpoint;
struct ReceiverRow {
std::string name;
std::string host;
std::uint16_t signaling_port = 0;
};
// Quality presets: CRF (visual quality), max bitrate (VBV cap), and frame
// rate cap. Each maps to a real-world CLI invocation. The frame rate is a
// critical bandwidth lever for desktop content — 15-25 fps is perfectly
// smooth for screencasting and cuts bandwidth 2-4x vs. monitor rate.
struct QualityPreset {
std::string label;
int crf;
int bitrate_kbps;
int fps; // 0 = no cap (use monitor rate)
};
const std::vector<QualityPreset>& quality_presets() {
static const std::vector<QualityPreset> presets = {
{"Low bandwidth", 26, 3000, 15},
{"Balanced", 22, 5000, 20},
{"Sharp", 18, 8000, 25},
{"Very sharp", 16, 12000, 30},
{"Maximum", 14, 20000, 0},
};
return presets;
}
class SenderWindow : public Gtk::ApplicationWindow {
public:
SenderWindow() {
set_title("screencast");
set_default_size(440, 440);
// GTK4 only supports themed icons: the PNG is installed into the
// hicolor theme (see meson.build) and picked up by name.
set_icon_name("screencast");
auto* box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::VERTICAL, 8);
set_child(*box);
box->set_margin(12);
// Receiver list header + refresh.
auto* header = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*header);
auto* title = Gtk::make_managed<Gtk::Label>();
title->set_text("Receivers");
title->set_hexpand(true);
title->set_halign(Gtk::Align::START);
header->append(*title);
refresh_button_ = Gtk::make_managed<Gtk::Button>();
refresh_button_->set_label("Refresh");
refresh_button_->signal_clicked().connect(sigc::mem_fun(*this, &SenderWindow::on_refresh));
header->append(*refresh_button_);
scrolled_ = Gtk::make_managed<Gtk::ScrolledWindow>();
scrolled_->set_policy(Gtk::PolicyType::NEVER, Gtk::PolicyType::AUTOMATIC);
scrolled_->set_vexpand(true);
box->append(*scrolled_);
receiver_list_ = Gtk::make_managed<Gtk::ListBox>();
receiver_list_->set_selection_mode(Gtk::SelectionMode::SINGLE);
receiver_list_->signal_row_selected().connect(
[this](Gtk::ListBoxRow*) { Glib::signal_idle().connect_once([this] { update_sensitivity(); }); });
scrolled_->set_child(*receiver_list_);
// Quality preset dropdown.
auto* quality_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*quality_box);
auto* quality_label = Gtk::make_managed<Gtk::Label>();
quality_label->set_text("Quality:");
quality_label->set_halign(Gtk::Align::START);
quality_box->append(*quality_label);
quality_combo_ = Gtk::make_managed<Gtk::DropDown>();
auto preset_list = Gtk::StringList::create({"placeholder"});
preset_list->remove(0);
for (const QualityPreset& preset : quality_presets()) {
preset_list->append(preset.label);
}
quality_combo_->set_model(preset_list);
quality_combo_->set_selected(1); // Standard
quality_combo_->property_selected().signal_changed().connect([this] { on_preset_changed(); });
quality_combo_->set_hexpand(true);
quality_box->append(*quality_combo_);
// Fine-tuning: bitrate cap and frame rate, updated by the preset.
auto* detail_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*detail_box);
bitrate_label_ = Gtk::make_managed<Gtk::Label>();
bitrate_label_->set_hexpand(true);
bitrate_label_->set_halign(Gtk::Align::START);
detail_box->append(*bitrate_label_);
bitrate_scale_ = Gtk::make_managed<Gtk::Scale>(Gtk::Orientation::HORIZONTAL);
bitrate_scale_->set_range(500.0, 20000.0);
bitrate_scale_->set_value(5000.0);
bitrate_scale_->set_increments(500.0, 1000.0);
bitrate_scale_->set_draw_value(false);
bitrate_scale_->set_hexpand(true);
bitrate_scale_->signal_value_changed().connect([this] { update_quality_labels(); });
detail_box->append(*bitrate_scale_);
auto* fps_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*fps_box);
auto* fps_label = Gtk::make_managed<Gtk::Label>();
fps_label->set_text("Frame rate:");
fps_label->set_halign(Gtk::Align::START);
fps_box->append(*fps_label);
auto fps_adjustment = Gtk::Adjustment::create(20.0, 5.0, 60.0, 1.0, 5.0, 0.0);
fps_spin_ = Gtk::make_managed<Gtk::SpinButton>(fps_adjustment, 1.0, 0);
fps_spin_->signal_value_changed().connect([this] { update_quality_labels(); });
fps_box->append(*fps_spin_);
auto* fps_hint = Gtk::make_managed<Gtk::Label>();
fps_hint->set_text("0 = uncapped (monitor rate)");
fps_hint->set_halign(Gtk::Align::END);
fps_hint->set_hexpand(true);
fps_hint->set_sensitive(false);
fps_box->append(*fps_hint);
start_button_ = Gtk::make_managed<Gtk::Button>();
start_button_->set_label("Start");
start_button_->signal_clicked().connect(sigc::mem_fun(*this, &SenderWindow::on_start_stop));
box->append(*start_button_);
status_label_ = Gtk::make_managed<Gtk::Label>();
status_label_->set_wrap(true);
status_label_->set_halign(Gtk::Align::START);
status_label_->set_valign(Gtk::Align::START);
status_label_->set_vexpand(true);
box->append(*status_label_);
// Per-second status refresh (elapsed time, liveness heartbeat).
Glib::signal_timeout().connect_seconds(
[this]() -> bool {
update_status();
return true;
},
1);
update_quality_labels();
update_sensitivity();
on_refresh();
}
~SenderWindow() override {
if (worker_.joinable()) {
worker_.join();
}
}
private:
int current_bitrate() const {
return static_cast<int>(bitrate_scale_->get_value());
}
int current_crf() const {
// CRF from the preset, adjusted by the bitrate slider's distance
// from the preset's default: moving the slider up from the preset
// means the user wants more headroom, so we keep the preset's CRF
// (the slider fine-tunes the cap, not the quality target).
const auto index = quality_combo_->get_selected();
if (index >= quality_presets().size()) {
return 22;
}
return quality_presets()[index].crf;
}
void on_preset_changed() {
const auto index = quality_combo_->get_selected();
if (index >= quality_presets().size()) {
return;
}
const QualityPreset& preset = quality_presets()[index];
bitrate_scale_->set_value(static_cast<double>(preset.bitrate_kbps));
fps_spin_->set_value(preset.fps);
update_quality_labels();
}
void update_quality_labels() {
bitrate_label_->set_text(std::format("Max bitrate: {} kbps · CRF {}", current_bitrate(), current_crf()));
}
int current_fps() const {
return static_cast<int>(fps_spin_->get_value());
}
void clear_receiver_rows() {
for (Gtk::Widget* row : receiver_rows_) {
receiver_list_->remove(*row);
}
receiver_rows_.clear();
receivers_.clear();
}
void on_refresh() {
refresh_button_->set_sensitive(false);
status("discovering receivers…");
clear_receiver_rows();
worker_ = std::jthread([this](std::stop_token stop_token) {
std::vector<DiscoveredPeer> peers;
std::mutex mutex;
std::string error;
auto discovery_result = sc::DiscoveryFactory::create_avahi();
if (sc::is_network_error(discovery_result)) {
error = sc::network_error(discovery_result).message;
} else {
auto discovery = std::move(sc::network_value(discovery_result));
if (!discovery->browse([&](const DiscoveredPeer& peer) {
std::lock_guard lock(mutex);
const bool known = std::any_of(peers.begin(), peers.end(), [&](const DiscoveredPeer& existing) {
return existing.service_name == peer.service_name &&
existing.signaling_port == peer.signaling_port;
});
if (!known) {
peers.push_back(peer);
}
})) {
error = discovery->last_error();
}
if (!stop_token.stop_requested()) {
std::this_thread::sleep_for(std::chrono::seconds(3));
}
discovery->stop();
}
Glib::signal_idle().connect_once([this, peers = std::move(peers), error = std::move(error)]() mutable {
receivers_ = group_receivers(std::move(peers));
for (const ReceiverRow& row : receivers_) {
auto* label = Gtk::make_managed<Gtk::Label>();
label->set_text(std::format("{}\n{}", row.name, row.host));
label->set_halign(Gtk::Align::START);
receiver_list_->append(*label);
receiver_rows_.push_back(label);
}
refresh_button_->set_sensitive(true);
update_sensitivity();
status(error.empty() ? (receivers_.empty() ? "no receivers found"
: std::format("{} receiver(s)", receivers_.size()))
: "discovery failed: " + error);
});
});
}
static std::vector<ReceiverRow> group_receivers(std::vector<DiscoveredPeer> peers) {
std::map<std::pair<std::string, std::uint16_t>, std::vector<std::string>> grouped;
for (DiscoveredPeer& peer : peers) {
grouped[{peer.service_name, peer.signaling_port}].push_back(std::move(peer.host));
}
std::vector<ReceiverRow> rows;
for (auto& [key, hosts] : grouped) {
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return sc::address_preference(lhs) < sc::address_preference(rhs);
});
ReceiverRow row;
row.name = key.first;
row.host = std::move(hosts.front());
row.signaling_port = key.second;
rows.push_back(std::move(row));
}
return rows;
}
void on_start_stop() {
if (session_.has_value()) {
auto session = std::move(*session_);
session_ = std::nullopt;
start_button_->set_sensitive(false);
status("stopping…");
worker_ = std::jthread([this, session = std::move(session)](std::stop_token) mutable {
session.stop();
Glib::signal_idle().connect_once([this] {
start_button_->set_label("Start");
update_sensitivity();
status("idle");
});
});
return;
}
const int index = selected_index();
if (index < 0) {
status("pick a receiver first");
return;
}
const ReceiverRow& receiver = receivers_[static_cast<std::size_t>(index)];
const Endpoint signaling{receiver.host, receiver.signaling_port};
const int bitrate = current_bitrate();
const int crf = current_crf();
const int fps = current_fps();
start_button_->set_sensitive(false);
status(std::format("connecting to {}… (choose a source in the portal dialog)", receiver.name));
worker_ = std::jthread([this, signaling, bitrate, crf, fps](std::stop_token) {
auto result = sc::SenderSession::start(signaling, bitrate, crf, fps, sc::CaptureTargetWholeScreen{});
if (auto* error = std::get_if<std::string>(&result)) {
Glib::signal_idle().connect_once([this, message = *error] {
start_button_->set_label("Start");
update_sensitivity();
status("failed: " + message);
});
return;
}
// sigc++ slots require copyable lambdas; the move-only session
// travels via shared_ptr.
auto session = std::make_shared<sc::SenderSession>(std::move(std::get<sc::SenderSession>(result)));
Glib::signal_idle().connect_once([this, session] {
const std::string receiver_text = session->receiver();
const std::string session_id = session->session_id();
session_ = std::move(*session);
start_button_->set_label("Stop");
update_sensitivity();
status(std::format("streaming to {} (session {})", receiver_text, session_id));
});
});
}
int selected_index() const {
Gtk::ListBoxRow* row = receiver_list_->get_selected_row();
return row != nullptr ? row->get_index() : -1;
}
void update_sensitivity() {
if (session_.has_value()) {
start_button_->set_sensitive(true);
refresh_button_->set_sensitive(false);
return;
}
refresh_button_->set_sensitive(true);
start_button_->set_sensitive(selected_index() >= 0);
}
void update_status() {
if (!session_.has_value()) {
return;
}
const auto state = sc::read_sender_state();
if (state.has_value()) {
const std::int64_t elapsed_ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::system_clock::now().time_since_epoch())
.count() -
state->started_epoch_ms;
status(std::format("streaming to {}\nsession {}\nelapsed {}m {:02}s",
state->receiver,
state->session_id,
elapsed_ms / 60000,
(elapsed_ms / 1000) % 60));
}
}
void status(const std::string& text) {
status_label_->set_text(text);
}
std::vector<ReceiverRow> receivers_;
std::vector<Gtk::Widget*> receiver_rows_;
std::optional<sc::SenderSession> session_;
std::jthread worker_;
Gtk::Button* refresh_button_ = nullptr;
Gtk::ScrolledWindow* scrolled_ = nullptr;
Gtk::ListBox* receiver_list_ = nullptr;
Gtk::DropDown* quality_combo_ = nullptr;
Gtk::Scale* bitrate_scale_ = nullptr;
Gtk::SpinButton* fps_spin_ = nullptr;
Gtk::Label* bitrate_label_ = nullptr;
Gtk::Button* start_button_ = nullptr;
Gtk::Label* status_label_ = nullptr;
};
} // namespace
int main(int argc, char** argv) {
auto app = Gtk::Application::create("io.github.screen_cast.panel");
return app->make_window_and_run<SenderWindow>(argc, argv);
}
+15
View File
@@ -0,0 +1,15 @@
# GTK4 sender panel, behind the `gui` option (default: false). The core
# CLI and tests never need GTK.
dep_gtkmm = dependency('gtkmm-4.0')
screencast_gui_sources = files(
'gui.cpp',
)
executable('screencast-gui',
screencast_gui_sources,
include_directories : [sc_core_inc, include_directories('../app')],
dependencies : [dep_gtkmm, sc_app_core_dep, sc_capture_dep, sc_codec_dep, sc_network_dep,
sc_render_dep],
install : true)
+14
View File
@@ -1,6 +1,16 @@
# Core public headers / include dependency.
sc_core_inc = include_directories('../include')
# Embed the application icon so the GUI window and the SDL window icon work
# from the build tree, after install, and on headless targets without any
# runtime path lookup.
icon_to_header = find_program('../scripts/icon_to_header.py')
icon_png_data = custom_target('icon_png_data',
input : files('../screencast_icon/screencast_256.png'),
output : 'icon_png_data.h',
command : [icon_to_header, '@INPUT@', '@OUTPUT@'])
sc_icon_dep = declare_dependency(sources : [icon_png_data])
# The sender needs the PipeWire / xdg-desktop-portal capture backend;
# receiver-only builds skip it entirely.
build_sender = get_option('sender')
@@ -32,3 +42,7 @@ subdir('network')
subdir('render')
subdir('app')
if get_option('gui')
subdir('gui')
endif
+9 -4
View File
@@ -139,7 +139,9 @@ void H264Packetizer::append_fu_a_packets(std::span<const std::byte> nal,
}
}
std::optional<std::vector<std::byte>> H264Depacketizer::depacketize(const RtpPacket& packet) {
DepacketizeResult H264Depacketizer::depacketize(const RtpPacket& packet) {
DepacketizeResult result;
// Track sequence continuity: a gap means packets were lost.
if (last_sequence_number_.has_value()) {
const std::uint16_t expected = static_cast<std::uint16_t>(*last_sequence_number_ + 1);
@@ -157,6 +159,7 @@ std::optional<std::vector<std::byte>> H264Depacketizer::depacketize(const RtpPac
// lost its tail and can no longer be recovered.
if (frame_started_ && packet.header.timestamp != frame_timestamp_) {
drop_frame();
result.frame_dropped = true;
}
if (!frame_started_) {
frame_started_ = true;
@@ -216,7 +219,7 @@ std::optional<std::vector<std::byte>> H264Depacketizer::depacketize(const RtpPac
}
if (!packet.header.marker) {
return std::nullopt;
return result;
}
if (fu_active_) {
@@ -226,9 +229,11 @@ std::optional<std::vector<std::byte>> H264Depacketizer::depacketize(const RtpPac
fu_nal_.clear();
}
std::optional<std::vector<std::byte>> result;
if (!frame_damaged_ && !access_unit_.empty()) {
result = std::move(access_unit_);
result.access_unit = std::move(access_unit_);
} else {
// The frame that just ended is unusable.
result.frame_dropped = true;
}
drop_frame();
return result;
+55
View File
@@ -133,4 +133,59 @@ std::optional<RtpPacket> RtpPacket::parse(std::span<const std::byte> in) noexcep
return packet;
}
RtpJitterBuffer::RtpJitterBuffer(std::size_t max_depth, std::chrono::milliseconds max_delay)
: max_depth_(max_depth), max_delay_(max_delay) {}
std::vector<RtpPacket> RtpJitterBuffer::push(RtpPacket packet) {
std::vector<RtpPacket> released;
const std::uint16_t sequence = packet.header.sequence_number;
const auto now = std::chrono::steady_clock::now();
std::lock_guard lock(mutex_);
if (!next_expected_.has_value()) {
next_expected_ = sequence;
}
// Serial-number comparison: a difference >= 32768 means the packet is
// older than what we already delivered (a duplicate or a late straggler).
const std::uint16_t distance = static_cast<std::uint16_t>(sequence - *next_expected_);
if (distance >= 32768) {
return released; // discard the straggler
}
buffer_[sequence] = {now, std::move(packet)};
// Release the consecutive run from the expected sequence.
while (true) {
const auto entry = buffer_.find(*next_expected_);
if (entry == buffer_.end()) {
break;
}
released.push_back(std::move(entry->second.second));
buffer_.erase(entry);
++(*next_expected_);
}
// A missing packet stalls the run: age out the backlog (or bound the
// buffer) and release what is there in order, so genuine loss reaches
// the depacketizer's gap detection rather than blocking forever.
if (!buffer_.empty()) {
const auto head_age = now - buffer_.begin()->second.first;
if (head_age > max_delay_ || buffer_.size() > max_depth_) {
for (auto& entry : buffer_) {
released.push_back(std::move(entry.second.second));
}
next_expected_ = static_cast<std::uint16_t>(buffer_.rbegin()->first + 1);
buffer_.clear();
}
}
return released;
}
void RtpJitterBuffer::clear() {
std::lock_guard lock(mutex_);
buffer_.clear();
next_expected_.reset();
}
} // namespace sc
+21 -5
View File
@@ -115,12 +115,17 @@ std::string serialize_message(const SignalingMessage& message) {
json["frame_rate_den"] = offer->frame_rate_den;
json["rtp_address"] = offer->rtp_endpoint.address;
json["rtp_port"] = offer->rtp_endpoint.port;
} else {
const SessionAnswer& answer = std::get<SessionAnswer>(message);
} else if (const SessionAnswer* answer = std::get_if<SessionAnswer>(&message)) {
json["type"] = "answer";
json["session_id"] = answer.session_id;
json["rtp_address"] = answer.rtp_endpoint.address;
json["rtp_port"] = answer.rtp_endpoint.port;
json["session_id"] = answer->session_id;
json["rtp_address"] = answer->rtp_endpoint.address;
json["rtp_port"] = answer->rtp_endpoint.port;
json["display_width"] = answer->display_width;
json["display_height"] = answer->display_height;
} else {
const SessionPli& pli = std::get<SessionPli>(message);
json["type"] = "pli";
json["session_id"] = pli.session_id;
}
return json.dump() + "\n";
}
@@ -169,8 +174,19 @@ std::optional<SignalingMessage> parse_message(std::string_view line) {
SessionAnswer answer;
answer.session_id = session_id;
answer.rtp_endpoint = rtp_endpoint;
if (json.contains("display_width") && json.at("display_width").is_number_integer()) {
answer.display_width = json.at("display_width").get<int>();
}
if (json.contains("display_height") && json.at("display_height").is_number_integer()) {
answer.display_height = json.at("display_height").get<int>();
}
return answer;
}
if (type == "pli") {
SessionPli pli;
pli.session_id = session_id;
return pli;
}
return std::nullopt;
}
+18 -2
View File
@@ -2,14 +2,30 @@
dep_sdl3 = dependency('sdl3')
# SDL3_image (window icon) is optional: no pkg-config file ships with it,
# so find the library directly. Builds without it (e.g. the Pi receiver)
# simply skip the window icon.
cc = meson.get_compiler('cpp')
dep_sdl3_image = cc.find_library('SDL3_image', required: false)
if not dep_sdl3_image.found()
dep_sdl3_image = disabler()
endif
sc_render_args = []
if dep_sdl3_image.found() and cc.has_header('SDL3_image/SDL_image.h', dependencies : dep_sdl3_image)
sc_render_args += ['-DSC_HAS_WINDOW_ICON=1']
endif
sc_render_sources = files('sdl_renderer.cpp')
sc_render = static_library('sc_render',
sc_render_sources,
include_directories : sc_core_inc,
dependencies : [dep_sdl3])
cpp_args : sc_render_args,
dependencies : [dep_sdl3, dep_sdl3_image, sc_icon_dep])
sc_render_dep = declare_dependency(
link_with : sc_render,
include_directories : sc_core_inc,
dependencies : [dep_sdl3])
compile_args : sc_render_args,
dependencies : [dep_sdl3, dep_sdl3_image])
+68 -7
View File
@@ -4,13 +4,23 @@
#include <string>
#ifdef SC_HAS_WINDOW_ICON
#include <SDL3_image/SDL_image.h>
#include <cstdio>
#include "icon_png_data.h"
#endif
namespace sc {
namespace {
// DecodedFrame pixels are AV_PIX_FMT_RGBA: memory order R, G, B, A. SDL
// names 32-bit formats MSB-first, so that byte order is SDL's ABGR8888 —
// using RGBA8888 would read the alpha byte as red and tint the image red.
constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_ABGR8888;
// DecodedFrame carries YUV420P planar data; SDL_PIXELFORMAT_IYUV is the
// matching SDL texture format. The GPU does the YUV→RGB conversion during
// rendering, eliminating a CPU-side swscale pass.
constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_IYUV;
class SdlRenderer final : public Renderer {
public:
@@ -52,6 +62,10 @@ class SdlRenderer final : public Renderer {
(void)SDL_HideCursor();
}
#ifdef SC_HAS_WINDOW_ICON
set_window_icon();
#endif
renderer_ = SDL_CreateRenderer(window_, nullptr);
if (renderer_ == nullptr) {
last_error_ = std::string{"SDL_CreateRenderer failed: "} + SDL_GetError();
@@ -59,6 +73,14 @@ class SdlRenderer final : public Renderer {
return false;
}
// Cache the display resolution: in fullscreen (KMSDRM on the Pi) this
// is the native monitor size; in windowed mode it is the window.
int window_width = 0;
int window_height = 0;
SDL_GetWindowSize(window_, &window_width, &window_height);
display_width_ = window_width;
display_height_ = window_height;
// Commit the surface once: on Wayland a window only becomes visible
// after the first present, and the receiver must be visible while it
// waits for the stream to start.
@@ -73,8 +95,16 @@ class SdlRenderer final : public Renderer {
return true;
}
int display_width() const override {
return display_width_;
}
int display_height() const override {
return display_height_;
}
bool present(const DecodedFrame& frame) override {
if (frame.width <= 0 || frame.height <= 0) {
if (frame.width <= 0 || frame.height <= 0 || frame.plane_y.empty()) {
return false;
}
if (frame.width != texture_width_ || frame.height != texture_height_) {
@@ -83,8 +113,14 @@ class SdlRenderer final : public Renderer {
}
}
const int pitch = frame.width * 4;
if (!SDL_UpdateTexture(texture_, nullptr, frame.rgba_pixels.data(), pitch)) {
if (!SDL_UpdateYUVTexture(texture_,
nullptr,
reinterpret_cast<const uint8_t*>(frame.plane_y.data()),
frame.stride_y,
reinterpret_cast<const uint8_t*>(frame.plane_u.data()),
frame.stride_u,
reinterpret_cast<const uint8_t*>(frame.plane_v.data()),
frame.stride_v)) {
return false;
}
// Clear first so the letterbox bars stay black between frames.
@@ -133,6 +169,29 @@ class SdlRenderer final : public Renderer {
}
private:
#ifdef SC_HAS_WINDOW_ICON
// The icon is embedded at build time (see icon_png_data.h), so it works
// without any runtime file lookup. Harmless under KMSDRM, where there is
// no window manager to display it.
void set_window_icon() {
auto* stream = SDL_IOFromConstMem(sc::app_icon_png.data(), sc::app_icon_png.size());
if (stream == nullptr) {
std::fprintf(stderr, "screencast: failed to create icon stream: %s\n", SDL_GetError());
return;
}
// closeio=true: SDL_image consumes the stream, success or failure.
SDL_Surface* icon_surface = IMG_Load_IO(stream, true);
if (icon_surface == nullptr) {
std::fprintf(stderr, "screencast: failed to load window icon: %s\n", SDL_GetError());
return;
}
if (SDL_SetWindowIcon(window_, icon_surface) != 0) {
std::fprintf(stderr, "screencast: failed to set window icon: %s\n", SDL_GetError());
}
SDL_DestroySurface(icon_surface);
}
#endif
bool recreate_texture(int width, int height) {
if (texture_ != nullptr) {
SDL_DestroyTexture(texture_);
@@ -164,6 +223,8 @@ class SdlRenderer final : public Renderer {
SDL_Texture* texture_ = nullptr;
int texture_width_ = 0;
int texture_height_ = 0;
int display_width_ = 0;
int display_height_ = 0;
};
} // namespace
+4 -4
View File
@@ -74,13 +74,13 @@ struct ReceiverSink {
int last_height = 0;
void on_packet(sc::RtpPacket packet) {
std::optional<std::vector<std::byte>> access_unit = depacketizer.depacketize(packet);
if (!access_unit.has_value()) {
const sc::DepacketizeResult result = depacketizer.depacketize(packet);
if (!result.access_unit.has_value()) {
return;
}
sc::EncodedFrame encoded;
encoded.data = std::move(*access_unit);
encoded.data = std::move(*result.access_unit);
encoded.rtp_timestamp = packet.header.timestamp;
auto decoded_result = decoder->decode(encoded);
@@ -90,7 +90,7 @@ struct ReceiverSink {
for (const sc::DecodedFrame& decoded : sc::codec_value(decoded_result)) {
last_width = decoded.width;
last_height = decoded.height;
saw_frame = decoded.width > 0 && !decoded.rgba_pixels.empty();
saw_frame = decoded.width > 0 && !decoded.plane_y.empty();
decoded_frames.fetch_add(1);
}
}
+43 -3
View File
@@ -1,7 +1,12 @@
#include "screencast/codec/decoder.h"
#include "screencast/codec/encoder.h"
#include <algorithm>
// swscale is a C library; without the extern wrapper its functions get
// C++ mangled and the linker cannot find them.
extern "C" {
#include <libswscale/swscale.h>
}
#include <cassert>
#include <cstdint>
#include <cstdlib>
@@ -47,10 +52,45 @@ sc::CapturedFrame make_frame(uint32_t index) {
return frame;
}
// Convert a decoded YUV420P frame back to RGBA for pixel comparison with
// the original capture. Test-only; the pipeline itself never converts.
std::vector<std::byte> decoded_to_rgba(const sc::DecodedFrame& decoded) {
const std::size_t rgba_size = static_cast<std::size_t>(decoded.width) * decoded.height * 4;
std::vector<std::byte> rgba(rgba_size);
const uint8_t* src_planes[4] = {
reinterpret_cast<const uint8_t*>(decoded.plane_y.data()),
reinterpret_cast<const uint8_t*>(decoded.plane_u.data()),
reinterpret_cast<const uint8_t*>(decoded.plane_v.data()),
nullptr,
};
const int src_strides[4] = {decoded.stride_y, decoded.stride_u, decoded.stride_v, 0};
uint8_t* dst_planes[4] = {reinterpret_cast<uint8_t*>(rgba.data()), nullptr, nullptr, nullptr};
const int dst_strides[4] = {decoded.width * 4, 0, 0, 0};
SwsContext* scaler = sws_getContext(decoded.width,
decoded.height,
AV_PIX_FMT_YUV420P,
decoded.width,
decoded.height,
AV_PIX_FMT_RGBA,
SWS_BILINEAR,
nullptr,
nullptr,
nullptr);
assert(scaler != nullptr);
(void)sws_scale(scaler, src_planes, src_strides, 0, decoded.height, dst_planes, dst_strides);
sws_freeContext(scaler);
return rgba;
}
int max_channel_difference(const sc::DecodedFrame& decoded, const sc::CapturedFrame& expected) {
const auto* decoded_pixels = reinterpret_cast<const std::uint8_t*>(decoded.rgba_pixels.data());
const auto decoded_rgba = decoded_to_rgba(decoded);
const auto* decoded_pixels = reinterpret_cast<const std::uint8_t*>(decoded_rgba.data());
const auto* expected_pixels = reinterpret_cast<const std::uint8_t*>(expected.pixels.data());
const std::size_t count = std::min(decoded.rgba_pixels.size(), expected.pixels.size());
const std::size_t count = std::min(decoded_rgba.size(), expected.pixels.size());
int max_diff = 0;
for (std::size_t i = 0; i < count; ++i) {

Some files were not shown because too many files have changed in this diff Show More