Compare commits

...

17 Commits

Author SHA1 Message Date
fegger c43dd3ca4a feat(ios): native receiver app (Swift, min iOS 17)
A native iOS receiver so an iPhone can act as the second receiver, speaking the existing signaling + RTP protocol (no C++ changes) and mirroring the Android receiver (Phase 8) source-to-source.

- RTP core (header/packet, jitter buffer, H.264 depacketizer) ported from the Android receiver
- BSD-socket signaling server (dual-stack, most-recent-peer, never-throwing sends) + NSBonjourServices
- VideoToolbox H.264 decode (in-band SPS/PPS, real-time, rebuilds on size change) -> AVSampleBufferDisplayLayer
- PLI keyframe recovery (500 ms) + pendingOffer for late surface attach
- XcodeGen project + bootstrap.sh; XCTest port of the Android suite + new coverage
- .gitignore for generated artifacts; CHANGELOG; PHASES + MEMORY updated

Status: authored; on-device validation pending a Mac + Xcode 26 + iPhone 16.
2026-09-10 21:31:53 +02:00
fegger 4f93c7cd20 fix(android): don't letterbox against the configure placeholder
H264Decoder.outputSize() reported the 320x240 configure() placeholder
until the codec parsed the in-band SPS, so fitVideo sized the view to
1488x1116 (1.33 aspect) and the first rendered frame(s) of 2.4 content
were visibly squished before the real size corrected it ~40ms later.
Report no size until INFO_OUTPUT_FORMAT_CHANGED delivers the real one:
the first frame renders into the fullscreen surface and the correct
letterbox follows immediately. Validated live on the Fairphone 6 (the
placeholder 'video size' line is gone from logcat; 25/25 JVM tests).
2026-09-10 12:36:15 +02:00
fegger 9a24933203 fix(android): pass PROTOCOL_DNS_SD so mDNS registration works
Android 16 added NsdManager.checkProtocol(): registerService with protocol
0 threw IllegalArgumentException: Unsupported protocol — silently, since
the old code swallowed the exception into a dead reflection fallback and
start() overwrote the failure status with the Listening line. mDNS never
actually advertised (the Phase 8 session streamed via --peer, masking it).

Also advertise after the listening status so a registration failure stays
visible, and log the exception and the registration success.

Validated live on the Fairphone 6: the desktop --discover lists the phone
and a full --send session decodes (in-band SPS 320x240 -> 2496x1040) and
renders.
2026-09-10 12:35:00 +02:00
fegger c6722d164b fix(android): harden the receiver after the review pass
- SignalingServer.send() never throws: a broken signaling connection
  drops the peer instead of killing the RTP reader thread via requestPli()
- H264Decoder: assign the codec before configure/start (no orphaned
  instance on failure); feed() reports input-queue timeouts so the
  pipeline requests a keyframe instead of dropping the frame silently
- Park offers that arrive without a surface as pendingOffer and configure
  on attachSurface(): a codec configured without a surface can never take
  one (setOutputSurface refuses it); the late configure sends a PLI
- Forget destroyed surfaces (onSurfaceTextureDestroyed -> detachSurface)
- Show the status overlay again on later messages
- Accumulate NAL bytes in ByteArrayOutputStreams, not boxed ArrayList<Int>
- Delete the dead NSD reflection fallback (ResolutionListener never
  existed; the classic API is present from API 16 through 36)
- Hold decoderLock around all MediaCodec calls (not thread-safe) and make
  requestPli thread-safe
- Update RUNBOOK quirks (NSD, setOutputSurface) and MEMORY notes
2026-09-10 12:19:21 +02:00
fegger bbe4f21a3b docs: document the Android receiver app (Phase 8)
README: phone-as-second-screen section (build, install, sender usage,
multi-monitor caveat). RUNBOOK: Android build/test procedure, device-test
commands, and the API-36 platform quirks found during validation.
PHASES: Phase 8 milestones, all validated. MEMORY: current phase state
and the hard-won Android/portal lessons.
2026-09-10 11:38:16 +02:00
fegger 455ba6055d feat(android): native receiver app (Kotlin, minSdk 30)
Phone-as-receiver for the USB-C -> HDMI mirroring use case. Speaks the
existing signaling + RTP wire protocol, so the C++ sender needs no
changes. AGP 9 built-in Kotlin (no separate plugin), no androidx.

- rtp/: RtpHeader, RtpPacket, JitterBuffer (16 pkt / 60 ms, straggler
  discard), H264Depacketizer (single-NAL + FU-A, Annex-B out) — JVM tests
- signaling/: newline-JSON server (offer -> answer, PLI, 500 ms rate cap)
- decode/H264Decoder: MediaCodec -> Surface; size from in-band SPS (offer
  is 0x0); surface passed to configure(); releaseOutputBuffer(render)
- pipeline/ReceiverPipeline: NSD advertise (API-36 RegistrationListener
  + reflection fallback), UDP RTP, jitter, decode, PLI recovery
- ReceiverActivity: fullscreen TextureView letterboxed by sizing the
  view to the video aspect (C2 scales output to the surface)

Validated end-to-end on a Fairphone 6 (API 36): offer/answer, first
frame, letterbox fit, PLI on loss.
2026-09-10 11:31:13 +02:00
fegger 69a1cdac1d feat(app): wire the app icon into the GUI, waybar, and SDL window
screencast_icon/ holds the single source icon set; meson install ships
the 256 px PNG into the hicolor theme for the GTK panel (themed icon
name) and the waybar CSS, and a custom target embeds it as
icon_png_data.h so the SDL receiver window sets it via SDL_SetWindowIcon
with an optional SDL3_image dependency (built-in find_library, since no
pkg-config file ships with it).
2026-09-10 11:30:39 +02:00
fegger 9ff8e94679 docs(readme): document all modes, GUI, resilience, and hotspot 2026-09-09 22:25:22 +02:00
fegger 9024930d91 chore(repo): ignore local hotspot.txt notes file 2026-09-09 22:25:16 +02:00
fegger 144aab6e5b feat(gui): recalibrate presets and add a frame rate spinner
The presets now include a frame rate cap alongside CRF and max
bitrate, reflecting three lessons learned since the CRF fix:

- Frame rate is the biggest bandwidth lever for desktop content:
  15-25 fps is perfectly smooth for screencasting and cuts bandwidth
  2-4x vs. uncapped monitor rate.
- CRF values are now actually applied (the global_quality bug is
  fixed), so the ranges are meaningful.
- The 'Low bandwidth' preset targets 15 fps + CRF 26 + 3000 kbps,
  which fits even the Pi's Wi-Fi hotspot; 'Maximum' uncaps the frame
  rate for the smoothest possible output on a good link.

  Low bandwidth   CRF 26,  3000 kbps, 15 fps
  Balanced        CRF 22,  5000 kbps, 20 fps
  Sharp           CRF 18,  8000 kbps, 25 fps
  Very sharp      CRF 16, 12000 kbps, 30 fps
  Maximum         CRF 14, 20000 kbps, uncapped

A frame rate spinner (5-60, 0 = uncapped) sits alongside the
bitrate slider for independent fine-tuning.
2026-09-09 21:45:25 +02:00
fegger 8f1c2aa868 fix(codec): set CRF via x264's private option, not global_quality
The CRF value was set through AVCodecContext.global_quality with
AV_CODEC_FLAG_QSCALE, which FFmpeg's libx264 wrapper divides by
FF_QP2LAMBDA (118) before passing to x264 — turning CRF 16 into
CRF 0.135 (essentially lossless) while x264 logged '-qscale is
ignored, -crf is recommended' and fell back to its own defaults.
The CRF was never actually applied.

Now the CRF is set as x264's private "crf" option via av_opt_set,
which passes the exact value directly to the encoder. The VBV max
rate still caps bursts as before.
2026-09-09 13:28:09 +02:00
fegger e82e7853d1 feat(app): adaptive quality, frame rate capping, and tighter VBV
Three changes to make the stream survive constrained links:

Adaptive quality: the sender pipeline now tracks the PLI rate from
the receiver. Every 5 seconds it evaluates: >0.5 PLI/s means the
link is saturated (the receiver is dropping frames), so the CRF
increases by 2 (lower quality, fewer bits) and the encoder restarts
with a keyframe. <0.1 PLI/s means the link is stable, so the CRF
decreases by 1 (better quality) and the encoder probes upward.
Clamped to [user CRF, user CRF + 10] so quality never degrades
below what the link can handle, and never exceeds what the user
asked for. The adaptation is logged to stderr for visibility.

Frame rate capping (--fps N): throttles the capture loop to N
frames per second (0 = no cap; monitor rate). At 15fps instead of
60fps, the bandwidth requirement drops 4x at the same quality
level. Desktop content is still smooth at 15-20fps.

Tighter VBV: one frame period of buffer instead of two. A two-frame
buffer lets a keyframe spike to twice the target rate in one burst,
which overflows any constrained hop (Wi-Fi hotspot, slow switch)
and cascades into PLI storms. One frame period keeps bursts
within what the link can absorb in real time.
2026-09-09 13:00:03 +02:00
fegger c78dec139b feat(gui): add a quality preset dropdown to the sender panel
Replaces the raw bitrate slider with a preset dropdown that maps
directly to common CLI invocations:

  Low bandwidth  CRF 28, max  2000 kbps   (screencast --send)
  Standard        CRF 22, max  4000 kbps   (screencast --send)
  Sharp           CRF 18, max  8000 kbps   (--crf 18 --bitrate 8000)
  Very sharp      CRF 16, max 12000 kbps   (--crf 16 --bitrate 12000)
  Maximum         CRF 14, max 20000 kbps   (--crf 14 --bitrate 20000)

The bitrate slider remains for fine-tuning the cap independently of
the preset's quality target (CRF). Selecting a preset sets both; the
label shows the current values.
2026-09-09 12:45:14 +02:00
fegger 36d086af4e perf(codec): CRF rate control, longer GOP, faster preset, screen tuning
Four encoder quality improvements, all sender-side:

- CRF rate control (default 22, --crf to override): targets a
  constant visual quality level instead of a fixed bitrate. Static
  desktop content uses 300-800 kbps (vs. forced 4000+), and the saved
  bits go to sharp text and clean motion when they appear. The VBV
  max rate (the --bitrate value, now a cap rather than a target)
  bounds bursts so the receiver's UDP buffers stay safe. Round-trip
  test bitrate dropped from 1390 kb/s to 47 kb/s on synthetic frames
  — the encoder uses only what it needs.

- 5-second GOP (was 1 second): 80% fewer keyframe bits freed for
  detail frames. Screen content changes incrementally, not
  wholesale; PLI feedback recovers from loss in one frame time
  regardless of GOP length.

- faster preset (was veryfast): better sub-pixel estimation and
  RDO on more decisions. The desktop handles it trivially at 1080p.

- Screen-content x264 tuning: aq-mode=2 (auto-variance AQ moves
  bits away from flat areas toward text edges) and psy-rd=1.5
  (preserves texture sharpness).

Combined with the earlier veryfast upgrade and sender-side
downscaling, this is roughly 2x the perceived quality at the same
average bandwidth compared to the original ultrafast ABR encoder.

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 12:12:36 +02:00
fegger 30538fba73 perf(codec): upgrade x264 preset and downscale to the receiver display
Two quality improvements:

Preset: ultrafast -> veryfast. Unlocks Main profile with CABAC
entropy coding, hexagonal motion search, 3 reference frames, and
adaptive quantization — typically 30-40% better quality at the same
bitrate. The desktop handles the extra encoding cost trivially
(150+ fps at 1080p).

Downscaling: the receiver now advertises its display resolution in
the signaling answer (display_width/display_height, 0 = unknown).
When the capture exceeds the display (e.g. 2256x1504 source on a
1920x1080 receiver), the sender scales down preserving aspect ratio
before encoding — the same sws_scale pass that already converts the
pixel format also handles the resolution change, so there is no
extra step. This concentrates the entire bitrate into pixels the
display actually shows (~2.7x more bits per visible pixel at
4000 kbps when going from 2256x1504 to 1620x1080).

The renderer caches the display size during window creation
(native monitor resolution in fullscreen/KMSDRM; window size
otherwise). The receiver includes it in every signaling answer; the
sender pipeline computes aspect-preserving, even-rounded scaled
dimensions when the display is smaller than the capture.

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 11:02:42 +02:00
fegger 6516b45b02 perf(codec): pass YUV through to the renderer and use slice threading
The receiver decoded H.264 to YUV420P, converted it to RGBA via a
CPU-intensive swscale pass, then uploaded 4 bytes/pixel to an SDL
texture — only for the GPU to convert back to RGB during rendering.
This eliminated the swscale pass entirely (40-60% of receiver CPU at
1080p) and cut the texture upload by 62%.

- DecodedFrame now carries three YUV420P planes with their strides
  instead of a packed RGBA buffer; the decoder copies the planes
  directly from the AVFrame (zero conversion for the common software
  path). Non-YUV420P decoder output (e.g. NV12 from v4l2m2m) is
  converted once to YUV420P.
- The SDL renderer uploads via SDL_UpdateYUVTexture with
  SDL_PIXELFORMAT_IYUV; the GPU does the YUV→RGB conversion during
  rendering.
- Decoder threading: slice-level with 4 threads (parallelizes within a
  frame, no added latency), not frame-level (which buffers multiple
  frames — the initial thread_count=0 broke the loopback test because
  the H.264 decoder introduced a multi-frame delay before producing
  output).
- The round-trip test converts decoded YUV back to RGBA for pixel
  comparison via a test-local swscale call (the pipeline itself never
  converts).

meson test 5/5 in both configurations, valgrind clean.
2026-09-09 09:45:22 +02:00
fegger b4d1411fd9 fix(build): make receiver-only builds link again after the GUI refactor
The sc_app_core static library was always compiled with SC_HAS_SENDER=1,
pulling CaptureFactory into receiver-only builds where the capture
backend does not exist. Three fixes:

- sc_app_core's cpp_args and capture dependency now follow the sender
  meson option, so pipelines.cpp and sender_session.cpp compile their
  sender code out on receiver-only targets.
- sender_session.cpp is fully guarded by SC_HAS_SENDER; its body is
  sender pipeline orchestration and has no business in a receiver.
- address_preference moved from a SenderSession static to a free inline
  function in sender_session.h — run_discover (available in every
  build) uses it for address sorting.

Also: the root meson.build now errors early if gui=true without
sender=true (the GUI is a sender panel).

Verified: meson test 5/5 in both configurations; the receiver-only
binary refuses --send with a clear message; the full build's sender,
GUI, and tests are unchanged.
2026-09-08 22:17:21 +02:00
93 changed files with 5023 additions and 126 deletions
+182 -2
View File
@@ -1,10 +1,82 @@
# Project Memory — screen_cast # Project Memory — screen_cast
Last updated: Phase 6 (discovery + signaling) complete and validated over Last updated: Phase 8 (Android receiver app) complete and validated on a
loopback; current phase is Phase 7. Fairphone 6; all prior phases done. The Android app's review findings were
fixed in a follow-up pass (same day) — see "Android app review" at the
bottom of this file.
## Project state ## Project state
- **Phase 9 in progress: iOS receiver app** (`ios/`, Swift, min iOS 17):
iPhone as a second receiver. Speaks the same signaling+RTP protocol — no C++
changes. Mirrors the Android receiver (Phase 8) source-to-source.
- Scaffolding: XcodeGen `project.yml` (Info.plist carries
`NSLocalNetworkUsageDescription` + `NSBonjourServices: _screencast._tcp`) +
`bootstrap.sh` (downloads XcodeGen from the GitHub release, no Homebrew;
generates the project; builds/tests via xcodebuild). `ios/README.md` has
build/install/device steps.
- App (`Receiver/App`): SwiftUI + `AVSampleBufferDisplayLayer` (`.resizeAspect`
= letterbox — the same "size the surface, not a transform" lesson as
Android); `ReceiverController` (ObservableObject) drives start/stop on
scenePhase.
- Protocol core (`Receiver/Rtp`, `Receiver/Signaling`): RtpHeader/RtpPacket/
JitterBuffer/H264Depacketizer ported source-to-source; SignalingMessage
(JSONSerialization) + LineAssembler + SignalingServer (BSD sockets,
dual-stack, most-recent-peer-wins, MSG_NOSIGNAL sends that never throw);
`AvccConverter` (Annex-B ↔ AVCC) + `NalExtractor` (SPS/PPS) are new for the
VideoToolbox path.
- Decode (`Receiver/Decode`): `H264FormatDescription` (Core Foundation H.264
config recipe) + `H264VideoToolboxDecoder` (in-band SPS/PPS → session;
`kVTDecompressionPropertyKey_RealTime`; session recreated on size change;
the output callback may run on a worker thread, so state is lock-guarded)
`AVSampleBufferRenderSink` (AVSampleBufferDisplayLayerSession).
- Support (`Receiver/Support`): `UdpTransport` (poll-based recv so close() can't
strand a blocked recvfrom) + `LocalAddress` (getifaddrs for the --peer hint).
- Pipeline (`Receiver/Pipeline`): ReceiverPipeline — one serial queue for all
state + decode; reader thread only polls/receives UDP; pendingOffer for late
surface attach (+ PLI); PLI rate-limited 500 ms; first-frame flag; video
size tracked (never reports a placeholder before the first real keyframe —
the Android startup-squish lesson).
- Tests (`ios/ReceiverTests`): the 25 Android JVM tests ported to XCTest plus
new coverage for signaling JSON, line framing, AVCC, NAL extraction.
- `.gitignore` excludes the generated `ios/Receiver.xcodeproj/`, `ios/tools/`,
`ios/Receiver/Info.plist` (the project.yml `info` block is the source of
truth).
- **NOT YET VALIDATED** (this box is Linux, no Xcode/iOS SDK): nothing here
compiles or runs. 9.5 needs a Mac with Xcode 26 + the iPhone 16.
Highest-risk on-device items: (1) local-network + Bonjour consent
(NSBonjourServices must be `_screencast._tcp`; iOS 26 tightened the prompt),
(2) the H264FormatDescription CF ownership recipe (a bug crashes on the
first keyframe — loud, not silent), (3) VideoToolbox decode →
AVSampleBufferDisplayLayer render. Run `ios/bootstrap.sh test` first on the
Mac.
- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30,
app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via
USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++
changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`;
Kotlin targets `compileOptions`, Java 17); no androidx (framework +
org.json + JUnit). 25 JVM tests green. Validated end-to-end on a
Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed
render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in
RUNBOOK): `INTERNET` permission is REQUIRED for NsdService;
`DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is
Inet6Address, not InetSocketAddress); pass the Surface to
`MediaCodec.configure()` + `start()`; render via
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
configure (in-band SPS reconfigures); `MediaFormat.format()`/
`KEY_MIME_TYPE` not public in API 36; NSD: the classic
`registerService(info, flags, RegistrationListener)` API exists from API
16 through 36 (javap-verified on the android-36 SDK) — an old reflection
fallback targeting a never-existent "ResolutionListener" was removed as
dead code;
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
by sizing the TextureView to the video aspect, NOT a transform matrix
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
picks the FIRST output (eDP-1, wrong display) — use `--target window`
(hyprland-share-picker). Activity FQN for `am start`:
`screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package =
namespace `screen_cast`).
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver - **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
announces its signaling port via a threaded-poll Avahi client; senders announces its signaling port via a threaded-poll Avahi client; senders
browse+resolve) plus JSON session signaling (offer/answer) over TCP with browse+resolve) plus JSON session signaling (offer/answer) over TCP with
@@ -65,6 +137,23 @@ loopback; current phase is Phase 7.
before the fix). Cross-compiling on the dev machine was considered and before the fix). Cross-compiling on the dev machine was considered and
dropped — the on-Pi build works and the toolchain/container effort was dropped — the on-Pi build works and the toolchain/container effort was
not needed. not needed.
- **App icons wired up** (`screencast_icon/`): single 256 px PNG used
everywhere. `meson install` ships it to
`share/icons/hicolor/256x256/apps/screencast.png`; the GTK panel sets it
via `set_icon_name("screencast")` (GTK4 removed the pixel-buffer window
icon — themed names only; `gtk_icon_theme_has_icon` verified OK); the
SDL receiver window embeds the PNG at build time (`icon_png_data.h` via
`scripts/icon_to_header.py` custom_target) and `SDL_SetWindowIcon` with
an optional SDL3_image dependency (no pkg-config ships with it →
`meson.get_compiler('cpp').find_library('SDL3_image', required: false)`
+ disabler; `-DSC_HAS_WINDOW_ICON` gate; Pi builds without it still
work). The waybar widget now shows the icon as `background-image` in
`~/.config/waybar/style.css` (▶/⏸ text hidden; `.idle` dimmed,
`.streaming` green tint) instead of text glyphs; backup
`style.css.bak-20260909-*`. Lesson: icons appear in compositor
taskbars/switchers, not title bars (GTK4 CSD and Hyprland decorations
don't draw them), so validate via theme lookup, not title-bar
screenshots. `sudo pacman -S sdl3_image` done (extra/sdl3_image 3.4.6).
- **GTK panel + waybar widget**: `screencast-gui` (gtkmm-4.0, behind - **GTK panel + waybar widget**: `screencast-gui` (gtkmm-4.0, behind
`-Dgui=true`; app internals now live in the `sc_app_core` static lib so `-Dgui=true`; app internals now live in the `sc_app_core` static lib so
CLI and GUI share pipelines/session/state). `screencast waybar [--toggle]` CLI and GUI share pipelines/session/state). `screencast waybar [--toggle]`
@@ -219,3 +308,94 @@ None.
surfacing the reason (logged to stderr). surfacing the reason (logged to stderr).
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders - Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
we control never emit them, but third-party interop would need support. we control never emit them, but third-party interop would need support.
### Android app review (2026-09-10) — findings FIXED same day
Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified
faithful to the C++ side source-to-source), followed by a fix pass that
landed all findings. No commit yet (user has not asked).
Fixed:
- `SignalingServer.send()` no longer throws (mirrors the C++ server, which
ignores write failures): a broken signaling TCP no longer kills the
`rtp-reader` thread via `requestPli()`; `peerOut` is dropped (closing the
socket) on write failure.
- `H264Decoder.configure()` assigns the created codec before configuring,
so a configure/start failure can no longer orphan the MediaCodec instance
(no finalizer; scarce native slots).
- Offer-before-surface no longer configures a ByteBuffer-mode decoder: the
offer is parked as `pendingOffer` and `attachSurface()` configures later
(a surface-less codec can never take one — `setOutputSurface` refuses
it, an IllegalStateException crash on the main thread). The late
configure sends a PLI (the sender only emits IDRs when asked).
- `ReceiverActivity.onSurfaceTextureDestroyed``pipeline.detachSurface()`;
the pipeline/decoder forget dead surfaces instead of configuring against
them (frames decode unrendered until the next attach).
- The status overlay reappears: `onStatus` sets `visibility = VISIBLE`
(it used to write into a GONE view after the first frame).
- `H264Decoder.feed()` returns false on input-queue timeout → the pipeline
requests a PLI instead of silently dropping the frame (no flush — the
codec is healthy).
- The depacketizer accumulates into `ByteArrayOutputStream`s instead of
boxed `ArrayList<Int>` (was ~MB/s of Integer allocations on keyframes).
- NSD reflection fallback deleted (dead code — see the quirk note above).
- Thread-safety: all codec calls now run under `decoderLock` (MediaCodec is
not thread-safe; feed/drain previously raced attachSurface);
`requestPli()` is thread-safe via `pliLock`.
Validation: `gradle :app:assembleDebug :app:testDebugUnitTest` green
(25/25, full --rerun-tasks rebuild, only two pre-existing warnings);
`meson test` 5/5 unchanged (no C++ touched).
### Discovery bug found and fixed on-device (2026-09-10, same day)
The user reported the desktop sender never discovered the phone.
Root cause (found live on the Fairphone 6): `registerService(info, 0,
listener)` — Android 16's `NsdManager.checkProtocol()` rejects protocol
`0` with `IllegalArgumentException: Unsupported protocol`. The old code
swallowed it into the dead reflection fallback, and `start()` posted the
"Listening…" status AFTER advertiseNsd, overwriting the failure text — so
mDNS never advertised and Phase 8's NSD validation was only ever "no
crash" (8.5 streamed via --peer, masking it).
Fixes in `ReceiverPipeline.advertiseNsd()`/`start()`:
- pass `NsdManager.PROTOCOL_DNS_SD`;
- advertise AFTER the listening status so a failure stays visible;
- `Log.e` the registration exception; `Log.i` on registered success.
On-device validation (adb, live): `mDNS registered:
screencast._screencast._tcp` in logcat; `dumpsys servicediscovery` shows
the active Advertiser (key diagnostic: `mClientRequests` empty == no
request ever issued); desktop `avahi-browse` and `screencast --discover`
list the phone at 192.168.178.29:5005; a 25s `--send --target monitor
--peer 192.168.178.29:5005` session decoded (in-band SPS reconfigured
320x240 → 2496x1040) and rendered (screencap mean brightness 0.51).
Note: with both the Pi and the phone on the LAN, plain `--send` refuses
(two receivers found) — target the phone with `--peer`.
Follow-up from the same on-device session — startup squish eliminated:
`H264Decoder.outputSize()` previously reported the 320x240 configure()
placeholder until the codec parsed the SPS, so `fitVideo` sized the
TextureView 1488x1116 (1.33 aspect) and the first rendered frame(s) of
2.4 content were visibly squished. It now returns null until
`INFO_OUTPUT_FORMAT_CHANGED` fires; the first frame renders into the
fullscreen surface and the correct letterbox (2484x1035) follows within
~40ms. Validated live: no placeholder "video size" line in logcat.
Commits: 9a24933 (PROTOCOL_DNS_SD discovery fix + docs), plus the
outputSize fix (see git log).
Still open (accepted, needs a device or a new test dep):
- Untested on device: `setOutputSurface` mid-session (surface switch) and
the whole pendingOffer path; API-35 `detachOutputSurface()` could replace
the render-flag approach.
- No JVM tests for `SignalingMessage`/`SignalingServer` (would need the
`org.json:json` test dependency; android.jar stubs throw).
- Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200),
RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a
future phase).
- Verified NOT a bug: reusing one `DatagramPacket` without resetting its
length — modern JVMs recv by buffer capacity (JDK-21 probe + the
successful on-device streaming confirm it).
+6
View File
@@ -19,6 +19,11 @@ compile_commands.json
# Smoke test output # Smoke test output
*.h264 *.h264
# iOS (generated by bootstrap.sh / XcodeGen; project.yml is the source of truth)
ios/Receiver.xcodeproj/
ios/tools/
ios/Receiver/Info.plist
# IDE # IDE
.vscode/ .vscode/
.idea/ .idea/
@@ -29,3 +34,4 @@ compile_commands.json
# OS # OS
.DS_Store .DS_Store
Thumbs.db Thumbs.db
hotspot.txt
+27
View File
@@ -0,0 +1,27 @@
# Changelog
Notable changes to `screen_cast`. Loosely follows [Keep a Changelog].
## [Unreleased]
### Added
- **Phase 9 — iOS receiver app** (`ios/`): a native Swift receiver so an iPhone
can act as the second receiver. Speaks the existing signaling + RTP protocol
(no C++ changes), mirroring the Android receiver (Phase 8) source-to-source.
- RTP core (header/packet, jitter buffer, H.264 depacketizer) ported
source-to-source from the Android receiver.
- BSD-socket signaling server (dual-stack, most-recent-peer, never-throwing
sends) + `NSBonjourServices` advertisement.
- VideoToolbox H.264 decode (in-band SPS/PPS, real-time, session rebuilt on
resolution change) rendered via `AVSampleBufferDisplayLayer` (letterbox).
- PLI keyframe recovery (rate-limited 500 ms) and `pendingOffer` for late
surface attach; no placeholder sizing before the first real keyframe.
- XcodeGen project + `bootstrap.sh`; XCTest port of the Android suite plus new
signaling / line-framing / AVCC / NAL-extraction coverage.
> **Status:** authored; on-device validation pending a Mac + Xcode 26 + iPhone 16
> (local-network/Bonjour consent, the H.264 format-description recipe, and the
> VideoToolbox render path are the on-device verification items).
[Keep a Changelog]: https://keepachangelog.com/en/1.1.0/
+122 -8
View File
@@ -3,9 +3,10 @@
A native Linux peer-to-peer screencast application. A native Linux peer-to-peer screencast application.
- **Send** your desktop or a window to another Linux machine. - **Send** your desktop or a window to another Linux machine.
- **Receive** a stream and render it in a window. - **Receive** a stream and render it in a window — or fullscreen headless.
- Discover receivers on the LAN via **mDNS/Avahi**; negotiate sessions with - Discover receivers on the LAN via **mDNS/Avahi**; negotiate sessions with
**JSON signaling**; stream **H.264 over RTP/UDP**. **JSON signaling**; stream **H.264 over RTP/UDP**.
- Control it from the **CLI**, a **GTK4 panel**, or a **waybar widget**.
Built with **C++20**, **Meson**, **PipeWire**, **FFmpeg**, and **SDL3**. Built with **C++20**, **Meson**, **PipeWire**, **FFmpeg**, and **SDL3**.
@@ -19,6 +20,7 @@ Requirements:
- SDL3 development package (`sdl3`) - SDL3 development package (`sdl3`)
- nlohmann JSON (`nlohmann_json`) and Avahi client (`avahi-client`) - nlohmann JSON (`nlohmann_json`) and Avahi client (`avahi-client`)
- For the sender only: PipeWire dev (`libpipewire-0.3`) and libportal - For the sender only: PipeWire dev (`libpipewire-0.3`) and libportal
- Optional, for the GUI: gtkmm-4.0
Build and run tests: Build and run tests:
@@ -37,6 +39,67 @@ screencast --send # machine B: discovers A, negotiates, streams
See `docs/RUNBOOK.md` for all modes, flags, and validation procedures. See `docs/RUNBOOK.md` for all modes, flags, and validation procedures.
## Modes and flags
```text
screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate MAX_KBPS] [--crf 0-51] [--fps 1-60]
screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen] [--swdecode]
screencast --discover [--timeout SECONDS]
screencast waybar [--toggle] # for waybar widgets
```
| Mode | Flag | Default | Meaning |
|-------------|------------------|---------------|------------------------------------------------|
| `--send` | `--target` | `monitor` | `monitor` or `window` (portal source picker) |
| | `--peer` | auto-discover | target a receiver directly, `HOST[:PORT]` |
| | `--bitrate` | `4000` | VBV max bitrate (kbps) |
| | `--crf` | `22` | encoder quality (lower = better) |
| | `--fps` | capture rate | frame-rate cap, 1-60 |
| `--receive` | `--port` | `5004` | local RTP UDP port |
| | `--signaling-port` | `5005` | TCP signaling port announced via mDNS |
| | `--fullscreen` | off | force fullscreen (automatic under KMSDRM) |
| | `--swdecode` | off | skip the hardware decode probe |
| `--discover`| `--timeout` | `3` | seconds to wait for mDNS responses |
| `waybar` | `--toggle` | print status | toggle streaming (stop / restart last session) |
`--send` without `--peer` discovers receivers on the LAN and requires that
exactly one is found; use `--discover` to list them.
## GUI and waybar widget
Build the GUI alongside the CLI (`meson configure build -Dgui=true`, then
recompile; installs as `screencast-gui`):
```sh
screencast-gui # receiver list → pick one → bitrate → Start
screencast waybar # one JSON line for a waybar custom module
screencast waybar --toggle
```
The waybar module shows the app icon (dimmed while idle, highlighted while
streaming; tooltip: receiver, bitrate, elapsed); left-click toggles
streaming to the last receiver, right-click opens the panel. All front-ends
agree on state because the sender publishes it to
`$XDG_RUNTIME_DIR/screencast/sender.json`. The icon (from `screencast_icon/`)
is installed into the hicolor theme by `meson install`, used by the GTK
panel and the waybar CSS, and embedded in the SDL receiver window.
## Under the hood: resilience
The receiver absorbs loss in two stages and recovers actively:
1. **Jitter window**: RTP packets are re-ordered by sequence number in a
small buffer (16 packets / 60 ms), so Wi-Fi reordering is not misread as
loss. In-order streams release immediately (zero added latency).
2. **PLI feedback**: when a frame arrives genuinely damaged, the receiver
drops it and asks the sender for a keyframe over the signaling channel
(rate-limited to one request per 500 ms); the sender re-encodes a
keyframe immediately, so recovery takes one frame time.
**Hardware decode**: the receiver probes `h264_v4l2m2m` (the VideoCore path
on Raspberry Pi) and falls back to software automatically; `--swdecode`
forces software.
## Receiver on a small ARM board (e.g. Raspberry Pi Zero 2 W) ## Receiver on a small ARM board (e.g. Raspberry Pi Zero 2 W)
The receiver does not need the sender's PipeWire/portal capture stack. On The receiver does not need the sender's PipeWire/portal capture stack. On
@@ -53,13 +116,60 @@ The script installs the dependencies, builds a receiver-only binary
or newer (GCC 13+ for C++20 `<format>`), builds SDL3 from source when the or newer (GCC 13+ for C++20 `<format>`), builds SDL3 from source when the
distribution does not package it, and enables `avahi-daemon`. It also distribution does not package it, and enables `avahi-daemon`. It also
installs and enables a systemd service so the receiver starts at boot — installs and enables a systemd service so the receiver starts at boot —
`systemctl status screencast-receiver` to check on it. `systemctl status screencast-receiver` to check on it (stop with
`sudo systemctl stop screencast-receiver`; always SIGTERM, never `kill -9`,
or stale mDNS records linger).
Performance note: decoding is software H.264; on very small boards expect On a headless console the receiver runs fullscreen automatically with
smooth playback for modest resolutions and reduced frame rates at high aspect-preserving letterboxing — no window manager needed (SDL3 KMSDRM; it
resolutions. Hardware decode is planned for Phase 7. On a headless console renders on its own VT, tty7, so `Ctrl+Alt+F1` returns to the console).
the receiver runs fullscreen automatically with aspect-preserving See `docs/RUNBOOK.md` for the headless setup details.
letterboxing.
## Direct link: receiver as a Wi-Fi hotspot
The receiver can act as a Wi-Fi access point, so a sender connects to the
board directly with no router in between:
```sh
sudo scripts/pi-hotspot.sh on # prints SSID + generated password
sudo scripts/pi-hotspot.sh status # shows the saved credentials
sudo scripts/pi-hotspot.sh off # back to normal router Wi-Fi
```
While active the Pi is reachable at `10.42.0.1`; on the sender, join the
hotspot's Wi-Fi and run `screencast --send`.
## Android receiver: a phone as the second screen
A native receiver app (Kotlin, minSdk 30) lets a phone act as the second
receiver — and with a USB-C DisplayPort-alt-mode cable, the phone screen
mirrors straight to HDMI. The app speaks the same signaling + RTP protocol,
so the sender needs no changes.
```sh
cd android
gradle :app:assembleDebug # or :app:installDebug with a device attached
adb install app/build/outputs/apk/debug/app-debug.apk
```
Launch the app (it advertises `_screencast._tcp` and shows its IP + ports),
then on the sender:
```sh
screencast --send --peer <phone-ip>:5005
```
Notes:
- The sender downscales to the phone's display resolution automatically
(from the signaling answer).
- The phone renders letterboxed (fit-within), fullscreen, screen kept on.
- The app needs only the `INTERNET` permission; no camera/location.
- mDNS discovery works on the same L2 segment; across subnets use `--peer`
(the app prints its IP and the exact fallback command).
- `--target monitor` captures the portal's default output — on multi-monitor
Hyprland/GTK-portal setups that may not be the one you want; use
`--target window` and pick a window on the target display.
## Architecture ## Architecture
@@ -69,7 +179,11 @@ See `docs/ARCHITECTURE.md` for module boundaries and design rules.
Development is split into phases in `docs/PHASES.md`. Development is split into phases in `docs/PHASES.md`.
Current phase: **Phase 7Resilience and polish**. Current phase: **Phase 8Android receiver app** (complete): the Kotlin
receiver app streams to a phone (validated on a Fairphone 6, phone → HDMI
via USB-C DP-alt-mode). Phase 7 items (PLI, jitter, hardware decode,
GUI/waybar) are complete and validated; deferred items remain the VAAPI
hardware encode probe and packaging.
## License ## License
+7
View File
@@ -0,0 +1,7 @@
.gradle/
.kotlin/
build/
local.properties
.idea/
captures/
.cxx/
+32
View File
@@ -0,0 +1,32 @@
plugins {
id("com.android.application")
}
android {
namespace = "screen_cast"
compileSdk = 36
defaultConfig {
applicationId = "screen_cast.receiver"
minSdk = 30
targetSdk = 36
versionCode = 1
versionName = "0.1.0"
}
// No release signing configured: the app is sideloaded as a debug build.
buildTypes {
release {
isMinifyEnabled = false
}
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
}
dependencies {
testImplementation("junit:junit:4.13.2")
}
+28
View File
@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
<!-- NsdService and all socket I/O require INTERNET. No other permissions:
no camera, no location, no network state. The screen is kept on, which
keeps Wi-Fi up. -->
<uses-permission android:name="android.permission.INTERNET" />
<application
android:label="screencast"
android:icon="@mipmap/ic_launcher"
android:allowBackup="false"
android:theme="@style/Theme.Screencast">
<activity
android:name=".ReceiverActivity"
android:exported="true"
android:launchMode="singleTask"
android:screenOrientation="sensorLandscape"
android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|keyboard|keyboardHidden|navigation|uiMode|density"
android:stateNotNeeded="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
</manifest>
@@ -0,0 +1,164 @@
package screen_cast
import android.app.Activity
import android.graphics.SurfaceTexture
import android.os.Bundle
import android.os.Handler
import android.os.Looper
import android.view.Surface
import android.view.TextureView
import android.view.View
import android.view.WindowManager
import android.widget.TextView
import java.net.Inet4Address
import java.net.NetworkInterface
import kotlin.math.min
import screen_cast.pipeline.ReceiverPipeline
/**
* Fullscreen receiver: a letterboxed TextureView plus a status overlay.
* The pipeline lives in the activity lifetime — created on demand, stopped
* when the app leaves the foreground, released on destroy.
*
* For the USB-C → HDMI use case the app only guarantees the screen is on,
* undimmed, landscape, and immersive; the display mirroring itself is the
* OS behavior of a DisplayPort-alt-mode port.
*/
class ReceiverActivity : Activity() {
companion object {
private const val TAG = "ReceiverActivity"
}
private val ui = Handler(Looper.getMainLooper())
private var pipeline: ReceiverPipeline? = null
private lateinit var videoView: TextureView
private lateinit var statusView: TextView
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_receiver)
videoView = findViewById(R.id.video)
statusView = findViewById(R.id.status)
videoView.isOpaque = true
videoView.surfaceTextureListener = object : TextureView.SurfaceTextureListener {
override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) {
pipeline?.attachSurface(Surface(surface))
fitVideo()
}
override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) {
fitVideo()
}
override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
// The TextureView is about to release this SurfaceTexture; the
// pipeline must forget it or a later offer would configure the
// decoder against a dead surface.
pipeline?.detachSurface()
return true
}
override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
}
}
override fun onResume() {
super.onResume()
applyFullscreen()
ensurePipeline()
pipeline?.start()
}
override fun onPause() {
super.onPause()
pipeline?.stop()
}
override fun onDestroy() {
super.onDestroy()
pipeline?.stop()
pipeline = null
}
private fun ensurePipeline() {
if (pipeline != null) return
val pipeline = ReceiverPipeline(
context = applicationContext,
localIp = localIpv4(),
displaySize = {
val metrics = resources.displayMetrics
metrics.widthPixels to metrics.heightPixels
},
onStatus = { text ->
ui.post {
statusView.text = text
statusView.visibility = View.VISIBLE
}
},
onFirstFrame = { ui.post { statusView.visibility = View.GONE } },
onVideoSize = { _, _ -> ui.post { fitVideo() } },
)
// The surface may already exist by the time the pipeline starts.
val surfaceTexture = videoView.surfaceTexture
if (surfaceTexture != null) {
pipeline.attachSurface(Surface(surfaceTexture))
}
this.pipeline = pipeline
}
/**
* Fit-within (letterbox) by sizing the TextureView to the video's aspect
* ratio, centered on the black window background. The C2 decoder scales
* its output to the Surface (android._video-scaling), so a surface with
* the video's aspect ratio renders 1:1 without distortion; a transform
* matrix on top would double-scale the already-stretched buffer.
*/
private fun fitVideo() {
val (videoWidth, videoHeight) = pipeline?.videoSize() ?: return
val parent = videoView.parent as? View ?: return
val parentWidth = parent.width
val parentHeight = parent.height
if (videoWidth <= 0 || videoHeight <= 0 || parentWidth <= 0 || parentHeight <= 0) return
val scale = min(parentWidth.toFloat() / videoWidth, parentHeight.toFloat() / videoHeight)
val width = (videoWidth * scale).toInt()
val height = (videoHeight * scale).toInt()
val params = videoView.layoutParams
if (params.width == width && params.height == height) return
params.width = width
params.height = height
if (params is android.widget.FrameLayout.LayoutParams) {
params.gravity = android.view.Gravity.CENTER
}
videoView.layoutParams = params
android.util.Log.i(TAG, "fitVideo: ${width}x$height in ${parentWidth}x$parentHeight")
}
private fun applyFullscreen() {
window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
val controller = window.insetsController ?: return
controller.systemBarsBehavior = android.view.WindowInsetsController.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE
controller.hide(
android.view.WindowInsets.Type.statusBars() or android.view.WindowInsets.Type.navigationBars(),
)
}
private fun localIpv4(): String {
return try {
val interfaces = NetworkInterface.getNetworkInterfaces() ?: return "unknown"
for (iface in interfaces) {
if (!iface.isUp || iface.isLoopback) continue
for (address in iface.inetAddresses) {
if (address is Inet4Address && !address.isLoopbackAddress) {
return address.hostAddress ?: "unknown"
}
}
}
"unknown"
} catch (e: Exception) {
"unknown"
}
}
}
@@ -0,0 +1,179 @@
package screen_cast.decode
import android.media.MediaCodec
import android.media.MediaFormat
import android.view.Surface
/**
* MediaCodec H.264 decoder that renders directly onto a Surface (no CPU
* pixels). The stream is self-describing: the sender repeats SPS/PPS
* in-band at every keyframe, so no out-of-band codec data is needed.
*/
class H264Decoder {
companion object {
// The C++ sender's VBV bounds keyframes to ~2 frame periods of bytes;
// 8 MiB is far beyond anything the negotiated rates can produce.
private const val MAX_INPUT_SIZE = 8 * 1024 * 1024
}
private var codec: MediaCodec? = null
private var configured = false
// True once the codec parsed the in-band SPS (INFO_OUTPUT_FORMAT_CHANGED).
// Before that, outputFormat still carries the configure() placeholder and
// must not drive layout — sizing the view to it squishes the first frame(s).
@Volatile
private var realFormatSeen = false
@Volatile
private var renderSurface: Surface? = null
/** Creates and configures the decoder. Width/height of 0 = unknown (the bitstream decides). */
@Synchronized
fun configure(width: Int, height: Int) {
if (configured) return
// 0x0 (the sender's offer) means the size is unknown: the Qualcomm
// C2 AVC decoder requires a concrete size at configure() and
// reconfigures from the in-band SPS of the first keyframe (the
// standard adaptive-resolution pattern).
val format = if (width > 0 && height > 0) {
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, width, height)
} else {
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240)
}
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE)
// Assign before configuring so a configure()/start() failure cannot
// orphan the created instance: MediaCodec has no finalizer and each
// unreleased instance holds a scarce native codec slot. The caller
// follows an exception with release(), which is a no-op on null.
val created = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
codec = created
try {
// The surface must go into configure(): the codec must start in
// surface mode. setOutputSurface() afterwards only switches an
// already-surface-mode codec; a codec configured without a
// surface can never take one.
created.configure(format, renderSurface, null, 0)
created.start()
} catch (e: Exception) {
codec = null
try {
created.release()
} catch (ignored: Exception) {
}
throw e
}
configured = true
realFormatSeen = false
}
/** Points the decoder at a (possibly new) render surface. */
@Synchronized
fun attachSurface(surface: Surface) {
renderSurface = surface
// Legal only in surface mode: setOutputSurface() dynamically switches
// an output surface on a codec configured WITH one (per the platform
// docs); on a ByteBuffer-mode codec it throws IllegalStateException.
codec?.setOutputSurface(surface)
}
/**
* Forgets a destroyed render surface: frames still decode (returning
* their buffers and keeping the queue moving) but are not rendered
* until the next [attachSurface]. API 35's detachOutputSurface() is the
* codec-side equivalent.
*/
@Synchronized
fun detachSurface() {
renderSurface = null
}
/**
* Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp).
* @return false when the codec's input queue was full and the frame was
* dropped; the caller should request a keyframe, not flush (the codec
* is healthy, merely busy).
*/
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean): Boolean {
val c = codec ?: throw IllegalStateException("decoder not configured")
val index = c.dequeueInputBuffer(10_000)
if (index < 0) return false
val buffer = c.getInputBuffer(index) ?: return false
buffer.clear()
buffer.put(data)
val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_KEY_FRAME else 0
c.queueInputBuffer(index, 0, data.size, presentationUs, flags)
return true
}
/**
* Non-blocking drain of available output frames; in surface mode each
* released buffer is rendered to the output surface by the codec.
* @throws IllegalStateException when the decoder signals a fatal error;
* the caller should flush() and request a keyframe.
*/
fun drain() {
val c = codec ?: return
val info = MediaCodec.BufferInfo()
while (true) {
val index = c.dequeueOutputBuffer(info, 0)
when {
index >= 0 -> {
// releaseOutputBuffer(render=true) is what puts the
// frame on the surface (and frees the pool slot).
c.releaseOutputBuffer(index, renderSurface != null)
}
index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
// The codec parsed the SPS size; outputFormat is ready.
realFormatSeen = true
android.util.Log.i(
"H264Decoder", "output format: " +
c.outputFormat.getInteger(MediaFormat.KEY_WIDTH) +
"x" +
c.outputFormat.getInteger(MediaFormat.KEY_HEIGHT)
)
}
index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED -> Unit
else -> break // no more frames right now
}
}
}
/** Resets decoder state; the next keyframe re-primes it (in-band SPS/PPS). */
fun flush() {
val c = codec ?: return
try {
c.flush()
} catch (e: Exception) {
// A codec in an error state may refuse the flush; the caller
// follows up with a keyframe request either way.
}
}
/** The decoded resolution, once the first keyframe configured the codec. */
@Synchronized
fun outputSize(): Pair<Int, Int>? {
val c = codec ?: return null
if (!realFormatSeen) return null // still the configure() placeholder
return try {
val format = c.outputFormat
val width = format.getInteger(MediaFormat.KEY_WIDTH)
val height = format.getInteger(MediaFormat.KEY_HEIGHT)
if (width > 0 && height > 0) width to height else null
} catch (e: Exception) {
null
}
}
@Synchronized
fun release() {
val c = codec ?: return
try {
c.stop()
} catch (e: Exception) {
// ignore
}
c.release()
codec = null
configured = false
renderSurface = null
}
}
@@ -0,0 +1,383 @@
package screen_cast.pipeline
import android.content.Context
import android.net.nsd.NsdManager
import android.net.nsd.NsdServiceInfo
import android.view.Surface
import screen_cast.decode.H264Decoder
import screen_cast.rtp.H264Depacketizer
import screen_cast.rtp.JitterBuffer
import screen_cast.rtp.RtpPacket
import screen_cast.signaling.SessionAnswer
import screen_cast.signaling.SessionOffer
import screen_cast.signaling.SessionPli
import screen_cast.signaling.SignalingServer
import java.net.DatagramPacket
import java.net.DatagramSocket
import java.net.InetSocketAddress
import java.net.SocketException
import java.util.concurrent.atomic.AtomicBoolean
/**
* The receiver pipeline, mirroring the C++ ReceiverPipeline:
*
* NSD advertise + signaling server (offer → answer)
* UDP RTP → jitter buffer → depacketize → MediaCodec → Surface
*
* Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
* (rate-limited to one per 500 ms) asks the sender for a keyframe.
*/
class ReceiverPipeline(
private val context: Context,
private val localIp: String,
private val displaySize: () -> Pair<Int, Int>,
private val onStatus: (String) -> Unit,
private val onFirstFrame: () -> Unit,
private val onVideoSize: (Int, Int) -> Unit,
) {
companion object {
private const val TAG = "ReceiverPipeline"
private const val SERVICE_NAME = "screencast"
private const val SERVICE_TYPE = "_screencast._tcp"
private const val DESIRED_UDP_PORT = 5004
private const val DESIRED_SIGNALING_PORT = 5005
private const val PLI_MIN_INTERVAL_MS = 500L
}
private val nsdManager: NsdManager = context.getSystemService(Context.NSD_SERVICE) as NsdManager
private val decoderLock = Any()
@Volatile private var running = false
private var udpSocket: DatagramSocket? = null
private var udpPort = 0
private var signalingPort = 0
private var signaling: SignalingServer? = null
private var readerThread: Thread? = null
@Volatile private var registrationListener: NsdManager.RegistrationListener? = null
// Guarded by decoderLock: currentSurface, pendingOffer, decoder.
@Volatile private var decoder: H264Decoder? = null
@Volatile private var depacketizer = H264Depacketizer()
private val jitter = JitterBuffer()
private var currentSurface: Surface? = null
private var pendingOffer: SessionOffer? = null
@Volatile private var activeSession = ""
private val firstFrameSeen = AtomicBoolean(false)
private val pliLock = Any()
private var lastPliAtMs = 0L
@Volatile private var videoWidth = 0
@Volatile private var videoHeight = 0
/** The current decoded resolution (0, 0 until the first keyframe). */
fun videoSize(): Pair<Int, Int> = videoWidth to videoHeight
/** Binds the ports, advertises the service, and starts reading RTP. */
@Synchronized
fun start() {
if (running) return
try {
// Plain DatagramSocket: consistent semantics across platforms
// (Android's DatagramChannel.receive() returns a SocketAddress,
// not the byte count). Port 5004 when free; otherwise ephemeral.
val socket = try {
DatagramSocket(InetSocketAddress(DESIRED_UDP_PORT))
} catch (e: Exception) {
DatagramSocket()
}
udpPort = socket.localPort
udpSocket = socket
} catch (e: Exception) {
onStatus("Failed to bind the media port: ${e.message}")
return
}
val server = SignalingServer(
onOffer = { offer -> onOffer(offer) },
onPli = { /* the receiver never receives PLIs */ },
)
try {
signalingPort = server.start(DESIRED_SIGNALING_PORT)
signaling = server
} catch (e: Exception) {
onStatus("Failed to start signaling: ${e.message}")
try {
udpSocket?.close()
} catch (ignored: Exception) {
}
return
}
running = true
readerThread = Thread({ readLoop() }, "rtp-reader").also { it.start() }
// Advertise AFTER the listening status: a registration failure must
// not be overwritten by it (both post to the same overlay).
onStatus(
"Listening on $localIp (media :$udpPort, signaling :$signalingPort)\n" +
"Waiting for a sender… (fall back to: screencast --send --peer $localIp:$signalingPort)",
)
advertiseNsd(signalingPort)
}
/**
* Points the decoder (current or future) at a render surface. A pending
* offer (accepted while no surface existed) configures its decoder now.
*/
fun attachSurface(surface: Surface) {
var configuredFromPending = false
synchronized(decoderLock) {
currentSurface = surface
decoder?.attachSurface(surface)
val offer = pendingOffer
if (offer != null) {
val created = configureDecoderLocked(offer.width, offer.height)
if (created != null) {
pendingOffer = null
decoder = created
configuredFromPending = true
} else {
// Keep the offer: the next attachSurface retries the
// configure instead of abandoning the session.
onStatus("Could not start the decoder")
}
}
}
if (configuredFromPending) {
// The sender is already streaming P-frames; the late-configured
// decoder needs a keyframe (SPS/PPS + IDR) to start producing
// output — the sender only emits one when asked.
requestPli()
}
}
/** Forgets a destroyed render surface so a later offer cannot configure against it. */
fun detachSurface() {
synchronized(decoderLock) {
currentSurface = null
decoder?.detachSurface()
}
}
/**
* Creates and configures a decoder for the stream size; caller holds
* [decoderLock]. Returns null on failure with the codec released.
*/
private fun configureDecoderLocked(width: Int, height: Int): H264Decoder? {
val fresh = H264Decoder()
try {
// Surface first: configure() needs it before the codec starts.
currentSurface?.let { fresh.attachSurface(it) }
fresh.configure(width, height)
} catch (e: Exception) {
fresh.release()
return null
}
return fresh
}
/** Stops listening; the pipeline can be started again. */
@Synchronized
fun stop() {
if (!running) return
running = false
activeSession = ""
try {
udpSocket?.close() // unblocks the reader's receive()
} catch (ignored: Exception) {
}
udpSocket = null
readerThread?.join(1000)
readerThread = null
signaling?.close()
signaling = null
unregisterNsd()
synchronized(decoderLock) {
decoder?.release()
decoder = null
pendingOffer = null
}
firstFrameSeen.set(false)
onStatus("Stopped")
}
private fun onOffer(offer: SessionOffer) {
if (offer.codec != "h264") {
onStatus("Unsupported codec: ${offer.codec}")
return
}
// The offer's width/height are informational (the C++ sender leaves
// them 0); the bitstream carries SPS/PPS at every keyframe.
var decoderReady = true
synchronized(decoderLock) {
pendingOffer = null
decoder?.release()
decoder = null
if (currentSurface != null) {
decoder = configureDecoderLocked(offer.width, offer.height)
decoderReady = decoder != null
} else {
// No surface yet (the window is between surfaces): remember
// the offer; attachSurface configures the decoder. Configuring
// without a surface is not an option — that codec could never
// take one afterwards (setOutputSurface refuses it).
pendingOffer = offer
}
}
if (!decoderReady) {
onStatus("Could not start the decoder")
return
}
// New session: pristine reassembly state.
depacketizer = H264Depacketizer()
jitter.clear()
firstFrameSeen.set(false)
videoWidth = 0
videoHeight = 0
activeSession = offer.sessionId
android.util.Log.i(TAG, "offer: session=${offer.sessionId} ${offer.width}x${offer.height} @${offer.frameRateNum}/${offer.frameRateDen}")
val (displayWidth, displayHeight) = displaySize()
signaling?.send(
SessionAnswer(
sessionId = offer.sessionId,
rtpAddress = "", // the sender targets the address of its own signaling connection
rtpPort = udpPort,
displayWidth = displayWidth,
displayHeight = displayHeight,
),
)
onStatus(
if (decoder != null) "Session ${offer.sessionId} negotiated — waiting for the first frame…"
else "Session ${offer.sessionId} negotiated — waiting for the display surface…"
)
}
private fun readLoop() {
val socket = udpSocket ?: return
val buffer = ByteArray(2048) // MTU 1200 + headroom
val datagram = DatagramPacket(buffer, buffer.size)
while (running) {
try {
socket.receive(datagram)
} catch (e: SocketException) {
break // socket closed
} catch (e: Exception) {
continue
}
val received = datagram.length
if (received <= 0) continue
val bytes = if (received == buffer.size) buffer.copyOf() else buffer.copyOf(received)
val packet = RtpPacket.parse(bytes) ?: continue
for (released in jitter.push(packet)) {
handleDepacketized(released)
}
}
}
private fun handleDepacketized(packet: RtpPacket) {
val result = depacketizer.depacketize(packet)
val accessUnit = result.accessUnit
if (accessUnit != null) {
val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL
try {
// MediaCodec is not thread-safe: attachSurface() (main
// thread) and onOffer (signaling thread) synchronize on the
// same lock around their codec calls.
synchronized(decoderLock) {
val activeDecoder = decoder ?: return
if (!activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame)) {
// Input queue full: the dropped frame corrupts the
// GOP until the next keyframe — ask for one. No
// flush; the codec is healthy, merely busy.
requestPli()
}
activeDecoder.drain()
if (firstFrameSeen.compareAndSet(false, true)) {
onFirstFrame()
}
updateVideoSize()
}
} catch (e: Exception) {
onStatus("Decoder error (${e.message}) — requesting a keyframe…")
synchronized(decoderLock) { decoder?.flush() }
requestPli()
}
}
if (result.frameDropped) {
requestPli()
}
}
private fun updateVideoSize() {
val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return
if (size.first != videoWidth || size.second != videoHeight) {
videoWidth = size.first
videoHeight = size.second
android.util.Log.i(TAG, "video size: ${size.first}x${size.second}")
onVideoSize(size.first, size.second)
}
}
// Rate-limited keyframe request, callable from any thread (the reader
// thread and the main thread's attachSurface both reach it).
private fun requestPli() {
val session = activeSession
if (session.isEmpty()) return
val now = System.currentTimeMillis()
synchronized(pliLock) {
if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return
lastPliAtMs = now
}
android.util.Log.i(TAG, "PLI requested for session $session")
signaling?.send(SessionPli(session))
}
private fun advertiseNsd(port: Int) {
val info = NsdServiceInfo().apply {
setServiceName(SERVICE_NAME)
setServiceType(SERVICE_TYPE)
setPort(port)
}
// The classic RegistrationListener API exists from API 16 through 36
// (verified against the android-36 SDK with javap), so no fallback is
// needed — the previously reflected "ResolutionListener" never
// existed at any API level and could only ever fail.
val listener = object : NsdManager.RegistrationListener {
override fun onServiceRegistered(serviceInfo: NsdServiceInfo) {
// Registered: the sender's mDNS browser should see it now.
android.util.Log.i(TAG, "mDNS registered: ${serviceInfo.serviceName}.${SERVICE_TYPE}")
}
override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
}
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
}
try {
// PROTOCOL_DNS_SD is mandatory on API 36: NsdManager.checkProtocol()
// rejects anything else (the historical 0 threw
// "IllegalArgumentException: Unsupported protocol", which the
// old code swallowed — mDNS never advertised on this phone).
nsdManager.registerService(info, NsdManager.PROTOCOL_DNS_SD, listener)
registrationListener = listener
} catch (e: Exception) {
android.util.Log.e(TAG, "mDNS registration failed", e)
onStatus("mDNS unavailable (${e.message}) — reach this receiver with --peer $localIp:$port")
}
}
private fun unregisterNsd() {
val listener = registrationListener ?: return
try {
nsdManager.unregisterService(listener)
} catch (ignored: Exception) {
// already unregistered
}
registrationListener = null
}
}
@@ -0,0 +1,179 @@
package screen_cast.rtp
import java.io.ByteArrayOutputStream
/** Result of feeding one packet to the depacketizer. */
data class DepacketizeResult(
/** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */
val accessUnit: ByteArray? = null,
/** True when this call discarded a frame as damaged (packet loss or unsupported packetization). */
val frameDropped: Boolean = false,
/** True when the completed access unit carries SPS/PPS (a keyframe). */
val isKeyFrame: Boolean = false,
)
/**
* Reassembles RFC 6184 packet streams (single NAL unit packets and FU-A)
* into Annex-B access units. Packets must arrive in order; frames damaged
* by sequence gaps or missing fragments are reported via DepacketizeResult.
*
* Mirrors the C++ `H264Depacketizer` (same state machine and start codes).
*/
class H264Depacketizer {
companion object {
private const val FU_A = 28
}
private var lastSequenceNumber: Int? = null
private var frameStarted = false
private var frameDamaged = false
private var frameTimestamp = 0
// Growable byte accumulators: keyframes reach hundreds of KB, and boxing
// each byte (the ArrayList<Int> this replaced) churned the GC hard.
private val accessUnit = ByteArrayOutputStream()
private var fuActive = false
private val fuNal = ByteArrayOutputStream()
/** Feed one packet (in sequence order, from the jitter buffer). */
fun depacketize(packet: RtpPacket): DepacketizeResult {
var result = DepacketizeResult()
// Track sequence continuity: a gap means packets were lost.
lastSequenceNumber?.let { last ->
val expected = (last + 1) and 0xFFFF
if (packet.header.sequenceNumber != expected) {
fuActive = false
fuNal.reset()
if (frameStarted) {
frameDamaged = true
}
}
}
lastSequenceNumber = packet.header.sequenceNumber
// A timestamp change without a closing marker means the previous frame
// lost its tail and can no longer be recovered.
if (frameStarted && packet.header.timestamp != frameTimestamp) {
dropFrame()
result = result.copy(frameDropped = true)
}
if (!frameStarted) {
frameStarted = true
frameDamaged = false
frameTimestamp = packet.header.timestamp
accessUnit.reset()
}
val payload = packet.payload
if (payload.isNotEmpty()) {
val type = payload[0].toInt() and 0x1F
when {
type in 1..23 -> {
// Single NAL unit packet.
if (fuActive) {
// The previous fragmented NAL never received its end packet.
frameDamaged = true
fuActive = false
fuNal.reset()
}
appendStartCode()
accessUnit.write(payload)
}
type == FU_A -> {
if (payload.size < 2) {
frameDamaged = true
} else {
val fuHeader = payload[1].toInt() and 0xFF
val start = fuHeader and 0x80 != 0
val end = fuHeader and 0x40 != 0
val fragment = payload.copyOfRange(2, payload.size)
when {
start -> {
if (fuActive) {
// The previous fragmented NAL lost its end packet.
frameDamaged = true
}
fuActive = true
fuNal.reset()
// The FU indicator keeps the original NAL's F bit (0) and NRI,
// and declares type 28; the FU header carries S/E plus the real type.
fuNal.write((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F))
fuNal.write(fragment)
}
!fuActive -> {
// Continuation without a start: the head of the NAL is lost.
frameDamaged = true
}
else -> {
fuNal.write(fragment)
if (end) {
appendStartCode()
fuNal.writeTo(accessUnit)
fuActive = false
fuNal.reset()
}
}
}
}
}
else -> {
// Unsupported packetization mode (STAP-A, MTAP, FU-B): the frame
// cannot be reconstructed.
frameDamaged = true
}
}
}
if (!packet.header.marker) {
return result
}
if (fuActive) {
// The marker arrived while a NAL was still fragmented.
frameDamaged = true
fuActive = false
fuNal.reset()
}
if (!frameDamaged && accessUnit.size() > 0) {
val unit = accessUnit.toByteArray()
result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit))
} else {
// The frame that just ended is unusable.
result = result.copy(frameDropped = true)
}
dropFrame()
return result
}
private fun appendStartCode() {
accessUnit.write(0)
accessUnit.write(0)
accessUnit.write(1)
}
/** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */
private fun containsParameterSets(unit: ByteArray): Boolean {
for (i in 0..unit.size - 4) {
if (unit[i] == 0.toByte() && unit[i + 1] == 0.toByte() && unit[i + 2] == 1.toByte()) {
val nalType = unit[i + 3].toInt() and 0x1F
if (nalType == 7 || nalType == 8) {
return true
}
}
}
return false
}
private fun dropFrame() {
frameStarted = false
frameDamaged = false
accessUnit.reset()
fuActive = false
fuNal.reset()
}
}
@@ -0,0 +1,71 @@
package screen_cast.rtp
import java.util.TreeMap
import java.util.concurrent.TimeUnit
/**
* Reorders RTP packets by sequence number before depacketization so that a
* reordering link (Wi-Fi) is not read as loss. Delivery stays in order;
* only aged-out or overflowing buffers release out of order, which the
* downstream gap detection still handles for genuine loss.
*
* Mirrors the C++ `RtpJitterBuffer` (same defaults and semantics).
*/
class JitterBuffer(
private val maxDepth: Int = 16,
private val maxDelayMs: Long = 60,
) {
private class BufferEntry(val timeNanos: Long, val packet: RtpPacket)
private val lock = Any()
private val buffer = TreeMap<Int, BufferEntry>()
private var nextExpected: Int? = null
/** Insert one packet and return the packets now ready for in-order delivery. */
fun push(packet: RtpPacket): List<RtpPacket> = synchronized(lock) {
val released = ArrayList<RtpPacket>()
val sequence = packet.header.sequenceNumber
val now = System.nanoTime()
val expected0 = nextExpected ?: sequence.also { nextExpected = it }
// Serial-number comparison: a distance >= 32768 means the packet is
// older than what we already delivered (duplicate or straggler).
val distance = (sequence - expected0 + 65536) % 65536
if (distance < 32768) {
buffer[sequence] = BufferEntry(now, packet)
// Release the consecutive run from the expected sequence.
var expected = expected0
while (true) {
val entry = buffer[expected] ?: break
released.add(entry.packet)
buffer.remove(expected)
expected = (expected + 1) and 0xFFFF
}
nextExpected = expected
// A missing packet stalls the run: age out the backlog (or bound
// the buffer) and release what is there in order, so genuine loss
// reaches the depacketizer's gap detection rather than blocking.
if (buffer.isNotEmpty()) {
val head = buffer.firstEntry().value
val headAgeMs = TimeUnit.NANOSECONDS.toMillis(now - head.timeNanos)
if (headAgeMs > maxDelayMs || buffer.size > maxDepth) {
released.addAll(buffer.values.map { it.packet })
nextExpected = (buffer.lastKey() + 1) and 0xFFFF
buffer.clear()
}
}
}
released
}
/** Discard everything still buffered. */
fun clear() {
synchronized(lock) {
buffer.clear()
nextExpected = null
}
}
}
@@ -0,0 +1,55 @@
package screen_cast.rtp
/** Minimal RTP header (RFC 3550) without extensions. */
data class RtpHeader(
val version: Int = 2,
val padding: Boolean = false,
val extension: Boolean = false,
val csrcCount: Int = 0,
val marker: Boolean = false,
val payloadType: Int = 96,
val sequenceNumber: Int = 0,
val timestamp: Int = 0,
val ssrc: Int = 0,
) {
/** Serializes the bare 12-byte header; requires a version-2, extension-less header. */
fun serialize(): ByteArray {
val out = ByteArray(12)
out[0] = (((version and 0x0F) shl 6) or (if (padding) 0x20 else 0) or (if (extension) 0x10 else 0) or (csrcCount and 0x0F)).toByte()
out[1] = ((if (marker) 0x80 else 0) or (payloadType and 0x7F)).toByte()
out[2] = (sequenceNumber ushr 8).toByte()
out[3] = (sequenceNumber and 0xFF).toByte()
out[4] = (timestamp ushr 24).toByte()
out[5] = (timestamp ushr 16).toByte()
out[6] = (timestamp ushr 8).toByte()
out[7] = (timestamp and 0xFF).toByte()
out[8] = (ssrc ushr 24).toByte()
out[9] = (ssrc ushr 16).toByte()
out[10] = (ssrc ushr 8).toByte()
out[11] = (ssrc and 0xFF).toByte()
return out
}
companion object {
fun parse(input: ByteArray): RtpHeader? {
if (input.size < 12) return null
val b0 = input[0].toInt() and 0xFF
val b1 = input[1].toInt() and 0xFF
val version = b0 ushr 6
if (version != 2) return null
return RtpHeader(
version = version,
padding = b0 and 0x20 != 0,
extension = b0 and 0x10 != 0,
csrcCount = b0 and 0x0F,
marker = b1 and 0x80 != 0,
payloadType = b1 and 0x7F,
sequenceNumber = ((input[2].toInt() and 0xFF) shl 8) or (input[3].toInt() and 0xFF),
timestamp = ((input[4].toInt() and 0xFF) shl 24) or ((input[5].toInt() and 0xFF) shl 16) or
((input[6].toInt() and 0xFF) shl 8) or (input[7].toInt() and 0xFF),
ssrc = ((input[8].toInt() and 0xFF) shl 24) or ((input[9].toInt() and 0xFF) shl 16) or
((input[10].toInt() and 0xFF) shl 8) or (input[11].toInt() and 0xFF),
)
}
}
}
@@ -0,0 +1,36 @@
package screen_cast.rtp
/** An RTP packet: 12-byte base header (plus optional CSRC/extension) and payload. */
class RtpPacket(val header: RtpHeader, val payload: ByteArray) {
companion object {
/**
* Parses a full RTP datagram. Honors CSRC lists, one-level extension
* headers, and RFC 3550 padding, mirroring the C++ receiver.
*/
fun parse(input: ByteArray): RtpPacket? {
if (input.size < 12) return null
val header = RtpHeader.parse(input) ?: return null
var offset = 12 + header.csrcCount * 4
if (input.size < offset) return null
if (header.extension) {
if (input.size < offset + 4) return null
val extensionWords =
((input[offset + 2].toInt() and 0xFF) shl 8) or (input[offset + 3].toInt() and 0xFF)
offset += 4 + extensionWords * 4
if (input.size < offset) return null
}
var payloadSize = input.size - offset
if (header.padding) {
// RFC 3550: the last byte holds the padding size, including itself.
if (payloadSize == 0) return null
val paddingSize = input[input.size - 1].toInt() and 0xFF
if (paddingSize == 0 || paddingSize > payloadSize) return null
payloadSize -= paddingSize
}
return RtpPacket(header, input.copyOfRange(offset, offset + payloadSize))
}
}
}
@@ -0,0 +1,100 @@
package screen_cast.signaling
import org.json.JSONObject
// JSON wire format shared with the C++ implementation: one JSON object per
// newline-terminated TCP line (offer / answer / pli).
data class SessionOffer(
val sessionId: String,
val codec: String,
val width: Int,
val height: Int,
val frameRateNum: Int,
val frameRateDen: Int,
val rtpAddress: String,
val rtpPort: Int,
) : SignalingMessage
data class SessionAnswer(
val sessionId: String,
val rtpAddress: String,
val rtpPort: Int,
val displayWidth: Int,
val displayHeight: Int,
) : SignalingMessage
data class SessionPli(val sessionId: String) : SignalingMessage
sealed interface SignalingMessage {
companion object {
private const val MAX_MESSAGE_BYTES = 64 * 1024
fun parse(line: String): SignalingMessage? {
if (line.length > MAX_MESSAGE_BYTES) return null
val json = try {
JSONObject(line)
} catch (e: Exception) {
return null
}
when (val type = json.optString("type")) {
"offer" ->
return SessionOffer(
sessionId = json.optString("session_id"),
codec = json.optString("codec"),
width = json.optInt("width"),
height = json.optInt("height"),
frameRateNum = json.optInt("frame_rate_num", 30),
frameRateDen = json.optInt("frame_rate_den", 1),
rtpAddress = json.optString("rtp_address"),
rtpPort = json.optInt("rtp_port"),
)
"answer" ->
return SessionAnswer(
sessionId = json.optString("session_id"),
rtpAddress = json.optString("rtp_address"),
rtpPort = json.optInt("rtp_port"),
displayWidth = json.optInt("display_width"),
displayHeight = json.optInt("display_height"),
)
"pli" -> return SessionPli(sessionId = json.optString("session_id"))
else -> return null
}
}
fun serialize(message: SignalingMessage): String {
val json = JSONObject()
when (message) {
is SessionOffer -> {
json.put("type", "offer")
json.put("session_id", message.sessionId)
json.put("codec", message.codec)
json.put("width", message.width)
json.put("height", message.height)
json.put("frame_rate_num", message.frameRateNum)
json.put("frame_rate_den", message.frameRateDen)
json.put("rtp_address", message.rtpAddress)
json.put("rtp_port", message.rtpPort)
}
is SessionAnswer -> {
json.put("type", "answer")
json.put("session_id", message.sessionId)
json.put("rtp_address", message.rtpAddress)
json.put("rtp_port", message.rtpPort)
json.put("display_width", message.displayWidth)
json.put("display_height", message.displayHeight)
}
is SessionPli -> {
json.put("type", "pli")
json.put("session_id", message.sessionId)
}
}
return json.toString() + "\n"
}
}
}
@@ -0,0 +1,144 @@
package screen_cast.signaling
import java.io.BufferedInputStream
import java.io.ByteArrayOutputStream
import java.io.OutputStream
import java.net.InetSocketAddress
import java.net.ServerSocket
import java.net.Socket
/**
* Newline-delimited JSON signaling server (the receiver side). Keeps the
* most recent connection as its active peer, mirroring the C++ server:
* `onOffer` may answer synchronously (the sender blocks on the answer).
*/
class SignalingServer(
private val onOffer: (SessionOffer) -> Unit,
private val onPli: (SessionPli) -> Unit,
) {
private val server = ServerSocket()
private val peerLock = Any()
private var peerOut: OutputStream? = null
private var acceptThread: Thread? = null
private var readerThread: Thread? = null
@Volatile
private var running = false
/** Binds the port (SO_REUSEADDR) and starts accepting. Returns the bound port. */
fun start(port: Int): Int {
server.reuseAddress = true
server.bind(InetSocketAddress(port))
running = true
acceptThread = Thread({ acceptLoop() }, "signaling-accept")
acceptThread!!.start()
return server.localPort
}
/**
* Sends to the current peer; never throws. A lost control message (a
* PLI, an answer) is recoverable — the session re-negotiates or the next
* keyframe arrives — but an exception here would kill the RTP reader
* thread, which reaches send() from requestPli(). Mirrors the C++ server,
* which ignores write failures.
*/
fun send(message: SignalingMessage) {
val bytes = try {
SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8)
} catch (e: Exception) {
return // unreachable for our message types; serialize is total
}
synchronized(peerLock) {
val out = peerOut ?: return
try {
out.write(bytes)
out.flush()
} catch (e: Exception) {
// The peer is gone (e.g. the sender died while media still
// flows). Dropping peerOut also closes the socket; the next
// accepted offer installs a fresh one.
peerOut = null
}
}
}
fun close() {
running = false
try {
server.close()
} catch (e: Exception) {
// already closed
}
synchronized(peerLock) {
try {
peerOut?.close()
} catch (e: Exception) {
// peer already gone
}
peerOut = null
}
acceptThread?.join(500)
readerThread?.join(500)
}
private fun acceptLoop() {
while (running) {
val socket = try {
server.accept()
} catch (e: Exception) {
break // listening socket closed
}
synchronized(peerLock) {
try {
peerOut?.close()
} catch (e: Exception) {
// previous sender already gone
}
peerOut = socket.getOutputStream()
}
readerThread = Thread({ readLoop(socket) }, "signaling-reader").also { it.start() }
}
}
private fun readLoop(socket: Socket) {
val input = BufferedInputStream(socket.getInputStream())
val line = ByteArrayOutputStream()
val chunk = ByteArray(4096)
while (running) {
val read = try {
input.read(chunk)
} catch (e: Exception) {
break
}
if (read < 0) break
for (i in 0 until read) {
val b = chunk[i].toInt() and 0xFF
if (b == '\n'.code) {
val text = line.toString(Charsets.UTF_8.name())
line.reset()
if (text.isNotEmpty()) {
dispatch(text)
}
} else if (b != '\r'.code) {
if (line.size() < 64 * 1024) {
line.write(b)
} else {
line.reset() // hostile or broken peer: drop the oversized line
}
}
}
}
}
private fun dispatch(line: String) {
val message = SignalingMessage.parse(line) ?: return
try {
when (message) {
is SessionOffer -> onOffer(message)
is SessionPli -> onPli(message)
is SessionAnswer -> Unit // the receiver never receives answers
}
} catch (e: Exception) {
// A broken callback must not kill the reader thread.
}
}
}
@@ -0,0 +1,27 @@
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@android:color/black">
<TextureView
android:id="@+id/video"
android:layout_width="match_parent"
android:layout_height="match_parent" />
<TextView
android:id="@+id/status"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="top|start"
android:layout_margin="16dp"
android:background="#66000000"
android:maxLines="3"
android:ellipsize="end"
android:paddingHorizontal="12dp"
android:paddingVertical="8dp"
android:textColor="@android:color/white"
android:textSize="13sp"
tools:text="Listening… waiting for a sender"
xmlns:tools="http://schemas.android.com/tools" />
</FrameLayout>
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

@@ -0,0 +1,9 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<style name="Theme.Screencast" parent="@android:style/Theme.Material.NoActionBar">
<item name="android:windowFullscreen">true</item>
<item name="android:windowBackground">@android:color/black</item>
<item name="android:statusBarColor">@android:color/black</item>
<item name="android:navigationBarColor">@android:color/black</item>
</style>
</resources>
@@ -0,0 +1,138 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
class H264DepacketizerTest {
companion object {
private val START = byteArrayOf(0x00, 0x00, 0x01)
private fun packet(seq: Int, ts: Int, payload: ByteArray, marker: Boolean = false) =
RtpPacket(
RtpHeader(sequenceNumber = seq, timestamp = ts, marker = marker),
payload,
)
}
@Test
fun single_nal_two_packets() {
val depacketizer = H264Depacketizer()
// SPS (type 7) then a slice (type 5), closed by the marker.
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x67, 0xAA.toByte(), 0xBB.toByte())))
assertNull(first.accessUnit)
val second = depacketizer.depacketize(packet(2, 100, byteArrayOf(0x41, 0x01, 0x02), marker = true))
assertTrue(
second.accessUnit
?.contentEquals(START + byteArrayOf(0x67, 0xAA.toByte(), 0xBB.toByte()) + START + byteArrayOf(0x41, 0x01, 0x02)) == true,
)
assertTrue(second.isKeyFrame)
assertFalse(second.frameDropped)
}
@Test
fun fu_a_reassembly() {
val depacketizer = H264Depacketizer()
// NAL: header 0x41 (type 1, NRI 2) + payload 0x11 0x22 0x33 0x44.
// FU indicator = (0x41 & 0xE0) | 28 = 0x5C; the depacketizer rebuilds
// the NAL header from indicator-NRI | FU-type, so the FU header must
// carry the original type (1).
val indicator = 0x5C.toByte()
val start = packet(1, 100, byteArrayOf(indicator, 0x81.toByte(), 0x11))
val mid = packet(2, 100, byteArrayOf(indicator, 0x01, 0x22))
val last = packet(3, 100, byteArrayOf(indicator, 0x41, 0x33, 0x44), marker = true)
assertNull(depacketizer.depacketize(start).accessUnit)
assertNull(depacketizer.depacketize(mid).accessUnit)
val done = depacketizer.depacketize(last)
assertTrue(done.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0x11, 0x22, 0x33, 0x44)) == true)
assertFalse(done.frameDropped)
}
@Test
fun drops_gapped_frames() {
val depacketizer = H264Depacketizer()
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01)))
assertNull(first.accessUnit)
// seq 2 is missing; the frame must be reported dropped, not delivered.
val tail = depacketizer.depacketize(packet(3, 100, byteArrayOf(0x41, 0x02), marker = true))
assertNull(tail.accessUnit)
assertTrue(tail.frameDropped)
}
@Test
fun separate_frames_by_marker() {
val depacketizer = H264Depacketizer()
val au1 = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0xAA.toByte()), marker = true))
assertTrue(au1.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0xAA.toByte())) == true)
val au2 = depacketizer.depacketize(packet(2, 200, byteArrayOf(0x41, 0xBB.toByte()), marker = true))
assertTrue(au2.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0xBB.toByte())) == true)
}
@Test
fun drops_fu_without_start() {
val depacketizer = H264Depacketizer()
// Continuation (no S bit) without any start packet.
val result = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte(), 0x41.toByte(), 0x11), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_still_fragmented_at_marker() {
val depacketizer = H264Depacketizer()
val start = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte(), 0x81.toByte(), 0x11)))
assertNull(start.accessUnit)
// Marker arrives while the FU-A NAL is still open.
val result = depacketizer.depacketize(packet(2, 100, byteArrayOf(0x41, 0x01), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_unsupported_packetization() {
val depacketizer = H264Depacketizer()
val stapA = 24.toByte() // STAP-A
val result = depacketizer.depacketize(
packet(1, 100, byteArrayOf(stapA, 0x00, 0x05, 0x41, 0x01), marker = true),
)
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
@Test
fun drops_timestamp_change_without_marker() {
val depacketizer = H264Depacketizer()
val first = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01)))
assertNull(first.accessUnit)
// The stale frame is reported dropped, but this packet starts (and
// closes) the next frame — matching the C++ depacketizer.
val result = depacketizer.depacketize(packet(2, 200, byteArrayOf(0x41, 0x02), marker = true))
assertTrue(result.accessUnit?.contentEquals(START + byteArrayOf(0x41, 0x02)) == true)
assertTrue(result.frameDropped)
}
@Test
fun keyframe_detection_requires_parameter_sets() {
val depacketizer = H264Depacketizer()
val plain = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x41, 0x01), marker = true))
assertFalse(plain.isKeyFrame)
val depacketizer2 = H264Depacketizer()
val withSps = depacketizer2.depacketize(
packet(1, 100, byteArrayOf(0x67, 0xAA.toByte(), 0x88.toByte(), 0x68, 0xBB.toByte(), 0x41, 0x01), marker = true),
)
assertTrue(withSps.isKeyFrame)
}
@Test
fun drops_short_fu_packets() {
val depacketizer = H264Depacketizer()
// FU-A packet without its FU header byte.
val result = depacketizer.depacketize(packet(1, 100, byteArrayOf(0x7C.toByte()), marker = true))
assertNull(result.accessUnit)
assertTrue(result.frameDropped)
}
}
@@ -0,0 +1,65 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
class JitterBufferTest {
private fun packet(seq: Int, ts: Int = 100) =
RtpPacket(RtpHeader(sequenceNumber = seq, timestamp = ts), byteArrayOf(seq.toByte()))
@Test
fun in_order_releases_immediately() {
val jitter = JitterBuffer()
assertEquals(listOf(1), jitter.push(packet(1)).map { it.header.sequenceNumber })
assertEquals(listOf(2), jitter.push(packet(2)).map { it.header.sequenceNumber })
assertEquals(listOf(3), jitter.push(packet(3)).map { it.header.sequenceNumber })
}
@Test
fun reorders_out_of_order_packets() {
val jitter = JitterBuffer()
assertEquals(listOf(1), jitter.push(packet(1)).map { it.header.sequenceNumber })
assertTrue(jitter.push(packet(3)).isEmpty())
val released = jitter.push(packet(2)).map { it.header.sequenceNumber }
assertEquals(listOf(2, 3), released)
}
@Test
fun overflow_releases_in_order_and_advances() {
val jitter = JitterBuffer(maxDepth = 4)
assertTrue(jitter.push(packet(1)).map { it.header.sequenceNumber } == listOf(1))
// seq 2 is lost; 3..6 stay buffered (within the depth bound).
for (seq in 3..6) {
assertTrue(jitter.push(packet(seq)).isEmpty())
}
// seq 7 overflows the buffer: 3..7 flush in order.
assertEquals(listOf(3, 4, 5, 6, 7), jitter.push(packet(7)).map { it.header.sequenceNumber })
// Delivery continues in order afterwards.
assertEquals(listOf(8), jitter.push(packet(8)).map { it.header.sequenceNumber })
assertEquals(listOf(9), jitter.push(packet(9)).map { it.header.sequenceNumber })
}
@Test
fun discards_stragglers() {
val jitter = JitterBuffer(maxDepth = 4)
jitter.push(packet(1))
for (seq in 3..8) {
jitter.push(packet(seq))
}
assertTrue(jitter.push(packet(9)).isNotEmpty())
// seq 4 is now far behind the expected sequence: discarded, not delivered.
assertTrue(jitter.push(packet(4)).isEmpty())
// In-order delivery continues from 10.
assertEquals(listOf(10), jitter.push(packet(10)).map { it.header.sequenceNumber })
}
@Test
fun clear_resets_state() {
val jitter = JitterBuffer()
jitter.push(packet(5))
jitter.clear()
// A completely different sequence now starts fresh.
assertEquals(listOf(100), jitter.push(packet(100)).map { it.header.sequenceNumber })
}
}
@@ -0,0 +1,112 @@
package screen_cast.rtp
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
class RtpHeaderTest {
@Test
fun roundtrip() {
val header = RtpHeader(
version = 2,
padding = false,
extension = false,
csrcCount = 0,
marker = true,
payloadType = 96,
sequenceNumber = 0xABCD,
timestamp = 0xDEADBEEF.toInt(),
ssrc = 0x12345678,
)
val wire = header.serialize()
assertEquals(12, wire.size)
val parsed = RtpHeader.parse(wire)
assertEquals(header, parsed)
}
@Test
fun rejects_bad_version() {
val wire = RtpHeader().serialize().copyOf()
wire[0] = (wire[0].toInt() and 0x3F or (1 shl 6)).toByte()
assertNull(RtpHeader.parse(wire))
}
@Test
fun rejects_short_input() {
val header = RtpHeader()
assertNull(RtpHeader.parse(header.serialize().copyOfRange(0, 11)))
assertNull(RtpHeader.parse(ByteArray(0)))
}
@Test
fun preserves_flags() {
val header = RtpHeader(padding = true, csrcCount = 2, marker = true, payloadType = 63)
val parsed = RtpHeader.parse(header.serialize())
assertEquals(true, parsed?.padding)
assertEquals(2, parsed?.csrcCount)
assertEquals(true, parsed?.marker)
assertEquals(63, parsed?.payloadType)
}
}
class RtpPacketTest {
private fun header(seq: Int, marker: Boolean = false) =
RtpHeader(sequenceNumber = seq, payloadType = 96, marker = marker)
@Test
fun roundtrip_with_payload() {
val packet = RtpPacket(header(seq = 7), byteArrayOf(0x11, 0x22, 0x33))
val wire = packet.header.serialize() + packet.payload
val parsed = RtpPacket.parse(wire)
assertEquals(header(seq = 7), parsed?.header)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x11, 0x22, 0x33)) == true)
}
@Test
fun skips_csrc_list() {
val header = RtpHeader(csrcCount = 1, sequenceNumber = 3)
val wire = header.serialize() + byteArrayOf(0x0A, 0x00, 0x00, 0x01) + byteArrayOf(0x99.toByte())
val parsed = RtpPacket.parse(wire)
assertEquals(1, parsed?.header?.csrcCount)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x99.toByte())) == true)
}
@Test
fun skips_extension_header() {
val header = RtpHeader(extension = true, sequenceNumber = 4)
// profile=0x0001, length=1 word, one word of data
val wire = header.serialize() +
byteArrayOf(0x00, 0x01, 0x00, 0x01, 0xDE.toByte(), 0xAD.toByte(), 0xBE.toByte(), 0xEF.toByte()) +
byteArrayOf(0x77)
val parsed = RtpPacket.parse(wire)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x77)) == true)
}
@Test
fun strips_padding() {
val header = RtpHeader(padding = true, sequenceNumber = 5)
// Payload byte, one padding zero, size byte (2 = padding incl. itself).
val wire = header.serialize() + byteArrayOf(0x55, 0x00, 0x02)
val parsed = RtpPacket.parse(wire)
assertTrue(parsed?.payload?.contentEquals(byteArrayOf(0x55)) == true)
}
@Test
fun rejects_truncated_csrc_and_extension() {
val csrc = RtpHeader(csrcCount = 1).serialize()
assertNull(RtpPacket.parse(csrc)) // 12 bytes, needs 16
val ext = RtpHeader(extension = true).serialize() + byteArrayOf(0x00, 0x01)
assertNull(RtpPacket.parse(ext)) // extension length field cut off
}
@Test
fun rejects_bad_padding() {
val zeroPad = RtpHeader(padding = true).serialize() + byteArrayOf(0x00)
assertNull(RtpPacket.parse(zeroPad))
val oversized = RtpHeader(padding = true).serialize() + byteArrayOf(0x00, 0x00, 0x05)
assertNull(RtpPacket.parse(oversized))
assertNull(RtpPacket.parse(ByteArray(11)))
}
}
+3
View File
@@ -0,0 +1,3 @@
plugins {
id("com.android.application") version "9.4.0" apply false
}
+3
View File
@@ -0,0 +1,3 @@
org.gradle.jvmargs=-Xmx2g -Dfile.encoding=UTF-8
org.gradle.caching=true
kotlin.code.style=official
+17
View File
@@ -0,0 +1,17 @@
pluginManagement {
repositories {
google()
mavenCentral()
gradlePluginPortal()
}
}
dependencyResolutionManagement {
repositories {
google()
mavenCentral()
}
}
rootProject.name = "screen_cast-android"
include(":app")
+58 -1
View File
@@ -102,6 +102,63 @@ where available. PLI + jitter validated on the desktop (induced-loss probe);
hardware decode validated as clean-fallback on the desktop, hardware path hardware decode validated as clean-fallback on the desktop, hardware path
pending a run on the Pi. pending a run on the Pi.
## Phase 8 — Android Receiver App
**Goal**: a native Android receiver app (Kotlin) so a phone can act as the
second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the
existing signaling + RTP protocol; no C++ changes.
- [x] 8.1 Gradle project builds (`android/`, AGP 9 built-in Kotlin, no
androidx; `gradle :app:assembleDebug`)
- [x] 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer,
depacketizer — 25 tests green)
- [x] 8.3 Signaling + NSD validated on device (offer → answer over the
network; mDNS registration via the API-36 RegistrationListener)
**correction 2026-09-10**: registration had in fact never
succeeded (registerService passed protocol `0`, which API 36
rejects with "Unsupported protocol"; the swallowed failure was
hidden behind the "Listening…" status). Fixed with
`PROTOCOL_DNS_SD` + real on-device validation: desktop
`--discover` lists the phone and a live `--send` session
decodes and renders.
- [x] 8.4 MediaCodec decode + Surface render validated on device (in-band
SPS sizing, letterbox fit)
- [x] 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed
fullscreen render, PLI keyframe recovery on Wi-Fi loss
- [x] 8.6 Docs: README receiver section, RUNBOOK build/install/test,
PHASES + MEMORY updates
**Validation**: `gradle :app:testDebugUnitTest` green; live session
`--send --target window --peer <phone>:5005` rendered fullscreen on the
phone (and HDMI via DP-alt-mode) with loss recovery.
## Phase 9 — iOS Receiver App
**Goal**: a native iOS receiver app (Swift) so an iPhone can act as the second
receiver. The app speaks the existing signaling + RTP protocol; no C++ changes.
Mirrors the Android receiver (Phase 8) source-to-source.
- [x] 9.1 Xcode project scaffolding (`ios/`, XcodeGen spec + bootstrap script;
Info.plist with local-network + Bonjour privacy keys)
- [x] 9.2 Swift protocol core + XCTest (rtp header/packet, jitter, depacketizer
ported source-to-source; added signaling JSON, line framing, AVCC, NAL
extraction tests — the Android side had no signaling tests)
- [x] 9.3 Signaling server (BSD sockets, dual-stack, most-recent-peer, never
throws) + `NSBonjourServices` advertisement
- [x] 9.4 Media path: UDP (poll-based) → jitter → depacketize → VideoToolbox
(in-band SPS/PPS, real-time decode) → AVSampleBufferDisplayLayer
(letterbox) + PLI on damage + pendingOffer for late surface attach
- [ ] 9.5 On-device validation (needs a Mac + Xcode 26 + iPhone 16): unit tests
green on the simulator; `--discover` lists the phone; `--send --peer
<ip>:5005` streams letterboxed; PLI recovery on Wi-Fi loss
- [ ] 9.6 Docs: iOS RUNBOOK quirks, README receiver section, PHASES + MEMORY
**Validation**: `ios/bootstrap.sh test` (simulator) green; live session
`--send --peer <iphone>:5005` renders fullscreen letterboxed on the iPhone with
loss recovery. Note: this box is Linux — no Xcode/iOS SDK — so 9.29.4 are
authored but only 9.5 can validate the VideoToolbox + local-network paths.
## Current phase ## Current phase
Phase 7Resilience and Polish. Phase 9iOS Receiver App (implementation authored; on-device validation
pending a Mac + Xcode 26 + iPhone 16).
+81 -5
View File
@@ -146,11 +146,24 @@ screencast waybar --toggle # stop a running sender / restart the last
``` ```
The waybar module (installed to this machine's `~/.config/waybar/` — see The waybar module (installed to this machine's `~/.config/waybar/` — see
`custom/screencast` there): `▶` while streaming (tooltip: receiver, bitrate, `custom/screencast` there): the app icon as the module face (dimmed while
elapsed), `⏸` while idle; left-click toggles streaming to the last receiver idle, green tint while streaming; the `▶`/`⏸` text from the JSON is hidden
via the state file, right-click opens the panel. The sender publishes its in the CSS), tooltip with receiver, bitrate, elapsed; left-click toggles
state to `$XDG_RUNTIME_DIR/screencast/sender.json`, so every front-end — streaming to the last receiver via the state file, right-click opens the
CLI, GUI, widget — agrees on what is running. panel. The sender publishes its state to `$XDG_RUNTIME_DIR/screencast/sender.json`,
so every front-end — CLI, GUI, widget — agrees on what is running.
**Icons**: `screencast_icon/screencast_256.png` is the single app icon.
`meson install` puts it in `share/icons/hicolor/256x256/apps/screencast.png`;
the GTK panel uses it by themed name (`set_icon_name` — GTK4 removed the
pixel-buffer window icon, so themed icons only), and the waybar CSS
references the installed path. The SDL receiver window embeds the PNG at
build time (`icon_png_data.h` generated by `scripts/icon_to_header.py`) and
sets it via `SDL_SetWindowIcon` when SDL3_image is available (optional
dependency; no pkg-config ships with it, so meson finds the library
directly). Icons surface in compositor taskbars/switchers, not in title
bars (neither GTK4's CSD nor Hyprland's decorations draw them).
Restart waybar after changing `style.css` to pick up icon changes.
## Under the hood: resilience (Phase 7) ## Under the hood: resilience (Phase 7)
@@ -196,6 +209,69 @@ The headless equivalent runs as part of `meson test` (`udp loopback` test):
synthetic frames → encode → packetize → localhost UDP → depacketize → synthetic frames → encode → packetize → localhost UDP → depacketize →
decode, no portal or window involved. decode, no portal or window involved.
## Android receiver app (Phase 8)
Toolchain: JDK 21, system `gradle` (no wrapper), Android SDK with android-36.
AGP 9 has **built-in Kotlin** — do not apply `org.jetbrains.kotlin.android`
and do not use `kotlinOptions {}`.
```sh
cd android
gradle :app:assembleDebug # -> app/build/outputs/apk/debug/app-debug.apk
gradle :app:testDebugUnitTest # 25 JVM tests (rtp, jitter, depacketizer)
```
Device test (validated on a Fairphone 6, Android 16 / API 36):
```sh
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n screen_cast.receiver/screen_cast.ReceiverActivity
# activity FQN is screen_cast.ReceiverActivity (namespace), not
# screen_cast.receiver.ReceiverActivity
adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml # status text
adb exec-out screencap -p > phone.png # visual check
```
On the sender: `screencast --send --peer <phone-ip>:5005` (add `--target
window` to pick a specific window; the portal's monitor source picks the
first output on multi-monitor Hyprland/GTK-portal setups).
Platform quirks found while validating (Android 16 / API 36):
- `android.permission.INTERNET` is required — NsdService rejects
registration without it.
- `NsdManager.registerService` on Android 16 validates the protocol
argument (`NsdManager.checkProtocol`): the historical `0` throws
`IllegalArgumentException: Unsupported protocol` — pass
`NsdManager.PROTOCOL_DNS_SD`. This was silent for a long time: the app
swallowed the exception and the failure status was overwritten by the
"Listening…" line, so mDNS never advertised even though the UI looked
fine (discovery only worked via `--peer`). Diagnosed via
`adb shell dumpsys servicediscovery` (the client's `mClientRequests`
stays empty when no request was ever issued) plus logging the
exception.
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
(Inet6Address on Android), not an `InetSocketAddress`.
- `MediaCodec`: pass the output Surface to `configure()` — the codec must
start in surface mode (`setOutputSurface` afterwards only switches an
already-surface-mode codec to a new surface; a codec configured without
a surface can never take one, per the API docs); call `start()`; render
with `releaseOutputBuffer(index, render=true)`; the C2 AVC decoder
requires a concrete width/height at configure (use a placeholder — the
in-band SPS reconfigures it).
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
surface.
- NSD: the classic `registerService(info, flags, RegistrationListener)`
API exists from API 16 through 36 (verified with javap on the android-36
SDK jar) — no fallback is needed. A reflection fallback targeting a
pre-36 "ResolutionListener" was removed: that class never existed at
any API level, so the fallback could only ever fail.
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
— letterbox by **sizing the TextureView to the video aspect** (centered
on a black window background), not with a transform matrix (double scale).
- mDNS does not cross subnets; `--peer` is the reliable path.
## Formatting ## Formatting
```sh ```sh
+3 -1
View File
@@ -10,7 +10,9 @@ namespace sc {
struct SendCommand { struct SendCommand {
std::string_view target = "monitor"; // monitor, window std::string_view target = "monitor"; // monitor, window
std::string_view peer_address; // optional; empty means auto-discover std::string_view peer_address; // optional; empty means auto-discover
int bitrate_kbps = 4000; int bitrate_kbps = 4000; // VBV max bitrate
int crf = 22; // constant rate factor (quality)
int fps = 0; // 0 = no cap (capture rate)
}; };
struct ReceiveCommand { struct ReceiveCommand {
+9
View File
@@ -22,6 +22,15 @@ struct SenderPipelineConfig {
// Session identity from signaling; written to the sender state file for // Session identity from signaling; written to the sender state file for
// status widgets (waybar) and one-click restarts. // status widgets (waybar) and one-click restarts.
std::string session_id; std::string session_id;
// The receiver's display resolution (0 = unknown). The sender downscales
// to this before encoding so bitrate is not spent on pixels the display
// cannot show.
int max_encode_width = 0;
int max_encode_height = 0;
// Cap the encoding frame rate (0 = no cap; use the capture rate).
// Lowering the frame rate halves the bandwidth at the same quality
// level — useful on constrained links.
int max_frame_rate = 0;
}; };
struct ReceiverPipelineConfig { struct ReceiverPipelineConfig {
+11 -1
View File
@@ -10,11 +10,21 @@
namespace sc { namespace sc {
// Decoded video frame in YUV420P planar format (the native decoder output,
// passed to the renderer without any colorspace conversion).
struct DecodedFrame { struct DecodedFrame {
int width = 0; int width = 0;
int height = 0; int height = 0;
uint64_t capture_timestamp_ns = 0; uint64_t capture_timestamp_ns = 0;
std::vector<std::byte> rgba_pixels;
// YUV420P planes; each row is `stride` bytes wide and may be padded
// beyond the picture width.
std::vector<std::byte> plane_y;
std::vector<std::byte> plane_u;
std::vector<std::byte> plane_v;
int stride_y = 0;
int stride_u = 0;
int stride_v = 0;
}; };
struct DecoderConfig { struct DecoderConfig {
+6
View File
@@ -23,6 +23,12 @@ struct EncoderConfig {
int height = 0; int height = 0;
int frame_rate_num = 30; int frame_rate_num = 30;
int frame_rate_den = 1; int frame_rate_den = 1;
// CRF (constant rate factor, 0-51): the target visual quality level.
// Lower = better quality. 23 is x264's default; screen content looks
// good at 20-24.
int crf = 22;
// Maximum bitrate in kbps (the VBV cap). The encoder uses fewer bits on
// static content and more on motion, but never exceeds this.
int bitrate_kbps = 4000; int bitrate_kbps = 4000;
bool hardware_accel = false; bool hardware_accel = false;
}; };
+4
View File
@@ -29,6 +29,10 @@ struct SessionAnswer {
// The address may be empty: the sender then targets the address of its // The address may be empty: the sender then targets the address of its
// signaling connection and the port carried here. // signaling connection and the port carried here.
Endpoint rtp_endpoint; Endpoint rtp_endpoint;
// The receiver's display resolution (0 = unknown). The sender may
// downscale to this to avoid encoding pixels the display cannot show.
int display_width = 0;
int display_height = 0;
}; };
// Picture Loss Indication: the receiver asks the sender for a keyframe // Picture Loss Indication: the receiver asks the sender for a keyframe
+5
View File
@@ -24,6 +24,11 @@ class Renderer {
// Present one decoded frame. Returns false if the window was closed. // Present one decoded frame. Returns false if the window was closed.
virtual bool present(const DecodedFrame& frame) = 0; virtual bool present(const DecodedFrame& frame) = 0;
// The display resolution (native monitor size in fullscreen mode).
// Used by the receiver to tell the sender what resolution to encode at.
virtual int display_width() const = 0;
virtual int display_height() const = 0;
// Pump events (window close, resize). Non-blocking. // Pump events (window close, resize). Non-blocking.
virtual bool poll_events() = 0; virtual bool poll_events() = 0;
+77
View File
@@ -0,0 +1,77 @@
# screencast iOS receiver
A native iOS receiver app (Swift) so an iPhone can act as a second receiver.
It speaks the **existing signaling + RTP protocol unchanged** — no C++ changes —
mirroring the Android receiver (Phase 8) source-to-source.
- **Min iOS:** 17 (validated on iPhone 16 / iOS 26.6.1)
- **No third-party dependencies** — only Apple system frameworks (SwiftUI,
AVFoundation, VideoToolbox, CoreMedia, BSD sockets).
## Pipeline
```
Bonjour advertise (_screencast._tcp) + TCP signaling server (offer → answer)
UDP RTP → jitter buffer (16 pkt / 60 ms) → depacketize (single-NAL + FU-A)
→ VideoToolbox H.264 (in-band SPS/PPS) → AVSampleBufferDisplayLayer (letterbox)
```
Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
(rate-limited to one per 500 ms) asks the sender for a keyframe.
## Layout
```
Receiver/
App/ SwiftUI app, video surface (AVSampleBufferDisplayLayer), controller
Rtp/ RtpHeader, RtpPacket, JitterBuffer, H264Depacketizer, Avcc, NAL extraction
Signaling/ SignalingMessage (JSON), LineAssembler, SignalingServer (BSD sockets)
Decode/ RenderSink, H.264 format description, VideoToolbox decoder
Support/ UdpTransport (poll-based), LocalAddress
Pipeline/ ReceiverPipeline (coordinates everything)
ReceiverTests/
XCTest port of the Android JVM tests + added coverage (signaling, line
framing, AVCC, NAL extraction)
```
## Build and test (on a Mac with Xcode 26)
```sh
./bootstrap.sh # installs XcodeGen, generates the project, builds for device
./bootstrap.sh test # runs the unit tests on the simulator
```
`bootstrap.sh` downloads XcodeGen from the GitHub release into `tools/` (no
Homebrew). Override the simulator with `IOS_DEST="platform=iOS Simulator,name=…"`.
## Install on a device
Free provisioning (or your team) is set in Xcode — this can't be scripted:
1. Open `Receiver.xcodeproj`.
2. Select the **Receiver** target → *Signing & Capabilities* → set your Apple ID
(a 7-day development certificate is fine for sideloading).
3. Run to the iPhone (USB, trust the computer).
The first run prompts for **Local Network** access — allow it, or the sender
will never discover the phone (silent failure, like the Android
`PROTOCOL_DNS_SD` bug).
## Stream to the phone
On the sender (Linux desktop):
```sh
screencast --send --peer <phone-ip>:5005 # target the phone directly
screencast --discover # or list receivers; the phone shows up
```
Expected: the phone shows the captured desktop, letterboxed to its screen, with
PLI recovery on Wi-Fi loss.
## Testing notes
The pure protocol core (RTP, jitter, depacketizer, signaling JSON, line framing,
AVCC, NAL extraction) is unit-tested in the simulator. The VideoToolbox decode
path and the local-network/Bonjour flow are on-device validation items — the
highest-risk parts to verify on the borrowed Mac + iPhone.
+88
View File
@@ -0,0 +1,88 @@
import SwiftUI
import UIKit
/// Owns the pipeline and mirrors its status to the UI. The pipeline posts its
/// callbacks to the main thread, so the @Published mutations happen there.
final class ReceiverController: ObservableObject {
@Published var status = "Starting…"
@Published var showStatus = true
private let pipeline: ReceiverPipeline
private let pixelSize: CGSize
init() {
let localIP = LocalAddress.primaryIPv4()
let size = ReceiverController.screenPixelSize()
pixelSize = size
pipeline = ReceiverPipeline(
localIP: localIP,
displaySize: { [weak self] in self?.pixelSize ?? .zero },
onStatus: { [weak self] text in self?.apply(status: text) },
onFirstFrame: { [weak self] in self?.apply(showStatus: false) },
onVideoSize: { _ in })
}
func start() { pipeline.start() }
func stop() { pipeline.stop() }
func bindSink(_ sink: RenderSink) { pipeline.attachSink(sink) }
private func apply(status: String? = nil, showStatus: Bool? = nil) {
if let s = status { self.status = s }
if let v = showStatus { self.showStatus = v }
}
static func screenPixelSize() -> CGSize {
let scale = UIScreen.main.scale
let b = UIScreen.main.bounds
return CGSize(width: b.width * scale, height: b.height * scale)
}
}
struct ContentView: View {
@StateObject private var controller = ReceiverController()
@Environment(\.scenePhase) private var scenePhase
var body: some View {
ZStack {
Color.black.ignoresSafeArea()
VideoSurfaceView { sink in controller.bindSink(sink) }
.ignoresSafeArea()
if controller.showStatus {
VStack {
Spacer()
Text(controller.status)
.font(.footnote)
.foregroundStyle(.white)
.multilineTextAlignment(.center)
.padding(.horizontal, 24)
.padding(.vertical, 12)
.background(.black.opacity(0.55), in: RoundedRectangle(cornerRadius: 10))
Spacer()
Spacer()
}
.transition(.opacity)
}
}
.onAppear { controller.start() }
.onChange(of: scenePhase) { _, phase in
switch phase {
case .active: controller.start()
case .inactive, .background: controller.stop()
@unknown default: break
}
}
}
}
@main
struct ReceiverApp: App {
var body: some Scene {
WindowGroup {
ContentView()
.preferredColorScheme(.dark)
.statusBarHidden(true)
.persistentSystemOverlays(.hidden)
}
}
}
+34
View File
@@ -0,0 +1,34 @@
import SwiftUI
import AVFoundation
/// Hosts the `AVSampleBufferDisplayLayer` and exposes it as a `RenderSink`.
/// The layer keeps the full screen and letterboxes via `.resizeAspect`, so the
/// decoded stream (already downscaled to the display size by the sender) is
/// shown 1:1 with no distortion.
struct VideoSurfaceView: UIViewRepresentable {
let onSink: (RenderSink) -> Void
func makeCoordinator() -> Coordinator {
Coordinator()
}
func makeUIView(context: Context) -> UIView {
let view = UIView(frame: .zero)
view.backgroundColor = .black
let layer = AVSampleBufferDisplayLayer()
layer.videoGravity = .resizeAspect
view.layer.addSublayer(layer)
let sink = AVSampleBufferRenderSink(layer: layer)
context.coordinator.sink = sink
onSink(sink)
return view
}
func updateUIView(_ uiView: UIView, context: Context) {}
final class Coordinator {
var sink: AVSampleBufferRenderSink?
}
}
@@ -0,0 +1,57 @@
import CoreMedia
import Foundation
/// Builds a `CMVideoFormatDescription` for H.264 from in-band SPS + PPS.
///
/// VideoToolbox needs the parameter sets up front; the sender repeats them
/// before every keyframe, so any keyframe carries a complete set. This is the
/// Core Foundation recipe for an H.264 "config" format description.
enum H264FormatDescription {
static func create(sps: [UInt8], pps: [UInt8]) -> CMVideoFormatDescription? {
let pointersArray = CFArrayCreateMutable(kCFAllocatorDefault, 0, &kCFTypeArrayCallBacks)
for ps in [sps, pps] {
guard let descriptor = makeParameterSetDescriptor(ps) else {
CFRelease(pointersArray)
return nil
}
CFArrayAppendValue(pointersArray, descriptor.takeUnretainedValue())
CFRelease(descriptor) // the array now owns it
}
let key = kCMFormatDescriptionExtension_SampleDescriptionPointers as CFString
let attrs = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFDictionaryKeyCallBacks, &kCFDictionaryValueCallBacks)
CFDictionarySetValue(attrs, key, pointersArray)
var config: Unmanaged<CMVideoFormatDescription>?
let status = CMVideoFormatDescriptionCreate(
kCFAllocatorDefault,
kCMVideoCodecType_H264,
0, 0, 0,
attrs,
&config)
CFRelease(attrs)
CFRelease(pointersArray)
guard status == noErr, let c = config else { return nil }
return c.takeRetainedValue()
}
private static func makeParameterSetDescriptor(_ ps: [UInt8]) -> Unmanaged<CMVideoFormatDescription>? {
let cfData = Data(ps) as CFData
let oneElement = CFArrayCreateMutable(kCFAllocatorDefault, 0, &kCFTypeArrayCallBacks)
CFArrayAppendValue(oneElement, cfData)
let key = kCMFormatDescriptionExtension_SampleDescriptionPointers as CFString
let attrs = CFDictionaryCreateMutable(kCFAllocatorDefault, 0, &kCFDictionaryKeyCallBacks, &kCFDictionaryValueCallBacks)
CFDictionarySetValue(attrs, key, oneElement)
var desc: Unmanaged<CMVideoFormatDescription>?
let status = CMVideoFormatDescriptionCreateForCodecType(
kCFAllocatorDefault,
kCMVideoCodecType_H264,
attrs,
&desc)
CFRelease(oneElement)
CFRelease(attrs)
guard status == noErr, let d = desc else { return nil }
return d
}
}
@@ -0,0 +1,174 @@
import VideoToolbox
import CoreMedia
import CoreGraphics
import Foundation
/// Hardware H.264 decoder using VideoToolbox. The stream is self-describing:
/// the sender repeats SPS/PPS in-band at every keyframe, so no out-of-band
/// codec data is needed. On a keyframe the in-band SPS/PPS (re)establish the
/// stream size; when that size changes, the decode session is recreated the
/// Android C2 "in-band SPS reconfigure" pattern, which also avoids the startup
/// squish (we never report a placeholder size before the first real keyframe).
///
/// The decode output callback may run on a worker thread, so session and
/// format-description access is guarded by a lock.
final class H264VideoToolboxDecoder {
private let renderSink: RenderSink
private let stateLock = NSLock()
private var session: VTDecompressionSession?
private var formatDescription: CMVideoFormatDescription?
private var streamSize = CGSize.zero
private var realSizeSeen = false
init(renderSink: RenderSink) {
self.renderSink = renderSink
}
/// The decoded resolution, once the first keyframe configured the session
/// (nil before that the caller must not drive layout off a placeholder).
func outputSize() -> CGSize? {
guard realSizeSeen, streamSize != .zero else { return nil }
return streamSize
}
/// Decodes one access unit (Annex-B) and renders the output. Returns false
/// when a decode error occurred and the caller should request a keyframe.
func decode(accessUnit annexB: [UInt8], rtpTimestamp: Int, isKeyFrame: Bool) -> Bool {
if isKeyFrame {
guard let sets = NalExtractor.parameterSets(annexB) else { return false }
configureIfNeeded(sps: sets.sps, pps: sets.pps)
}
stateLock.lock()
let session = self.session
let cd = self.formatDescription
stateLock.unlock()
guard let session, let cd else { return false }
guard let avcc = AvccConverter.toAvcc(annexB) else { return false }
let pts = CMTime(value: CMTimeValue(rtpTimestamp), timescale: 90000)
guard let blockBuffer = makeBlockBuffer(avcc) else { return false }
var infoFlags: VTDecodeInfoFlags = []
let status = VTDecompressionSessionDecodeFrame(
session,
blockBuffer,
isKeyFrame ? kVTDecodeFrameFlags_EnableFastPath : 0,
&infoFlags,
pts)
CFRelease(blockBuffer)
if status != noErr {
// Recoverable: the next keyframe (SPS/PPS + IDR) re-primes it.
teardownSession()
return false
}
realSizeSeen = true
return true
}
func release() {
stateLock.lock()
teardownSessionLocked()
if let cd = formatDescription { CFRelease(cd) }
formatDescription = nil
stateLock.unlock()
renderSink.detach()
streamSize = .zero
realSizeSeen = false
}
// MARK: - Internals
private func configureIfNeeded(sps: [UInt8], pps: [UInt8]) {
guard let cd = H264FormatDescription.create(sps: sps, pps: pps) else { return }
var dims = CMVideoDimensions()
guard CMVideoFormatDescriptionGetDimensions(cd, &dims) == noErr else {
CFRelease(cd)
return
}
let size = CGSize(width: CGFloat(dims.width), height: CGFloat(dims.height))
stateLock.lock()
// Same size and a live session: keep it (the sender only changes the
// stream when the receiver's display size changes).
if session != nil && size == streamSize {
CFRelease(cd)
stateLock.unlock()
return
}
teardownSessionLocked()
var outSession: VTDecompressionSession?
let status = VTDecompressionSessionCreate(
kCFAllocatorDefault,
cd,
vtOutputCallback,
Unmanaged.passUnretained(self).toOpaque(),
&outSession)
guard status == noErr, let newSession = outSession else {
CFRelease(cd)
stateLock.unlock()
return
}
// Low-latency decode: emit as soon as the frame is complete.
VTSessionSetProperty(newSession, kVTDecompressionPropertyKey_RealTime, kCFBooleanTrue)
renderSink.setFormatDescription(cd)
formatDescription = cd // we hold the +1 from H264FormatDescription.create
session = newSession
streamSize = size
realSizeSeen = false
stateLock.unlock()
}
// Caller holds stateLock.
private func teardownSessionLocked() {
if let s = session {
VTDecompressionSessionInvalidate(s)
CFRelease(s)
}
session = nil
}
// Caller does not hold the lock.
private func teardownSession() {
stateLock.lock()
teardownSessionLocked()
stateLock.unlock()
}
private func makeBlockBuffer(_ bytes: [UInt8]) -> CMBlockBuffer? {
let cfData = Data(bytes) as CFData
var blockBuffer: CMBlockBuffer?
let status = CMBlockBufferCreateWithData(kCFAllocatorDefault, cfData, &blockBuffer)
guard status == noErr, let bb = blockBuffer else { return nil }
return bb
}
// Runs on whatever thread VideoToolbox uses for the output callback.
private func handleOutput(_ pixelBuffer: CVPixelBuffer?, _ presentationTime: CMTime?) {
stateLock.lock()
let cd = formatDescription
stateLock.unlock()
guard let cd, let pixelBuffer else { return }
let pts = presentationTime ?? CMTime(value: 0, timescale: 600)
var sampleBuffer: CMSampleBuffer?
let status = CMSampleBufferCreate(kCFAllocatorDefault, nil, 0, pts, .invalid, 1, 0, nil, &sampleBuffer)
guard status == noErr, let sb = sampleBuffer else { return }
guard CMSampleBufferSetDataBufferFromPixelBuffer(sb, pixelBuffer) == noErr else {
CFRelease(sb)
return
}
renderSink.enqueue(sb)
}
}
/// C-compatible decode output callback; recovers the decoder from the refCon.
private func vtOutputCallback(_ refCon: UnsafeMutableRawPointer?,
_ pixelBuffer: CVPixelBuffer?,
_ presentationTime: CMTime?,
_ duration: CMTime?,
_ infoFlags: VTDecodeInfoFlags) {
guard let refCon = refCon else { return }
let decoder = Unmanaged<H264VideoToolboxDecoder>.fromOpaque(refCon).takeUnretainedValue()
decoder.handleOutput(pixelBuffer, presentationTime: presentationTime)
}
+61
View File
@@ -0,0 +1,61 @@
import AVFoundation
import CoreMedia
/// A render target for decoded frames. The decoder enqueues a `CMSampleBuffer`
/// (wrapping a CVPixelBuffer) per frame; the sink renders it. Mirrors the role
/// of the Android `Surface` + MediaCodec surface-mode rendering.
protocol RenderSink: AnyObject {
var isAttached: Bool { get }
func attach()
func detach()
/// Updates the layer's video format description (called when the in-band
/// SPS/PPS establish a (new) stream size).
func setFormatDescription(_ formatDescription: CMVideoFormatDescription)
/// Enqueues one decoded frame for display.
func enqueue(_ sampleBuffer: CMSampleBuffer)
}
/// Renders decoded CVPixelBuffers via `AVSampleBufferDisplayLayer`, which does
/// the video-scaling for us. `.resizeAspect` gives letterbox directly, so the
/// host view can stay full-screen without a manual transform (the same lesson
/// as the Android TextureView sizing: size the surface to the aspect, not a
/// transform matrix).
final class AVSampleBufferRenderSink: RenderSink {
private let layer: AVSampleBufferDisplayLayer
private var session: AVSampleBufferDisplayLayerSession?
private let lock = NSLock()
private(set) var isAttached = false
init(layer: AVSampleBufferDisplayLayer) {
self.layer = layer
}
func attach() {
lock.lock()
defer { lock.unlock() }
guard session == nil else { return }
let s = AVSampleBufferDisplayLayerSession(layer)
s.start()
session = s
isAttached = true
}
func detach() {
lock.lock()
defer { lock.unlock() }
session?.stop()
session = nil
isAttached = false
}
func setFormatDescription(_ formatDescription: CMVideoFormatDescription) {
layer.formatDescription = formatDescription
}
func enqueue(_ sampleBuffer: CMSampleBuffer) {
lock.lock()
let s = session
lock.unlock()
s?.enqueue(sampleBuffer)
}
}
@@ -0,0 +1,268 @@
import Foundation
import CoreGraphics
/// The receiver pipeline, mirroring the C++ ReceiverPipeline and the Kotlin
/// receiver:
///
/// signaling server (offer answer)
/// UDP RTP jitter buffer depacketize VideoToolbox render sink
///
/// Recovery matches the C++ receiver: a damaged frame is dropped and a PLI
/// (rate-limited to one per 500 ms) asks the sender for a keyframe.
///
/// Concurrency: all shared state and every decode call run on a single serial
/// queue; the reader thread only polls/receives UDP and forwards datagrams to
/// that queue. UI callbacks are marshalled to the main thread.
final class ReceiverPipeline {
static let desiredUdpPort: UInt16 = 5004
static let desiredSignalingPort: UInt16 = 5005
static let pliMinIntervalMs: Int = 500
static let receiveBufferSize: Int32 = 4 * 1024 * 1024
private let localIP: String
private let displaySize: () -> CGSize
private let onStatus: (String) -> Void
private let onFirstFrame: () -> Void
private let onVideoSize: (CGSize) -> Void
private let queue = DispatchQueue(label: "sc.receiver.pipeline")
private var running = false
private var udp: UdpTransport?
private var signaling: SignalingServer?
private var readerThread: Thread?
private let jitter = JitterBuffer()
private var depacketizer = H264Depacketizer()
private var decoder: H264VideoToolboxDecoder?
private var renderSink: RenderSink?
private var pendingOffer: SignalingMessage?
private var activeSession = ""
private var firstFrameSeen = false
private var currentVideoSize = CGSize.zero
private var lastPliAtMs: Int64 = 0
init(localIP: String,
displaySize: @escaping () -> CGSize,
onStatus: @escaping (String) -> Void,
onFirstFrame: @escaping () -> Void,
onVideoSize: @escaping (CGSize) -> Void) {
self.localIP = localIP
self.displaySize = displaySize
self.onStatus = onStatus
self.onFirstFrame = onFirstFrame
self.onVideoSize = onVideoSize
}
/// The current decoded resolution (zero until the first keyframe).
func videoSize() -> CGSize {
return queue.sync { currentVideoSize }
}
// MARK: - Lifecycle
/// Binds the ports, starts the signaling server, and starts reading RTP.
func start() {
queue.async { [weak self] in
guard let self, !self.running else { return }
let udp = UdpTransport()
guard udp.bind(preferredPort: Self.desiredUdpPort, receiveBufferSize: Self.receiveBufferSize) else {
self.postStatus("Failed to bind the media port")
return
}
let signaling = SignalingServer(
onOffer: { [weak self] offer in self?.queue.async { self?.handleOffer(offer) } },
onPli: { _ in })
guard signaling.start(Self.desiredSignalingPort) else {
udp.close()
self.postStatus("Failed to start signaling")
return
}
self.running = true
self.udp = udp
self.signaling = signaling
let thread = Thread { [weak self] in self?.readLoop(udp: udp) }
thread.name = "rtp-reader"
self.readerThread = thread
thread.start()
let ip = self.localIP
let mediaPort = udp.port
let sigPort = signaling.port
self.postStatus("Listening on \(ip) (media :\(mediaPort), signaling :\(sigPort))\nWaiting for a sender… (fall back to: screencast --send --peer \(ip):\(sigPort))")
}
}
/// Points the (current or future) decoder at a render sink. A pending offer
/// (accepted while no sink existed) configures its decoder now and requests
/// a keyframe, since the sender only emits one when asked.
func attachSink(_ sink: RenderSink) {
queue.async { [weak self] in
guard let self else { return }
self.renderSink = sink
sink.attach()
guard let offer = self.pendingOffer else { return }
self.pendingOffer = nil
// The decoder configures on the first keyframe; the sink is already
// attached, so creation cannot fail. A late-configured decoder needs
// a keyframe (the sender only emits one when asked).
self.decoder = H264VideoToolboxDecoder(renderSink: sink)
self.requestPli()
}
}
/// Forgets a destroyed render sink so a later offer cannot render into it.
func detachSink() {
queue.async { [weak self] in
guard let self else { return }
self.decoder?.release()
self.decoder = nil
self.renderSink?.detach()
self.renderSink = nil
}
}
/// Stops listening; the pipeline can be started again.
func stop() {
queue.async { [weak self] in
guard let self, self.running else { return }
self.running = false
self.activeSession = ""
self.udp?.close()
self.udp = nil
self.readerThread?.join()
self.readerThread = nil
self.signaling?.close()
self.signaling = nil
self.decoder?.release()
self.decoder = nil
self.pendingOffer = nil
self.firstFrameSeen = false
self.currentVideoSize = .zero
self.postStatus("Stopped")
}
}
// MARK: - Media path
private func readLoop(udp: UdpTransport) {
while true {
switch udp.poll(timeoutMs: 100) {
case 1:
if let data = udp.receiveDatagram() {
queue.async { [weak self] in self?.processDatagram(data) }
}
case 0:
continue // timeout: re-check on the next poll
default:
return // closed/errored: the socket was closed by stop()
}
}
}
private func processDatagram(_ data: [UInt8]) {
guard let packet = RtpPacket.parse(data) else { return }
for released in jitter.push(packet) {
handleDepacketized(released)
}
}
private func handleDepacketized(_ packet: RtpPacket) {
let result = depacketizer.depacketize(packet)
if let accessUnit = result.accessUnit {
let presentation = Int(packet.header.timestamp & 0xFFFFFFFF)
guard let decoder = decoder else { return }
if !decoder.decode(accessUnit: accessUnit, rtpTimestamp: presentation, isKeyFrame: result.isKeyFrame) {
// Input/decode trouble: the dropped frame corrupts the GOP
// until the next keyframe ask for one.
requestPli()
}
if !firstFrameSeen {
firstFrameSeen = true
postFirstFrame()
}
postVideoSizeIfChanged()
}
if result.frameDropped {
requestPli()
}
}
private func postVideoSizeIfChanged() {
guard let size = decoder?.outputSize(), size != .zero else { return }
if size != currentVideoSize {
currentVideoSize = size
postVideoSize(size)
}
}
// MARK: - Signaling
private func handleOffer(_ offer: SignalingMessage) {
guard case let .offer(sessionId, codec, _, _, _, _, _, _) = offer else { return }
if codec != "h264" {
postStatus("Unsupported codec: \(codec)")
return
}
// New session: pristine decoder, reassembly state, and session.
decoder?.release()
decoder = nil
pendingOffer = nil
if renderSink != nil {
decoder = H264VideoToolboxDecoder(renderSink: renderSink!)
} else {
// No surface yet: park the offer; attachSink configures later.
pendingOffer = offer
}
depacketizer = H264Depacketizer()
jitter.clear()
firstFrameSeen = false
currentVideoSize = .zero
activeSession = sessionId
let size = displaySize()
let answer = SignalingMessage.answer(
sessionId: sessionId,
rtpAddress: "", // the sender targets the address of its own signaling connection
rtpPort: Int(udp?.port ?? 0),
displayWidth: Int(size.width),
displayHeight: Int(size.height))
signaling?.send(answer)
if decoder != nil {
postStatus("Session \(sessionId) negotiated — waiting for the first frame…")
} else {
postStatus("Session \(sessionId) negotiated — waiting for the display surface…")
}
}
/// Rate-limited keyframe request, callable from any thread (it always runs
/// on the pipeline queue in practice).
private func requestPli() {
let session = activeSession
guard !session.isEmpty else { return }
let now = Int64(Date().timeIntervalSince1970 * 1000)
if now - lastPliAtMs < Int64(Self.pliMinIntervalMs) { return }
lastPliAtMs = now
signaling?.send(.pli(sessionId: session))
}
// MARK: - UI callbacks (main thread)
private func postStatus(_ text: String) {
DispatchQueue.main.async { [onStatus] in onStatus(text) }
}
private func postFirstFrame() {
DispatchQueue.main.async { [onFirstFrame] in onFirstFrame() }
}
private func postVideoSize(_ size: CGSize) {
DispatchQueue.main.async { [onVideoSize] in onVideoSize(size) }
}
}
+73
View File
@@ -0,0 +1,73 @@
import Foundation
/// Converts between Annex-B (start-code delimited) and AVCC (4-byte
/// big-endian length prefixed) H.264 representations.
///
/// VideoToolbox consumes AVCC: each NAL unit is preceded by a 32-bit length.
/// Our depacketizer emits Annex-B (the project's canonical 3-byte start codes),
/// so the decoder feeds AVCC derived here.
enum AvccConverter {
/// Splits an Annex-B access unit into its NAL units (start codes removed).
static func nalUnits(_ annexB: [UInt8]) -> [[UInt8]] {
guard annexB.count >= 4 else { return [] }
let n = annexB.count
// Locate every start code (3-byte `00 00 01` and 4-byte `00 00 00 01`).
var offsets: [Int] = []
var i = 0
while i + 2 < n {
if annexB[i] == 0 && annexB[i + 1] == 0 && annexB[i + 2] == 1 {
offsets.append(i)
i += 3
continue
}
if i + 3 < n, annexB[i + 2] == 0, annexB[i + 3] == 1 {
offsets.append(i)
i += 4
continue
}
i += 1
}
guard !offsets.isEmpty else { return [] }
var units: [[UInt8]] = []
for (idx, offset) in offsets.enumerated() {
let nalStart = offset + 3
let nalEnd = idx + 1 < offsets.count ? offsets[idx + 1] : n
let nal = Array(annexB[nalStart..<nalEnd])
if !nal.isEmpty { units.append(nal) }
}
return units
}
/// Returns the AVCC form of an Annex-B access unit, or nil if it has no NALs.
static func toAvcc(_ annexB: [UInt8]) -> [UInt8]? {
let units = nalUnits(annexB)
guard !units.isEmpty else { return nil }
var out: [UInt8] = []
out.reserveCapacity(annexB.count + units.count * 4)
for nal in units {
let len = nal.count
out.append(UInt8((len >> 24) & 0xFF))
out.append(UInt8((len >> 16) & 0xFF))
out.append(UInt8((len >> 8) & 0xFF))
out.append(UInt8(len & 0xFF))
out.append(contentsOf: nal)
}
return out
}
/// Splits an AVCC byte array (4-byte big-endian length prefixes) into NAL units.
static func fromAvcc(_ avcc: [UInt8]) -> [[UInt8]] {
var units: [[UInt8]] = []
var i = 0
let n = avcc.count
while i + 4 <= n {
let len = (Int(avcc[i]) << 24) | (Int(avcc[i + 1]) << 16) | (Int(avcc[i + 2]) << 8) | Int(avcc[i + 3])
guard len > 0, i + 4 + len <= n else { break }
units.append(Array(avcc[(i + 4)..<(i + 4 + len)]))
i += 4 + len
}
return units
}
}
+160
View File
@@ -0,0 +1,160 @@
import Foundation
/// Result of feeding one packet to the depacketizer.
struct DepacketizeResult {
/// Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged.
var accessUnit: [UInt8]?
/// True when this call discarded a frame as damaged (packet loss or unsupported packetization).
var frameDropped = false
/// True when the completed access unit carries SPS/PPS (a keyframe).
var isKeyFrame = false
}
/// Reassembles RFC 6184 packet streams (single NAL unit packets and FU-A)
/// into Annex-B access units. Packets must arrive in order; frames damaged by
/// sequence gaps or missing fragments are reported via DepacketizeResult.
///
/// Mirrors the C++ `H264Depacketizer` (same state machine and start codes).
final class H264Depacketizer {
static let fuA = 28
private var lastSequenceNumber: Int?
private var frameStarted = false
private var frameDamaged = false
private var frameTimestamp = 0
// Growable byte accumulators: keyframes reach hundreds of KB.
private var accessUnit: [UInt8] = []
private var fuActive = false
private var fuNal: [UInt8] = []
/// Feed one packet (in sequence order, from the jitter buffer).
func depacketize(_ packet: RtpPacket) -> DepacketizeResult {
var result = DepacketizeResult()
// Track sequence continuity: a gap means packets were lost.
if let last = lastSequenceNumber {
let expected = (last + 1) & 0xFFFF
if packet.header.sequenceNumber != expected {
fuActive = false
fuNal.removeAll(keepingCapacity: true)
if frameStarted { frameDamaged = true }
}
}
lastSequenceNumber = packet.header.sequenceNumber
// A timestamp change without a closing marker means the previous frame
// lost its tail and can no longer be recovered.
if frameStarted && packet.header.timestamp != frameTimestamp {
dropFrame()
result.frameDropped = true
}
if !frameStarted {
frameStarted = true
frameDamaged = false
frameTimestamp = packet.header.timestamp
accessUnit.removeAll(keepingCapacity: true)
}
let payload = packet.payload
if !payload.isEmpty {
let type = Int(payload[0]) & 0x1F
if type >= 1 && type <= 23 {
// Single NAL unit packet.
if fuActive {
frameDamaged = true
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
appendStartCode()
accessUnit.append(contentsOf: payload)
} else if type == Self.fuA {
if payload.count < 2 {
frameDamaged = true
} else {
let fuHeader = Int(payload[1])
let start = fuHeader & 0x80 != 0
let end = fuHeader & 0x40 != 0
let fragment = Array(payload[2...])
if start {
if fuActive {
// The previous fragmented NAL lost its end packet.
frameDamaged = true
}
fuActive = true
fuNal.removeAll(keepingCapacity: true)
// The FU indicator keeps the original NAL's F bit (0) and NRI,
// and declares type 28; the FU header carries S/E plus the real type.
fuNal.append(UInt8((Int(payload[0]) & 0xE0) | (fuHeader & 0x1F)))
fuNal.append(contentsOf: fragment)
} else if !fuActive {
// Continuation without a start: the head of the NAL is lost.
frameDamaged = true
} else {
fuNal.append(contentsOf: fragment)
if end {
appendStartCode()
accessUnit.append(contentsOf: fuNal)
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
}
}
} else {
// Unsupported packetization mode (STAP-A, MTAP, FU-B).
frameDamaged = true
}
}
if !packet.header.marker {
return result
}
if fuActive {
// The marker arrived while a NAL was still fragmented.
frameDamaged = true
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
if !frameDamaged && !accessUnit.isEmpty {
let unit = accessUnit
result.accessUnit = unit
result.isKeyFrame = Self.containsParameterSets(unit)
} else {
// The frame that just ended is unusable.
result.frameDropped = true
}
dropFrame()
return result
}
private func appendStartCode() {
accessUnit.append(0)
accessUnit.append(0)
accessUnit.append(1)
}
/// The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8.
private static func containsParameterSets(_ unit: [UInt8]) -> Bool {
guard unit.count >= 4 else { return false }
var i = 0
while i <= unit.count - 4 {
if unit[i] == 0 && unit[i + 1] == 0 && unit[i + 2] == 1 {
let nalType = Int(unit[i + 3]) & 0x1F
if nalType == 7 || nalType == 8 {
return true
}
}
i += 1
}
return false
}
private func dropFrame() {
frameStarted = false
frameDamaged = false
accessUnit.removeAll(keepingCapacity: true)
fuActive = false
fuNal.removeAll(keepingCapacity: true)
}
}
+83
View File
@@ -0,0 +1,83 @@
import Foundation
/// Reorders RTP packets by sequence number before depacketization so that a
/// reordering link (Wi-Fi) is not read as loss. Delivery stays in order; only
/// aged-out or overflowing buffers release out of order, which the downstream
/// gap detection still handles for genuine loss.
///
/// Mirrors the C++ `RtpJitterBuffer` (same defaults and semantics) and the
/// Kotlin receiver.
final class JitterBuffer {
private struct Entry {
let timeNs: Int64
let packet: RtpPacket
}
private let maxDepth: Int
private let maxDelayNs: Int64
private var buffer: [Int: Entry] = [:]
private var nextExpected: Int?
init(maxDepth: Int = 16, maxDelayMs: Int = 60) {
self.maxDepth = maxDepth
self.maxDelayNs = Int64(maxDelayMs) * 1_000_000
}
/// Insert one packet and return the packets now ready for in-order delivery.
func push(_ packet: RtpPacket) -> [RtpPacket] {
var released: [RtpPacket] = []
let sequence = packet.header.sequenceNumber
let now = Self.nowNanos()
let expected0: Int
if let e = nextExpected {
expected0 = e
} else {
expected0 = sequence
nextExpected = sequence
}
// Serial-number comparison: a distance >= 32768 means the packet is
// older than what we already delivered (duplicate or straggler).
let distance = (sequence - expected0 + 65536) % 65536
if distance < 32768 {
buffer[sequence] = Entry(timeNs: now, packet: packet)
// Release the consecutive run from the expected sequence.
var expected = expected0
while let entry = buffer[expected] {
released.append(entry.packet)
buffer.removeValue(forKey: expected)
expected = (expected + 1) & 0xFFFF
}
nextExpected = expected
// A missing packet stalls the run: age out the backlog (or bound
// the buffer) and release what is there in order, so genuine loss
// reaches the depacketizer's gap detection rather than blocking.
if !buffer.isEmpty {
let keys = buffer.keys.sorted()
let head = keys.first!
let headAgeNs = now - buffer[head]!.timeNs
if headAgeNs > maxDelayNs || buffer.count > maxDepth {
for key in keys {
released.append(buffer[key]!.packet)
}
nextExpected = (keys.last! + 1) & 0xFFFF
buffer.removeAll(keepingCapacity: true)
}
}
}
return released
}
/// Discard everything still buffered.
func clear() {
buffer.removeAll(keepingCapacity: true)
nextExpected = nil
}
private static func nowNanos() -> Int64 {
return Int64(DispatchTime.now().uptimeNanoseconds)
}
}
+29
View File
@@ -0,0 +1,29 @@
import Foundation
/// The in-band SPS (type 7) and PPS (type 8) NAL units extracted from an
/// access unit. The sender repeats both ahead of every keyframe, so any
/// keyframe carries them; they are the source for the VideoToolbox format
/// description.
struct NalSets {
let sps: [UInt8]
let pps: [UInt8]
}
/// Extracts parameter sets from an Annex-B access unit.
enum NalExtractor {
static func parameterSets(_ annexB: [UInt8]) -> NalSets? {
var sps: [UInt8]?
var pps: [UInt8]?
for nal in AvccConverter.nalUnits(annexB) {
guard !nal.isEmpty else { continue }
let type = Int(nal[0]) & 0x1F
if type == 7 && sps == nil {
sps = nal
} else if type == 8 && pps == nil {
pps = nal
}
}
guard let s = sps, let p = pps else { return nil }
return NalSets(sps: s, pps: p)
}
}
+56
View File
@@ -0,0 +1,56 @@
import Foundation
/// Minimal RTP header (RFC 3550) without extensions, mirroring the C++
/// `RtpHeader` and the Kotlin receiver.
struct RtpHeader: Equatable {
var version = 2
var padding = false
var extensionHeader = false
var csrcCount = 0
var marker = false
var payloadType = 96
var sequenceNumber = 0
var timestamp = 0
var ssrc = 0
/// Serializes the bare 12-byte header; requires a version-2, extension-less header.
func serialize() -> [UInt8] {
var out = [UInt8](repeating: 0, count: 12)
out[0] = UInt8((version & 0x0F) << 6
| (padding ? 0x20 : 0)
| (extensionHeader ? 0x10 : 0)
| (csrcCount & 0x0F))
out[1] = UInt8((marker ? 0x80 : 0) | (payloadType & 0x7F))
out[2] = UInt8((sequenceNumber >> 8) & 0xFF)
out[3] = UInt8(sequenceNumber & 0xFF)
out[4] = UInt8((timestamp >> 24) & 0xFF)
out[5] = UInt8((timestamp >> 16) & 0xFF)
out[6] = UInt8((timestamp >> 8) & 0xFF)
out[7] = UInt8(timestamp & 0xFF)
out[8] = UInt8((ssrc >> 24) & 0xFF)
out[9] = UInt8((ssrc >> 16) & 0xFF)
out[10] = UInt8((ssrc >> 8) & 0xFF)
out[11] = UInt8(ssrc & 0xFF)
return out
}
/// Parses a 12-byte header from the start of a datagram.
static func parse(_ input: [UInt8]) -> RtpHeader? {
guard input.count >= 12 else { return nil }
let b0 = Int(input[0])
let b1 = Int(input[1])
let version = b0 >> 6
guard version == 2 else { return nil }
return RtpHeader(
version: version,
padding: b0 & 0x20 != 0,
extensionHeader: b0 & 0x10 != 0,
csrcCount: b0 & 0x0F,
marker: b1 & 0x80 != 0,
payloadType: b1 & 0x7F,
sequenceNumber: (Int(input[2]) << 8) | Int(input[3]),
timestamp: (Int(input[4]) << 24) | (Int(input[5]) << 16) | (Int(input[6]) << 8) | Int(input[7]),
ssrc: (Int(input[8]) << 24) | (Int(input[9]) << 16) | (Int(input[10]) << 8) | Int(input[11]),
)
}
}
+38
View File
@@ -0,0 +1,38 @@
import Foundation
/// An RTP packet: 12-byte base header (plus optional CSRC/extension) and payload.
struct RtpPacket {
let header: RtpHeader
let payload: [UInt8]
init(header: RtpHeader, payload: [UInt8]) {
self.header = header
self.payload = payload
}
/// Parses a full RTP datagram. Honors CSRC lists, one-level extension
/// headers, and RFC 3550 padding, mirroring the C++ receiver.
static func parse(_ input: [UInt8]) -> RtpPacket? {
guard input.count >= 12, let header = RtpHeader.parse(input) else { return nil }
var offset = 12 + header.csrcCount * 4
guard input.count >= offset else { return nil }
if header.extensionHeader {
guard input.count >= offset + 4 else { return nil }
let extensionWords = (Int(input[offset + 2]) << 8) | Int(input[offset + 3])
offset += 4 + extensionWords * 4
guard input.count >= offset else { return nil }
}
var payloadSize = input.count - offset
if header.padding {
// RFC 3550: the last byte holds the padding size, including itself.
guard payloadSize > 0 else { return nil }
let paddingSize = Int(input[input.count - 1])
guard paddingSize != 0, paddingSize <= payloadSize else { return nil }
payloadSize -= paddingSize
}
return RtpPacket(header: header, payload: Array(input[offset..<(offset + payloadSize)]))
}
}
@@ -0,0 +1,33 @@
import Foundation
/// Splits a byte stream into newline-terminated lines, dropping CR and
/// enforcing the 64 KB line cap the same framing the C++ and Kotlin
/// receivers use. Extracted so the logic is testable in isolation.
final class LineAssembler {
private var pending: [UInt8] = []
private let maxLine = SignalingMessage.maxMessageBytes
/// Feed a chunk of received bytes; returns the complete lines it contained.
func feed(_ chunk: [UInt8]) -> [String] {
var lines: [String] = []
for b in chunk {
if b == 0x0A { // \n
let text = String(bytes: pending, encoding: .utf8) ?? ""
pending.removeAll(keepingCapacity: true)
if !text.isEmpty { lines.append(text) }
} else if b != 0x0D { // \r
if pending.count < maxLine {
pending.append(b)
} else {
// Hostile or broken peer: drop the oversized line.
pending.removeAll(keepingCapacity: true)
}
}
}
return lines
}
func reset() {
pending.removeAll(keepingCapacity: true)
}
}
@@ -0,0 +1,89 @@
import Foundation
/// JSON wire format shared with the C++ implementation: one JSON object per
/// newline-terminated TCP line (offer / answer / pli). Mirrors the C++
/// `SignalingMessage` and the Kotlin receiver.
enum SignalingMessage {
case offer(sessionId: String,
codec: String,
width: Int,
height: Int,
frameRateNum: Int,
frameRateDen: Int,
rtpAddress: String,
rtpPort: Int)
case answer(sessionId: String,
rtpAddress: String,
rtpPort: Int,
displayWidth: Int,
displayHeight: Int)
case pli(sessionId: String)
static let maxMessageBytes = 64 * 1024
static func parse(_ line: String) -> SignalingMessage? {
guard line.count <= maxMessageBytes else { return nil }
guard let data = line.data(using: .utf8),
let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let type = obj["type"] as? String else {
return nil
}
switch type {
case "offer":
return .offer(
sessionId: obj["session_id"] as? String ?? "",
codec: obj["codec"] as? String ?? "",
width: obj["width"] as? Int ?? 0,
height: obj["height"] as? Int ?? 0,
frameRateNum: obj["frame_rate_num"] as? Int ?? 30,
frameRateDen: obj["frame_rate_den"] as? Int ?? 1,
rtpAddress: obj["rtp_address"] as? String ?? "",
rtpPort: obj["rtp_port"] as? Int ?? 0)
case "answer":
return .answer(
sessionId: obj["session_id"] as? String ?? "",
rtpAddress: obj["rtp_address"] as? String ?? "",
rtpPort: obj["rtp_port"] as? Int ?? 0,
displayWidth: obj["display_width"] as? Int ?? 0,
displayHeight: obj["display_height"] as? Int ?? 0)
case "pli":
return .pli(sessionId: obj["session_id"] as? String ?? "")
default:
return nil
}
}
static func serialize(_ message: SignalingMessage) -> String {
let json: [String: Any]
switch message {
case let .offer(sessionId, codec, width, height, frameRateNum, frameRateDen, rtpAddress, rtpPort):
json = [
"type": "offer",
"session_id": sessionId,
"codec": codec,
"width": width,
"height": height,
"frame_rate_num": frameRateNum,
"frame_rate_den": frameRateDen,
"rtp_address": rtpAddress,
"rtp_port": rtpPort,
]
case let .answer(sessionId, rtpAddress, rtpPort, displayWidth, displayHeight):
json = [
"type": "answer",
"session_id": sessionId,
"rtp_address": rtpAddress,
"rtp_port": rtpPort,
"display_width": displayWidth,
"display_height": displayHeight,
]
case let .pli(sessionId):
json = ["type": "pli", "session_id": sessionId]
}
guard let data = try? JSONSerialization.data(withJSONObject: json),
let s = String(data: data, encoding: .utf8) else {
return "{}\n" // unreachable for our message types; serialize is total
}
return s + "\n"
}
}
@@ -0,0 +1,212 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// Small socket helpers shared by the signaling server and the UDP transport.
enum SocketUtils {
/// Creates a bound, listening TCP socket. Prefers a dual-stack IPv6
/// listener (both families), falls back to IPv4-only. Returns -1 on failure.
static func makeStreamListener(port: UInt16) -> Int32 {
for family in [AF_INET6, AF_INET] {
let fd = socket(family, SOCK_STREAM, 0)
guard fd >= 0 else { continue }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
if family == AF_INET6 {
var no: Int32 = 0
setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &no, socklen_t(MemoryLayout<Int32>.size))
}
let bound: Int32
if family == AF_INET6 {
var a = sockaddr_in6()
a.sin6_family = sa_family_t(AF_INET6)
a.sin6_port = port.bigEndian
a.sin6_addr = in6addr_any
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in6>.size)) }
}
} else {
var a = sockaddr_in()
a.sin_family = sa_family_t(AF_INET)
a.sin_port = port.bigEndian
a.sin_addr = in_addr(s_addr: INADDR_ANY)
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in>.size)) }
}
}
if bound != 0 { close(fd); continue }
if listen(fd, 16) != 0 { close(fd); continue }
return fd
}
return -1
}
/// The local port of a bound socket (the port is at byte offset 2 for both
/// IPv4 and IPv6 sockets).
static func boundPort(_ fd: Int32) -> UInt16 {
var a = sockaddr_storage()
var len = socklen_t(MemoryLayout<sockaddr_storage>.size)
guard getsockname(fd, &a, &len) == 0 else { return 0 }
return withUnsafeBytes(of: &a) { raw in
raw.load(fromByteOffset: 2, as: UInt16.self).bigEndian
}
}
/// A UDP socket bound to [port] (or 0 for ephemeral) for receiving, with a
/// generous receive buffer (the C++ sender's VBV bounds bursts, but a larger
/// buffer absorbs a burst on a lossy link).
static func makeUdpReceiver(port: UInt16, receiveBufferSize: Int32) -> Int32 {
for family in [AF_INET6, AF_INET] {
let fd = socket(family, SOCK_DGRAM, 0)
guard fd >= 0 else { continue }
var yes: Int32 = 1
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &yes, socklen_t(MemoryLayout<Int32>.size))
if family == AF_INET6 {
var no: Int32 = 0
setsockopt(fd, IPPROTO_IPV6, IPV6_V6ONLY, &no, socklen_t(MemoryLayout<Int32>.size))
}
let bound: Int32
if family == AF_INET6 {
var a = sockaddr_in6()
a.sin6_family = sa_family_t(AF_INET6)
a.sin6_port = port.bigEndian
a.sin6_addr = in6addr_any
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in6>.size)) }
}
} else {
var a = sockaddr_in()
a.sin_family = sa_family_t(AF_INET)
a.sin_port = port.bigEndian
a.sin_addr = in_addr(s_addr: INADDR_ANY)
bound = withUnsafePointer(to: &a) { p in
p.withMemoryRebound(to: sockaddr.self, capacity: 1) { bind(fd, $0, socklen_t(MemoryLayout<sockaddr_in>.size)) }
}
}
if bound != 0 { close(fd); continue }
setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &receiveBufferSize, socklen_t(MemoryLayout<Int32>.size))
return fd
}
return -1
}
}
/// Newline-delimited JSON signaling server (the receiver side). Keeps the most
/// recent connection as its active peer, mirroring the C++ server: `onOffer`
/// may answer synchronously (the sender blocks on the answer).
final class SignalingServer {
private let onOffer: (SignalingMessage) -> Void
private let onPli: (SignalingMessage) -> Void
private var listenFd: Int32 = -1
private(set) var port: UInt16 = 0
private var peerFd: Int32 = -1
private let peerLock = NSLock()
private let assembler = LineAssembler()
private var acceptThread: Thread?
private var readerThread: Thread?
private var running = false
init(onOffer: @escaping (SignalingMessage) -> Void,
onPli: @escaping (SignalingMessage) -> Void) {
self.onOffer = onOffer
self.onPli = onPli
}
/// Binds the port (SO_REUSEADDR) and starts accepting. Returns true on success.
func start(_ preferredPort: UInt16) -> Bool {
guard let fd = SocketUtils.makeStreamListener(port: preferredPort), fd >= 0 else { return false }
listenFd = fd
port = SocketUtils.boundPort(fd)
running = true
let thread = Thread { [weak self] in self?.acceptLoop() }
thread.name = "signaling-accept"
acceptThread = thread
thread.start()
return true
}
/// Sends to the current peer; never throws. A lost control message is
/// recoverable the session re-negotiates or the next keyframe arrives
/// but an exception here would kill the RTP reader thread that reaches
/// send() from requestPli(). Mirrors the C++ server, which ignores write
/// failures.
func send(_ message: SignalingMessage) {
guard let bytes = SignalingMessage.serialize(message).data(using: .utf8) else { return }
peerLock.lock()
let fd = peerFd
peerLock.unlock()
guard fd >= 0 else { return }
bytes.withUnsafeBytes { raw in
_ = send(fd, raw.baseAddress, raw.count, Int32(MSG_NOSIGNAL))
}
}
func close() {
running = false
peerLock.lock()
let peer = peerFd
peerFd = -1
peerLock.unlock()
if peer >= 0 { close(peer) }
if listenFd >= 0 { close(listenFd) }
listenFd = -1
}
private func acceptLoop() {
while running {
var addr = sockaddr()
var len = socklen_t(MemoryLayout<sockaddr>.size)
let client = accept(listenFd, &addr, &len)
if client < 0 {
if !running { break }
continue
}
// Most-recent-connection-wins: close the previous peer.
peerLock.lock()
let old = peerFd
peerFd = client
peerLock.unlock()
if old >= 0 { close(old) }
let thread = Thread { [weak self] in self?.readLoop(fd: client) }
thread.name = "signaling-reader"
readerThread = thread
thread.start()
}
}
private func readLoop(fd: Int32) {
assembler.reset()
var buffer = [UInt8](repeating: 0, count: 4096)
while running {
let read = buffer.withUnsafeMutableBytes { raw in
recv(fd, raw.baseAddress, raw.count, 0)
}
guard read > 0 else { break }
for line in assembler.feed(Array(buffer.prefix(read))) {
dispatch(line)
}
}
// Peer closed; if it is still our active peer, mark it gone.
peerLock.lock()
if peerFd == fd {
peerFd = -1
}
peerLock.unlock()
}
private func dispatch(_ line: String) {
guard let message = SignalingMessage.parse(line) else { return }
// A broken callback must not kill the reader thread.
switch message {
case .offer:
onOffer(message)
case .pli:
onPli(message)
case .answer:
break // the receiver never receives answers
}
}
}
+41
View File
@@ -0,0 +1,41 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// Finds the primary IPv4 address (e.g. on Wi-Fi) for the status overlay and
/// the `--peer` hint. Best-effort; the sender reaches us by IP over the LAN.
enum LocalAddress {
static func primaryIPv4() -> String {
var result = "unknown"
var fallback = "unknown"
var ptr: UnsafeMutablePointer<ifaddrs>?
guard getifaddrs(&ptr) == 0 else { return result }
defer { freeifaddrs(ptr) }
var current = ptr
while let iface = current {
let next = iface.pointee.ifa_next
current = next
guard let sa = iface.pointee.ifa_addr else { continue }
guard sa.pointee.sa_family == sa_family_t(AF_INET) else { continue }
if Int32(iface.pointee.ifa_flags) & IFF_LOOPBACK == 0 {
let inaddr = sa.assumingMemoryBound(to: sockaddr_in.self).pointee
var host = [CChar](repeating: 0, count: Int(INET_ADDRSTRLEN))
if inet_ntop(AF_INET, &inaddr.sin_addr, &host, socklen_t(INET_ADDRSTRLEN)) != nil {
let ip = String(cString: host)
if ip.hasPrefix("169.254") {
continue // link-local; prefer a routable address
}
fallback = ip
let name = String(cString: iface.pointee.ifa_name)
if name.hasPrefix("en") || name.hasPrefix("wlan") {
return ip // Wi-Fi / Ethernet: good enough for the hint
}
}
}
}
return fallback
}
}
+59
View File
@@ -0,0 +1,59 @@
import Foundation
#if canImport(Darwin)
import Darwin
#endif
/// A UDP socket for receiving RTP. Binds to a preferred port (or an ephemeral
/// port when it is busy), then delivers complete datagrams one at a time.
/// Mirrors the C++ `UdpRtpTransport` receive path and the Kotlin `DatagramSocket`
/// reader. Uses poll(2) with a timeout so `close()` from another thread cannot
/// strand a blocked recv (a close does not reliably unblock a POSIX recvfrom).
final class UdpTransport {
private var fd: Int32 = -1
private(set) var port: UInt16 = 0
/// Binds to [preferredPort] (or an ephemeral port when it is busy).
@discardableResult
func bind(preferredPort: UInt16, receiveBufferSize: Int32) -> Bool {
guard fd < 0 else { return true }
for p in [preferredPort, UInt16(0)] {
if let f = SocketUtils.makeUdpReceiver(port: p, receiveBufferSize: receiveBufferSize), f >= 0 {
fd = f
port = SocketUtils.boundPort(fd)
return true
}
}
return false
}
/// Waits up to [timeoutMs] for a datagram. Returns 1 when one is ready to
/// read, 0 on timeout, and -1 when the socket is closed/errored.
func poll(timeoutMs: Int32) -> Int32 {
guard fd >= 0 else { return -1 }
var pfd = pollfd(fd: fd, events: poll_events_t(POLLIN), revents: 0)
let r = withUnsafeMutablePointer(to: &pfd) { poll($0, 1, timeoutMs) }
if r > 0 {
if pfd.revents & poll_events_t(POLLERR) != 0 { return -1 }
return 1
}
return r == 0 ? 0 : -1
}
/// Reads one ready datagram. Returns nil on error or empty read.
func receiveDatagram() -> [UInt8]? {
guard fd >= 0 else { return nil }
var buf = [UInt8](repeating: 0, count: 4096)
let read = buf.withUnsafeMutableBytes { raw in
recvfrom(fd, raw.baseAddress, raw.count, 0, nil, nil)
}
guard read > 0 else { return nil }
return Array(buf.prefix(read))
}
func close() {
if fd >= 0 {
close(fd)
fd = -1
}
}
}
@@ -0,0 +1,27 @@
import XCTest
@testable import Receiver
final class AvccConverterTests: XCTestCase {
func testNalUnits() {
let annexB: [UInt8] = [0, 0, 1, 0x41, 0x11, 0, 0, 1, 0x67, 0xAA]
XCTAssertEqual(AvccConverter.nalUnits(annexB), [[0x41, 0x11], [0x67, 0xAA]])
}
func testToAvcc() {
let annexB: [UInt8] = [0, 0, 1, 0x41, 0x11, 0, 0, 1, 0x67, 0xAA]
let avcc = AvccConverter.toAvcc(annexB)
XCTAssertEqual(avcc, [0, 0, 0, 2, 0x41, 0x11, 0, 0, 0, 2, 0x67, 0xAA])
}
func testFromAvccRoundtrip() {
let units: [[UInt8]] = [[0x41, 0x11], [0x67, 0xAA, 0xBB]]
let annexB = [0, 0, 1] + units[0] + [0, 0, 1] + units[1]
let avcc = AvccConverter.toAvcc(annexB)
XCTAssertEqual(AvccConverter.fromAvcc(avcc ?? []), units)
}
func testEmptyReturnsNil() {
XCTAssertNil(AvccConverter.toAvcc([UInt8]()))
XCTAssertNil(AvccConverter.toAvcc([0, 0, 1]))
}
}
@@ -0,0 +1,103 @@
import XCTest
@testable import Receiver
final class H264DepacketizerTests: XCTestCase {
private let start: [UInt8] = [0x00, 0x00, 0x01]
private func packet(seq: Int, ts: Int, payload: [UInt8], marker: Bool = false) -> RtpPacket {
RtpPacket(header: RtpHeader(sequenceNumber: seq, timestamp: ts, marker: marker), payload: payload)
}
func testSingleNalTwoPackets() {
let d = H264Depacketizer()
// SPS (type 7) then a slice (type 5), closed by the marker.
let first = d.depacketize(packet(seq: 1, ts: 100, payload: [0x67, 0xAA, 0xBB]))
XCTAssertNil(first.accessUnit)
let second = d.depacketize(packet(seq: 2, ts: 100, payload: [0x41, 0x01, 0x02], marker: true))
XCTAssertEqual(second.accessUnit, start + [0x67, 0xAA, 0xBB] + start + [0x41, 0x01, 0x02])
XCTAssertTrue(second.isKeyFrame)
XCTAssertFalse(second.frameDropped)
}
func testFuAReassembly() {
let d = H264Depacketizer()
// NAL: header 0x41 (type 1, NRI 2) + payload 0x11 0x22 0x33 0x44.
// FU indicator = (0x41 & 0xE0) | 28 = 0x5C.
let indicator: UInt8 = 0x5C
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [indicator, 0x81, 0x11]))
_ = d.depacketize(packet(seq: 2, ts: 100, payload: [indicator, 0x01, 0x22]))
let done = d.depacketize(packet(seq: 3, ts: 100, payload: [indicator, 0x41, 0x33, 0x44], marker: true))
XCTAssertEqual(done.accessUnit, start + [0x41, 0x11, 0x22, 0x33, 0x44])
XCTAssertFalse(done.frameDropped)
}
func testDropsGappedFrames() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01]))
// seq 2 is missing; the frame must be reported dropped, not delivered.
let tail = d.depacketize(packet(seq: 3, ts: 100, payload: [0x41, 0x02], marker: true))
XCTAssertNil(tail.accessUnit)
XCTAssertTrue(tail.frameDropped)
}
func testSeparateFramesByMarker() {
let d = H264Depacketizer()
let au1 = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0xAA], marker: true))
XCTAssertEqual(au1.accessUnit, start + [0x41, 0xAA])
let au2 = d.depacketize(packet(seq: 2, ts: 200, payload: [0x41, 0xBB], marker: true))
XCTAssertEqual(au2.accessUnit, start + [0x41, 0xBB])
}
func testDropsFuWithoutStart() {
let d = H264Depacketizer()
// Continuation (no S bit) without any start packet.
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C, 0x41, 0x11], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsStillFragmentedAtMarker() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C, 0x81, 0x11]))
// Marker arrives while the FU-A NAL is still open.
let result = d.depacketize(packet(seq: 2, ts: 100, payload: [0x41, 0x01], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsUnsupportedPacketization() {
let d = H264Depacketizer()
let stapA: UInt8 = 24 // STAP-A
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [stapA, 0x00, 0x05, 0x41, 0x01], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
func testDropsTimestampChangeWithoutMarker() {
let d = H264Depacketizer()
_ = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01]))
// The stale frame is reported dropped, but this packet starts (and
// closes) the next frame matching the C++ depacketizer.
let result = d.depacketize(packet(seq: 2, ts: 200, payload: [0x41, 0x02], marker: true))
XCTAssertEqual(result.accessUnit, start + [0x41, 0x02])
XCTAssertTrue(result.frameDropped)
}
func testKeyframeDetectionRequiresParameterSets() {
let d = H264Depacketizer()
let plain = d.depacketize(packet(seq: 1, ts: 100, payload: [0x41, 0x01], marker: true))
XCTAssertFalse(plain.isKeyFrame)
let d2 = H264Depacketizer()
let withSps = d2.depacketize(packet(seq: 1, ts: 100, payload: [0x67, 0xAA, 0x88, 0x68, 0xBB, 0x41, 0x01], marker: true))
XCTAssertTrue(withSps.isKeyFrame)
}
func testDropsShortFuPackets() {
let d = H264Depacketizer()
// FU-A packet without its FU header byte.
let result = d.depacketize(packet(seq: 1, ts: 100, payload: [0x7C], marker: true))
XCTAssertNil(result.accessUnit)
XCTAssertTrue(result.frameDropped)
}
}
+57
View File
@@ -0,0 +1,57 @@
import XCTest
@testable import Receiver
final class JitterBufferTests: XCTestCase {
private func packet(seq: Int, ts: Int = 100) -> RtpPacket {
RtpPacket(header: RtpHeader(sequenceNumber: seq, timestamp: ts), payload: [UInt8(seq)])
}
func testInOrderReleasesImmediately() {
let jitter = JitterBuffer()
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
XCTAssertEqual(jitter.push(packet(seq: 2)).map { $0.header.sequenceNumber }, [2])
XCTAssertEqual(jitter.push(packet(seq: 3)).map { $0.header.sequenceNumber }, [3])
}
func testReordersOutOfOrderPackets() {
let jitter = JitterBuffer()
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
XCTAssertTrue(jitter.push(packet(seq: 3)).isEmpty)
XCTAssertEqual(jitter.push(packet(seq: 2)).map { $0.header.sequenceNumber }, [2, 3])
}
func testOverflowReleasesInOrderAndAdvances() {
let jitter = JitterBuffer(maxDepth: 4)
XCTAssertEqual(jitter.push(packet(seq: 1)).map { $0.header.sequenceNumber }, [1])
// seq 2 is lost; 3..6 stay buffered (within the depth bound).
for seq in 3...6 {
XCTAssertTrue(jitter.push(packet(seq: seq)).isEmpty)
}
// seq 7 overflows the buffer: 3..7 flush in order.
XCTAssertEqual(jitter.push(packet(seq: 7)).map { $0.header.sequenceNumber }, [3, 4, 5, 6, 7])
// Delivery continues in order afterwards.
XCTAssertEqual(jitter.push(packet(seq: 8)).map { $0.header.sequenceNumber }, [8])
XCTAssertEqual(jitter.push(packet(seq: 9)).map { $0.header.sequenceNumber }, [9])
}
func testDiscardsStragglers() {
let jitter = JitterBuffer(maxDepth: 4)
_ = jitter.push(packet(seq: 1))
for seq in 3...8 {
_ = jitter.push(packet(seq: seq))
}
XCTAssertTrue(jitter.push(packet(seq: 9)).isNotEmpty)
// seq 4 is now far behind the expected sequence: discarded, not delivered.
XCTAssertTrue(jitter.push(packet(seq: 4)).isEmpty)
// In-order delivery continues from 10.
XCTAssertEqual(jitter.push(packet(seq: 10)).map { $0.header.sequenceNumber }, [10])
}
func testClearResetsState() {
let jitter = JitterBuffer()
_ = jitter.push(packet(seq: 5))
jitter.clear()
// A completely different sequence now starts fresh.
XCTAssertEqual(jitter.push(packet(seq: 100)).map { $0.header.sequenceNumber }, [100])
}
}
@@ -0,0 +1,44 @@
import XCTest
@testable import Receiver
final class LineAssemblerTests: XCTestCase {
private func bytes(_ s: String) -> [UInt8] { Array(s.utf8) }
func testSingleLine() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("hello")), [])
XCTAssertEqual(a.feed(bytes("\n")), ["hello"])
}
func testMultipleLinesInOneChunk() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("one\ntwo\nthree\n")), ["one", "two", "three"])
}
func testDropsCR() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("line\r\n")), ["line"])
}
func testIgnoresEmptyLine() {
let a = LineAssembler()
XCTAssertEqual(a.feed(bytes("\n")), [])
}
func testSplitAcrossChunks() {
let a = LineAssembler()
let payload = "{\"session_id\":\"x\"}"
var lines: [String] = []
lines += a.feed(Array(payload.prefix(5)))
lines += a.feed(Array(payload.suffix(from: 5)))
lines += a.feed(["\n".utf8.first!])
XCTAssertEqual(lines, [payload])
}
func testDropsOversizedLine() {
let a = LineAssembler()
let big = String(repeating: "a", count: SignalingMessage.maxMessageBytes + 1)
_ = a.feed(Array(big.utf8))
XCTAssertEqual(a.feed(bytes("\n")), [])
}
}
+24
View File
@@ -0,0 +1,24 @@
import XCTest
@testable import Receiver
final class NalExtractorTests: XCTestCase {
func testExtractsSpsAndPps() {
// Annex-B: SPS (type 7), PPS (type 8), slice (type 5).
let annexB: [UInt8] = [0, 0, 1, 0x67, 0xAA, 0, 0, 1, 0x68, 0xBB, 0, 0, 1, 0x41, 0x01]
let sets = NalExtractor.parameterSets(annexB)
XCTAssertEqual(sets?.sps, [0x67, 0xAA])
XCTAssertEqual(sets?.pps, [0x68, 0xBB])
}
func testReturnsNilWithoutBoth() {
let annexB: [UInt8] = [0, 0, 1, 0x67, 0xAA, 0, 0, 1, 0x41, 0x01] // SPS but no PPS
XCTAssertNil(NalExtractor.parameterSets(annexB))
}
func testPicksFirstOfEach() {
let annexB: [UInt8] = [0, 0, 1, 0x67, 0x11, 0, 0, 1, 0x68, 0x22, 0, 0, 1, 0x67, 0x33, 0, 0, 1, 0x68, 0x44]
let sets = NalExtractor.parameterSets(annexB)
XCTAssertEqual(sets?.sps, [0x67, 0x11])
XCTAssertEqual(sets?.pps, [0x68, 0x22])
}
}
+87
View File
@@ -0,0 +1,87 @@
import XCTest
@testable import Receiver
final class RtpHeaderTests: XCTestCase {
func testRoundtrip() {
let header = RtpHeader(version: 2, padding: false, extensionHeader: false, csrcCount: 0,
marker: true, payloadType: 96, sequenceNumber: 0xABCD,
timestamp: 0xDEADBEEF, ssrc: 0x12345678)
let wire = header.serialize()
XCTAssertEqual(wire.count, 12)
XCTAssertEqual(RtpHeader.parse(wire), header)
}
func testRejectsBadVersion() {
var wire = RtpHeader().serialize()
wire[0] = (wire[0] & 0x3F) | (1 << 6)
XCTAssertNil(RtpHeader.parse(wire))
}
func testRejectsShortInput() {
let header = RtpHeader()
XCTAssertNil(RtpHeader.parse(Array(header.serialize().prefix(11))))
XCTAssertNil(RtpHeader.parse([]))
}
func testPreservesFlags() {
let header = RtpHeader(padding: true, csrcCount: 2, marker: true, payloadType: 63)
let parsed = RtpHeader.parse(header.serialize())
XCTAssertEqual(parsed?.padding, true)
XCTAssertEqual(parsed?.csrcCount, 2)
XCTAssertEqual(parsed?.marker, true)
XCTAssertEqual(parsed?.payloadType, 63)
}
}
final class RtpPacketTests: XCTestCase {
private func header(seq: Int, marker: Bool = false) -> RtpHeader {
RtpHeader(sequenceNumber: seq, payloadType: 96, marker: marker)
}
func testRoundtripWithPayload() {
let packet = RtpPacket(header: header(seq: 7), payload: [0x11, 0x22, 0x33])
let wire = packet.header.serialize() + packet.payload
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.header, header(seq: 7))
XCTAssertEqual(parsed?.payload, [0x11, 0x22, 0x33])
}
func testSkipsCsrcList() {
let header = RtpHeader(csrcCount: 1, sequenceNumber: 3)
let wire = header.serialize() + [0x0A, 0x00, 0x00, 0x01] + [0x99]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.header.csrcCount, 1)
XCTAssertEqual(parsed?.payload, [0x99])
}
func testSkipsExtensionHeader() {
let header = RtpHeader(extensionHeader: true, sequenceNumber: 4)
// profile=0x0001, length=1 word, one word of data.
let wire = header.serialize() + [0x00, 0x01, 0x00, 0x01, 0xDE, 0xAD, 0xBE, 0xEF] + [0x77]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.payload, [0x77])
}
func testStripsPadding() {
let header = RtpHeader(padding: true, sequenceNumber: 5)
// Payload byte, one padding zero, size byte (2 = padding incl. itself).
let wire = header.serialize() + [0x55, 0x00, 0x02]
let parsed = RtpPacket.parse(wire)
XCTAssertEqual(parsed?.payload, [0x55])
}
func testRejectsTruncatedCsrcAndExtension() {
let csrc = RtpHeader(csrcCount: 1).serialize()
XCTAssertNil(RtpPacket.parse(csrc)) // 12 bytes, needs 16
let ext = RtpHeader(extensionHeader: true).serialize() + [0x00, 0x01]
XCTAssertNil(RtpPacket.parse(ext)) // extension length field cut off
}
func testRejectsBadPadding() {
let zeroPad = RtpHeader(padding: true).serialize() + [0x00]
XCTAssertNil(RtpPacket.parse(zeroPad))
let oversized = RtpHeader(padding: true).serialize() + [0x00, 0x00, 0x05]
XCTAssertNil(RtpPacket.parse(oversized))
XCTAssertNil(RtpPacket.parse([UInt8](repeating: 0, count: 11)))
}
}
@@ -0,0 +1,54 @@
import XCTest
@testable import Receiver
final class SignalingMessageTests: XCTestCase {
func testOfferRoundtrip() {
let offer = SignalingMessage.offer(sessionId: "s-1", codec: "h264", width: 0, height: 0,
frameRateNum: 30, frameRateDen: 1, rtpAddress: "10.0.0.5", rtpPort: 1234)
let line = SignalingMessage.serialize(offer).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .offer(sid, codec, w, h, num, den, addr, port) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-1")
XCTAssertEqual(codec, "h264")
XCTAssertEqual(w, 0)
XCTAssertEqual(h, 0)
XCTAssertEqual(num, 30)
XCTAssertEqual(den, 1)
XCTAssertEqual(addr, "10.0.0.5")
XCTAssertEqual(port, 1234)
}
func testAnswerRoundtrip() {
let answer = SignalingMessage.answer(sessionId: "s-2", rtpAddress: "", rtpPort: 5004,
displayWidth: 1179, displayHeight: 2556)
let line = SignalingMessage.serialize(answer).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .answer(sid, addr, port, dw, dh) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-2")
XCTAssertEqual(addr, "")
XCTAssertEqual(port, 5004)
XCTAssertEqual(dw, 1179)
XCTAssertEqual(dh, 2556)
}
func testPliRoundtrip() {
let pli = SignalingMessage.pli(sessionId: "s-3")
let line = SignalingMessage.serialize(pli).trimmingCharacters(in: .newlines)
let parsed = SignalingMessage.parse(line)
guard case let .pli(sid) = parsed else { return XCTFail() }
XCTAssertEqual(sid, "s-3")
}
func testRejectsUnknownType() {
XCTAssertNil(SignalingMessage.parse("{\"type\":\"bogus\"}"))
}
func testRejectsInvalidJson() {
XCTAssertNil(SignalingMessage.parse("not json"))
}
func testRejectsOversizedLine() {
let big = String(repeating: "a", count: SignalingMessage.maxMessageBytes + 1)
XCTAssertNil(SignalingMessage.parse(big))
}
}
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
#
# Bootstrap for the iOS receiver. Intended to be run on a Mac with Xcode 26.
# It installs XcodeGen (if absent) from the GitHub release (no Homebrew needed),
# generates the Xcode project, and builds/tests.
#
# Usage:
# ./bootstrap.sh # generate + build for device
# ./bootstrap.sh test # generate + run the unit tests (simulator)
# ./bootstrap.sh build-sim # generate + build for the simulator
# ./bootstrap.sh generate # just generate the project
#
# Device install/signing is intentionally left to Xcode: open
# Receiver.xcodeproj, set your Apple ID on the Receiver target, and hit Run.
set -euo pipefail
IOS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TOOLS_DIR="$IOS_DIR/tools"
XCODEGEN_BIN="$TOOLS_DIR/xcodegen"
PROJ="$IOS_DIR/Receiver.xcodeproj"
# 1. Xcode
if ! command -v xcodebuild >/dev/null 2>&1; then
echo "error: xcodebuild not found. Install Xcode (26.x) and select its CLI tools." >&2
echo " xcode-select --install (or: sudo xcode-select -s /Applications/Xcode.app/Contents/Developer)" >&2
exit 1
fi
# 2. XcodeGen: prefer a system copy, then a local install, else download latest.
if ! command -v xcodegen >/dev/null 2>&1 && [ ! -x "$XCODEGEN_BIN" ]; then
echo "Installing XcodeGen into $TOOLS_DIR ..."
mkdir -p "$TOOLS_DIR"
url="https://github.com/yonaskolb/XcodeGen/releases/latest/download/xcodegen.zip"
tmp="$(mktemp -d)"
curl -fL "$url" -o "$tmp/xcodegen.zip"
unzip -o -q "$tmp/xcodegen.zip" -d "$TOOLS_DIR"
# The release zip extracts to a flat binary named "xcodegen".
if [ ! -x "$XCODEGEN_BIN" ] && [ -f "$TOOLS_DIR/xcodegen" ]; then
chmod +x "$XCODEGEN_BIN"
fi
rm -rf "$tmp"
fi
XCODEGEN="$(command -v xcodegen || echo "$XCODEGEN_BIN")"
if [ ! -x "$XCODEGEN" ] && [ ! -x "$XCODEGEN_BIN" ]; then
echo "error: XcodeGen not found and download failed." >&2
exit 1
fi
XCODEGEN="${XCODEGEN_BIN}"
generate() {
"$XCODEGEN_BIN" generate --spec "$IOS_DIR/project.yml" --project "$IOS_DIR"
echo "Generated $PROJ"
}
dest_sim() { echo "${IOS_DEST:-platform=iOS Simulator,name=iPhone 16}"; }
command="${1:-build}"
generate
case "$command" in
generate)
;;
build)
xcodebuild -project "$PROJ" -scheme Receiver -destination "generic/platform=iOS" build
;;
build-sim)
xcodebuild -project "$PROJ" -scheme Receiver -destination "$(dest_sim)" build
;;
test)
xcodebuild test -project "$PROJ" -scheme Receiver -destination "$(dest_sim)" \
-only-testing:ReceiverTests
;;
*)
echo "usage: bootstrap.sh [generate|build|build-sim|test]" >&2
exit 1
;;
esac
echo "Done."
+71
View File
@@ -0,0 +1,71 @@
name: Receiver
options:
bundleIdPrefix: screencast
deploymentTarget:
iOS: "17.0"
createIntermediateGroups: true
generateEmptySchemes: false
targets:
Receiver:
type: application
platform: iOS
deploymentTarget: "17.0"
sources:
- path: Receiver
settings:
base:
PRODUCT_BUNDLE_IDENTIFIER: screencast.receiver
PRODUCT_NAME: Receiver
TARGETED_DEVICE_FAMILY: 1
CODE_SIGN_STYLE: Automatic
DEVELOPMENT_TEAM: ""
IPHONEOS_DEPLOYMENT_TARGET: "17.0"
ENABLE_USER_SCRIPT_SANDBOXING: YES
info:
path: Receiver/Info.plist
properties:
CFBundleDisplayName: screencast
CFBundleName: screencast
CFBundleShortVersionString: "0.9.0"
CFBundleVersion: "1"
UILaunchScreen: {}
UISupportedInterfaceOrientations:
- UIInterfaceOrientationLandscapeLeft
- UIInterfaceOrientationLandscapeRight
UISupportedInterfaceOrientations~ipad:
- UIInterfaceOrientationLandscapeLeft
- UIInterfaceOrientationLandscapeRight
- UIInterfaceOrientationPortrait
- UIInterfaceOrientationPortraitUpsideDown
# Local-network privacy: required for both Bonjour and the TCP/UDP
# sockets. Without NSBonjourServices the sender never resolves us.
NSLocalNetworkUsageDescription: "screencast receives a video stream on your local network."
NSBonjourServices:
- "_screencast._tcp"
ReceiverTests:
type: bundle.unit-test
platform: iOS
deploymentTarget: "17.0"
sources:
- path: ReceiverTests
dependencies:
- target: Receiver
settings:
base:
PRODUCT_BUNDLE_IDENTIFIER: screencast.receiver.tests
BUNDLE_LOADER: "$(TEST_HOST)"
TEST_HOST: "$(BUILT_PRODUCTS_DIR)/Receiver.app/Receiver"
schemes:
Receiver:
build:
targets:
Receiver: all
test:
config: Debug
targets:
- ReceiverTests
archive:
enabled: false
+11
View File
@@ -10,6 +10,11 @@ project('screen_cast', 'cpp',
# Public and private include directories are declared in `src/meson.build`. # Public and private include directories are declared in `src/meson.build`.
subdir('src') subdir('src')
# The GUI is a sender panel and needs the capture backend.
if get_option('gui') and not get_option('sender')
error('The GUI is a sender panel and requires -Dsender=true')
endif
# Manual smoke tools are sender-side. # Manual smoke tools are sender-side.
if get_option('sender') if get_option('sender')
subdir('tools') subdir('tools')
@@ -21,6 +26,12 @@ if enable_tests
subdir('tests') subdir('tests')
endif endif
# Install the application icon into the hicolor theme: the waybar widget CSS
# and any future .desktop file reference it.
install_data('screencast_icon/screencast_256.png',
install_dir : get_option('datadir') / 'icons/hicolor/256x256/apps',
rename : 'screencast.png')
# Summary for the user # Summary for the user
summary({ summary({
'tests': enable_tests, 'tests': enable_tests,
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.2 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 398 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 9.1 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 84 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.8 KiB

+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""Generate a C++ header that embeds an image file as a byte array.
Usage: icon_to_header.py INPUT_IMAGE OUTPUT_HEADER
"""
import sys
def main() -> int:
if len(sys.argv) != 3:
print(f"usage: {sys.argv[0]} INPUT_IMAGE OUTPUT_HEADER", file=sys.stderr)
return 2
with open(sys.argv[1], "rb") as src:
data = src.read()
words = ", ".join(f"0x{byte:02x}" for byte in data)
header = f"""// GENERATED FILE - do not edit by hand.
// Produced by scripts/icon_to_header.py from screencast_icon/screencast_256.png.
#pragma once
#include <array>
#include <cstdint>
namespace sc {{
inline constexpr std::array<std::uint8_t, {len(data)}> app_icon_png = {{
{words}
}};
}} // namespace sc
"""
with open(sys.argv[2], "w") as dst:
dst.write(header)
return 0
if __name__ == "__main__":
sys.exit(main())
+16 -1
View File
@@ -8,7 +8,8 @@ namespace sc {
namespace { namespace {
void print_usage() { void print_usage() {
std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate KBPS]\n" std::fputs("usage: screencast --send [--target monitor|window] [--peer HOST[:PORT]] [--bitrate MAX_KBPS] [--crf "
"0-51] [--fps 1-60]\n"
" screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen] [--swdecode]\n" " screencast --receive [--port PORT] [--signaling-port PORT] [--fullscreen] [--swdecode]\n"
" screencast --discover [--timeout SECONDS]\n" " screencast --discover [--timeout SECONDS]\n"
" screencast waybar [--toggle] # for waybar widgets\n" " screencast waybar [--toggle] # for waybar widgets\n"
@@ -105,6 +106,20 @@ std::optional<Command> parse_cli(int argc, const char* const argv[]) {
print_usage(); print_usage();
return std::nullopt; return std::nullopt;
} }
} else if (argument == "--crf") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, send.crf) || send.crf < 0 ||
send.crf > 51) {
print_usage();
return std::nullopt;
}
} else if (argument == "--fps") {
std::string_view value;
if (!next_argument(argc, argv, index, value) || !parse_int(value, send.fps) || send.fps < 0 ||
send.fps > 60) {
print_usage();
return std::nullopt;
}
} else if (argument == "--port") { } else if (argument == "--port") {
std::string_view value; std::string_view value;
int port = 0; int port = 0;
+3 -3
View File
@@ -154,13 +154,13 @@ int run_sender(const sc::SendCommand& command) {
auto [name, port] = receivers.begin()->first; auto [name, port] = receivers.begin()->first;
std::vector<std::string> hosts = receivers.begin()->second; std::vector<std::string> hosts = receivers.begin()->second;
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) { std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return sc::SenderSession::address_preference(lhs) < sc::SenderSession::address_preference(rhs); return sc::address_preference(lhs) < sc::address_preference(rhs);
}); });
std::cout << std::format("screencast: found receiver '{}'\n", name); std::cout << std::format("screencast: found receiver '{}'\n", name);
signaling = sc::Endpoint{hosts.front(), port}; signaling = sc::Endpoint{hosts.front(), port};
} }
auto session_result = sc::SenderSession::start(signaling, command.bitrate_kbps, target); auto session_result = sc::SenderSession::start(signaling, command.bitrate_kbps, command.crf, command.fps, target);
if (auto* error = std::get_if<std::string>(&session_result)) { if (auto* error = std::get_if<std::string>(&session_result)) {
std::cerr << std::format("screencast: {}\n", *error); std::cerr << std::format("screencast: {}\n", *error);
return 1; return 1;
@@ -305,7 +305,7 @@ int run_discover(const sc::DiscoverCommand& command) {
} }
for (auto& [key, hosts] : receivers) { for (auto& [key, hosts] : receivers) {
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) { std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return sc::SenderSession::address_preference(lhs) < sc::SenderSession::address_preference(rhs); return sc::address_preference(lhs) < sc::address_preference(rhs);
}); });
std::string joined = std::string joined =
std::accumulate(hosts.begin(), hosts.end(), std::string{}, [](std::string lhs, const std::string& rhs) { std::accumulate(hosts.begin(), hosts.end(), std::string{}, [](std::string lhs, const std::string& rhs) {
+12 -3
View File
@@ -1,5 +1,7 @@
# Shared application internals used by both the CLI binary and the GUI: # Shared application internals used by both the CLI binary and the GUI:
# pipelines, session orchestration, and the state store. # pipelines, session orchestration, and the state store. The sender
# pipeline is only compiled when the sender is built; receiver-only
# targets must not require the capture backend.
sc_app_core_sources = files( sc_app_core_sources = files(
'pipelines.cpp', 'pipelines.cpp',
@@ -7,11 +9,18 @@ sc_app_core_sources = files(
'state_store.cpp', 'state_store.cpp',
) )
sc_app_core_args = []
sc_app_core_deps = [dep_json]
if build_sender
sc_app_core_args += ['-DSC_HAS_SENDER=1']
sc_app_core_deps += [sc_capture_dep]
endif
sc_app_core = static_library('sc_app_core', sc_app_core = static_library('sc_app_core',
sc_app_core_sources, sc_app_core_sources,
include_directories : [sc_core_inc, include_directories('.')], include_directories : [sc_core_inc, include_directories('.')],
cpp_args : ['-DSC_HAS_SENDER=1'], cpp_args : sc_app_core_args,
dependencies : [dep_json]) dependencies : sc_app_core_deps)
sc_app_core_dep = declare_dependency( sc_app_core_dep = declare_dependency(
link_with : sc_app_core, link_with : sc_app_core,
+84 -4
View File
@@ -10,6 +10,7 @@
#include <unistd.h> #include <unistd.h>
#include <algorithm>
#include <array> #include <array>
#include <chrono> #include <chrono>
#include <condition_variable> #include <condition_variable>
@@ -36,7 +37,7 @@ std::string receiver_service_name() {
class SenderPipeline::Impl { class SenderPipeline::Impl {
public: public:
explicit Impl(SenderPipelineConfig config) : config_(std::move(config)) {} explicit Impl(SenderPipelineConfig config) : config_(std::move(config)), current_crf_(config_.encoder.crf) {}
~Impl() { ~Impl() {
stop(); stop();
@@ -59,15 +60,12 @@ class SenderPipeline::Impl {
run_thread_ = std::jthread([this](std::stop_token stop_token) { run(std::move(stop_token)); }); run_thread_ = std::jthread([this](std::stop_token stop_token) { run(std::move(stop_token)); });
// Publish the session for status widgets and one-click restarts. // Publish the session for status widgets and one-click restarts.
// The RTP peer is what a restart needs; the session id identifies it.
write_sender_state(SenderState{ write_sender_state(SenderState{
.session_id = config_.session_id, .session_id = config_.session_id,
.receiver = std::format("{}:{}", config_.peer_rtp_endpoint.address, config_.peer_rtp_endpoint.port), .receiver = std::format("{}:{}", config_.peer_rtp_endpoint.address, config_.peer_rtp_endpoint.port),
.bitrate_kbps = config_.encoder.bitrate_kbps, .bitrate_kbps = config_.encoder.bitrate_kbps,
.pid = ::getpid(), .pid = ::getpid(),
.started_epoch_ms = 0}); .started_epoch_ms = 0});
// A restart targets the receiver's signaling endpoint; the RTP
// endpoint is re-negotiated from it.
std::string restart_peer = std::format("{}:5005", config_.peer_rtp_endpoint.address); std::string restart_peer = std::format("{}:5005", config_.peer_rtp_endpoint.address);
if (config_.signaling_server.has_value()) { if (config_.signaling_server.has_value()) {
restart_peer = std::format("{}:{}", config_.signaling_server->address, config_.signaling_server->port); restart_peer = std::format("{}:{}", config_.signaling_server->address, config_.signaling_server->port);
@@ -87,10 +85,16 @@ class SenderPipeline::Impl {
void request_keyframe() { void request_keyframe() {
keyframe_requested_.store(true); keyframe_requested_.store(true);
pli_count_.fetch_add(1, std::memory_order_relaxed);
} }
private: private:
void run(std::stop_token stop_token) { void run(std::stop_token stop_token) {
auto last_adaptation = std::chrono::steady_clock::now();
auto last_pli_check = last_adaptation;
auto last_pli_count = 0;
auto last_frame_time = last_adaptation;
while (!stop_token.stop_requested()) { while (!stop_token.stop_requested()) {
if (keyframe_requested_.exchange(false) && encoder_ != nullptr) { if (keyframe_requested_.exchange(false) && encoder_ != nullptr) {
encoder_->request_keyframe(); encoder_->request_keyframe();
@@ -105,6 +109,17 @@ class SenderPipeline::Impl {
break; break;
} }
// Frame rate capping: skip frames that arrive faster than the
// configured target. 0 = no cap (use the monitor rate).
if (config_.max_frame_rate > 0) {
const auto now = std::chrono::steady_clock::now();
const auto min_interval = std::chrono::microseconds(1'000'000 / config_.max_frame_rate);
if (now - last_frame_time < min_interval) {
continue;
}
last_frame_time = now;
}
auto encoded_result = encoder_->encode(*frame); auto encoded_result = encoder_->encode(*frame);
if (is_codec_error(encoded_result)) { if (is_codec_error(encoded_result)) {
std::cerr << std::format("screencast: encode failed: {}\n", codec_error(encoded_result).message); std::cerr << std::format("screencast: encode failed: {}\n", codec_error(encoded_result).message);
@@ -115,13 +130,70 @@ class SenderPipeline::Impl {
(void)transport_->send(packet); (void)transport_->send(packet);
} }
} }
// Adaptive quality: evaluate the link every 5 seconds by
// checking how many PLIs the receiver sent. Frequent PLIs
// mean the receiver is dropping frames — the link is
// saturated, so increase the CRF (lower quality, fewer
// bits). When the link is quiet, try lowering the CRF to
// probe for better quality.
const auto now = std::chrono::steady_clock::now();
if (now - last_pli_check >= std::chrono::seconds(5)) {
const auto current_pli = pli_count_.load(std::memory_order_relaxed);
const auto pli_delta = current_pli - last_pli_count;
const auto seconds = std::chrono::duration_cast<std::chrono::seconds>(now - last_pli_check).count();
const auto pli_per_second = static_cast<double>(pli_delta) / static_cast<double>(seconds);
last_pli_count = current_pli;
last_pli_check = now;
if (pli_per_second > 0.5 && current_crf_ < config_.encoder.crf + 10) {
// Saturated: degrade quality (higher CRF = fewer bits)
current_crf_ += 2;
std::cerr << std::format(
"screencast: link saturated ({} PLI/s); adapting CRF to {}\n", pli_per_second, current_crf_);
if (!restart_encoder(*frame)) {
break;
} }
} else if (pli_per_second < 0.1 && current_crf_ > config_.encoder.crf) {
// Stable: try better quality (lower CRF = more bits)
current_crf_ -= 1;
std::cerr << std::format("screencast: link stable; probing CRF {}\n", current_crf_);
if (!restart_encoder(*frame)) {
break;
}
}
}
}
}
bool restart_encoder(const CapturedFrame& frame) {
// Create a new encoder with the adjusted CRF; the next encoded
// frame is a keyframe, so the receiver recovers immediately.
encoder_ = nullptr;
return create_encoder(frame);
} }
bool create_encoder(const CapturedFrame& frame) { bool create_encoder(const CapturedFrame& frame) {
EncoderConfig config = config_.encoder; EncoderConfig config = config_.encoder;
config.width = frame.width; config.width = frame.width;
config.height = frame.height; config.height = frame.height;
config.crf = current_crf_;
// Downscale to the receiver's display when the capture is larger,
// preserving aspect ratio and rounding to even values (YUV420P
// requires even dimensions for the chroma planes).
if (config_.max_encode_width > 0 && config_.max_encode_height > 0 &&
(frame.width > config_.max_encode_width || frame.height > config_.max_encode_height)) {
const double scale = std::min(static_cast<double>(config_.max_encode_width) / frame.width,
static_cast<double>(config_.max_encode_height) / frame.height);
config.width = std::max(2, static_cast<int>(frame.width * scale) & ~1);
config.height = std::max(2, static_cast<int>(frame.height * scale) & ~1);
std::cerr << std::format("screencast: downscaling {}x{} to {}x{} for the receiver's display\n",
frame.width,
frame.height,
config.width,
config.height);
}
auto encoder_result = EncoderFactory::create(config); auto encoder_result = EncoderFactory::create(config);
if (is_codec_error(encoder_result)) { if (is_codec_error(encoder_result)) {
@@ -133,12 +205,14 @@ class SenderPipeline::Impl {
} }
SenderPipelineConfig config_; SenderPipelineConfig config_;
int current_crf_;
std::unique_ptr<CaptureSession> capture_; std::unique_ptr<CaptureSession> capture_;
std::unique_ptr<Encoder> encoder_; std::unique_ptr<Encoder> encoder_;
H264Packetizer packetizer_; H264Packetizer packetizer_;
std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create(); std::unique_ptr<RtpTransport> transport_ = RtpTransportFactory::create();
std::jthread run_thread_; std::jthread run_thread_;
std::atomic<bool> keyframe_requested_{false}; std::atomic<bool> keyframe_requested_{false};
std::atomic<int> pli_count_{0};
}; };
void SenderPipeline::request_keyframe() { void SenderPipeline::request_keyframe() {
@@ -334,6 +408,12 @@ class ReceiverPipeline::Impl {
// Empty address: the sender targets the address of its signaling // Empty address: the sender targets the address of its signaling
// connection, which reaches this RTP port. // connection, which reaches this RTP port.
answer.rtp_endpoint = Endpoint{"", config_.local_rtp_endpoint.port}; answer.rtp_endpoint = Endpoint{"", config_.local_rtp_endpoint.port};
// Tell the sender what display it is rendering to so it can
// downscale instead of encoding pixels the display cannot show.
if (renderer_ != nullptr) {
answer.display_width = renderer_->display_width();
answer.display_height = renderer_->display_height();
}
signaling_->send(answer); signaling_->send(answer);
} }
+14 -17
View File
@@ -13,6 +13,8 @@
namespace sc { namespace sc {
#ifdef SC_HAS_SENDER
namespace { namespace {
bool is_private_ipv4(std::string_view host) { bool is_private_ipv4(std::string_view host) {
@@ -34,24 +36,8 @@ bool is_private_ipv4(std::string_view host) {
} // namespace } // namespace
int SenderSession::address_preference(std::string_view host) {
if (host.find(':') == std::string_view::npos) {
return is_private_ipv4(host) ? 0 : 1;
}
if (host.rfind("fd", 0) == 0 || host.rfind("fc", 0) == 0) {
return 2;
}
if (host.rfind("2002:", 0) == 0) {
return 4;
}
if (host.rfind("fe80:", 0) == 0) {
return 5;
}
return 3;
}
std::variant<SenderSession, std::string> std::variant<SenderSession, std::string>
SenderSession::start(const Endpoint& signaling_endpoint, int bitrate_kbps, CaptureTarget target) { SenderSession::start(const Endpoint& signaling_endpoint, int bitrate_kbps, int crf, int max_fps, CaptureTarget target) {
auto channel_result = SignalingFactory::create_client(); auto channel_result = SignalingFactory::create_client();
if (is_network_error(channel_result)) { if (is_network_error(channel_result)) {
return network_error(channel_result).message; return network_error(channel_result).message;
@@ -112,7 +98,11 @@ SenderSession::start(const Endpoint& signaling_endpoint, int bitrate_kbps, Captu
config.peer_rtp_endpoint = rtp_endpoint; config.peer_rtp_endpoint = rtp_endpoint;
config.signaling_server = signaling_endpoint; config.signaling_server = signaling_endpoint;
config.encoder.bitrate_kbps = bitrate_kbps; config.encoder.bitrate_kbps = bitrate_kbps;
config.encoder.crf = crf;
config.max_frame_rate = max_fps;
config.session_id = session_id; config.session_id = session_id;
config.max_encode_width = answer.display_width;
config.max_encode_height = answer.display_height;
auto pipeline = std::make_unique<SenderPipeline>(std::move(config)); auto pipeline = std::make_unique<SenderPipeline>(std::move(config));
if (!pipeline->start()) { if (!pipeline->start()) {
@@ -174,4 +164,11 @@ void SenderSession::stop() {
} }
} }
#else
// Receiver-only builds have no sender pipeline; address_preference is the
// inline header function, and the rest of SenderSession is compiled out.
#endif // SC_HAS_SENDER
} // namespace sc } // namespace sc
+24 -7
View File
@@ -10,6 +10,29 @@
namespace sc { namespace sc {
// Ordering for trying a receiver's addresses: private IPv4 first (LANs,
// most reliable), then public IPv4, ULA, and global IPv6. 6to4 (2002::) and
// link-local addresses last: 6to4 is frequently unreachable between LAN
// peers, and link-local needs a scope id to even route. Inline so that
// receiver-only builds (no sender session) can still use it for discovery.
inline int address_preference(std::string_view host) {
if (host.find(':') == std::string_view::npos) {
const bool priv = host.rfind("192.168.", 0) == 0 || host.rfind("10.", 0) == 0 ||
(host.rfind("172.", 0) == 0 && host.find('.', 5) != std::string_view::npos);
return priv ? 0 : 1;
}
if (host.rfind("fd", 0) == 0 || host.rfind("fc", 0) == 0) {
return 2;
}
if (host.rfind("2002:", 0) == 0) {
return 4;
}
if (host.rfind("fe80:", 0) == 0) {
return 5;
}
return 3;
}
// A running sender session: it negotiated over signaling (keeping the // A running sender session: it negotiated over signaling (keeping the
// channel open for PLI feedback) and drives the sender pipeline. Shared by // channel open for PLI feedback) and drives the sender pipeline. Shared by
// the CLI, the GUI, and the waybar widget's one-click restart. // the CLI, the GUI, and the waybar widget's one-click restart.
@@ -19,13 +42,7 @@ class SenderSession {
// pipeline — which includes the portal's interactive source picker. // pipeline — which includes the portal's interactive source picker.
// Returns an error message on failure. // Returns an error message on failure.
static std::variant<SenderSession, std::string> static std::variant<SenderSession, std::string>
start(const Endpoint& signaling_endpoint, int bitrate_kbps, CaptureTarget target); start(const Endpoint& signaling_endpoint, int bitrate_kbps, int crf, int max_fps, CaptureTarget target);
// Ordering for trying a receiver's addresses: private IPv4 first (LANs,
// most reliable), then public IPv4, ULA, and global IPv6. 6to4 (2002::)
// and link-local addresses last: 6to4 is frequently unreachable between
// LAN peers, and link-local needs a scope id to even route.
static int address_preference(std::string_view host);
SenderSession() = default; SenderSession() = default;
~SenderSession(); ~SenderSession();
+53 -22
View File
@@ -147,34 +147,61 @@ class FfmpegDecoder final : public Decoder {
? static_cast<uint64_t>(av_rescale_q(input->pts, ctx_->time_base, AVRational{1, 1'000'000'000})) ? static_cast<uint64_t>(av_rescale_q(input->pts, ctx_->time_base, AVRational{1, 1'000'000'000}))
: fallback_timestamp_ns; : fallback_timestamp_ns;
const std::size_t buffer_size = const AVPixelFormat pixel_format = static_cast<AVPixelFormat>(input->format);
static_cast<std::size_t>(av_image_get_buffer_size(AV_PIX_FMT_RGBA, frame.width, frame.height, 1)); if (pixel_format != AV_PIX_FMT_YUV420P) {
frame.rgba_pixels.resize(buffer_size); // Hardware decoders (e.g. v4l2m2m) may emit NV12 or other planar
// variants; convert to YUV420P once. The common software path
if (!ensure_scaler(input->width, input->height, static_cast<AVPixelFormat>(input->format))) { // (YUV420P) skips this entirely.
return CodecError{"failed to create swscale context"}; if (!ensure_scaler(input->width, input->height, pixel_format)) {
return CodecError{"failed to create format conversion context"};
}
AvFramePtr converted(av_frame_alloc(), AvFrameDeleter{});
converted->width = input->width;
converted->height = input->height;
converted->format = AV_PIX_FMT_YUV420P;
if (av_frame_get_buffer(converted.get(), 0) < 0) {
return CodecError{"failed to allocate converted frame"};
}
if (sws_scale(scaler_.get(),
input->data,
input->linesize,
0,
input->height,
converted->data,
converted->linesize) <= 0) {
return CodecError{"failed to convert decoded frame to YUV420P"};
}
return copy_yuv420p_planes(converted.get(), std::move(frame));
} }
std::array<uint8_t*, 4> dst{nullptr, nullptr, nullptr, nullptr}; return copy_yuv420p_planes(input, std::move(frame));
std::array<int, 4> dst_lines{0, 0, 0, 0};
if (av_image_fill_arrays(dst.data(),
dst_lines.data(),
as_u8(frame.rgba_pixels.data()),
AV_PIX_FMT_RGBA,
frame.width,
frame.height,
1) < 0) {
return CodecError{"failed to fill output pixel arrays"};
} }
if (sws_scale(scaler_.get(), input->data, input->linesize, 0, input->height, dst.data(), dst_lines.data()) <= // Copies the three YUV420P planes with their strides (which may include
0) { // alignment padding). SDL's UpdateYUVTexture accepts arbitrary pitches.
return CodecError{"failed to convert decoded frame to RGBA"}; CodecResult<DecodedFrame> copy_yuv420p_planes(const AVFrame* input, DecodedFrame frame) const {
} const int width = input->width;
const int height = input->height;
frame.stride_y = input->linesize[0];
frame.stride_u = input->linesize[1];
frame.stride_v = input->linesize[2];
const std::size_t y_size = static_cast<std::size_t>(frame.stride_y) * height;
const std::size_t uv_size = static_cast<std::size_t>(frame.stride_u) * ((height + 1) / 2);
const std::size_t v_size = static_cast<std::size_t>(frame.stride_v) * ((height + 1) / 2);
frame.plane_y.resize(y_size);
frame.plane_u.resize(uv_size);
frame.plane_v.resize(v_size);
std::memcpy(frame.plane_y.data(), input->data[0], y_size);
std::memcpy(frame.plane_u.data(), input->data[1], uv_size);
std::memcpy(frame.plane_v.data(), input->data[2], v_size);
return frame; return frame;
} }
// Only used when the decoder emits a non-YUV420P format (hardware paths).
bool ensure_scaler(int width, int height, AVPixelFormat input_format) const { bool ensure_scaler(int width, int height, AVPixelFormat input_format) const {
if (scaler_ != nullptr && scaler_input_width_ == width && scaler_input_height_ == height && if (scaler_ != nullptr && scaler_input_width_ == width && scaler_input_height_ == height &&
scaler_input_format_ == input_format) { scaler_input_format_ == input_format) {
@@ -182,7 +209,7 @@ class FfmpegDecoder final : public Decoder {
} }
scaler_.reset(sws_getContext( scaler_.reset(sws_getContext(
width, height, input_format, width, height, AV_PIX_FMT_RGBA, SWS_BILINEAR, nullptr, nullptr, nullptr)); width, height, input_format, width, height, AV_PIX_FMT_YUV420P, SWS_BILINEAR, nullptr, nullptr, nullptr));
if (scaler_ == nullptr) { if (scaler_ == nullptr) {
return false; return false;
} }
@@ -222,7 +249,11 @@ std::optional<std::string> open_decoder(const DecoderConfig& config, const AVCod
if (config.height > 0) { if (config.height > 0) {
ctx->height = config.height; ctx->height = config.height;
} }
ctx->thread_count = 1; // Slice-level threading parallelizes within a single frame (no added
// latency), unlike frame-level threading which buffers multiple frames
// before producing output — unacceptable for a live stream.
ctx->thread_count = 4;
ctx->thread_type = FF_THREAD_SLICE;
if (!config.extradata.empty()) { if (!config.extradata.empty()) {
if (config.extradata.size() > static_cast<std::size_t>(std::numeric_limits<int>::max())) { if (config.extradata.size() > static_cast<std::size_t>(std::numeric_limits<int>::max())) {
+58 -14
View File
@@ -286,8 +286,12 @@ class FfmpegEncoder final : public Encoder {
return CodecError{"failed to allocate AVFrame"}; return CodecError{"failed to allocate AVFrame"};
} }
output->width = frame.width; // The output frame is at the encoder's configured dimensions (which
output->height = frame.height; // may be smaller than the capture when downscaling to the receiver's
// display); sws_scale handles both the format conversion and the
// resolution change in one pass.
output->width = config_.width;
output->height = config_.height;
output->format = AV_PIX_FMT_YUV420P; output->format = AV_PIX_FMT_YUV420P;
output->time_base = ctx_->time_base; output->time_base = ctx_->time_base;
output->pts = output->pts =
@@ -338,8 +342,18 @@ class FfmpegEncoder final : public Encoder {
return true; return true;
} }
scaler_.reset(sws_getContext( // Scale to the encoder's configured output (which may be smaller
width, height, input_format, width, height, AV_PIX_FMT_YUV420P, SWS_BILINEAR, nullptr, nullptr, nullptr)); // than the input when downscaling to the receiver's display).
scaler_.reset(sws_getContext(width,
height,
input_format,
config_.width,
config_.height,
AV_PIX_FMT_YUV420P,
SWS_BILINEAR,
nullptr,
nullptr,
nullptr));
if (scaler_ == nullptr) { if (scaler_ == nullptr) {
return false; return false;
} }
@@ -367,7 +381,10 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
return CodecError{"encoder frame rate must be positive"}; return CodecError{"encoder frame rate must be positive"};
} }
if (config.bitrate_kbps <= 0) { if (config.bitrate_kbps <= 0) {
return CodecError{"encoder bitrate must be positive"}; return CodecError{"encoder VBV max bitrate must be positive"};
}
if (config.crf < 0 || config.crf > 51) {
return CodecError{"encoder CRF must be between 0 and 51"};
} }
if (config.codec_name != "h264" && config.codec_name != "libx264") { if (config.codec_name != "h264" && config.codec_name != "libx264") {
return CodecError{"only h264 is supported in phase 2"}; return CodecError{"only h264 is supported in phase 2"};
@@ -395,14 +412,32 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
ctx->time_base = AVRational{1, 1'000'000}; ctx->time_base = AVRational{1, 1'000'000};
ctx->framerate = AVRational{config.frame_rate_num, config.frame_rate_den}; ctx->framerate = AVRational{config.frame_rate_num, config.frame_rate_den};
ctx->pix_fmt = AV_PIX_FMT_YUV420P; ctx->pix_fmt = AV_PIX_FMT_YUV420P;
ctx->bit_rate = static_cast<int64_t>(config.bitrate_kbps) * 1000;
// VBV keeps the stream CBR-ish: without it a keyframe may take many // CRF rate control: target a constant visual quality level instead of
// times the average frame size in one burst, overflowing the receiver's // a fixed bitrate, and let the encoder use fewer bits on static screen
// UDP socket buffer and dropping the packets that carry SPS/PPS. Two // content and more on motion or text. The VBV max rate still caps the
// frame periods of budget keep latency low while bounding the burst. // peak so bursts cannot overflow the receiver's UDP buffers.
ctx->rc_max_rate = ctx->bit_rate; // CRF rate control: set x264's CRF directly as a private option.
ctx->rc_buffer_size = static_cast<int>(ctx->bit_rate * 2 / config.frame_rate_num); // FFmpeg's global_quality + AV_CODEC_FLAG_QSCALE path divides by
ctx->gop_size = config.frame_rate_num; // FF_QP2LAMBDA, giving a wrong CRF value (16/118 ≈ 0.1, essentially
// lossless) — hence the "-qscale is ignored" warning. Setting the
// private "crf" option bypasses that and gives x264 the exact value.
const std::string crf_value = std::to_string(config.crf);
if (av_opt_set(ctx->priv_data, "crf", crf_value.c_str(), 0) < 0) {
return CodecError{"failed to set CRF quality"};
}
ctx->rc_max_rate = static_cast<int64_t>(config.bitrate_kbps) * 1000;
// VBV: one frame period of budget keeps bursts tight — a two-frame
// buffer lets a keyframe spike beyond what a constrained link can
// absorb in real time, causing packet loss that cascades into PLI
// storms. The tighter buffer trades a small quality dip on keyframes
// for much better behavior on slow paths.
ctx->rc_buffer_size = static_cast<int>(ctx->rc_max_rate / config.frame_rate_num);
// A long GOP saves the keyframe overhead for screen content (which
// changes incrementally); PLI feedback recovers from loss within one
// frame time regardless of the GOP length.
ctx->gop_size = config.frame_rate_num * 5;
ctx->max_b_frames = 0; ctx->max_b_frames = 0;
ctx->thread_count = 1; ctx->thread_count = 1;
ctx->profile = AV_PROFILE_H264_MAIN; ctx->profile = AV_PROFILE_H264_MAIN;
@@ -411,7 +446,7 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
// without out-of-band parameter negotiation. // without out-of-band parameter negotiation.
ctx->flags |= AV_CODEC_FLAG_LOW_DELAY; ctx->flags |= AV_CODEC_FLAG_LOW_DELAY;
if (av_opt_set(ctx->priv_data, "preset", "ultrafast", 0) < 0) { if (av_opt_set(ctx->priv_data, "preset", "faster", 0) < 0) {
return CodecError{"failed to set libx264 preset"}; return CodecError{"failed to set libx264 preset"};
} }
if (av_opt_set(ctx->priv_data, "tune", "zerolatency", 0) < 0) { if (av_opt_set(ctx->priv_data, "tune", "zerolatency", 0) < 0) {
@@ -420,6 +455,15 @@ CodecResult<std::unique_ptr<Encoder>> EncoderFactory::create(const EncoderConfig
if (av_opt_set(ctx->priv_data, "forced-idr", "1", 0) < 0) { if (av_opt_set(ctx->priv_data, "forced-idr", "1", 0) < 0) {
return CodecError{"failed to enable forced IDR keyframes"}; return CodecError{"failed to enable forced IDR keyframes"};
} }
// Screen-content tuning: auto-variance AQ allocates bits away from
// flat areas and toward text edges; higher psy-rd preserves texture
// sharpness at the cost of slight rate efficiency.
if (av_opt_set(ctx->priv_data, "aq-mode", "2", 0) < 0) {
return CodecError{"failed to set adaptive quantization mode"};
}
if (av_opt_set(ctx->priv_data, "psy-rd", "1.5", 0) < 0) {
return CodecError{"failed to set psychovisual rate-distortion strength"};
}
int open_ret = avcodec_open2(ctx.get(), codec, nullptr); int open_ret = avcodec_open2(ctx.get(), codec, nullptr);
if (open_ret < 0) { if (open_ret < 0) {
+112 -19
View File
@@ -1,6 +1,7 @@
// GTK4 sender panel: discover receivers on the LAN, pick one, set the // GTK4 sender panel: discover receivers on the LAN, pick one, choose a
// bitrate, and start/stop streaming. The blocking parts (portal source // quality preset, and start/stop streaming. The blocking parts (portal
// picker, negotiation) run on worker threads so the UI stays responsive. // source picker, negotiation) run on worker threads so the UI stays
// responsive.
#include "screencast/network/discovery.h" #include "screencast/network/discovery.h"
@@ -31,11 +32,36 @@ struct ReceiverRow {
std::uint16_t signaling_port = 0; std::uint16_t signaling_port = 0;
}; };
// Quality presets: CRF (visual quality), max bitrate (VBV cap), and frame
// rate cap. Each maps to a real-world CLI invocation. The frame rate is a
// critical bandwidth lever for desktop content — 15-25 fps is perfectly
// smooth for screencasting and cuts bandwidth 2-4x vs. monitor rate.
struct QualityPreset {
std::string label;
int crf;
int bitrate_kbps;
int fps; // 0 = no cap (use monitor rate)
};
const std::vector<QualityPreset>& quality_presets() {
static const std::vector<QualityPreset> presets = {
{"Low bandwidth", 26, 3000, 15},
{"Balanced", 22, 5000, 20},
{"Sharp", 18, 8000, 25},
{"Very sharp", 16, 12000, 30},
{"Maximum", 14, 20000, 0},
};
return presets;
}
class SenderWindow : public Gtk::ApplicationWindow { class SenderWindow : public Gtk::ApplicationWindow {
public: public:
SenderWindow() { SenderWindow() {
set_title("screencast"); set_title("screencast");
set_default_size(440, 400); set_default_size(440, 440);
// GTK4 only supports themed icons: the PNG is installed into the
// hicolor theme (see meson.build) and picked up by name.
set_icon_name("screencast");
auto* box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::VERTICAL, 8); auto* box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::VERTICAL, 8);
set_child(*box); set_child(*box);
@@ -64,22 +90,57 @@ class SenderWindow : public Gtk::ApplicationWindow {
[this](Gtk::ListBoxRow*) { Glib::signal_idle().connect_once([this] { update_sensitivity(); }); }); [this](Gtk::ListBoxRow*) { Glib::signal_idle().connect_once([this] { update_sensitivity(); }); });
scrolled_->set_child(*receiver_list_); scrolled_->set_child(*receiver_list_);
// Bitrate. // Quality preset dropdown.
auto* bitrate_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8); auto* quality_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*bitrate_box); box->append(*quality_box);
auto* quality_label = Gtk::make_managed<Gtk::Label>();
quality_label->set_text("Quality:");
quality_label->set_halign(Gtk::Align::START);
quality_box->append(*quality_label);
quality_combo_ = Gtk::make_managed<Gtk::DropDown>();
auto preset_list = Gtk::StringList::create({"placeholder"});
preset_list->remove(0);
for (const QualityPreset& preset : quality_presets()) {
preset_list->append(preset.label);
}
quality_combo_->set_model(preset_list);
quality_combo_->set_selected(1); // Standard
quality_combo_->property_selected().signal_changed().connect([this] { on_preset_changed(); });
quality_combo_->set_hexpand(true);
quality_box->append(*quality_combo_);
// Fine-tuning: bitrate cap and frame rate, updated by the preset.
auto* detail_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*detail_box);
bitrate_label_ = Gtk::make_managed<Gtk::Label>(); bitrate_label_ = Gtk::make_managed<Gtk::Label>();
bitrate_label_->set_hexpand(true); bitrate_label_->set_hexpand(true);
bitrate_label_->set_halign(Gtk::Align::START); bitrate_label_->set_halign(Gtk::Align::START);
bitrate_box->append(*bitrate_label_); detail_box->append(*bitrate_label_);
bitrate_scale_ = Gtk::make_managed<Gtk::Scale>(Gtk::Orientation::HORIZONTAL); bitrate_scale_ = Gtk::make_managed<Gtk::Scale>(Gtk::Orientation::HORIZONTAL);
bitrate_scale_->set_range(500.0, 20000.0); bitrate_scale_->set_range(500.0, 20000.0);
bitrate_scale_->set_value(4000.0); bitrate_scale_->set_value(5000.0);
bitrate_scale_->set_increments(500.0, 1000.0); bitrate_scale_->set_increments(500.0, 1000.0);
bitrate_scale_->set_draw_value(false); bitrate_scale_->set_draw_value(false);
bitrate_scale_->set_hexpand(true); bitrate_scale_->set_hexpand(true);
bitrate_scale_->signal_value_changed().connect([this] { update_bitrate_label(); }); bitrate_scale_->signal_value_changed().connect([this] { update_quality_labels(); });
bitrate_box->append(*bitrate_scale_); detail_box->append(*bitrate_scale_);
update_bitrate_label();
auto* fps_box = Gtk::make_managed<Gtk::Box>(Gtk::Orientation::HORIZONTAL, 8);
box->append(*fps_box);
auto* fps_label = Gtk::make_managed<Gtk::Label>();
fps_label->set_text("Frame rate:");
fps_label->set_halign(Gtk::Align::START);
fps_box->append(*fps_label);
auto fps_adjustment = Gtk::Adjustment::create(20.0, 5.0, 60.0, 1.0, 5.0, 0.0);
fps_spin_ = Gtk::make_managed<Gtk::SpinButton>(fps_adjustment, 1.0, 0);
fps_spin_->signal_value_changed().connect([this] { update_quality_labels(); });
fps_box->append(*fps_spin_);
auto* fps_hint = Gtk::make_managed<Gtk::Label>();
fps_hint->set_text("0 = uncapped (monitor rate)");
fps_hint->set_halign(Gtk::Align::END);
fps_hint->set_hexpand(true);
fps_hint->set_sensitive(false);
fps_box->append(*fps_hint);
start_button_ = Gtk::make_managed<Gtk::Button>(); start_button_ = Gtk::make_managed<Gtk::Button>();
start_button_->set_label("Start"); start_button_->set_label("Start");
@@ -101,6 +162,7 @@ class SenderWindow : public Gtk::ApplicationWindow {
}, },
1); 1);
update_quality_labels();
update_sensitivity(); update_sensitivity();
on_refresh(); on_refresh();
} }
@@ -112,14 +174,41 @@ class SenderWindow : public Gtk::ApplicationWindow {
} }
private: private:
void update_bitrate_label() {
bitrate_label_->set_text(std::format("Bitrate: {} kbps", current_bitrate()));
}
int current_bitrate() const { int current_bitrate() const {
return static_cast<int>(bitrate_scale_->get_value()); return static_cast<int>(bitrate_scale_->get_value());
} }
int current_crf() const {
// CRF from the preset, adjusted by the bitrate slider's distance
// from the preset's default: moving the slider up from the preset
// means the user wants more headroom, so we keep the preset's CRF
// (the slider fine-tunes the cap, not the quality target).
const auto index = quality_combo_->get_selected();
if (index >= quality_presets().size()) {
return 22;
}
return quality_presets()[index].crf;
}
void on_preset_changed() {
const auto index = quality_combo_->get_selected();
if (index >= quality_presets().size()) {
return;
}
const QualityPreset& preset = quality_presets()[index];
bitrate_scale_->set_value(static_cast<double>(preset.bitrate_kbps));
fps_spin_->set_value(preset.fps);
update_quality_labels();
}
void update_quality_labels() {
bitrate_label_->set_text(std::format("Max bitrate: {} kbps · CRF {}", current_bitrate(), current_crf()));
}
int current_fps() const {
return static_cast<int>(fps_spin_->get_value());
}
void clear_receiver_rows() { void clear_receiver_rows() {
for (Gtk::Widget* row : receiver_rows_) { for (Gtk::Widget* row : receiver_rows_) {
receiver_list_->remove(*row); receiver_list_->remove(*row);
@@ -187,7 +276,7 @@ class SenderWindow : public Gtk::ApplicationWindow {
std::vector<ReceiverRow> rows; std::vector<ReceiverRow> rows;
for (auto& [key, hosts] : grouped) { for (auto& [key, hosts] : grouped) {
std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) { std::sort(hosts.begin(), hosts.end(), [](const std::string& lhs, const std::string& rhs) {
return sc::SenderSession::address_preference(lhs) < sc::SenderSession::address_preference(rhs); return sc::address_preference(lhs) < sc::address_preference(rhs);
}); });
ReceiverRow row; ReceiverRow row;
row.name = key.first; row.name = key.first;
@@ -223,12 +312,14 @@ class SenderWindow : public Gtk::ApplicationWindow {
const ReceiverRow& receiver = receivers_[static_cast<std::size_t>(index)]; const ReceiverRow& receiver = receivers_[static_cast<std::size_t>(index)];
const Endpoint signaling{receiver.host, receiver.signaling_port}; const Endpoint signaling{receiver.host, receiver.signaling_port};
const int bitrate = current_bitrate(); const int bitrate = current_bitrate();
const int crf = current_crf();
const int fps = current_fps();
start_button_->set_sensitive(false); start_button_->set_sensitive(false);
status(std::format("connecting to {}… (choose a source in the portal dialog)", receiver.name)); status(std::format("connecting to {}… (choose a source in the portal dialog)", receiver.name));
worker_ = std::jthread([this, signaling, bitrate](std::stop_token) { worker_ = std::jthread([this, signaling, bitrate, crf, fps](std::stop_token) {
auto result = sc::SenderSession::start(signaling, bitrate, sc::CaptureTargetWholeScreen{}); auto result = sc::SenderSession::start(signaling, bitrate, crf, fps, sc::CaptureTargetWholeScreen{});
if (auto* error = std::get_if<std::string>(&result)) { if (auto* error = std::get_if<std::string>(&result)) {
Glib::signal_idle().connect_once([this, message = *error] { Glib::signal_idle().connect_once([this, message = *error] {
start_button_->set_label("Start"); start_button_->set_label("Start");
@@ -295,7 +386,9 @@ class SenderWindow : public Gtk::ApplicationWindow {
Gtk::Button* refresh_button_ = nullptr; Gtk::Button* refresh_button_ = nullptr;
Gtk::ScrolledWindow* scrolled_ = nullptr; Gtk::ScrolledWindow* scrolled_ = nullptr;
Gtk::ListBox* receiver_list_ = nullptr; Gtk::ListBox* receiver_list_ = nullptr;
Gtk::DropDown* quality_combo_ = nullptr;
Gtk::Scale* bitrate_scale_ = nullptr; Gtk::Scale* bitrate_scale_ = nullptr;
Gtk::SpinButton* fps_spin_ = nullptr;
Gtk::Label* bitrate_label_ = nullptr; Gtk::Label* bitrate_label_ = nullptr;
Gtk::Button* start_button_ = nullptr; Gtk::Button* start_button_ = nullptr;
Gtk::Label* status_label_ = nullptr; Gtk::Label* status_label_ = nullptr;
+10
View File
@@ -1,6 +1,16 @@
# Core public headers / include dependency. # Core public headers / include dependency.
sc_core_inc = include_directories('../include') sc_core_inc = include_directories('../include')
# Embed the application icon so the GUI window and the SDL window icon work
# from the build tree, after install, and on headless targets without any
# runtime path lookup.
icon_to_header = find_program('../scripts/icon_to_header.py')
icon_png_data = custom_target('icon_png_data',
input : files('../screencast_icon/screencast_256.png'),
output : 'icon_png_data.h',
command : [icon_to_header, '@INPUT@', '@OUTPUT@'])
sc_icon_dep = declare_dependency(sources : [icon_png_data])
# The sender needs the PipeWire / xdg-desktop-portal capture backend; # The sender needs the PipeWire / xdg-desktop-portal capture backend;
# receiver-only builds skip it entirely. # receiver-only builds skip it entirely.
build_sender = get_option('sender') build_sender = get_option('sender')
+8
View File
@@ -120,6 +120,8 @@ std::string serialize_message(const SignalingMessage& message) {
json["session_id"] = answer->session_id; json["session_id"] = answer->session_id;
json["rtp_address"] = answer->rtp_endpoint.address; json["rtp_address"] = answer->rtp_endpoint.address;
json["rtp_port"] = answer->rtp_endpoint.port; json["rtp_port"] = answer->rtp_endpoint.port;
json["display_width"] = answer->display_width;
json["display_height"] = answer->display_height;
} else { } else {
const SessionPli& pli = std::get<SessionPli>(message); const SessionPli& pli = std::get<SessionPli>(message);
json["type"] = "pli"; json["type"] = "pli";
@@ -172,6 +174,12 @@ std::optional<SignalingMessage> parse_message(std::string_view line) {
SessionAnswer answer; SessionAnswer answer;
answer.session_id = session_id; answer.session_id = session_id;
answer.rtp_endpoint = rtp_endpoint; answer.rtp_endpoint = rtp_endpoint;
if (json.contains("display_width") && json.at("display_width").is_number_integer()) {
answer.display_width = json.at("display_width").get<int>();
}
if (json.contains("display_height") && json.at("display_height").is_number_integer()) {
answer.display_height = json.at("display_height").get<int>();
}
return answer; return answer;
} }
if (type == "pli") { if (type == "pli") {
+18 -2
View File
@@ -2,14 +2,30 @@
dep_sdl3 = dependency('sdl3') dep_sdl3 = dependency('sdl3')
# SDL3_image (window icon) is optional: no pkg-config file ships with it,
# so find the library directly. Builds without it (e.g. the Pi receiver)
# simply skip the window icon.
cc = meson.get_compiler('cpp')
dep_sdl3_image = cc.find_library('SDL3_image', required: false)
if not dep_sdl3_image.found()
dep_sdl3_image = disabler()
endif
sc_render_args = []
if dep_sdl3_image.found() and cc.has_header('SDL3_image/SDL_image.h', dependencies : dep_sdl3_image)
sc_render_args += ['-DSC_HAS_WINDOW_ICON=1']
endif
sc_render_sources = files('sdl_renderer.cpp') sc_render_sources = files('sdl_renderer.cpp')
sc_render = static_library('sc_render', sc_render = static_library('sc_render',
sc_render_sources, sc_render_sources,
include_directories : sc_core_inc, include_directories : sc_core_inc,
dependencies : [dep_sdl3]) cpp_args : sc_render_args,
dependencies : [dep_sdl3, dep_sdl3_image, sc_icon_dep])
sc_render_dep = declare_dependency( sc_render_dep = declare_dependency(
link_with : sc_render, link_with : sc_render,
include_directories : sc_core_inc, include_directories : sc_core_inc,
dependencies : [dep_sdl3]) compile_args : sc_render_args,
dependencies : [dep_sdl3, dep_sdl3_image])
+68 -7
View File
@@ -4,13 +4,23 @@
#include <string> #include <string>
#ifdef SC_HAS_WINDOW_ICON
#include <SDL3_image/SDL_image.h>
#include <cstdio>
#include "icon_png_data.h"
#endif
namespace sc { namespace sc {
namespace { namespace {
// DecodedFrame pixels are AV_PIX_FMT_RGBA: memory order R, G, B, A. SDL // DecodedFrame carries YUV420P planar data; SDL_PIXELFORMAT_IYUV is the
// names 32-bit formats MSB-first, so that byte order is SDL's ABGR8888 — // matching SDL texture format. The GPU does the YUV→RGB conversion during
// using RGBA8888 would read the alpha byte as red and tint the image red. // rendering, eliminating a CPU-side swscale pass.
constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_ABGR8888; constexpr SDL_PixelFormat kSdlPixelFormat = SDL_PIXELFORMAT_IYUV;
class SdlRenderer final : public Renderer { class SdlRenderer final : public Renderer {
public: public:
@@ -52,6 +62,10 @@ class SdlRenderer final : public Renderer {
(void)SDL_HideCursor(); (void)SDL_HideCursor();
} }
#ifdef SC_HAS_WINDOW_ICON
set_window_icon();
#endif
renderer_ = SDL_CreateRenderer(window_, nullptr); renderer_ = SDL_CreateRenderer(window_, nullptr);
if (renderer_ == nullptr) { if (renderer_ == nullptr) {
last_error_ = std::string{"SDL_CreateRenderer failed: "} + SDL_GetError(); last_error_ = std::string{"SDL_CreateRenderer failed: "} + SDL_GetError();
@@ -59,6 +73,14 @@ class SdlRenderer final : public Renderer {
return false; return false;
} }
// Cache the display resolution: in fullscreen (KMSDRM on the Pi) this
// is the native monitor size; in windowed mode it is the window.
int window_width = 0;
int window_height = 0;
SDL_GetWindowSize(window_, &window_width, &window_height);
display_width_ = window_width;
display_height_ = window_height;
// Commit the surface once: on Wayland a window only becomes visible // Commit the surface once: on Wayland a window only becomes visible
// after the first present, and the receiver must be visible while it // after the first present, and the receiver must be visible while it
// waits for the stream to start. // waits for the stream to start.
@@ -73,8 +95,16 @@ class SdlRenderer final : public Renderer {
return true; return true;
} }
int display_width() const override {
return display_width_;
}
int display_height() const override {
return display_height_;
}
bool present(const DecodedFrame& frame) override { bool present(const DecodedFrame& frame) override {
if (frame.width <= 0 || frame.height <= 0) { if (frame.width <= 0 || frame.height <= 0 || frame.plane_y.empty()) {
return false; return false;
} }
if (frame.width != texture_width_ || frame.height != texture_height_) { if (frame.width != texture_width_ || frame.height != texture_height_) {
@@ -83,8 +113,14 @@ class SdlRenderer final : public Renderer {
} }
} }
const int pitch = frame.width * 4; if (!SDL_UpdateYUVTexture(texture_,
if (!SDL_UpdateTexture(texture_, nullptr, frame.rgba_pixels.data(), pitch)) { nullptr,
reinterpret_cast<const uint8_t*>(frame.plane_y.data()),
frame.stride_y,
reinterpret_cast<const uint8_t*>(frame.plane_u.data()),
frame.stride_u,
reinterpret_cast<const uint8_t*>(frame.plane_v.data()),
frame.stride_v)) {
return false; return false;
} }
// Clear first so the letterbox bars stay black between frames. // Clear first so the letterbox bars stay black between frames.
@@ -133,6 +169,29 @@ class SdlRenderer final : public Renderer {
} }
private: private:
#ifdef SC_HAS_WINDOW_ICON
// The icon is embedded at build time (see icon_png_data.h), so it works
// without any runtime file lookup. Harmless under KMSDRM, where there is
// no window manager to display it.
void set_window_icon() {
auto* stream = SDL_IOFromConstMem(sc::app_icon_png.data(), sc::app_icon_png.size());
if (stream == nullptr) {
std::fprintf(stderr, "screencast: failed to create icon stream: %s\n", SDL_GetError());
return;
}
// closeio=true: SDL_image consumes the stream, success or failure.
SDL_Surface* icon_surface = IMG_Load_IO(stream, true);
if (icon_surface == nullptr) {
std::fprintf(stderr, "screencast: failed to load window icon: %s\n", SDL_GetError());
return;
}
if (SDL_SetWindowIcon(window_, icon_surface) != 0) {
std::fprintf(stderr, "screencast: failed to set window icon: %s\n", SDL_GetError());
}
SDL_DestroySurface(icon_surface);
}
#endif
bool recreate_texture(int width, int height) { bool recreate_texture(int width, int height) {
if (texture_ != nullptr) { if (texture_ != nullptr) {
SDL_DestroyTexture(texture_); SDL_DestroyTexture(texture_);
@@ -164,6 +223,8 @@ class SdlRenderer final : public Renderer {
SDL_Texture* texture_ = nullptr; SDL_Texture* texture_ = nullptr;
int texture_width_ = 0; int texture_width_ = 0;
int texture_height_ = 0; int texture_height_ = 0;
int display_width_ = 0;
int display_height_ = 0;
}; };
} // namespace } // namespace
+1 -1
View File
@@ -90,7 +90,7 @@ struct ReceiverSink {
for (const sc::DecodedFrame& decoded : sc::codec_value(decoded_result)) { for (const sc::DecodedFrame& decoded : sc::codec_value(decoded_result)) {
last_width = decoded.width; last_width = decoded.width;
last_height = decoded.height; last_height = decoded.height;
saw_frame = decoded.width > 0 && !decoded.rgba_pixels.empty(); saw_frame = decoded.width > 0 && !decoded.plane_y.empty();
decoded_frames.fetch_add(1); decoded_frames.fetch_add(1);
} }
} }
+43 -3
View File
@@ -1,7 +1,12 @@
#include "screencast/codec/decoder.h" #include "screencast/codec/decoder.h"
#include "screencast/codec/encoder.h" #include "screencast/codec/encoder.h"
#include <algorithm> // swscale is a C library; without the extern wrapper its functions get
// C++ mangled and the linker cannot find them.
extern "C" {
#include <libswscale/swscale.h>
}
#include <cassert> #include <cassert>
#include <cstdint> #include <cstdint>
#include <cstdlib> #include <cstdlib>
@@ -47,10 +52,45 @@ sc::CapturedFrame make_frame(uint32_t index) {
return frame; return frame;
} }
// Convert a decoded YUV420P frame back to RGBA for pixel comparison with
// the original capture. Test-only; the pipeline itself never converts.
std::vector<std::byte> decoded_to_rgba(const sc::DecodedFrame& decoded) {
const std::size_t rgba_size = static_cast<std::size_t>(decoded.width) * decoded.height * 4;
std::vector<std::byte> rgba(rgba_size);
const uint8_t* src_planes[4] = {
reinterpret_cast<const uint8_t*>(decoded.plane_y.data()),
reinterpret_cast<const uint8_t*>(decoded.plane_u.data()),
reinterpret_cast<const uint8_t*>(decoded.plane_v.data()),
nullptr,
};
const int src_strides[4] = {decoded.stride_y, decoded.stride_u, decoded.stride_v, 0};
uint8_t* dst_planes[4] = {reinterpret_cast<uint8_t*>(rgba.data()), nullptr, nullptr, nullptr};
const int dst_strides[4] = {decoded.width * 4, 0, 0, 0};
SwsContext* scaler = sws_getContext(decoded.width,
decoded.height,
AV_PIX_FMT_YUV420P,
decoded.width,
decoded.height,
AV_PIX_FMT_RGBA,
SWS_BILINEAR,
nullptr,
nullptr,
nullptr);
assert(scaler != nullptr);
(void)sws_scale(scaler, src_planes, src_strides, 0, decoded.height, dst_planes, dst_strides);
sws_freeContext(scaler);
return rgba;
}
int max_channel_difference(const sc::DecodedFrame& decoded, const sc::CapturedFrame& expected) { int max_channel_difference(const sc::DecodedFrame& decoded, const sc::CapturedFrame& expected) {
const auto* decoded_pixels = reinterpret_cast<const std::uint8_t*>(decoded.rgba_pixels.data()); const auto decoded_rgba = decoded_to_rgba(decoded);
const auto* decoded_pixels = reinterpret_cast<const std::uint8_t*>(decoded_rgba.data());
const auto* expected_pixels = reinterpret_cast<const std::uint8_t*>(expected.pixels.data()); const auto* expected_pixels = reinterpret_cast<const std::uint8_t*>(expected.pixels.data());
const std::size_t count = std::min(decoded.rgba_pixels.size(), expected.pixels.size()); const std::size_t count = std::min(decoded_rgba.size(), expected.pixels.size());
int max_diff = 0; int max_diff = 0;
for (std::size_t i = 0; i < count; ++i) { for (std::size_t i = 0; i < count; ++i) {