diff --git a/.agents/MEMORY.md b/.agents/MEMORY.md index 1b5850b..69d7ba3 100644 --- a/.agents/MEMORY.md +++ b/.agents/MEMORY.md @@ -1,10 +1,33 @@ # Project Memory — screen_cast -Last updated: Phase 6 (discovery + signaling) complete and validated over -loopback; current phase is Phase 7. +Last updated: Phase 8 (Android receiver app) complete and validated on a +Fairphone 6; all prior phases done. ## Project state +- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30, + app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via + USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++ + changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`; + Kotlin targets `compileOptions`, Java 17); no androidx (framework + + org.json + JUnit). 25 JVM tests green. Validated end-to-end on a + Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed + render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in + RUNBOOK): `INTERNET` permission is REQUIRED for NsdService; + `DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is + Inet6Address, not InetSocketAddress); pass the Surface to + `MediaCodec.configure()` + `start()`; render via + `releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at + configure (in-band SPS reconfigures); `MediaFormat.format()`/ + `KEY_MIME_TYPE` not public in API 36; NSD `RegistrationListener` replaced + `ResolutionListener` (reflection fallback for older devices); + **`android._video-scaling`: C2 scales output to the Surface** → letterbox + by sizing the TextureView to the video aspect, NOT a transform matrix + (double scale). Sender-side: Hyprland + GTK portal's `--target monitor` + picks the FIRST output (eDP-1, wrong display) — use `--target window` + (hyprland-share-picker). Activity FQN for `am start`: + `screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package = + namespace `screen_cast`). - **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver announces its signaling port via a threaded-poll Avahi client; senders browse+resolve) plus JSON session signaling (offer/answer) over TCP with diff --git a/README.md b/README.md index 8b44e3c..b9e0de7 100644 --- a/README.md +++ b/README.md @@ -139,6 +139,38 @@ sudo scripts/pi-hotspot.sh off # back to normal router Wi-Fi While active the Pi is reachable at `10.42.0.1`; on the sender, join the hotspot's Wi-Fi and run `screencast --send`. +## Android receiver: a phone as the second screen + +A native receiver app (Kotlin, minSdk 30) lets a phone act as the second +receiver — and with a USB-C DisplayPort-alt-mode cable, the phone screen +mirrors straight to HDMI. The app speaks the same signaling + RTP protocol, +so the sender needs no changes. + +```sh +cd android +gradle :app:assembleDebug # or :app:installDebug with a device attached +adb install app/build/outputs/apk/debug/app-debug.apk +``` + +Launch the app (it advertises `_screencast._tcp` and shows its IP + ports), +then on the sender: + +```sh +screencast --send --peer :5005 +``` + +Notes: + +- The sender downscales to the phone's display resolution automatically + (from the signaling answer). +- The phone renders letterboxed (fit-within), fullscreen, screen kept on. +- The app needs only the `INTERNET` permission; no camera/location. +- mDNS discovery works on the same L2 segment; across subnets use `--peer` + (the app prints its IP and the exact fallback command). +- `--target monitor` captures the portal's default output — on multi-monitor + Hyprland/GTK-portal setups that may not be the one you want; use + `--target window` and pick a window on the target display. + ## Architecture See `docs/ARCHITECTURE.md` for module boundaries and design rules. @@ -147,11 +179,11 @@ See `docs/ARCHITECTURE.md` for module boundaries and design rules. Development is split into phases in `docs/PHASES.md`. -Current phase: **Phase 7 — Resilience and polish**. PLI recovery, the -jitter buffer, hardware decode, and the GUI/waybar front-ends are complete -and validated on a real two-machine setup (desktop → Raspberry Pi Zero 2 W -over Wi-Fi); deferred items are the VAAPI hardware encode probe and -packaging. +Current phase: **Phase 8 — Android receiver app** (complete): the Kotlin +receiver app streams to a phone (validated on a Fairphone 6, phone → HDMI +via USB-C DP-alt-mode). Phase 7 items (PLI, jitter, hardware decode, +GUI/waybar) are complete and validated; deferred items remain the VAAPI +hardware encode probe and packaging. ## License diff --git a/docs/PHASES.md b/docs/PHASES.md index e050fe8..bcbc76c 100644 --- a/docs/PHASES.md +++ b/docs/PHASES.md @@ -102,6 +102,29 @@ where available. PLI + jitter validated on the desktop (induced-loss probe); hardware decode validated as clean-fallback on the desktop, hardware path pending a run on the Pi. +## Phase 8 — Android Receiver App + +**Goal**: a native Android receiver app (Kotlin) so a phone can act as the +second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the +existing signaling + RTP protocol; no C++ changes. + +- [x] 8.1 Gradle project builds (`android/`, AGP 9 built-in Kotlin, no + androidx; `gradle :app:assembleDebug`) +- [x] 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer, + depacketizer — 25 tests green) +- [x] 8.3 Signaling + NSD validated on device (offer → answer over the + network; mDNS registration via the API-36 RegistrationListener) +- [x] 8.4 MediaCodec decode + Surface render validated on device (in-band + SPS sizing, letterbox fit) +- [x] 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed + fullscreen render, PLI keyframe recovery on Wi-Fi loss +- [x] 8.6 Docs: README receiver section, RUNBOOK build/install/test, + PHASES + MEMORY updates + +**Validation**: `gradle :app:testDebugUnitTest` green; live session +`--send --target window --peer :5005` rendered fullscreen on the +phone (and HDMI via DP-alt-mode) with loss recovery. + ## Current phase -Phase 7 — Resilience and Polish. +Phase 8 — Android Receiver App (complete). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index fa7e5ec..12422b0 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -209,6 +209,55 @@ The headless equivalent runs as part of `meson test` (`udp loopback` test): synthetic frames → encode → packetize → localhost UDP → depacketize → decode, no portal or window involved. +## Android receiver app (Phase 8) + +Toolchain: JDK 21, system `gradle` (no wrapper), Android SDK with android-36. +AGP 9 has **built-in Kotlin** — do not apply `org.jetbrains.kotlin.android` +and do not use `kotlinOptions {}`. + +```sh +cd android +gradle :app:assembleDebug # -> app/build/outputs/apk/debug/app-debug.apk +gradle :app:testDebugUnitTest # 25 JVM tests (rtp, jitter, depacketizer) +``` + +Device test (validated on a Fairphone 6, Android 16 / API 36): + +```sh +adb install -r app/build/outputs/apk/debug/app-debug.apk +adb shell am start -n screen_cast.receiver/screen_cast.ReceiverActivity +# activity FQN is screen_cast.ReceiverActivity (namespace), not +# screen_cast.receiver.ReceiverActivity +adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml # status text +adb exec-out screencap -p > phone.png # visual check +``` + +On the sender: `screencast --send --peer :5005` (add `--target +window` to pick a specific window; the portal's monitor source picks the +first output on multi-monitor Hyprland/GTK-portal setups). + +Platform quirks found while validating (Android 16 / API 36): + +- `android.permission.INTERNET` is required — NsdService rejects + registration without it. +- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for + unconnected datagram sockets, and `.localAddress` is an `InetAddress` + (Inet6Address on Android), not an `InetSocketAddress`. +- `MediaCodec`: pass the output Surface to `configure()` (`setOutputSurface` + is illegal after configure); call `start()`; render with + `releaseOutputBuffer(index, render=true)`; the C2 AVC decoder requires a + concrete width/height at configure (use a placeholder — the in-band SPS + reconfigures it). +- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public + surface. +- NSD: API 36 replaced `ResolutionListener` registration with + `registerService(info, flags, RegistrationListener)`; the app falls back + to reflection on older devices. +- The C2 decoder scales its output to the Surface (`android._video-scaling`) + — letterbox by **sizing the TextureView to the video aspect** (centered + on a black window background), not with a transform matrix (double scale). +- mDNS does not cross subnets; `--peer` is the reliable path. + ## Formatting ```sh