docs: document the Android receiver app (Phase 8)
README: phone-as-second-screen section (build, install, sender usage, multi-monitor caveat). RUNBOOK: Android build/test procedure, device-test commands, and the API-36 platform quirks found during validation. PHASES: Phase 8 milestones, all validated. MEMORY: current phase state and the hard-won Android/portal lessons.
This commit is contained in:
+25
-2
@@ -1,10 +1,33 @@
|
|||||||
# Project Memory — screen_cast
|
# Project Memory — screen_cast
|
||||||
|
|
||||||
Last updated: Phase 6 (discovery + signaling) complete and validated over
|
Last updated: Phase 8 (Android receiver app) complete and validated on a
|
||||||
loopback; current phase is Phase 7.
|
Fairphone 6; all prior phases done.
|
||||||
|
|
||||||
## Project state
|
## Project state
|
||||||
|
|
||||||
|
- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30,
|
||||||
|
app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via
|
||||||
|
USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++
|
||||||
|
changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`;
|
||||||
|
Kotlin targets `compileOptions`, Java 17); no androidx (framework +
|
||||||
|
org.json + JUnit). 25 JVM tests green. Validated end-to-end on a
|
||||||
|
Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed
|
||||||
|
render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in
|
||||||
|
RUNBOOK): `INTERNET` permission is REQUIRED for NsdService;
|
||||||
|
`DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is
|
||||||
|
Inet6Address, not InetSocketAddress); pass the Surface to
|
||||||
|
`MediaCodec.configure()` + `start()`; render via
|
||||||
|
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
|
||||||
|
configure (in-band SPS reconfigures); `MediaFormat.format()`/
|
||||||
|
`KEY_MIME_TYPE` not public in API 36; NSD `RegistrationListener` replaced
|
||||||
|
`ResolutionListener` (reflection fallback for older devices);
|
||||||
|
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
|
||||||
|
by sizing the TextureView to the video aspect, NOT a transform matrix
|
||||||
|
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
|
||||||
|
picks the FIRST output (eDP-1, wrong display) — use `--target window`
|
||||||
|
(hyprland-share-picker). Activity FQN for `am start`:
|
||||||
|
`screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package =
|
||||||
|
namespace `screen_cast`).
|
||||||
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
|
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
|
||||||
announces its signaling port via a threaded-poll Avahi client; senders
|
announces its signaling port via a threaded-poll Avahi client; senders
|
||||||
browse+resolve) plus JSON session signaling (offer/answer) over TCP with
|
browse+resolve) plus JSON session signaling (offer/answer) over TCP with
|
||||||
|
|||||||
@@ -139,6 +139,38 @@ sudo scripts/pi-hotspot.sh off # back to normal router Wi-Fi
|
|||||||
While active the Pi is reachable at `10.42.0.1`; on the sender, join the
|
While active the Pi is reachable at `10.42.0.1`; on the sender, join the
|
||||||
hotspot's Wi-Fi and run `screencast --send`.
|
hotspot's Wi-Fi and run `screencast --send`.
|
||||||
|
|
||||||
|
## Android receiver: a phone as the second screen
|
||||||
|
|
||||||
|
A native receiver app (Kotlin, minSdk 30) lets a phone act as the second
|
||||||
|
receiver — and with a USB-C DisplayPort-alt-mode cable, the phone screen
|
||||||
|
mirrors straight to HDMI. The app speaks the same signaling + RTP protocol,
|
||||||
|
so the sender needs no changes.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd android
|
||||||
|
gradle :app:assembleDebug # or :app:installDebug with a device attached
|
||||||
|
adb install app/build/outputs/apk/debug/app-debug.apk
|
||||||
|
```
|
||||||
|
|
||||||
|
Launch the app (it advertises `_screencast._tcp` and shows its IP + ports),
|
||||||
|
then on the sender:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
screencast --send --peer <phone-ip>:5005
|
||||||
|
```
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- The sender downscales to the phone's display resolution automatically
|
||||||
|
(from the signaling answer).
|
||||||
|
- The phone renders letterboxed (fit-within), fullscreen, screen kept on.
|
||||||
|
- The app needs only the `INTERNET` permission; no camera/location.
|
||||||
|
- mDNS discovery works on the same L2 segment; across subnets use `--peer`
|
||||||
|
(the app prints its IP and the exact fallback command).
|
||||||
|
- `--target monitor` captures the portal's default output — on multi-monitor
|
||||||
|
Hyprland/GTK-portal setups that may not be the one you want; use
|
||||||
|
`--target window` and pick a window on the target display.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
See `docs/ARCHITECTURE.md` for module boundaries and design rules.
|
See `docs/ARCHITECTURE.md` for module boundaries and design rules.
|
||||||
@@ -147,11 +179,11 @@ See `docs/ARCHITECTURE.md` for module boundaries and design rules.
|
|||||||
|
|
||||||
Development is split into phases in `docs/PHASES.md`.
|
Development is split into phases in `docs/PHASES.md`.
|
||||||
|
|
||||||
Current phase: **Phase 7 — Resilience and polish**. PLI recovery, the
|
Current phase: **Phase 8 — Android receiver app** (complete): the Kotlin
|
||||||
jitter buffer, hardware decode, and the GUI/waybar front-ends are complete
|
receiver app streams to a phone (validated on a Fairphone 6, phone → HDMI
|
||||||
and validated on a real two-machine setup (desktop → Raspberry Pi Zero 2 W
|
via USB-C DP-alt-mode). Phase 7 items (PLI, jitter, hardware decode,
|
||||||
over Wi-Fi); deferred items are the VAAPI hardware encode probe and
|
GUI/waybar) are complete and validated; deferred items remain the VAAPI
|
||||||
packaging.
|
hardware encode probe and packaging.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+24
-1
@@ -102,6 +102,29 @@ where available. PLI + jitter validated on the desktop (induced-loss probe);
|
|||||||
hardware decode validated as clean-fallback on the desktop, hardware path
|
hardware decode validated as clean-fallback on the desktop, hardware path
|
||||||
pending a run on the Pi.
|
pending a run on the Pi.
|
||||||
|
|
||||||
|
## Phase 8 — Android Receiver App
|
||||||
|
|
||||||
|
**Goal**: a native Android receiver app (Kotlin) so a phone can act as the
|
||||||
|
second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the
|
||||||
|
existing signaling + RTP protocol; no C++ changes.
|
||||||
|
|
||||||
|
- [x] 8.1 Gradle project builds (`android/`, AGP 9 built-in Kotlin, no
|
||||||
|
androidx; `gradle :app:assembleDebug`)
|
||||||
|
- [x] 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer,
|
||||||
|
depacketizer — 25 tests green)
|
||||||
|
- [x] 8.3 Signaling + NSD validated on device (offer → answer over the
|
||||||
|
network; mDNS registration via the API-36 RegistrationListener)
|
||||||
|
- [x] 8.4 MediaCodec decode + Surface render validated on device (in-band
|
||||||
|
SPS sizing, letterbox fit)
|
||||||
|
- [x] 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed
|
||||||
|
fullscreen render, PLI keyframe recovery on Wi-Fi loss
|
||||||
|
- [x] 8.6 Docs: README receiver section, RUNBOOK build/install/test,
|
||||||
|
PHASES + MEMORY updates
|
||||||
|
|
||||||
|
**Validation**: `gradle :app:testDebugUnitTest` green; live session
|
||||||
|
`--send --target window --peer <phone>:5005` rendered fullscreen on the
|
||||||
|
phone (and HDMI via DP-alt-mode) with loss recovery.
|
||||||
|
|
||||||
## Current phase
|
## Current phase
|
||||||
|
|
||||||
Phase 7 — Resilience and Polish.
|
Phase 8 — Android Receiver App (complete).
|
||||||
|
|||||||
@@ -209,6 +209,55 @@ The headless equivalent runs as part of `meson test` (`udp loopback` test):
|
|||||||
synthetic frames → encode → packetize → localhost UDP → depacketize →
|
synthetic frames → encode → packetize → localhost UDP → depacketize →
|
||||||
decode, no portal or window involved.
|
decode, no portal or window involved.
|
||||||
|
|
||||||
|
## Android receiver app (Phase 8)
|
||||||
|
|
||||||
|
Toolchain: JDK 21, system `gradle` (no wrapper), Android SDK with android-36.
|
||||||
|
AGP 9 has **built-in Kotlin** — do not apply `org.jetbrains.kotlin.android`
|
||||||
|
and do not use `kotlinOptions {}`.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd android
|
||||||
|
gradle :app:assembleDebug # -> app/build/outputs/apk/debug/app-debug.apk
|
||||||
|
gradle :app:testDebugUnitTest # 25 JVM tests (rtp, jitter, depacketizer)
|
||||||
|
```
|
||||||
|
|
||||||
|
Device test (validated on a Fairphone 6, Android 16 / API 36):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
adb install -r app/build/outputs/apk/debug/app-debug.apk
|
||||||
|
adb shell am start -n screen_cast.receiver/screen_cast.ReceiverActivity
|
||||||
|
# activity FQN is screen_cast.ReceiverActivity (namespace), not
|
||||||
|
# screen_cast.receiver.ReceiverActivity
|
||||||
|
adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml # status text
|
||||||
|
adb exec-out screencap -p > phone.png # visual check
|
||||||
|
```
|
||||||
|
|
||||||
|
On the sender: `screencast --send --peer <phone-ip>:5005` (add `--target
|
||||||
|
window` to pick a specific window; the portal's monitor source picks the
|
||||||
|
first output on multi-monitor Hyprland/GTK-portal setups).
|
||||||
|
|
||||||
|
Platform quirks found while validating (Android 16 / API 36):
|
||||||
|
|
||||||
|
- `android.permission.INTERNET` is required — NsdService rejects
|
||||||
|
registration without it.
|
||||||
|
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
|
||||||
|
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
|
||||||
|
(Inet6Address on Android), not an `InetSocketAddress`.
|
||||||
|
- `MediaCodec`: pass the output Surface to `configure()` (`setOutputSurface`
|
||||||
|
is illegal after configure); call `start()`; render with
|
||||||
|
`releaseOutputBuffer(index, render=true)`; the C2 AVC decoder requires a
|
||||||
|
concrete width/height at configure (use a placeholder — the in-band SPS
|
||||||
|
reconfigures it).
|
||||||
|
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
|
||||||
|
surface.
|
||||||
|
- NSD: API 36 replaced `ResolutionListener` registration with
|
||||||
|
`registerService(info, flags, RegistrationListener)`; the app falls back
|
||||||
|
to reflection on older devices.
|
||||||
|
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
|
||||||
|
— letterbox by **sizing the TextureView to the video aspect** (centered
|
||||||
|
on a black window background), not with a transform matrix (double scale).
|
||||||
|
- mDNS does not cross subnets; `--peer` is the reliable path.
|
||||||
|
|
||||||
## Formatting
|
## Formatting
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
|||||||
Reference in New Issue
Block a user