docs: document the Android receiver app (Phase 8)
README: phone-as-second-screen section (build, install, sender usage, multi-monitor caveat). RUNBOOK: Android build/test procedure, device-test commands, and the API-36 platform quirks found during validation. PHASES: Phase 8 milestones, all validated. MEMORY: current phase state and the hard-won Android/portal lessons.
This commit is contained in:
+25
-2
@@ -1,10 +1,33 @@
|
||||
# Project Memory — screen_cast
|
||||
|
||||
Last updated: Phase 6 (discovery + signaling) complete and validated over
|
||||
loopback; current phase is Phase 7.
|
||||
Last updated: Phase 8 (Android receiver app) complete and validated on a
|
||||
Fairphone 6; all prior phases done.
|
||||
|
||||
## Project state
|
||||
|
||||
- **Phase 8 done: Android receiver app** (`android/`, Kotlin, minSdk 30,
|
||||
app id `screen_cast.receiver`): phone as second receiver (screen → HDMI via
|
||||
USB-C DP-alt-mode). Speaks the existing signaling+RTP protocol — no C++
|
||||
changes. AGP 9 **built-in Kotlin** (no kotlin plugin, no `kotlinOptions`;
|
||||
Kotlin targets `compileOptions`, Java 17); no androidx (framework +
|
||||
org.json + JUnit). 25 JVM tests green. Validated end-to-end on a
|
||||
Fairphone 6 (API 36): offer/answer → MediaCodec → fullscreen letterboxed
|
||||
render; PLI recovery on real Wi-Fi loss. Hard-won API-36 quirks (in
|
||||
RUNBOOK): `INTERNET` permission is REQUIRED for NsdService;
|
||||
`DatagramSocket.localPort` (`.port` is -1 unconnected; `.localAddress` is
|
||||
Inet6Address, not InetSocketAddress); pass the Surface to
|
||||
`MediaCodec.configure()` + `start()`; render via
|
||||
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
|
||||
configure (in-band SPS reconfigures); `MediaFormat.format()`/
|
||||
`KEY_MIME_TYPE` not public in API 36; NSD `RegistrationListener` replaced
|
||||
`ResolutionListener` (reflection fallback for older devices);
|
||||
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
|
||||
by sizing the TextureView to the video aspect, NOT a transform matrix
|
||||
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
|
||||
picks the FIRST output (eDP-1, wrong display) — use `--target window`
|
||||
(hyprland-share-picker). Activity FQN for `am start`:
|
||||
`screen_cast.receiver/screen_cast.ReceiverActivity` (Kotlin package =
|
||||
namespace `screen_cast`).
|
||||
- **Phase 6 done**: Avahi mDNS discovery (`_screencast._tcp` — receiver
|
||||
announces its signaling port via a threaded-poll Avahi client; senders
|
||||
browse+resolve) plus JSON session signaling (offer/answer) over TCP with
|
||||
|
||||
@@ -139,6 +139,38 @@ sudo scripts/pi-hotspot.sh off # back to normal router Wi-Fi
|
||||
While active the Pi is reachable at `10.42.0.1`; on the sender, join the
|
||||
hotspot's Wi-Fi and run `screencast --send`.
|
||||
|
||||
## Android receiver: a phone as the second screen
|
||||
|
||||
A native receiver app (Kotlin, minSdk 30) lets a phone act as the second
|
||||
receiver — and with a USB-C DisplayPort-alt-mode cable, the phone screen
|
||||
mirrors straight to HDMI. The app speaks the same signaling + RTP protocol,
|
||||
so the sender needs no changes.
|
||||
|
||||
```sh
|
||||
cd android
|
||||
gradle :app:assembleDebug # or :app:installDebug with a device attached
|
||||
adb install app/build/outputs/apk/debug/app-debug.apk
|
||||
```
|
||||
|
||||
Launch the app (it advertises `_screencast._tcp` and shows its IP + ports),
|
||||
then on the sender:
|
||||
|
||||
```sh
|
||||
screencast --send --peer <phone-ip>:5005
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- The sender downscales to the phone's display resolution automatically
|
||||
(from the signaling answer).
|
||||
- The phone renders letterboxed (fit-within), fullscreen, screen kept on.
|
||||
- The app needs only the `INTERNET` permission; no camera/location.
|
||||
- mDNS discovery works on the same L2 segment; across subnets use `--peer`
|
||||
(the app prints its IP and the exact fallback command).
|
||||
- `--target monitor` captures the portal's default output — on multi-monitor
|
||||
Hyprland/GTK-portal setups that may not be the one you want; use
|
||||
`--target window` and pick a window on the target display.
|
||||
|
||||
## Architecture
|
||||
|
||||
See `docs/ARCHITECTURE.md` for module boundaries and design rules.
|
||||
@@ -147,11 +179,11 @@ See `docs/ARCHITECTURE.md` for module boundaries and design rules.
|
||||
|
||||
Development is split into phases in `docs/PHASES.md`.
|
||||
|
||||
Current phase: **Phase 7 — Resilience and polish**. PLI recovery, the
|
||||
jitter buffer, hardware decode, and the GUI/waybar front-ends are complete
|
||||
and validated on a real two-machine setup (desktop → Raspberry Pi Zero 2 W
|
||||
over Wi-Fi); deferred items are the VAAPI hardware encode probe and
|
||||
packaging.
|
||||
Current phase: **Phase 8 — Android receiver app** (complete): the Kotlin
|
||||
receiver app streams to a phone (validated on a Fairphone 6, phone → HDMI
|
||||
via USB-C DP-alt-mode). Phase 7 items (PLI, jitter, hardware decode,
|
||||
GUI/waybar) are complete and validated; deferred items remain the VAAPI
|
||||
hardware encode probe and packaging.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
+24
-1
@@ -102,6 +102,29 @@ where available. PLI + jitter validated on the desktop (induced-loss probe);
|
||||
hardware decode validated as clean-fallback on the desktop, hardware path
|
||||
pending a run on the Pi.
|
||||
|
||||
## Phase 8 — Android Receiver App
|
||||
|
||||
**Goal**: a native Android receiver app (Kotlin) so a phone can act as the
|
||||
second receiver — screen on, USB-C DP-alt-mode to HDMI. The app speaks the
|
||||
existing signaling + RTP protocol; no C++ changes.
|
||||
|
||||
- [x] 8.1 Gradle project builds (`android/`, AGP 9 built-in Kotlin, no
|
||||
androidx; `gradle :app:assembleDebug`)
|
||||
- [x] 8.2 Kotlin sources + JVM unit tests (rtp framing, jitter buffer,
|
||||
depacketizer — 25 tests green)
|
||||
- [x] 8.3 Signaling + NSD validated on device (offer → answer over the
|
||||
network; mDNS registration via the API-36 RegistrationListener)
|
||||
- [x] 8.4 MediaCodec decode + Surface render validated on device (in-band
|
||||
SPS sizing, letterbox fit)
|
||||
- [x] 8.5 End-to-end on a Fairphone 6 (Android 16): streaming, letterboxed
|
||||
fullscreen render, PLI keyframe recovery on Wi-Fi loss
|
||||
- [x] 8.6 Docs: README receiver section, RUNBOOK build/install/test,
|
||||
PHASES + MEMORY updates
|
||||
|
||||
**Validation**: `gradle :app:testDebugUnitTest` green; live session
|
||||
`--send --target window --peer <phone>:5005` rendered fullscreen on the
|
||||
phone (and HDMI via DP-alt-mode) with loss recovery.
|
||||
|
||||
## Current phase
|
||||
|
||||
Phase 7 — Resilience and Polish.
|
||||
Phase 8 — Android Receiver App (complete).
|
||||
|
||||
@@ -209,6 +209,55 @@ The headless equivalent runs as part of `meson test` (`udp loopback` test):
|
||||
synthetic frames → encode → packetize → localhost UDP → depacketize →
|
||||
decode, no portal or window involved.
|
||||
|
||||
## Android receiver app (Phase 8)
|
||||
|
||||
Toolchain: JDK 21, system `gradle` (no wrapper), Android SDK with android-36.
|
||||
AGP 9 has **built-in Kotlin** — do not apply `org.jetbrains.kotlin.android`
|
||||
and do not use `kotlinOptions {}`.
|
||||
|
||||
```sh
|
||||
cd android
|
||||
gradle :app:assembleDebug # -> app/build/outputs/apk/debug/app-debug.apk
|
||||
gradle :app:testDebugUnitTest # 25 JVM tests (rtp, jitter, depacketizer)
|
||||
```
|
||||
|
||||
Device test (validated on a Fairphone 6, Android 16 / API 36):
|
||||
|
||||
```sh
|
||||
adb install -r app/build/outputs/apk/debug/app-debug.apk
|
||||
adb shell am start -n screen_cast.receiver/screen_cast.ReceiverActivity
|
||||
# activity FQN is screen_cast.ReceiverActivity (namespace), not
|
||||
# screen_cast.receiver.ReceiverActivity
|
||||
adb shell uiautomator dump /sdcard/ui.xml && adb shell cat /sdcard/ui.xml # status text
|
||||
adb exec-out screencap -p > phone.png # visual check
|
||||
```
|
||||
|
||||
On the sender: `screencast --send --peer <phone-ip>:5005` (add `--target
|
||||
window` to pick a specific window; the portal's monitor source picks the
|
||||
first output on multi-monitor Hyprland/GTK-portal setups).
|
||||
|
||||
Platform quirks found while validating (Android 16 / API 36):
|
||||
|
||||
- `android.permission.INTERNET` is required — NsdService rejects
|
||||
registration without it.
|
||||
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
|
||||
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
|
||||
(Inet6Address on Android), not an `InetSocketAddress`.
|
||||
- `MediaCodec`: pass the output Surface to `configure()` (`setOutputSurface`
|
||||
is illegal after configure); call `start()`; render with
|
||||
`releaseOutputBuffer(index, render=true)`; the C2 AVC decoder requires a
|
||||
concrete width/height at configure (use a placeholder — the in-band SPS
|
||||
reconfigures it).
|
||||
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
|
||||
surface.
|
||||
- NSD: API 36 replaced `ResolutionListener` registration with
|
||||
`registerService(info, flags, RegistrationListener)`; the app falls back
|
||||
to reflection on older devices.
|
||||
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
|
||||
— letterbox by **sizing the TextureView to the video aspect** (centered
|
||||
on a black window background), not with a transform matrix (double scale).
|
||||
- mDNS does not cross subnets; `--peer` is the reliable path.
|
||||
|
||||
## Formatting
|
||||
|
||||
```sh
|
||||
|
||||
Reference in New Issue
Block a user