fix(android): harden the receiver after the review pass
- SignalingServer.send() never throws: a broken signaling connection drops the peer instead of killing the RTP reader thread via requestPli() - H264Decoder: assign the codec before configure/start (no orphaned instance on failure); feed() reports input-queue timeouts so the pipeline requests a keyframe instead of dropping the frame silently - Park offers that arrive without a surface as pendingOffer and configure on attachSurface(): a codec configured without a surface can never take one (setOutputSurface refuses it); the late configure sends a PLI - Forget destroyed surfaces (onSurfaceTextureDestroyed -> detachSurface) - Show the status overlay again on later messages - Accumulate NAL bytes in ByteArrayOutputStreams, not boxed ArrayList<Int> - Delete the dead NSD reflection fallback (ResolutionListener never existed; the classic API is present from API 16 through 36) - Hold decoderLock around all MediaCodec calls (not thread-safe) and make requestPli thread-safe - Update RUNBOOK quirks (NSD, setOutputSurface) and MEMORY notes
This commit is contained in:
+62
-4
@@ -1,7 +1,9 @@
|
|||||||
# Project Memory — screen_cast
|
# Project Memory — screen_cast
|
||||||
|
|
||||||
Last updated: Phase 8 (Android receiver app) complete and validated on a
|
Last updated: Phase 8 (Android receiver app) complete and validated on a
|
||||||
Fairphone 6; all prior phases done.
|
Fairphone 6; all prior phases done. The Android app's review findings were
|
||||||
|
fixed in a follow-up pass (same day) — see "Android app review" at the
|
||||||
|
bottom of this file.
|
||||||
|
|
||||||
## Project state
|
## Project state
|
||||||
|
|
||||||
@@ -19,8 +21,11 @@ Fairphone 6; all prior phases done.
|
|||||||
`MediaCodec.configure()` + `start()`; render via
|
`MediaCodec.configure()` + `start()`; render via
|
||||||
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
|
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
|
||||||
configure (in-band SPS reconfigures); `MediaFormat.format()`/
|
configure (in-band SPS reconfigures); `MediaFormat.format()`/
|
||||||
`KEY_MIME_TYPE` not public in API 36; NSD `RegistrationListener` replaced
|
`KEY_MIME_TYPE` not public in API 36; NSD: the classic
|
||||||
`ResolutionListener` (reflection fallback for older devices);
|
`registerService(info, flags, RegistrationListener)` API exists from API
|
||||||
|
16 through 36 (javap-verified on the android-36 SDK) — an old reflection
|
||||||
|
fallback targeting a never-existent "ResolutionListener" was removed as
|
||||||
|
dead code;
|
||||||
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
|
**`android._video-scaling`: C2 scales output to the Surface** → letterbox
|
||||||
by sizing the TextureView to the video aspect, NOT a transform matrix
|
by sizing the TextureView to the video aspect, NOT a transform matrix
|
||||||
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
|
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
|
||||||
@@ -258,4 +263,57 @@ None.
|
|||||||
- `CaptureSession::next_frame()` returns `nullopt` on stream error without
|
- `CaptureSession::next_frame()` returns `nullopt` on stream error without
|
||||||
surfacing the reason (logged to stderr).
|
surfacing the reason (logged to stderr).
|
||||||
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
|
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
|
||||||
we control never emit them, but third-party interop would need support.
|
we control never emit them, but third-party interop would need support.
|
||||||
|
|
||||||
|
### Android app review (2026-09-10) — findings FIXED same day
|
||||||
|
|
||||||
|
Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified
|
||||||
|
faithful to the C++ side source-to-source), followed by a fix pass that
|
||||||
|
landed all findings. No commit yet (user has not asked).
|
||||||
|
|
||||||
|
Fixed:
|
||||||
|
|
||||||
|
- `SignalingServer.send()` no longer throws (mirrors the C++ server, which
|
||||||
|
ignores write failures): a broken signaling TCP no longer kills the
|
||||||
|
`rtp-reader` thread via `requestPli()`; `peerOut` is dropped (closing the
|
||||||
|
socket) on write failure.
|
||||||
|
- `H264Decoder.configure()` assigns the created codec before configuring,
|
||||||
|
so a configure/start failure can no longer orphan the MediaCodec instance
|
||||||
|
(no finalizer; scarce native slots).
|
||||||
|
- Offer-before-surface no longer configures a ByteBuffer-mode decoder: the
|
||||||
|
offer is parked as `pendingOffer` and `attachSurface()` configures later
|
||||||
|
(a surface-less codec can never take one — `setOutputSurface` refuses
|
||||||
|
it, an IllegalStateException crash on the main thread). The late
|
||||||
|
configure sends a PLI (the sender only emits IDRs when asked).
|
||||||
|
- `ReceiverActivity.onSurfaceTextureDestroyed` → `pipeline.detachSurface()`;
|
||||||
|
the pipeline/decoder forget dead surfaces instead of configuring against
|
||||||
|
them (frames decode unrendered until the next attach).
|
||||||
|
- The status overlay reappears: `onStatus` sets `visibility = VISIBLE`
|
||||||
|
(it used to write into a GONE view after the first frame).
|
||||||
|
- `H264Decoder.feed()` returns false on input-queue timeout → the pipeline
|
||||||
|
requests a PLI instead of silently dropping the frame (no flush — the
|
||||||
|
codec is healthy).
|
||||||
|
- The depacketizer accumulates into `ByteArrayOutputStream`s instead of
|
||||||
|
boxed `ArrayList<Int>` (was ~MB/s of Integer allocations on keyframes).
|
||||||
|
- NSD reflection fallback deleted (dead code — see the quirk note above).
|
||||||
|
- Thread-safety: all codec calls now run under `decoderLock` (MediaCodec is
|
||||||
|
not thread-safe; feed/drain previously raced attachSurface);
|
||||||
|
`requestPli()` is thread-safe via `pliLock`.
|
||||||
|
|
||||||
|
Validation: `gradle :app:assembleDebug :app:testDebugUnitTest` green
|
||||||
|
(25/25, full --rerun-tasks rebuild, only two pre-existing warnings);
|
||||||
|
`meson test` 5/5 unchanged (no C++ touched).
|
||||||
|
|
||||||
|
Still open (accepted, needs a device or a new test dep):
|
||||||
|
|
||||||
|
- Untested on device: `setOutputSurface` mid-session (surface switch) and
|
||||||
|
the whole pendingOffer path; API-35 `detachOutputSurface()` could replace
|
||||||
|
the render-flag approach.
|
||||||
|
- No JVM tests for `SignalingMessage`/`SignalingServer` (would need the
|
||||||
|
`org.json:json` test dependency; android.jar stubs throw).
|
||||||
|
- Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200),
|
||||||
|
RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a
|
||||||
|
future phase).
|
||||||
|
- Verified NOT a bug: reusing one `DatagramPacket` without resetting its
|
||||||
|
length — modern JVMs recv by buffer capacity (JDK-21 probe + the
|
||||||
|
successful on-device streaming confirm it).
|
||||||
@@ -52,7 +52,13 @@ class ReceiverActivity : Activity() {
|
|||||||
fitVideo()
|
fitVideo()
|
||||||
}
|
}
|
||||||
|
|
||||||
override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean = true
|
override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
|
||||||
|
// The TextureView is about to release this SurfaceTexture; the
|
||||||
|
// pipeline must forget it or a later offer would configure the
|
||||||
|
// decoder against a dead surface.
|
||||||
|
pipeline?.detachSurface()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
|
override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
|
||||||
}
|
}
|
||||||
@@ -85,7 +91,12 @@ class ReceiverActivity : Activity() {
|
|||||||
val metrics = resources.displayMetrics
|
val metrics = resources.displayMetrics
|
||||||
metrics.widthPixels to metrics.heightPixels
|
metrics.widthPixels to metrics.heightPixels
|
||||||
},
|
},
|
||||||
onStatus = { text -> ui.post { statusView.text = text } },
|
onStatus = { text ->
|
||||||
|
ui.post {
|
||||||
|
statusView.text = text
|
||||||
|
statusView.visibility = View.VISIBLE
|
||||||
|
}
|
||||||
|
},
|
||||||
onFirstFrame = { ui.post { statusView.visibility = View.GONE } },
|
onFirstFrame = { ui.post { statusView.visibility = View.GONE } },
|
||||||
onVideoSize = { _, _ -> ui.post { fitVideo() } },
|
onVideoSize = { _, _ -> ui.post { fitVideo() } },
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -35,11 +35,26 @@ class H264Decoder {
|
|||||||
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240)
|
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240)
|
||||||
}
|
}
|
||||||
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE)
|
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE)
|
||||||
codec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC).also { c ->
|
// Assign before configuring so a configure()/start() failure cannot
|
||||||
// The surface must be passed to configure(); setOutputSurface()
|
// orphan the created instance: MediaCodec has no finalizer and each
|
||||||
// is only legal before configuration.
|
// unreleased instance holds a scarce native codec slot. The caller
|
||||||
c.configure(format, renderSurface, null, 0)
|
// follows an exception with release(), which is a no-op on null.
|
||||||
c.start()
|
val created = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
|
||||||
|
codec = created
|
||||||
|
try {
|
||||||
|
// The surface must go into configure(): the codec must start in
|
||||||
|
// surface mode. setOutputSurface() afterwards only switches an
|
||||||
|
// already-surface-mode codec; a codec configured without a
|
||||||
|
// surface can never take one.
|
||||||
|
created.configure(format, renderSurface, null, 0)
|
||||||
|
created.start()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
codec = null
|
||||||
|
try {
|
||||||
|
created.release()
|
||||||
|
} catch (ignored: Exception) {
|
||||||
|
}
|
||||||
|
throw e
|
||||||
}
|
}
|
||||||
configured = true
|
configured = true
|
||||||
}
|
}
|
||||||
@@ -48,19 +63,39 @@ class H264Decoder {
|
|||||||
@Synchronized
|
@Synchronized
|
||||||
fun attachSurface(surface: Surface) {
|
fun attachSurface(surface: Surface) {
|
||||||
renderSurface = surface
|
renderSurface = surface
|
||||||
|
// Legal only in surface mode: setOutputSurface() dynamically switches
|
||||||
|
// an output surface on a codec configured WITH one (per the platform
|
||||||
|
// docs); on a ByteBuffer-mode codec it throws IllegalStateException.
|
||||||
codec?.setOutputSurface(surface)
|
codec?.setOutputSurface(surface)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp). */
|
/**
|
||||||
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean) {
|
* Forgets a destroyed render surface: frames still decode (returning
|
||||||
|
* their buffers and keeping the queue moving) but are not rendered
|
||||||
|
* until the next [attachSurface]. API 35's detachOutputSurface() is the
|
||||||
|
* codec-side equivalent.
|
||||||
|
*/
|
||||||
|
@Synchronized
|
||||||
|
fun detachSurface() {
|
||||||
|
renderSurface = null
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp).
|
||||||
|
* @return false when the codec's input queue was full and the frame was
|
||||||
|
* dropped; the caller should request a keyframe, not flush (the codec
|
||||||
|
* is healthy, merely busy).
|
||||||
|
*/
|
||||||
|
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean): Boolean {
|
||||||
val c = codec ?: throw IllegalStateException("decoder not configured")
|
val c = codec ?: throw IllegalStateException("decoder not configured")
|
||||||
val index = c.dequeueInputBuffer(10_000)
|
val index = c.dequeueInputBuffer(10_000)
|
||||||
if (index < 0) return
|
if (index < 0) return false
|
||||||
val buffer = c.getInputBuffer(index) ?: return
|
val buffer = c.getInputBuffer(index) ?: return false
|
||||||
buffer.clear()
|
buffer.clear()
|
||||||
buffer.put(data)
|
buffer.put(data)
|
||||||
val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_SYNC_FRAME else 0
|
val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_KEY_FRAME else 0
|
||||||
c.queueInputBuffer(index, 0, data.size, presentationUs, flags)
|
c.queueInputBuffer(index, 0, data.size, presentationUs, flags)
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ import screen_cast.signaling.SessionAnswer
|
|||||||
import screen_cast.signaling.SessionOffer
|
import screen_cast.signaling.SessionOffer
|
||||||
import screen_cast.signaling.SessionPli
|
import screen_cast.signaling.SessionPli
|
||||||
import screen_cast.signaling.SignalingServer
|
import screen_cast.signaling.SignalingServer
|
||||||
import java.lang.reflect.Proxy
|
|
||||||
import java.net.DatagramPacket
|
import java.net.DatagramPacket
|
||||||
import java.net.DatagramSocket
|
import java.net.DatagramSocket
|
||||||
import java.net.InetSocketAddress
|
import java.net.InetSocketAddress
|
||||||
@@ -54,16 +53,17 @@ class ReceiverPipeline(
|
|||||||
private var signalingPort = 0
|
private var signalingPort = 0
|
||||||
private var signaling: SignalingServer? = null
|
private var signaling: SignalingServer? = null
|
||||||
private var readerThread: Thread? = null
|
private var readerThread: Thread? = null
|
||||||
private var nsdInfo: NsdServiceInfo? = null
|
|
||||||
@Volatile private var registrationListener: NsdManager.RegistrationListener? = null
|
@Volatile private var registrationListener: NsdManager.RegistrationListener? = null
|
||||||
@Volatile private var legacyNsdListener: Any? = null
|
|
||||||
|
|
||||||
|
// Guarded by decoderLock: currentSurface, pendingOffer, decoder.
|
||||||
@Volatile private var decoder: H264Decoder? = null
|
@Volatile private var decoder: H264Decoder? = null
|
||||||
@Volatile private var depacketizer = H264Depacketizer()
|
@Volatile private var depacketizer = H264Depacketizer()
|
||||||
private val jitter = JitterBuffer()
|
private val jitter = JitterBuffer()
|
||||||
@Volatile private var currentSurface: Surface? = null
|
private var currentSurface: Surface? = null
|
||||||
|
private var pendingOffer: SessionOffer? = null
|
||||||
@Volatile private var activeSession = ""
|
@Volatile private var activeSession = ""
|
||||||
private val firstFrameSeen = AtomicBoolean(false)
|
private val firstFrameSeen = AtomicBoolean(false)
|
||||||
|
private val pliLock = Any()
|
||||||
private var lastPliAtMs = 0L
|
private var lastPliAtMs = 0L
|
||||||
@Volatile private var videoWidth = 0
|
@Volatile private var videoWidth = 0
|
||||||
@Volatile private var videoHeight = 0
|
@Volatile private var videoHeight = 0
|
||||||
@@ -117,10 +117,60 @@ class ReceiverPipeline(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Points the decoder (current or future) at a render surface. */
|
/**
|
||||||
|
* Points the decoder (current or future) at a render surface. A pending
|
||||||
|
* offer (accepted while no surface existed) configures its decoder now.
|
||||||
|
*/
|
||||||
fun attachSurface(surface: Surface) {
|
fun attachSurface(surface: Surface) {
|
||||||
currentSurface = surface
|
var configuredFromPending = false
|
||||||
synchronized(decoderLock) { decoder?.attachSurface(surface) }
|
synchronized(decoderLock) {
|
||||||
|
currentSurface = surface
|
||||||
|
decoder?.attachSurface(surface)
|
||||||
|
val offer = pendingOffer
|
||||||
|
if (offer != null) {
|
||||||
|
val created = configureDecoderLocked(offer.width, offer.height)
|
||||||
|
if (created != null) {
|
||||||
|
pendingOffer = null
|
||||||
|
decoder = created
|
||||||
|
configuredFromPending = true
|
||||||
|
} else {
|
||||||
|
// Keep the offer: the next attachSurface retries the
|
||||||
|
// configure instead of abandoning the session.
|
||||||
|
onStatus("Could not start the decoder")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (configuredFromPending) {
|
||||||
|
// The sender is already streaming P-frames; the late-configured
|
||||||
|
// decoder needs a keyframe (SPS/PPS + IDR) to start producing
|
||||||
|
// output — the sender only emits one when asked.
|
||||||
|
requestPli()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Forgets a destroyed render surface so a later offer cannot configure against it. */
|
||||||
|
fun detachSurface() {
|
||||||
|
synchronized(decoderLock) {
|
||||||
|
currentSurface = null
|
||||||
|
decoder?.detachSurface()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Creates and configures a decoder for the stream size; caller holds
|
||||||
|
* [decoderLock]. Returns null on failure with the codec released.
|
||||||
|
*/
|
||||||
|
private fun configureDecoderLocked(width: Int, height: Int): H264Decoder? {
|
||||||
|
val fresh = H264Decoder()
|
||||||
|
try {
|
||||||
|
// Surface first: configure() needs it before the codec starts.
|
||||||
|
currentSurface?.let { fresh.attachSurface(it) }
|
||||||
|
fresh.configure(width, height)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
fresh.release()
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
return fresh
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Stops listening; the pipeline can be started again. */
|
/** Stops listening; the pipeline can be started again. */
|
||||||
@@ -142,6 +192,7 @@ class ReceiverPipeline(
|
|||||||
synchronized(decoderLock) {
|
synchronized(decoderLock) {
|
||||||
decoder?.release()
|
decoder?.release()
|
||||||
decoder = null
|
decoder = null
|
||||||
|
pendingOffer = null
|
||||||
}
|
}
|
||||||
firstFrameSeen.set(false)
|
firstFrameSeen.set(false)
|
||||||
onStatus("Stopped")
|
onStatus("Stopped")
|
||||||
@@ -154,23 +205,23 @@ class ReceiverPipeline(
|
|||||||
}
|
}
|
||||||
// The offer's width/height are informational (the C++ sender leaves
|
// The offer's width/height are informational (the C++ sender leaves
|
||||||
// them 0); the bitstream carries SPS/PPS at every keyframe.
|
// them 0); the bitstream carries SPS/PPS at every keyframe.
|
||||||
val fresh = H264Decoder()
|
var decoderReady = true
|
||||||
val surface = currentSurface
|
|
||||||
var ok = true
|
|
||||||
synchronized(decoderLock) {
|
synchronized(decoderLock) {
|
||||||
|
pendingOffer = null
|
||||||
decoder?.release()
|
decoder?.release()
|
||||||
try {
|
decoder = null
|
||||||
// Surface first: configure() needs it before the codec starts.
|
if (currentSurface != null) {
|
||||||
surface?.let { fresh.attachSurface(it) }
|
decoder = configureDecoderLocked(offer.width, offer.height)
|
||||||
fresh.configure(offer.width, offer.height)
|
decoderReady = decoder != null
|
||||||
decoder = fresh
|
} else {
|
||||||
} catch (e: Exception) {
|
// No surface yet (the window is between surfaces): remember
|
||||||
ok = false
|
// the offer; attachSurface configures the decoder. Configuring
|
||||||
decoder = null
|
// without a surface is not an option — that codec could never
|
||||||
|
// take one afterwards (setOutputSurface refuses it).
|
||||||
|
pendingOffer = offer
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (!ok) {
|
if (!decoderReady) {
|
||||||
fresh.release()
|
|
||||||
onStatus("Could not start the decoder")
|
onStatus("Could not start the decoder")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -193,7 +244,10 @@ class ReceiverPipeline(
|
|||||||
displayHeight = displayHeight,
|
displayHeight = displayHeight,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
onStatus("Session ${offer.sessionId} negotiated — waiting for the first frame…")
|
onStatus(
|
||||||
|
if (decoder != null) "Session ${offer.sessionId} negotiated — waiting for the first frame…"
|
||||||
|
else "Session ${offer.sessionId} negotiated — waiting for the display surface…"
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun readLoop() {
|
private fun readLoop() {
|
||||||
@@ -224,19 +278,28 @@ class ReceiverPipeline(
|
|||||||
|
|
||||||
val accessUnit = result.accessUnit
|
val accessUnit = result.accessUnit
|
||||||
if (accessUnit != null) {
|
if (accessUnit != null) {
|
||||||
val activeDecoder = synchronized(decoderLock) { decoder }
|
val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL
|
||||||
if (activeDecoder == null) return
|
|
||||||
try {
|
try {
|
||||||
val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL
|
// MediaCodec is not thread-safe: attachSurface() (main
|
||||||
activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame)
|
// thread) and onOffer (signaling thread) synchronize on the
|
||||||
activeDecoder.drain()
|
// same lock around their codec calls.
|
||||||
if (firstFrameSeen.compareAndSet(false, true)) {
|
synchronized(decoderLock) {
|
||||||
onFirstFrame()
|
val activeDecoder = decoder ?: return
|
||||||
|
if (!activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame)) {
|
||||||
|
// Input queue full: the dropped frame corrupts the
|
||||||
|
// GOP until the next keyframe — ask for one. No
|
||||||
|
// flush; the codec is healthy, merely busy.
|
||||||
|
requestPli()
|
||||||
|
}
|
||||||
|
activeDecoder.drain()
|
||||||
|
if (firstFrameSeen.compareAndSet(false, true)) {
|
||||||
|
onFirstFrame()
|
||||||
|
}
|
||||||
|
updateVideoSize()
|
||||||
}
|
}
|
||||||
updateVideoSize()
|
|
||||||
} catch (e: Exception) {
|
} catch (e: Exception) {
|
||||||
onStatus("Decoder error (${e.message}) — requesting a keyframe…")
|
onStatus("Decoder error (${e.message}) — requesting a keyframe…")
|
||||||
activeDecoder.flush()
|
synchronized(decoderLock) { decoder?.flush() }
|
||||||
requestPli()
|
requestPli()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -247,7 +310,7 @@ class ReceiverPipeline(
|
|||||||
|
|
||||||
private fun updateVideoSize() {
|
private fun updateVideoSize() {
|
||||||
val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return
|
val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return
|
||||||
if (size != null && (size.first != videoWidth || size.second != videoHeight)) {
|
if (size.first != videoWidth || size.second != videoHeight) {
|
||||||
videoWidth = size.first
|
videoWidth = size.first
|
||||||
videoHeight = size.second
|
videoHeight = size.second
|
||||||
android.util.Log.i(TAG, "video size: ${size.first}x${size.second}")
|
android.util.Log.i(TAG, "video size: ${size.first}x${size.second}")
|
||||||
@@ -255,15 +318,18 @@ class ReceiverPipeline(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Rate-limited keyframe request (single-threaded access from the reader).
|
// Rate-limited keyframe request, callable from any thread (the reader
|
||||||
|
// thread and the main thread's attachSurface both reach it).
|
||||||
private fun requestPli() {
|
private fun requestPli() {
|
||||||
val session = activeSession
|
val session = activeSession
|
||||||
if (session.isEmpty()) return
|
if (session.isEmpty()) return
|
||||||
val now = System.currentTimeMillis()
|
val now = System.currentTimeMillis()
|
||||||
if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return
|
synchronized(pliLock) {
|
||||||
lastPliAtMs = now
|
if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return
|
||||||
|
lastPliAtMs = now
|
||||||
|
}
|
||||||
android.util.Log.i(TAG, "PLI requested for session $session")
|
android.util.Log.i(TAG, "PLI requested for session $session")
|
||||||
signaling?.send(screen_cast.signaling.SessionPli(session))
|
signaling?.send(SessionPli(session))
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun advertiseNsd(port: Int) {
|
private fun advertiseNsd(port: Int) {
|
||||||
@@ -272,70 +338,38 @@ class ReceiverPipeline(
|
|||||||
setServiceType(SERVICE_TYPE)
|
setServiceType(SERVICE_TYPE)
|
||||||
setPort(port)
|
setPort(port)
|
||||||
}
|
}
|
||||||
nsdInfo = info
|
// The classic RegistrationListener API exists from API 16 through 36
|
||||||
// The NSD registration API was replaced in API 36 (ResolutionListener
|
// (verified against the android-36 SDK with javap), so no fallback is
|
||||||
// removed). Try the new API first, then the legacy one via reflection
|
// needed — the previously reflected "ResolutionListener" never
|
||||||
// so the same build works on both.
|
// existed at any API level and could only ever fail.
|
||||||
try {
|
val listener = object : NsdManager.RegistrationListener {
|
||||||
val listener = object : NsdManager.RegistrationListener {
|
override fun onServiceRegistered(serviceInfo: NsdServiceInfo) {
|
||||||
override fun onServiceRegistered(serviceInfo: NsdServiceInfo) {
|
// Registered: the sender's mDNS browser should see it now.
|
||||||
// Registered: the sender's mDNS browser should see it now.
|
|
||||||
}
|
|
||||||
|
|
||||||
override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
|
|
||||||
|
|
||||||
override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
|
|
||||||
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
|
|
||||||
}
|
|
||||||
|
|
||||||
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
|
|
||||||
}
|
}
|
||||||
registrationListener = listener
|
|
||||||
nsdManager.registerService(info, 0, listener)
|
|
||||||
} catch (e: Throwable) {
|
|
||||||
try {
|
|
||||||
registerNsdLegacy(info, port)
|
|
||||||
} catch (e2: Throwable) {
|
|
||||||
onStatus("mDNS unavailable (${e2.message}) — reach this receiver with --peer $localIp:$port")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Pre-API-36 registration API (removed from the API 36 SDK; reflection). */
|
override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
|
||||||
private fun registerNsdLegacy(info: NsdServiceInfo, port: Int) {
|
|
||||||
val listenerClass = Class.forName("android.net.nsd.NsdManager\$ResolutionListener")
|
override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
|
||||||
val proxy = Proxy.newProxyInstance(
|
|
||||||
listenerClass.classLoader,
|
|
||||||
arrayOf(listenerClass),
|
|
||||||
) { _, method, _ ->
|
|
||||||
if (method.name == "onResolutionFailed") {
|
|
||||||
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
|
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
|
||||||
}
|
}
|
||||||
null
|
|
||||||
|
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
nsdManager.registerService(info, 0, listener)
|
||||||
|
registrationListener = listener
|
||||||
|
} catch (e: Exception) {
|
||||||
|
onStatus("mDNS unavailable (${e.message}) — reach this receiver with --peer $localIp:$port")
|
||||||
}
|
}
|
||||||
legacyNsdListener = proxy
|
|
||||||
NsdManager::class.java
|
|
||||||
.getMethod("registerService", NsdServiceInfo::class.java, Int::class.javaPrimitiveType, listenerClass)
|
|
||||||
.invoke(nsdManager, info, 0, proxy)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private fun unregisterNsd() {
|
private fun unregisterNsd() {
|
||||||
val newListener = registrationListener
|
val listener = registrationListener ?: return
|
||||||
val legacyListener = legacyNsdListener
|
|
||||||
val info = nsdInfo
|
|
||||||
try {
|
try {
|
||||||
when {
|
nsdManager.unregisterService(listener)
|
||||||
newListener != null -> nsdManager.unregisterService(newListener)
|
} catch (ignored: Exception) {
|
||||||
legacyListener != null && info != null -> NsdManager::class.java
|
|
||||||
.getMethod("unregisterService", NsdServiceInfo::class.java)
|
|
||||||
.invoke(nsdManager, info)
|
|
||||||
else -> return
|
|
||||||
}
|
|
||||||
} catch (ignored: Throwable) {
|
|
||||||
// already unregistered
|
// already unregistered
|
||||||
}
|
}
|
||||||
registrationListener = null
|
registrationListener = null
|
||||||
legacyNsdListener = null
|
|
||||||
nsdInfo = null
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
package screen_cast.rtp
|
package screen_cast.rtp
|
||||||
|
|
||||||
|
import java.io.ByteArrayOutputStream
|
||||||
|
|
||||||
/** Result of feeding one packet to the depacketizer. */
|
/** Result of feeding one packet to the depacketizer. */
|
||||||
data class DepacketizeResult(
|
data class DepacketizeResult(
|
||||||
/** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */
|
/** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */
|
||||||
@@ -26,9 +28,11 @@ class H264Depacketizer {
|
|||||||
private var frameStarted = false
|
private var frameStarted = false
|
||||||
private var frameDamaged = false
|
private var frameDamaged = false
|
||||||
private var frameTimestamp = 0
|
private var frameTimestamp = 0
|
||||||
private val accessUnit = ArrayList<Int>()
|
// Growable byte accumulators: keyframes reach hundreds of KB, and boxing
|
||||||
|
// each byte (the ArrayList<Int> this replaced) churned the GC hard.
|
||||||
|
private val accessUnit = ByteArrayOutputStream()
|
||||||
private var fuActive = false
|
private var fuActive = false
|
||||||
private val fuNal = ArrayList<Int>()
|
private val fuNal = ByteArrayOutputStream()
|
||||||
|
|
||||||
/** Feed one packet (in sequence order, from the jitter buffer). */
|
/** Feed one packet (in sequence order, from the jitter buffer). */
|
||||||
fun depacketize(packet: RtpPacket): DepacketizeResult {
|
fun depacketize(packet: RtpPacket): DepacketizeResult {
|
||||||
@@ -39,7 +43,7 @@ class H264Depacketizer {
|
|||||||
val expected = (last + 1) and 0xFFFF
|
val expected = (last + 1) and 0xFFFF
|
||||||
if (packet.header.sequenceNumber != expected) {
|
if (packet.header.sequenceNumber != expected) {
|
||||||
fuActive = false
|
fuActive = false
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
if (frameStarted) {
|
if (frameStarted) {
|
||||||
frameDamaged = true
|
frameDamaged = true
|
||||||
}
|
}
|
||||||
@@ -57,7 +61,7 @@ class H264Depacketizer {
|
|||||||
frameStarted = true
|
frameStarted = true
|
||||||
frameDamaged = false
|
frameDamaged = false
|
||||||
frameTimestamp = packet.header.timestamp
|
frameTimestamp = packet.header.timestamp
|
||||||
accessUnit.clear()
|
accessUnit.reset()
|
||||||
}
|
}
|
||||||
|
|
||||||
val payload = packet.payload
|
val payload = packet.payload
|
||||||
@@ -70,10 +74,10 @@ class H264Depacketizer {
|
|||||||
// The previous fragmented NAL never received its end packet.
|
// The previous fragmented NAL never received its end packet.
|
||||||
frameDamaged = true
|
frameDamaged = true
|
||||||
fuActive = false
|
fuActive = false
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
}
|
}
|
||||||
appendStartCode()
|
appendStartCode()
|
||||||
payload.forEach { accessUnit.add(it.toInt() and 0xFF) }
|
accessUnit.write(payload)
|
||||||
}
|
}
|
||||||
|
|
||||||
type == FU_A -> {
|
type == FU_A -> {
|
||||||
@@ -91,11 +95,11 @@ class H264Depacketizer {
|
|||||||
frameDamaged = true
|
frameDamaged = true
|
||||||
}
|
}
|
||||||
fuActive = true
|
fuActive = true
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
// The FU indicator keeps the original NAL's F bit (0) and NRI,
|
// The FU indicator keeps the original NAL's F bit (0) and NRI,
|
||||||
// and declares type 28; the FU header carries S/E plus the real type.
|
// and declares type 28; the FU header carries S/E plus the real type.
|
||||||
fuNal.add((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F))
|
fuNal.write((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F))
|
||||||
fragment.forEach { fuNal.add(it.toInt() and 0xFF) }
|
fuNal.write(fragment)
|
||||||
}
|
}
|
||||||
|
|
||||||
!fuActive -> {
|
!fuActive -> {
|
||||||
@@ -104,12 +108,12 @@ class H264Depacketizer {
|
|||||||
}
|
}
|
||||||
|
|
||||||
else -> {
|
else -> {
|
||||||
fragment.forEach { fuNal.add(it.toInt() and 0xFF) }
|
fuNal.write(fragment)
|
||||||
if (end) {
|
if (end) {
|
||||||
appendStartCode()
|
appendStartCode()
|
||||||
accessUnit.addAll(fuNal)
|
fuNal.writeTo(accessUnit)
|
||||||
fuActive = false
|
fuActive = false
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -132,11 +136,11 @@ class H264Depacketizer {
|
|||||||
// The marker arrived while a NAL was still fragmented.
|
// The marker arrived while a NAL was still fragmented.
|
||||||
frameDamaged = true
|
frameDamaged = true
|
||||||
fuActive = false
|
fuActive = false
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!frameDamaged && accessUnit.isNotEmpty()) {
|
if (!frameDamaged && accessUnit.size() > 0) {
|
||||||
val unit = ByteArray(accessUnit.size) { accessUnit[it].toByte() }
|
val unit = accessUnit.toByteArray()
|
||||||
result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit))
|
result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit))
|
||||||
} else {
|
} else {
|
||||||
// The frame that just ended is unusable.
|
// The frame that just ended is unusable.
|
||||||
@@ -147,9 +151,9 @@ class H264Depacketizer {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private fun appendStartCode() {
|
private fun appendStartCode() {
|
||||||
accessUnit.add(0)
|
accessUnit.write(0)
|
||||||
accessUnit.add(0)
|
accessUnit.write(0)
|
||||||
accessUnit.add(1)
|
accessUnit.write(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */
|
/** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */
|
||||||
@@ -168,8 +172,8 @@ class H264Depacketizer {
|
|||||||
private fun dropFrame() {
|
private fun dropFrame() {
|
||||||
frameStarted = false
|
frameStarted = false
|
||||||
frameDamaged = false
|
frameDamaged = false
|
||||||
accessUnit.clear()
|
accessUnit.reset()
|
||||||
fuActive = false
|
fuActive = false
|
||||||
fuNal.clear()
|
fuNal.reset()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -34,12 +34,29 @@ class SignalingServer(
|
|||||||
return server.localPort
|
return server.localPort
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Sends to the current peer; never throws. A lost control message (a
|
||||||
|
* PLI, an answer) is recoverable — the session re-negotiates or the next
|
||||||
|
* keyframe arrives — but an exception here would kill the RTP reader
|
||||||
|
* thread, which reaches send() from requestPli(). Mirrors the C++ server,
|
||||||
|
* which ignores write failures.
|
||||||
|
*/
|
||||||
fun send(message: SignalingMessage) {
|
fun send(message: SignalingMessage) {
|
||||||
val bytes = SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8)
|
val bytes = try {
|
||||||
|
SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8)
|
||||||
|
} catch (e: Exception) {
|
||||||
|
return // unreachable for our message types; serialize is total
|
||||||
|
}
|
||||||
synchronized(peerLock) {
|
synchronized(peerLock) {
|
||||||
peerOut?.let { out ->
|
val out = peerOut ?: return
|
||||||
|
try {
|
||||||
out.write(bytes)
|
out.write(bytes)
|
||||||
out.flush()
|
out.flush()
|
||||||
|
} catch (e: Exception) {
|
||||||
|
// The peer is gone (e.g. the sender died while media still
|
||||||
|
// flows). Dropping peerOut also closes the socket; the next
|
||||||
|
// accepted offer installs a fresh one.
|
||||||
|
peerOut = null
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+12
-8
@@ -243,16 +243,20 @@ Platform quirks found while validating (Android 16 / API 36):
|
|||||||
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
|
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
|
||||||
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
|
unconnected datagram sockets, and `.localAddress` is an `InetAddress`
|
||||||
(Inet6Address on Android), not an `InetSocketAddress`.
|
(Inet6Address on Android), not an `InetSocketAddress`.
|
||||||
- `MediaCodec`: pass the output Surface to `configure()` (`setOutputSurface`
|
- `MediaCodec`: pass the output Surface to `configure()` — the codec must
|
||||||
is illegal after configure); call `start()`; render with
|
start in surface mode (`setOutputSurface` afterwards only switches an
|
||||||
`releaseOutputBuffer(index, render=true)`; the C2 AVC decoder requires a
|
already-surface-mode codec to a new surface; a codec configured without
|
||||||
concrete width/height at configure (use a placeholder — the in-band SPS
|
a surface can never take one, per the API docs); call `start()`; render
|
||||||
reconfigures it).
|
with `releaseOutputBuffer(index, render=true)`; the C2 AVC decoder
|
||||||
|
requires a concrete width/height at configure (use a placeholder — the
|
||||||
|
in-band SPS reconfigures it).
|
||||||
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
|
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
|
||||||
surface.
|
surface.
|
||||||
- NSD: API 36 replaced `ResolutionListener` registration with
|
- NSD: the classic `registerService(info, flags, RegistrationListener)`
|
||||||
`registerService(info, flags, RegistrationListener)`; the app falls back
|
API exists from API 16 through 36 (verified with javap on the android-36
|
||||||
to reflection on older devices.
|
SDK jar) — no fallback is needed. A reflection fallback targeting a
|
||||||
|
pre-36 "ResolutionListener" was removed: that class never existed at
|
||||||
|
any API level, so the fallback could only ever fail.
|
||||||
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
|
- The C2 decoder scales its output to the Surface (`android._video-scaling`)
|
||||||
— letterbox by **sizing the TextureView to the video aspect** (centered
|
— letterbox by **sizing the TextureView to the video aspect** (centered
|
||||||
on a black window background), not with a transform matrix (double scale).
|
on a black window background), not with a transform matrix (double scale).
|
||||||
|
|||||||
Reference in New Issue
Block a user