fix(android): harden the receiver after the review pass

- SignalingServer.send() never throws: a broken signaling connection
  drops the peer instead of killing the RTP reader thread via requestPli()
- H264Decoder: assign the codec before configure/start (no orphaned
  instance on failure); feed() reports input-queue timeouts so the
  pipeline requests a keyframe instead of dropping the frame silently
- Park offers that arrive without a surface as pendingOffer and configure
  on attachSurface(): a codec configured without a surface can never take
  one (setOutputSurface refuses it); the late configure sends a PLI
- Forget destroyed surfaces (onSurfaceTextureDestroyed -> detachSurface)
- Show the status overlay again on later messages
- Accumulate NAL bytes in ByteArrayOutputStreams, not boxed ArrayList<Int>
- Delete the dead NSD reflection fallback (ResolutionListener never
  existed; the classic API is present from API 16 through 36)
- Hold decoderLock around all MediaCodec calls (not thread-safe) and make
  requestPli thread-safe
- Update RUNBOOK quirks (NSD, setOutputSurface) and MEMORY notes
This commit is contained in:
2026-09-10 12:19:21 +02:00
parent bbe4f21a3b
commit c6722d164b
7 changed files with 297 additions and 134 deletions
+62 -4
View File
@@ -1,7 +1,9 @@
# Project Memory — screen_cast # Project Memory — screen_cast
Last updated: Phase 8 (Android receiver app) complete and validated on a Last updated: Phase 8 (Android receiver app) complete and validated on a
Fairphone 6; all prior phases done. Fairphone 6; all prior phases done. The Android app's review findings were
fixed in a follow-up pass (same day) — see "Android app review" at the
bottom of this file.
## Project state ## Project state
@@ -19,8 +21,11 @@ Fairphone 6; all prior phases done.
`MediaCodec.configure()` + `start()`; render via `MediaCodec.configure()` + `start()`; render via
`releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at `releaseOutputBuffer(render=true)`; C2 AVC needs a concrete size at
configure (in-band SPS reconfigures); `MediaFormat.format()`/ configure (in-band SPS reconfigures); `MediaFormat.format()`/
`KEY_MIME_TYPE` not public in API 36; NSD `RegistrationListener` replaced `KEY_MIME_TYPE` not public in API 36; NSD: the classic
`ResolutionListener` (reflection fallback for older devices); `registerService(info, flags, RegistrationListener)` API exists from API
16 through 36 (javap-verified on the android-36 SDK) — an old reflection
fallback targeting a never-existent "ResolutionListener" was removed as
dead code;
**`android._video-scaling`: C2 scales output to the Surface** → letterbox **`android._video-scaling`: C2 scales output to the Surface** → letterbox
by sizing the TextureView to the video aspect, NOT a transform matrix by sizing the TextureView to the video aspect, NOT a transform matrix
(double scale). Sender-side: Hyprland + GTK portal's `--target monitor` (double scale). Sender-side: Hyprland + GTK portal's `--target monitor`
@@ -258,4 +263,57 @@ None.
- `CaptureSession::next_frame()` returns `nullopt` on stream error without - `CaptureSession::next_frame()` returns `nullopt` on stream error without
surfacing the reason (logged to stderr). surfacing the reason (logged to stderr).
- Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders - Receiver ignores unknown packetization modes (STAP-A/MTAP/FU-B); senders
we control never emit them, but third-party interop would need support. we control never emit them, but third-party interop would need support.
### Android app review (2026-09-10) — findings FIXED same day
Review pass (25 JVM tests re-run green; rtp/jitter/depacketizer verified
faithful to the C++ side source-to-source), followed by a fix pass that
landed all findings. No commit yet (user has not asked).
Fixed:
- `SignalingServer.send()` no longer throws (mirrors the C++ server, which
ignores write failures): a broken signaling TCP no longer kills the
`rtp-reader` thread via `requestPli()`; `peerOut` is dropped (closing the
socket) on write failure.
- `H264Decoder.configure()` assigns the created codec before configuring,
so a configure/start failure can no longer orphan the MediaCodec instance
(no finalizer; scarce native slots).
- Offer-before-surface no longer configures a ByteBuffer-mode decoder: the
offer is parked as `pendingOffer` and `attachSurface()` configures later
(a surface-less codec can never take one — `setOutputSurface` refuses
it, an IllegalStateException crash on the main thread). The late
configure sends a PLI (the sender only emits IDRs when asked).
- `ReceiverActivity.onSurfaceTextureDestroyed``pipeline.detachSurface()`;
the pipeline/decoder forget dead surfaces instead of configuring against
them (frames decode unrendered until the next attach).
- The status overlay reappears: `onStatus` sets `visibility = VISIBLE`
(it used to write into a GONE view after the first frame).
- `H264Decoder.feed()` returns false on input-queue timeout → the pipeline
requests a PLI instead of silently dropping the frame (no flush — the
codec is healthy).
- The depacketizer accumulates into `ByteArrayOutputStream`s instead of
boxed `ArrayList<Int>` (was ~MB/s of Integer allocations on keyframes).
- NSD reflection fallback deleted (dead code — see the quirk note above).
- Thread-safety: all codec calls now run under `decoderLock` (MediaCodec is
not thread-safe; feed/drain previously raced attachSurface);
`requestPli()` is thread-safe via `pliLock`.
Validation: `gradle :app:assembleDebug :app:testDebugUnitTest` green
(25/25, full --rerun-tasks rebuild, only two pre-existing warnings);
`meson test` 5/5 unchanged (no C++ touched).
Still open (accepted, needs a device or a new test dep):
- Untested on device: `setOutputSurface` mid-session (surface switch) and
the whole pendingOffer path; API-35 `detachOutputSurface()` could replace
the render-flag approach.
- No JVM tests for `SignalingMessage`/`SignalingServer` (would need the
`org.json:json` test dependency; android.jar stubs throw).
- Interop caveats by design: 2048-byte datagram buffer (our MTU is 1200),
RTP timestamps (90 kHz) fed as µs, unauthenticated offers (SRTP is a
future phase).
- Verified NOT a bug: reusing one `DatagramPacket` without resetting its
length — modern JVMs recv by buffer capacity (JDK-21 probe + the
successful on-device streaming confirm it).
@@ -52,7 +52,13 @@ class ReceiverActivity : Activity() {
fitVideo() fitVideo()
} }
override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean = true override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
// The TextureView is about to release this SurfaceTexture; the
// pipeline must forget it or a later offer would configure the
// decoder against a dead surface.
pipeline?.detachSurface()
return true
}
override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
} }
@@ -85,7 +91,12 @@ class ReceiverActivity : Activity() {
val metrics = resources.displayMetrics val metrics = resources.displayMetrics
metrics.widthPixels to metrics.heightPixels metrics.widthPixels to metrics.heightPixels
}, },
onStatus = { text -> ui.post { statusView.text = text } }, onStatus = { text ->
ui.post {
statusView.text = text
statusView.visibility = View.VISIBLE
}
},
onFirstFrame = { ui.post { statusView.visibility = View.GONE } }, onFirstFrame = { ui.post { statusView.visibility = View.GONE } },
onVideoSize = { _, _ -> ui.post { fitVideo() } }, onVideoSize = { _, _ -> ui.post { fitVideo() } },
) )
@@ -35,11 +35,26 @@ class H264Decoder {
MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240) MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 320, 240)
} }
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE) format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_INPUT_SIZE)
codec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC).also { c -> // Assign before configuring so a configure()/start() failure cannot
// The surface must be passed to configure(); setOutputSurface() // orphan the created instance: MediaCodec has no finalizer and each
// is only legal before configuration. // unreleased instance holds a scarce native codec slot. The caller
c.configure(format, renderSurface, null, 0) // follows an exception with release(), which is a no-op on null.
c.start() val created = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC)
codec = created
try {
// The surface must go into configure(): the codec must start in
// surface mode. setOutputSurface() afterwards only switches an
// already-surface-mode codec; a codec configured without a
// surface can never take one.
created.configure(format, renderSurface, null, 0)
created.start()
} catch (e: Exception) {
codec = null
try {
created.release()
} catch (ignored: Exception) {
}
throw e
} }
configured = true configured = true
} }
@@ -48,19 +63,39 @@ class H264Decoder {
@Synchronized @Synchronized
fun attachSurface(surface: Surface) { fun attachSurface(surface: Surface) {
renderSurface = surface renderSurface = surface
// Legal only in surface mode: setOutputSurface() dynamically switches
// an output surface on a codec configured WITH one (per the platform
// docs); on a ByteBuffer-mode codec it throws IllegalStateException.
codec?.setOutputSurface(surface) codec?.setOutputSurface(surface)
} }
/** Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp). */ /**
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean) { * Forgets a destroyed render surface: frames still decode (returning
* their buffers and keeping the queue moving) but are not rendered
* until the next [attachSurface]. API 35's detachOutputSurface() is the
* codec-side equivalent.
*/
@Synchronized
fun detachSurface() {
renderSurface = null
}
/**
* Queues one access unit. [presentationUs] must be monotonic (the RTP timestamp).
* @return false when the codec's input queue was full and the frame was
* dropped; the caller should request a keyframe, not flush (the codec
* is healthy, merely busy).
*/
fun feed(data: ByteArray, presentationUs: Long, isKeyFrame: Boolean): Boolean {
val c = codec ?: throw IllegalStateException("decoder not configured") val c = codec ?: throw IllegalStateException("decoder not configured")
val index = c.dequeueInputBuffer(10_000) val index = c.dequeueInputBuffer(10_000)
if (index < 0) return if (index < 0) return false
val buffer = c.getInputBuffer(index) ?: return val buffer = c.getInputBuffer(index) ?: return false
buffer.clear() buffer.clear()
buffer.put(data) buffer.put(data)
val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_SYNC_FRAME else 0 val flags = if (isKeyFrame) MediaCodec.BUFFER_FLAG_KEY_FRAME else 0
c.queueInputBuffer(index, 0, data.size, presentationUs, flags) c.queueInputBuffer(index, 0, data.size, presentationUs, flags)
return true
} }
/** /**
@@ -12,7 +12,6 @@ import screen_cast.signaling.SessionAnswer
import screen_cast.signaling.SessionOffer import screen_cast.signaling.SessionOffer
import screen_cast.signaling.SessionPli import screen_cast.signaling.SessionPli
import screen_cast.signaling.SignalingServer import screen_cast.signaling.SignalingServer
import java.lang.reflect.Proxy
import java.net.DatagramPacket import java.net.DatagramPacket
import java.net.DatagramSocket import java.net.DatagramSocket
import java.net.InetSocketAddress import java.net.InetSocketAddress
@@ -54,16 +53,17 @@ class ReceiverPipeline(
private var signalingPort = 0 private var signalingPort = 0
private var signaling: SignalingServer? = null private var signaling: SignalingServer? = null
private var readerThread: Thread? = null private var readerThread: Thread? = null
private var nsdInfo: NsdServiceInfo? = null
@Volatile private var registrationListener: NsdManager.RegistrationListener? = null @Volatile private var registrationListener: NsdManager.RegistrationListener? = null
@Volatile private var legacyNsdListener: Any? = null
// Guarded by decoderLock: currentSurface, pendingOffer, decoder.
@Volatile private var decoder: H264Decoder? = null @Volatile private var decoder: H264Decoder? = null
@Volatile private var depacketizer = H264Depacketizer() @Volatile private var depacketizer = H264Depacketizer()
private val jitter = JitterBuffer() private val jitter = JitterBuffer()
@Volatile private var currentSurface: Surface? = null private var currentSurface: Surface? = null
private var pendingOffer: SessionOffer? = null
@Volatile private var activeSession = "" @Volatile private var activeSession = ""
private val firstFrameSeen = AtomicBoolean(false) private val firstFrameSeen = AtomicBoolean(false)
private val pliLock = Any()
private var lastPliAtMs = 0L private var lastPliAtMs = 0L
@Volatile private var videoWidth = 0 @Volatile private var videoWidth = 0
@Volatile private var videoHeight = 0 @Volatile private var videoHeight = 0
@@ -117,10 +117,60 @@ class ReceiverPipeline(
) )
} }
/** Points the decoder (current or future) at a render surface. */ /**
* Points the decoder (current or future) at a render surface. A pending
* offer (accepted while no surface existed) configures its decoder now.
*/
fun attachSurface(surface: Surface) { fun attachSurface(surface: Surface) {
currentSurface = surface var configuredFromPending = false
synchronized(decoderLock) { decoder?.attachSurface(surface) } synchronized(decoderLock) {
currentSurface = surface
decoder?.attachSurface(surface)
val offer = pendingOffer
if (offer != null) {
val created = configureDecoderLocked(offer.width, offer.height)
if (created != null) {
pendingOffer = null
decoder = created
configuredFromPending = true
} else {
// Keep the offer: the next attachSurface retries the
// configure instead of abandoning the session.
onStatus("Could not start the decoder")
}
}
}
if (configuredFromPending) {
// The sender is already streaming P-frames; the late-configured
// decoder needs a keyframe (SPS/PPS + IDR) to start producing
// output — the sender only emits one when asked.
requestPli()
}
}
/** Forgets a destroyed render surface so a later offer cannot configure against it. */
fun detachSurface() {
synchronized(decoderLock) {
currentSurface = null
decoder?.detachSurface()
}
}
/**
* Creates and configures a decoder for the stream size; caller holds
* [decoderLock]. Returns null on failure with the codec released.
*/
private fun configureDecoderLocked(width: Int, height: Int): H264Decoder? {
val fresh = H264Decoder()
try {
// Surface first: configure() needs it before the codec starts.
currentSurface?.let { fresh.attachSurface(it) }
fresh.configure(width, height)
} catch (e: Exception) {
fresh.release()
return null
}
return fresh
} }
/** Stops listening; the pipeline can be started again. */ /** Stops listening; the pipeline can be started again. */
@@ -142,6 +192,7 @@ class ReceiverPipeline(
synchronized(decoderLock) { synchronized(decoderLock) {
decoder?.release() decoder?.release()
decoder = null decoder = null
pendingOffer = null
} }
firstFrameSeen.set(false) firstFrameSeen.set(false)
onStatus("Stopped") onStatus("Stopped")
@@ -154,23 +205,23 @@ class ReceiverPipeline(
} }
// The offer's width/height are informational (the C++ sender leaves // The offer's width/height are informational (the C++ sender leaves
// them 0); the bitstream carries SPS/PPS at every keyframe. // them 0); the bitstream carries SPS/PPS at every keyframe.
val fresh = H264Decoder() var decoderReady = true
val surface = currentSurface
var ok = true
synchronized(decoderLock) { synchronized(decoderLock) {
pendingOffer = null
decoder?.release() decoder?.release()
try { decoder = null
// Surface first: configure() needs it before the codec starts. if (currentSurface != null) {
surface?.let { fresh.attachSurface(it) } decoder = configureDecoderLocked(offer.width, offer.height)
fresh.configure(offer.width, offer.height) decoderReady = decoder != null
decoder = fresh } else {
} catch (e: Exception) { // No surface yet (the window is between surfaces): remember
ok = false // the offer; attachSurface configures the decoder. Configuring
decoder = null // without a surface is not an option — that codec could never
// take one afterwards (setOutputSurface refuses it).
pendingOffer = offer
} }
} }
if (!ok) { if (!decoderReady) {
fresh.release()
onStatus("Could not start the decoder") onStatus("Could not start the decoder")
return return
} }
@@ -193,7 +244,10 @@ class ReceiverPipeline(
displayHeight = displayHeight, displayHeight = displayHeight,
), ),
) )
onStatus("Session ${offer.sessionId} negotiated — waiting for the first frame…") onStatus(
if (decoder != null) "Session ${offer.sessionId} negotiated — waiting for the first frame…"
else "Session ${offer.sessionId} negotiated — waiting for the display surface…"
)
} }
private fun readLoop() { private fun readLoop() {
@@ -224,19 +278,28 @@ class ReceiverPipeline(
val accessUnit = result.accessUnit val accessUnit = result.accessUnit
if (accessUnit != null) { if (accessUnit != null) {
val activeDecoder = synchronized(decoderLock) { decoder } val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL
if (activeDecoder == null) return
try { try {
val presentationUs = packet.header.timestamp.toLong() and 0xFFFFFFFFL // MediaCodec is not thread-safe: attachSurface() (main
activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame) // thread) and onOffer (signaling thread) synchronize on the
activeDecoder.drain() // same lock around their codec calls.
if (firstFrameSeen.compareAndSet(false, true)) { synchronized(decoderLock) {
onFirstFrame() val activeDecoder = decoder ?: return
if (!activeDecoder.feed(accessUnit, presentationUs, result.isKeyFrame)) {
// Input queue full: the dropped frame corrupts the
// GOP until the next keyframe — ask for one. No
// flush; the codec is healthy, merely busy.
requestPli()
}
activeDecoder.drain()
if (firstFrameSeen.compareAndSet(false, true)) {
onFirstFrame()
}
updateVideoSize()
} }
updateVideoSize()
} catch (e: Exception) { } catch (e: Exception) {
onStatus("Decoder error (${e.message}) — requesting a keyframe…") onStatus("Decoder error (${e.message}) — requesting a keyframe…")
activeDecoder.flush() synchronized(decoderLock) { decoder?.flush() }
requestPli() requestPli()
} }
} }
@@ -247,7 +310,7 @@ class ReceiverPipeline(
private fun updateVideoSize() { private fun updateVideoSize() {
val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return val size = synchronized(decoderLock) { decoder?.outputSize() } ?: return
if (size != null && (size.first != videoWidth || size.second != videoHeight)) { if (size.first != videoWidth || size.second != videoHeight) {
videoWidth = size.first videoWidth = size.first
videoHeight = size.second videoHeight = size.second
android.util.Log.i(TAG, "video size: ${size.first}x${size.second}") android.util.Log.i(TAG, "video size: ${size.first}x${size.second}")
@@ -255,15 +318,18 @@ class ReceiverPipeline(
} }
} }
// Rate-limited keyframe request (single-threaded access from the reader). // Rate-limited keyframe request, callable from any thread (the reader
// thread and the main thread's attachSurface both reach it).
private fun requestPli() { private fun requestPli() {
val session = activeSession val session = activeSession
if (session.isEmpty()) return if (session.isEmpty()) return
val now = System.currentTimeMillis() val now = System.currentTimeMillis()
if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return synchronized(pliLock) {
lastPliAtMs = now if (now - lastPliAtMs < PLI_MIN_INTERVAL_MS) return
lastPliAtMs = now
}
android.util.Log.i(TAG, "PLI requested for session $session") android.util.Log.i(TAG, "PLI requested for session $session")
signaling?.send(screen_cast.signaling.SessionPli(session)) signaling?.send(SessionPli(session))
} }
private fun advertiseNsd(port: Int) { private fun advertiseNsd(port: Int) {
@@ -272,70 +338,38 @@ class ReceiverPipeline(
setServiceType(SERVICE_TYPE) setServiceType(SERVICE_TYPE)
setPort(port) setPort(port)
} }
nsdInfo = info // The classic RegistrationListener API exists from API 16 through 36
// The NSD registration API was replaced in API 36 (ResolutionListener // (verified against the android-36 SDK with javap), so no fallback is
// removed). Try the new API first, then the legacy one via reflection // needed — the previously reflected "ResolutionListener" never
// so the same build works on both. // existed at any API level and could only ever fail.
try { val listener = object : NsdManager.RegistrationListener {
val listener = object : NsdManager.RegistrationListener { override fun onServiceRegistered(serviceInfo: NsdServiceInfo) {
override fun onServiceRegistered(serviceInfo: NsdServiceInfo) { // Registered: the sender's mDNS browser should see it now.
// Registered: the sender's mDNS browser should see it now.
}
override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
}
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
} }
registrationListener = listener
nsdManager.registerService(info, 0, listener)
} catch (e: Throwable) {
try {
registerNsdLegacy(info, port)
} catch (e2: Throwable) {
onStatus("mDNS unavailable (${e2.message}) — reach this receiver with --peer $localIp:$port")
}
}
}
/** Pre-API-36 registration API (removed from the API 36 SDK; reflection). */ override fun onServiceUnregistered(serviceInfo: NsdServiceInfo) = Unit
private fun registerNsdLegacy(info: NsdServiceInfo, port: Int) {
val listenerClass = Class.forName("android.net.nsd.NsdManager\$ResolutionListener") override fun onRegistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {
val proxy = Proxy.newProxyInstance(
listenerClass.classLoader,
arrayOf(listenerClass),
) { _, method, _ ->
if (method.name == "onResolutionFailed") {
onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port") onStatus("mDNS registration failed — reach this receiver with --peer $localIp:$port")
} }
null
override fun onUnregistrationFailed(serviceInfo: NsdServiceInfo, errorCode: Int) = Unit
}
try {
nsdManager.registerService(info, 0, listener)
registrationListener = listener
} catch (e: Exception) {
onStatus("mDNS unavailable (${e.message}) — reach this receiver with --peer $localIp:$port")
} }
legacyNsdListener = proxy
NsdManager::class.java
.getMethod("registerService", NsdServiceInfo::class.java, Int::class.javaPrimitiveType, listenerClass)
.invoke(nsdManager, info, 0, proxy)
} }
private fun unregisterNsd() { private fun unregisterNsd() {
val newListener = registrationListener val listener = registrationListener ?: return
val legacyListener = legacyNsdListener
val info = nsdInfo
try { try {
when { nsdManager.unregisterService(listener)
newListener != null -> nsdManager.unregisterService(newListener) } catch (ignored: Exception) {
legacyListener != null && info != null -> NsdManager::class.java
.getMethod("unregisterService", NsdServiceInfo::class.java)
.invoke(nsdManager, info)
else -> return
}
} catch (ignored: Throwable) {
// already unregistered // already unregistered
} }
registrationListener = null registrationListener = null
legacyNsdListener = null
nsdInfo = null
} }
} }
@@ -1,5 +1,7 @@
package screen_cast.rtp package screen_cast.rtp
import java.io.ByteArrayOutputStream
/** Result of feeding one packet to the depacketizer. */ /** Result of feeding one packet to the depacketizer. */
data class DepacketizeResult( data class DepacketizeResult(
/** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */ /** Completed access unit (Annex-B with 3-byte start codes) when the frame closed undamaged. */
@@ -26,9 +28,11 @@ class H264Depacketizer {
private var frameStarted = false private var frameStarted = false
private var frameDamaged = false private var frameDamaged = false
private var frameTimestamp = 0 private var frameTimestamp = 0
private val accessUnit = ArrayList<Int>() // Growable byte accumulators: keyframes reach hundreds of KB, and boxing
// each byte (the ArrayList<Int> this replaced) churned the GC hard.
private val accessUnit = ByteArrayOutputStream()
private var fuActive = false private var fuActive = false
private val fuNal = ArrayList<Int>() private val fuNal = ByteArrayOutputStream()
/** Feed one packet (in sequence order, from the jitter buffer). */ /** Feed one packet (in sequence order, from the jitter buffer). */
fun depacketize(packet: RtpPacket): DepacketizeResult { fun depacketize(packet: RtpPacket): DepacketizeResult {
@@ -39,7 +43,7 @@ class H264Depacketizer {
val expected = (last + 1) and 0xFFFF val expected = (last + 1) and 0xFFFF
if (packet.header.sequenceNumber != expected) { if (packet.header.sequenceNumber != expected) {
fuActive = false fuActive = false
fuNal.clear() fuNal.reset()
if (frameStarted) { if (frameStarted) {
frameDamaged = true frameDamaged = true
} }
@@ -57,7 +61,7 @@ class H264Depacketizer {
frameStarted = true frameStarted = true
frameDamaged = false frameDamaged = false
frameTimestamp = packet.header.timestamp frameTimestamp = packet.header.timestamp
accessUnit.clear() accessUnit.reset()
} }
val payload = packet.payload val payload = packet.payload
@@ -70,10 +74,10 @@ class H264Depacketizer {
// The previous fragmented NAL never received its end packet. // The previous fragmented NAL never received its end packet.
frameDamaged = true frameDamaged = true
fuActive = false fuActive = false
fuNal.clear() fuNal.reset()
} }
appendStartCode() appendStartCode()
payload.forEach { accessUnit.add(it.toInt() and 0xFF) } accessUnit.write(payload)
} }
type == FU_A -> { type == FU_A -> {
@@ -91,11 +95,11 @@ class H264Depacketizer {
frameDamaged = true frameDamaged = true
} }
fuActive = true fuActive = true
fuNal.clear() fuNal.reset()
// The FU indicator keeps the original NAL's F bit (0) and NRI, // The FU indicator keeps the original NAL's F bit (0) and NRI,
// and declares type 28; the FU header carries S/E plus the real type. // and declares type 28; the FU header carries S/E plus the real type.
fuNal.add((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F)) fuNal.write((payload[0].toInt() and 0xE0) or (fuHeader and 0x1F))
fragment.forEach { fuNal.add(it.toInt() and 0xFF) } fuNal.write(fragment)
} }
!fuActive -> { !fuActive -> {
@@ -104,12 +108,12 @@ class H264Depacketizer {
} }
else -> { else -> {
fragment.forEach { fuNal.add(it.toInt() and 0xFF) } fuNal.write(fragment)
if (end) { if (end) {
appendStartCode() appendStartCode()
accessUnit.addAll(fuNal) fuNal.writeTo(accessUnit)
fuActive = false fuActive = false
fuNal.clear() fuNal.reset()
} }
} }
} }
@@ -132,11 +136,11 @@ class H264Depacketizer {
// The marker arrived while a NAL was still fragmented. // The marker arrived while a NAL was still fragmented.
frameDamaged = true frameDamaged = true
fuActive = false fuActive = false
fuNal.clear() fuNal.reset()
} }
if (!frameDamaged && accessUnit.isNotEmpty()) { if (!frameDamaged && accessUnit.size() > 0) {
val unit = ByteArray(accessUnit.size) { accessUnit[it].toByte() } val unit = accessUnit.toByteArray()
result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit)) result = result.copy(accessUnit = unit, isKeyFrame = containsParameterSets(unit))
} else { } else {
// The frame that just ended is unusable. // The frame that just ended is unusable.
@@ -147,9 +151,9 @@ class H264Depacketizer {
} }
private fun appendStartCode() { private fun appendStartCode() {
accessUnit.add(0) accessUnit.write(0)
accessUnit.add(0) accessUnit.write(0)
accessUnit.add(1) accessUnit.write(1)
} }
/** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */ /** The sender repeats SPS/PPS in-band at every keyframe; sniff for NAL types 7/8. */
@@ -168,8 +172,8 @@ class H264Depacketizer {
private fun dropFrame() { private fun dropFrame() {
frameStarted = false frameStarted = false
frameDamaged = false frameDamaged = false
accessUnit.clear() accessUnit.reset()
fuActive = false fuActive = false
fuNal.clear() fuNal.reset()
} }
} }
@@ -34,12 +34,29 @@ class SignalingServer(
return server.localPort return server.localPort
} }
/**
* Sends to the current peer; never throws. A lost control message (a
* PLI, an answer) is recoverable — the session re-negotiates or the next
* keyframe arrives — but an exception here would kill the RTP reader
* thread, which reaches send() from requestPli(). Mirrors the C++ server,
* which ignores write failures.
*/
fun send(message: SignalingMessage) { fun send(message: SignalingMessage) {
val bytes = SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8) val bytes = try {
SignalingMessage.serialize(message).toByteArray(Charsets.UTF_8)
} catch (e: Exception) {
return // unreachable for our message types; serialize is total
}
synchronized(peerLock) { synchronized(peerLock) {
peerOut?.let { out -> val out = peerOut ?: return
try {
out.write(bytes) out.write(bytes)
out.flush() out.flush()
} catch (e: Exception) {
// The peer is gone (e.g. the sender died while media still
// flows). Dropping peerOut also closes the socket; the next
// accepted offer installs a fresh one.
peerOut = null
} }
} }
} }
+12 -8
View File
@@ -243,16 +243,20 @@ Platform quirks found while validating (Android 16 / API 36):
- `DatagramSocket.localPort` gives the bound port; `.port` is -1 for - `DatagramSocket.localPort` gives the bound port; `.port` is -1 for
unconnected datagram sockets, and `.localAddress` is an `InetAddress` unconnected datagram sockets, and `.localAddress` is an `InetAddress`
(Inet6Address on Android), not an `InetSocketAddress`. (Inet6Address on Android), not an `InetSocketAddress`.
- `MediaCodec`: pass the output Surface to `configure()` (`setOutputSurface` - `MediaCodec`: pass the output Surface to `configure()` — the codec must
is illegal after configure); call `start()`; render with start in surface mode (`setOutputSurface` afterwards only switches an
`releaseOutputBuffer(index, render=true)`; the C2 AVC decoder requires a already-surface-mode codec to a new surface; a codec configured without
concrete width/height at configure (use a placeholder the in-band SPS a surface can never take one, per the API docs); call `start()`; render
reconfigures it). with `releaseOutputBuffer(index, render=true)`; the C2 AVC decoder
requires a concrete width/height at configure (use a placeholder — the
in-band SPS reconfigures it).
- `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public - `MediaFormat.format()` / `KEY_MIME_TYPE` are not in the API-36 public
surface. surface.
- NSD: API 36 replaced `ResolutionListener` registration with - NSD: the classic `registerService(info, flags, RegistrationListener)`
`registerService(info, flags, RegistrationListener)`; the app falls back API exists from API 16 through 36 (verified with javap on the android-36
to reflection on older devices. SDK jar) — no fallback is needed. A reflection fallback targeting a
pre-36 "ResolutionListener" was removed: that class never existed at
any API level, so the fallback could only ever fail.
- The C2 decoder scales its output to the Surface (`android._video-scaling`) - The C2 decoder scales its output to the Surface (`android._video-scaling`)
— letterbox by **sizing the TextureView to the video aspect** (centered — letterbox by **sizing the TextureView to the video aspect** (centered
on a black window background), not with a transform matrix (double scale). on a black window background), not with a transform matrix (double scale).