Discussion on Optimizing Scrcpy Performance

[mengyanshou]

I am currently trying to port Sunshine to the Android platform. In Sunshine, the streaming part is implemented entirely in C/C++, with Java mainly responsible for passing the Surface to C/C++. During the porting process, I began to wonder if the performance of scrcpy could be further optimized by implementing the decoding, streaming, and even event handling in C/C++.

For example, when I use scrcpy to play games, extreme performance and lower latency become more important. Through this issue, I would like to discuss some methods to optimize performance further. We can keep this issue open for a while, and later, I will share an Android-to-Android mirroring application implemented with scrcpy.

I have conducted several tests.

Using Raw IP + Port

This is when connecting to an Android device via wireless adb. I wanted to remove the adb forward process and connect directly via IP + Port. Here is a method to get the file descriptor of a Socket using Java reflection.

public static FileDescriptor getFileDescriptor(Socket socket) throws IOException {
    try {
        // 获取SocketImpl
        Field implField = Socket.class.getDeclaredField("impl");
        implField.setAccessible(true);
        Object socketImpl = implField.get(socket);

        // 获取文件描述符
        Field fdField = socketImpl.getClass().getDeclaredField("fd");
        fdField.setAccessible(true);
        return (FileDescriptor) fdField.get(socketImpl);
    } catch (NoSuchFieldException | IllegalAccessException e) {
        throw new IOException("Failed to get FileDescriptor from Socket", e);
    }
}

I modified DesktopConnection to test this.

    public static DesktopConnection open(int scid, boolean tunnelForward, boolean video, boolean audio, boolean control, boolean sendDummyByte)
            throws IOException {
        String socketName = getSocketName(scid);
        boolean usePortSocket = false;

        LocalSocket videoSocket = null;
        LocalSocket audioSocket = null;
        LocalSocket controlSocket = null;
        Socket videoPortSocket = null;
        Socket audioPortSocket = null;
        Socket controlPortSocket = null;

        try {
            if (usePortSocket) {
                ServerSocket serverSocket = new ServerSocket(9999);
                videoPortSocket = serverSocket.accept();
                if (sendDummyByte) {
                    // send one byte so the client may read() to detect a connection error
                    videoPortSocket.getOutputStream().write(0);
                    sendDummyByte = false;
                }
                DataOutputStream out = new DataOutputStream(videoPortSocket.getOutputStream());
                DataInputStream in = new DataInputStream(videoPortSocket.getInputStream());
                test(in, out);
                audioPortSocket = serverSocket.accept();
                controlPortSocket = serverSocket.accept();

I continuously send one byte from the client and the server replies with one byte to calculate the latency. However, during my tests, the latency fluctuated by a few milliseconds, making it difficult to determine if it was truly faster.

Implementing Encoding and Streaming in C/C++

I implemented encoding using NDK at the C/C++ level. Here is all the code I tested with.

#include <media/NdkMediaCodec.h>
#include <media/NdkMediaFormat.h>
#include "stdio.h"
// include android log
#include <android/log.h>
#include <jni.h>
#include <unistd.h>
#include <string.h>
#include <android/native_window.h>
#include <android/native_window_jni.h>
#include <time.h>
// include mutex
#include <mutex>


#define PACKET_FLAG_CONFIG (1ULL << 63)    // 使用无符号64位整数，设置最高位
#define PACKET_FLAG_KEY_FRAME (1ULL << 62) // 使用无符号64位整数，设置次高位
static AMediaCodec *g_codec;

AMediaFormat *createFormat(int width, int height) {
    AMediaFormat *format = AMediaFormat_new();
    AMediaFormat_setString(format, AMEDIAFORMAT_KEY_MIME, "video/avc");
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_BIT_RATE, 80000000);
    // 2135033992 is
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_COLOR_FORMAT, 2130708361);
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_FRAME_RATE, 10);
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_I_FRAME_INTERVAL, 100000);
//    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_LATENCY, 0);
//    AMediaFormat_setInt32(format, "max-bframes", 0);
    // set width and height
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_WIDTH, width);
    AMediaFormat_setInt32(format, AMEDIAFORMAT_KEY_HEIGHT, height);
    // KEY_REPEAT_PREVIOUS_FRAME_AFTER 100ms
    AMediaFormat_setInt32(format, "repeat-previous-frame-after", 100000);
    AMediaFormat_setInt32(format, "max-fps-to-encoder", 120);
    return format;
}

AMediaCodec *createMediaCodec(AMediaFormat *format) {
    AMediaCodec *codec = AMediaCodec_createEncoderByType("video/avc");

    // Configure encoder
    media_status_t status = AMediaCodec_configure(
            codec,
            format,
            nullptr,
            nullptr,
            AMEDIACODEC_CONFIGURE_FLAG_ENCODE
    );
    if (status != AMEDIA_OK) {
        // Android log error
        __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Failed to configure codec");
        AMediaCodec_delete(codec);
        AMediaFormat_delete(format);
        return nullptr;
    }
    return codec;
}

// Define a simple streamer structure
typedef struct {
    int fd;
} Streamer;

ssize_t writeFully(int fd, const void *buffer, size_t size) {
    // log size
//    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "writeFully size: %d", size);
    size_t totalWritten = 0;
    while (totalWritten < size) {
        ssize_t written = write(fd, (const uint8_t *) buffer + totalWritten, size - totalWritten);
        // log current write
//        __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "writeFully written: %d", written);
        if (written < 0) {
            perror("write");
            return -1; // 写入失败
        }
        totalWritten += written;
    }
    // log totalWritten
//    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "writeFully totalWritten: %d", totalWritten);
    return totalWritten;
}

/**
 * Writes a value to a buffer in big-endian format
 * @param buffer The target buffer
 * @param value The value to write
 * @param byteLength Number of bytes to write (1-8)
 */
void writeBigEndian(uint8_t *buffer, uint64_t value, int byteLength) {
    for (int i = 0; i < byteLength; i++) {
        buffer[i] = (value >> ((byteLength - 1 - i) * 8)) & 0xFF;
    }
}

// C 实现的 writeFrameMeta 函数
void writeFrameMeta(int fd, int packetSize, int64_t pts, bool config, bool keyFrame, int encodeTimeMs) {
    // 8 字节的 ptsAndFlags + 4 字节的 packetSize
    uint8_t headerBuffer[16];
    int64_t ptsAndFlags;

    // 设置 ptsAndFlags
    if (config) {
        ptsAndFlags = PACKET_FLAG_CONFIG; // 非媒体数据包
    } else {
        ptsAndFlags = pts;
        if (keyFrame) {
            ptsAndFlags |= PACKET_FLAG_KEY_FRAME;
        }
    }

    // Write ptsAndFlags in big-endian format
    writeBigEndian(headerBuffer, ptsAndFlags, 8);

    // Write encodeTimeMs in big-endian format
    writeBigEndian(headerBuffer + 8, encodeTimeMs, 4);

    // Write packetSize in big-endian format
    writeBigEndian(headerBuffer + 12, packetSize, 4);


    // Write to file descriptor
    if (writeFully(fd, headerBuffer, sizeof(headerBuffer)) < 0) {
        fprintf(stderr, "Failed to write frame metadata\n");
    }
}

static void streamer_write_packet(
        Streamer *streamer,
        const uint8_t *buffer,
        const AMediaCodecBufferInfo *info,
        const int encodeTimeMs
) {
    if (!streamer || !buffer || !info || info->size <= 0) {
        return;
    }

    writeFrameMeta(
            streamer->fd,
            info->size,
            info->presentationTimeUs,
            info->flags & AMEDIACODEC_BUFFER_FLAG_CODEC_CONFIG,
            info->flags & AMEDIACODEC_BUFFER_FLAG_KEY_FRAME,
            encodeTimeMs
    );

    writeFully(streamer->fd, buffer, info->size);
}

// Helper function to calculate elapsed time in milliseconds
double getElapsedTimeMs(struct timespec start, struct timespec end) {
    return (end.tv_sec - start.tv_sec) * 1000.0 +
           (end.tv_nsec - start.tv_nsec) / 1000000.0;
}


static Streamer g_streamer;
static std::mutex g_mutex;
static std::condition_variable g_condition;
static bool g_encodingFinished = false;

// 异步回调函数
static void onAsyncInputAvailable(AMediaCodec *codec, void *userdata, int32_t bufferId) {
    // 当前代码不需要处理输入缓冲区
}

static void onAsyncOutputAvailable(AMediaCodec *codec, void *userdata, int32_t bufferId, AMediaCodecBufferInfo *info) {
    // log
    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Output available bufferId=%d, size=%d, flags=%d", bufferId, info->size, info->flags);
    if (bufferId >= 0) {
        // 检查是否为流结束标志
        bool eos = (info->flags & AMEDIACODEC_BUFFER_FLAG_END_OF_STREAM) != 0;

        if (info->size > 0) {
            // 获取输出 buffer
            uint8_t *codecBuffer = AMediaCodec_getOutputBuffer(codec, bufferId, nullptr);
            if (codecBuffer != nullptr) {
                // 写入数据
//                streamer_write_packet(&g_streamer, codecBuffer, info);
            }
        }

        // 释放输出 buffer
        AMediaCodec_releaseOutputBuffer(codec, bufferId, false);

        if (eos) {
            // 停止编码器
            AMediaCodec_stop(codec);
            AMediaCodec_delete(codec);
            close(g_streamer.fd);

            // 通知编码结束
            {
                std::lock_guard<std::mutex> lock(g_mutex);
                g_encodingFinished = true;
            }
            g_condition.notify_one();

            __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Encoding finished");
        }
    }
}

static void onAsyncFormatChanged(AMediaCodec *codec, void *userdata, AMediaFormat *format) {
    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Output format changed");
}

static void onAsyncError(AMediaCodec *codec, void *userdata, media_status_t error, int32_t actionCode, const char *detail) {
    __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Codec error: %d, actionCode: %d, detail: %s", error, actionCode, detail);
}


extern "C"
JNIEXPORT void JNICALL
Java_com_genymobile_scrcpy_JNIBridge_startEncodingAsync(JNIEnv *env, jclass clazz, jobject fileDescriptor) {
    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "startEncoding invoked");

    jclass fdClass = env->GetObjectClass(fileDescriptor);
    jfieldID fdField = env->GetFieldID(fdClass, "descriptor", "I");
    jint fd = env->GetIntField(fileDescriptor, fdField);
    int fdInt = (int) fd;

    // 设置全局 streamer
    g_streamer.fd = fdInt;

    // 设置异步回调
    AMediaCodecOnAsyncNotifyCallback callback = {
            .onAsyncInputAvailable = onAsyncInputAvailable,
            .onAsyncOutputAvailable = onAsyncOutputAvailable,
            .onAsyncFormatChanged = onAsyncFormatChanged,
            .onAsyncError = onAsyncError,
    };
    media_status_t status = AMediaCodec_setAsyncNotifyCallback(g_codec, callback, nullptr);
    if (status != AMEDIA_OK) {
        __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Failed to set async callback, error code: %d", status);
        close(fdInt);
        return;
    }

    // 启动编码器
    status = AMediaCodec_start(g_codec);
    if (status != AMEDIA_OK) {
        __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Failed to start encoder, error code: %d", status);
        close(fdInt);
        return;
    }

    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Async encoding started");
    // 等待编码结束
//    {
//        std::unique_lock<std::mutex> lock(g_mutex);
//        g_condition.wait(lock, [] { return g_encodingFinished; });
//    }

    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Encoding process completed");

}

extern "C"
JNIEXPORT void JNICALL
Java_com_genymobile_scrcpy_JNIBridge_startEncoding(JNIEnv *env, jclass clazz, jobject fileDescriptor) {

    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "startEncoding invokesadasdas");
    jclass fdClass = (env)->GetObjectClass(fileDescriptor);
    jfieldID fdField = (env)->GetFieldID(fdClass, "descriptor", "I");
    jint fd = (env)->GetIntField(fileDescriptor, fdField);
    // convert jint to int
    int fdInt = (int) fd;
    // log file descriptor
    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "File Descriptor: %d", fdInt);
    // Start encoder
    media_status_t status = AMediaCodec_start(g_codec);
    if (status != AMEDIA_OK) {
        // Failed to start encoder, error code:
        __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Failed to start encoder, error code: %d", status);
        return;
    }
    // Create streamer with the file descriptor
    Streamer streamer = {.fd = fdInt};

    AMediaCodecBufferInfo bufferInfo;
    bool eos = false;

    while (!eos) {
        // 获取可用的输出 buffer
        struct timespec start_dequeue, end_dequeue;
        clock_gettime(CLOCK_MONOTONIC, &start_dequeue);
        int outputBufferId = AMediaCodec_dequeueOutputBuffer(g_codec, &bufferInfo, -1);
        clock_gettime(CLOCK_MONOTONIC, &end_dequeue);
        double dequeue_time = getElapsedTimeMs(start_dequeue, end_dequeue);
        // log output buffer id
//        __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Output Buffer ID: %d", outputBufferId);

        if (outputBufferId >= 0) {
            // 检查是否为流结束标志
            eos = (bufferInfo.flags & AMEDIACODEC_BUFFER_FLAG_END_OF_STREAM) != 0;

            // 检查 buffer 大小是否大于 0
            if (bufferInfo.size > 0) {
                // Measure time for AMediaCodec_getOutputBuffer
                struct timespec start_getbuffer, end_getbuffer;
                clock_gettime(CLOCK_MONOTONIC, &start_getbuffer);
                uint8_t *codecBuffer = nullptr;
                size_t bufferSize = 0;

                // 获取输出 buffer
                codecBuffer = AMediaCodec_getOutputBuffer(g_codec, outputBufferId, &bufferSize);
                clock_gettime(CLOCK_MONOTONIC, &end_getbuffer);
                double getbuffer_time = getElapsedTimeMs(start_getbuffer, end_getbuffer);
                // log bufferSize outputBufferId
                if (codecBuffer != nullptr) {
                    AMEDIACODEC_INFO_TRY_AGAIN_LATER;
                    AMEDIACODEC_INFO_OUTPUT_FORMAT_CHANGED;
                    AMEDIACODEC_INFO_OUTPUT_BUFFERS_CHANGED;
                    // Measure time for streamer_write_packet
                    struct timespec start_write, end_write;
                    clock_gettime(CLOCK_MONOTONIC, &start_write);
                    streamer_write_packet(&streamer, codecBuffer, &bufferInfo, (int)(dequeue_time * 100));
                    clock_gettime(CLOCK_MONOTONIC, &end_write);
                    double write_time = getElapsedTimeMs(start_write, end_write);

                    __android_log_print(ANDROID_LOG_INFO, "Scrcpy",
                                        "queue=%0.2f,getBuffer=%0.2f ms, writePacket=%0.2f ms, size=%d flags=%d",
                                        dequeue_time, getbuffer_time, write_time, bufferInfo.size, bufferInfo.flags);
                }
            }
            // 释放输出 buffer
            AMediaCodec_releaseOutputBuffer(g_codec, outputBufferId, false);
        } else if (outputBufferId == AMEDIACODEC_INFO_OUTPUT_FORMAT_CHANGED) {
            __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Output format changed");
        } else if (outputBufferId == AMEDIACODEC_INFO_OUTPUT_BUFFERS_CHANGED) {
            __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Output buffers changed");
        } else if (outputBufferId == AMEDIACODEC_INFO_TRY_AGAIN_LATER) {
            // Just retry
        }
    }

// Clean up
    AMediaCodec_stop(g_codec);
    AMediaCodec_delete(g_codec);
//    AMediaFormat_delete(format);
    close(fd);
}


extern "C"
JNIEXPORT jobject JNICALL
Java_com_genymobile_scrcpy_JNIBridge_createSurface(JNIEnv *env, jclass clazz, int width, int height) {
    // log width and height
    __android_log_print(ANDROID_LOG_INFO, "Scrcpy", "Width: %d, Height: %d", width, height);
    AMediaFormat *format = createFormat(width, height);
    AMediaCodec *codec = createMediaCodec(format);
    if (codec == nullptr) {
        return nullptr;
    }

    // Get input Surface
    ANativeWindow *inputSurface;
    media_status_t status = AMediaCodec_createInputSurface(codec, &inputSurface);
    if (status != AMEDIA_OK) {
        // Failed to create input Surface, error code: " << status
        __android_log_print(ANDROID_LOG_ERROR, "Scrcpy", "Failed to create input Surface, error code: %d", status);
        AMediaCodec_delete(codec);
        AMediaFormat_delete(format);
        return nullptr;
    }
    jobject surface = ANativeWindow_toSurface(env, inputSurface);
    g_codec = codec;
    return env->NewGlobalRef(reinterpret_cast<jobject>(surface));
}

JNIBridge.java

package com.genymobile.scrcpy;

import android.view.Surface;

import java.io.FileDescriptor;

public class JNIBridge {

    static {
        System.load("/data/local/tmp/libscrcpy.so");
    }

    public static native void startEncoding(FileDescriptor fd);
    public static native void startEncodingAsync(FileDescriptor fd);

    public static native Surface createSurface(int width, int height);
}

package com.genymobile.scrcpy.video;

import com.genymobile.scrcpy.AndroidVersions;
import com.genymobile.scrcpy.AsyncProcessor;
import com.genymobile.scrcpy.JNIBridge;
import com.genymobile.scrcpy.Options;
import com.genymobile.scrcpy.device.ConfigurationException;
import com.genymobile.scrcpy.device.Size;
import com.genymobile.scrcpy.device.Streamer;
import com.genymobile.scrcpy.util.Codec;
import com.genymobile.scrcpy.util.CodecOption;
import com.genymobile.scrcpy.util.CodecUtils;
import com.genymobile.scrcpy.util.IO;
import com.genymobile.scrcpy.util.Ln;
import com.genymobile.scrcpy.util.LogUtils;

import android.graphics.Canvas;
import android.graphics.Color;
import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.os.Build;
import android.os.Looper;
import android.os.SystemClock;
import android.view.Surface;

import java.io.FileDescriptor;
import java.io.IOException;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;

public class SurfaceEncoder implements AsyncProcessor {

    private static final int DEFAULT_I_FRAME_INTERVAL = 10; // seconds
    private static final int REPEAT_FRAME_DELAY_US = 100_000; // repeat after 100ms
    private static final String KEY_MAX_FPS_TO_ENCODER = "max-fps-to-encoder";

    // Keep the values in descending order
    private static final int[] MAX_SIZE_FALLBACK = {2560, 1920, 1600, 1280, 1024, 800};
    private static final int MAX_CONSECUTIVE_ERRORS = 3;

    private final SurfaceCapture capture;
    private final Streamer streamer;
    private final String encoderName;
    private final List<CodecOption> codecOptions;
    private final int videoBitRate;
    private final float maxFps;
    private final boolean downsizeOnError;

    private boolean firstFrameSent;
    private int consecutiveErrors;

    private Thread thread;
    private final AtomicBoolean stopped = new AtomicBoolean();

    private final CaptureReset reset = new CaptureReset();
    private boolean useNativeEncoder = true;

    public SurfaceEncoder(SurfaceCapture capture, Streamer streamer, Options options) {
        this.capture = capture;
        this.streamer = streamer;
        this.videoBitRate = options.getVideoBitRate();
        this.maxFps = options.getMaxFps();
        this.codecOptions = options.getVideoCodecOptions();
        this.encoderName = options.getVideoEncoder();
        this.downsizeOnError = options.getDownsizeOnError();
    }

    private void streamCapture() throws IOException, ConfigurationException {
        if (useNativeEncoder) {
            capture.init(reset);
            capture.prepare();
            Size size = capture.getSize();
            streamer.writeVideoHeader(size);
            Surface surface = JNIBridge.createSurface(size.getWidth(), size.getHeight());
            capture.start(surface);
            JNIBridge.startEncoding(streamer.fd);
            return;
        }
        Codec codec = streamer.getCodec();
        MediaCodec mediaCodec = createMediaCodec(codec, encoderName);
        Ln.i("videoBitRate -> " + videoBitRate);
        MediaFormat format = null;
        if (!useNativeEncoder) {
            format = createFormat(codec.getMimeType(), videoBitRate, maxFps, codecOptions);
        }

        capture.init(reset);

        try {
            boolean alive;
            boolean headerWritten = false;

            do {
                reset.consumeReset(); // If a capture reset was requested, it is implicitly fulfilled
                capture.prepare();
                Size size = capture.getSize();
                if (!headerWritten) {
                    streamer.writeVideoHeader(size);
                    headerWritten = true;
                }
                if (!useNativeEncoder) {
                    assert format != null;
                    format.setInteger(MediaFormat.KEY_WIDTH, size.getWidth());
                    format.setInteger(MediaFormat.KEY_HEIGHT, size.getHeight());
                }

                Surface surface = null;
                boolean mediaCodecStarted = false;
                boolean captureStarted = false;
                try {
                    if (!useNativeEncoder) {
                        mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
                        surface = mediaCodec.createInputSurface();
                    } else {
                        surface = JNIBridge.createSurface(size.getWidth(), size.getHeight());
                    }


                    capture.start(surface);
                    captureStarted = true;
                    if (!useNativeEncoder) {
                        mediaCodec.start();
                    }
                    mediaCodecStarted = true;

                    // Set the MediaCodec instance to "interrupt" (by signaling an EOS) on reset
                    reset.setRunningMediaCodec(mediaCodec);

                    if (stopped.get()) {
                        alive = false;
                    } else {
                        boolean resetRequested = reset.consumeReset();
                        if (!resetRequested) {
                            // If a reset is requested during encode(), it will interrupt the encoding by an EOS
                            if (!useNativeEncoder) {
                                encode(mediaCodec, streamer);
                            } else {
                                JNIBridge.startEncoding(streamer.fd);
                            }
                        }
                        // The capture might have been closed internally (for example if the camera is disconnected)
                        alive = !stopped.get() && !capture.isClosed();
                    }
                } catch (IllegalStateException | IllegalArgumentException | IOException e) {
                    if (IO.isBrokenPipe(e)) {
                        // Do not retry on broken pipe, which is expected on close because the socket is closed by the client
                        throw e;
                    }
                    Ln.e("Capture/encoding error: " + e.getClass().getName() + ": " + e.getMessage());
                    if (!prepareRetry(size)) {
                        throw e;
                    }
                    alive = true;
                } finally {
                    reset.setRunningMediaCodec(null);
                    if (captureStarted) {
                        capture.stop();
                    }
                    if (mediaCodecStarted) {
                        try {
                            mediaCodec.stop();
                        } catch (IllegalStateException e) {
                            // ignore (just in case)
                        }
                    }
                    mediaCodec.reset();
                    if (surface != null) {
                        surface.release();
                    }
                }
            } while (alive);
        } finally {
            mediaCodec.release();
            capture.release();
        }
    }

    private boolean prepareRetry(Size currentSize) {
        if (firstFrameSent) {
            ++consecutiveErrors;
            if (consecutiveErrors >= MAX_CONSECUTIVE_ERRORS) {
                // Definitively fail
                return false;
            }

            // Wait a bit to increase the probability that retrying will fix the problem
            SystemClock.sleep(50);
            return true;
        }

        if (!downsizeOnError) {
            // Must fail immediately
            return false;
        }

        // Downsizing on error is only enabled if an encoding failure occurs before the first frame (downsizing later could be surprising)

        int newMaxSize = chooseMaxSizeFallback(currentSize);
        if (newMaxSize == 0) {
            // Must definitively fail
            return false;
        }

        boolean accepted = capture.setMaxSize(newMaxSize);
        if (!accepted) {
            return false;
        }

        // Retry with a smaller size
        Ln.i("Retrying with -m" + newMaxSize + "...");
        return true;
    }

    private static int chooseMaxSizeFallback(Size failedSize) {
        int currentMaxSize = Math.max(failedSize.getWidth(), failedSize.getHeight());
        for (int value : MAX_SIZE_FALLBACK) {
            if (value < currentMaxSize) {
                // We found a smaller value to reduce the video size
                return value;
            }
        }
        // No fallback, fail definitively
        return 0;
    }

    private void encode(MediaCodec codec, Streamer streamer) throws IOException {
        MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();

        boolean eos;
        do {
            long startDequeue = SystemClock.elapsedRealtimeNanos();
            int outputBufferId = codec.dequeueOutputBuffer(bufferInfo, -1);
            long endDequeue = SystemClock.elapsedRealtimeNanos();
            double dequeueTimeMs = (endDequeue - startDequeue) / 1_000_000.0;
            try {
                eos = (bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0;
                // On EOS, there might be data or not, depending on bufferInfo.size
                if (outputBufferId >= 0 && bufferInfo.size > 0) {
                    long startGetBuffer = SystemClock.elapsedRealtimeNanos();
                    ByteBuffer codecBuffer = codec.getOutputBuffer(outputBufferId);
                    long endGetBuffer = SystemClock.elapsedRealtimeNanos();
                    double getBufferTimeMs = (endGetBuffer - startGetBuffer) / 1_000_000.0;

                    boolean isConfig = (bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0;
                    if (!isConfig) {
                        // If this is not a config packet, then it contains a frame
                        firstFrameSent = true;
                        consecutiveErrors = 0;
                    }
                    long startWrite = SystemClock.elapsedRealtimeNanos();
//                    Ln.i("dequeueTimeMs -> " + dequeueTimeMs + ", getBufferTimeMs -> " + getBufferTimeMs + ", size -> " + bufferInfo.size + ", flags -> " + bufferInfo.flags);
                    streamer.writePacket(codecBuffer, bufferInfo, (int) (dequeueTimeMs * 100));
                    long endWrite = SystemClock.elapsedRealtimeNanos();
                    double writeTimeMs = (endWrite - startWrite) / 1_000_000.0;

//                    Ln.i(String.format("dequeueBuffer=%.2f ms, getBuffer=%.2f ms, size=%d flags=%d write=%.2f ms",
//                            dequeueTimeMs, getBufferTimeMs, bufferInfo.size, bufferInfo.flags, writeTimeMs));
                }
            } finally {
                if (outputBufferId >= 0) {
                    codec.releaseOutputBuffer(outputBufferId, false);
                }
            }
        } while (!eos);
    }

    private static MediaCodec createMediaCodec(Codec codec, String encoderName) throws IOException, ConfigurationException {
        if (encoderName != null) {
            Ln.d("Creating encoder by name: '" + encoderName + "'");
            try {
                MediaCodec mediaCodec = MediaCodec.createByCodecName(encoderName);
                String mimeType = Codec.getMimeType(mediaCodec);
                if (!codec.getMimeType().equals(mimeType)) {
                    Ln.e("Video encoder type for \"" + encoderName + "\" (" + mimeType + ") does not match codec type (" + codec.getMimeType() + ")");
                    throw new ConfigurationException("Incorrect encoder type: " + encoderName);
                }
                return mediaCodec;
            } catch (IllegalArgumentException e) {
                Ln.e("Video encoder '" + encoderName + "' for " + codec.getName() + " not found\n" + LogUtils.buildVideoEncoderListMessage());
                throw new ConfigurationException("Unknown encoder: " + encoderName);
            } catch (IOException e) {
                Ln.e("Could not create video encoder '" + encoderName + "' for " + codec.getName() + "\n" + LogUtils.buildVideoEncoderListMessage());
                throw e;
            }
        }

        try {
            MediaCodec mediaCodec = MediaCodec.createEncoderByType(codec.getMimeType());
            Ln.d("Using video encoder: '" + mediaCodec.getName() + "'");
            return mediaCodec;
        } catch (IOException | IllegalArgumentException e) {
            Ln.e("Could not create default video encoder for " + codec.getName() + "\n" + LogUtils.buildVideoEncoderListMessage());
            throw e;
        }
    }

    private static MediaFormat createFormat(String videoMimeType, int bitRate, float maxFps, List<CodecOption> codecOptions) {
        MediaFormat format = new MediaFormat();
        format.setString(MediaFormat.KEY_MIME, videoMimeType);
        format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
        // must be present to configure the encoder, but does not impact the actual frame rate, which is variable
        format.setInteger(MediaFormat.KEY_FRAME_RATE, 60);
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            format.setInteger(MediaFormat.KEY_LATENCY, 0);
        }
        format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
        if (Build.VERSION.SDK_INT >= AndroidVersions.API_24_ANDROID_7_0) {
            format.setInteger(MediaFormat.KEY_COLOR_RANGE, MediaFormat.COLOR_RANGE_LIMITED);
        }
        format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, DEFAULT_I_FRAME_INTERVAL);
        // display the very first frame, and recover from bad quality when no new frames
        format.setLong(MediaFormat.KEY_REPEAT_PREVIOUS_FRAME_AFTER, REPEAT_FRAME_DELAY_US); // µs
        if (maxFps > 0) {
            // The key existed privately before Android 10:
            // <https://android.googlesource.com/platform/frameworks/base/+/625f0aad9f7a259b6881006ad8710adce57d1384%5E%21/>
            // <https://github.com/Genymobile/scrcpy/issues/488#issuecomment-567321437>
            format.setFloat(KEY_MAX_FPS_TO_ENCODER, maxFps);
        }

        if (codecOptions != null) {
            for (CodecOption option : codecOptions) {
                String key = option.getKey();
                Object value = option.getValue();
                CodecUtils.setCodecOption(format, key, value);
                Ln.d("Video codec option set: " + key + " (" + value.getClass().getSimpleName() + ") = " + value);
            }
        }

        return format;
    }

    @Override
    public void start(TerminationListener listener) {
        thread = new Thread(() -> {
            // Some devices (Meizu) deadlock if the video encoding thread has no Looper
            // <https://github.com/Genymobile/scrcpy/issues/4143>
            Looper.prepare();

            try {
                streamCapture();
            } catch (ConfigurationException e) {
                // Do not print stack trace, a user-friendly error-message has already been logged
            } catch (IOException e) {
                // Broken pipe is expected on close, because the socket is closed by the client
                if (!IO.isBrokenPipe(e)) {
                    Ln.e("Video encoding error", e);
                }
            } finally {
                Ln.i("Screen streaming stopped");
                listener.onTerminated(true);
            }
        }, "video");
        thread.start();
    }

    @Override
    public void stop() {
        if (thread != null) {
            stopped.set(true);
            reset.reset();
        }
    }

    @Override
    public void join() throws InterruptedException {
        if (thread != null) {
            thread.join();
        }
    }
}

Notice the useNativeEncoder variable in the code.

I have run this test successfully but have not found concrete methods to measure performance accurately. Relying solely on frame rate seems insufficient. Despite sending additional information to calculate performance differences, external factors and latency fluctuations make it challenging to obtain usable results.

Do you have any good ideas for this?

[rom1v]

Thank you for your work and your interest.

First, a general (meaningful) method is to measure end-to-end latency (for example #646), but to get precise results it requires a high-framerate camera.

I wanted to remove the adb forward process and connect directly via IP + Port.

It's interesting to see if adb forward adds latency.

However, note that the adb connection between the computer and the device is authenticated, so using adb forward allows to benefit from the authentication.

In the past, I also tested to use a local (UNIX) socket on both side (when the computer runs on linux) instead of using a TCP socket on the computer side. But in practice I could not get any measurable improvement, so it was not worth the added complexity to work differently on different platforms.

However, during my tests, the latency fluctuated by a few milliseconds, making it difficult to determine if it was truly faster.

To get useful results, you should also test real use cases (typically not send a single byte, but full video packets).

One possible interesting test would be to get a timestamp when you write to the socket (on the device) and a timestamp when you read from the socket (on the client), and measure the difference for a lot of data, and produce statistics (mean & variance). Of course, the two clocks are different, so you have to convert the timestamps from one side to the clock of the other size, using a simple NTP-like protocol (a single UDP packet round-trip with 2 timestamps from the client and 2 timestamps from the server, cf §The clock synchronization algorithm). Due to clock drift, you must synchronize quite often (every few seconds, depending on the accuracy/precision you want). In practice, depending on many factors, the clock drift might be in the order of ~4 seconds per day (~50ppm).

Implementing Encoding and Streaming in C/C++

It's not trivial to compare, especially since with MediaCodec we receive encoded packets directly: there is not a "capture step" and then an "encoding step", make it more difficult to compare different approaches (Java vs C for example).

But the code executed in Java or C here is "minimal", the "hard" work is done by the encoder. In both cases there is JNI involved to communicate between C and Java. The most offending part in Java would be to pass buffer data from C to Java to C, but the ByteBuffer used is direct, and is written directly to the socket using its file descriptor and Os.write(fd, from), so data is not copied.

Therefore, I would expect that writing the code which calls MediaCodec in C or Java would not make any significant difference.

[mengyanshou]

Thank you for your reply. Recently, I have been delving deeper into encoding and decoding, such as whether the low-latency flag is also used by the encoder. Regarding the testing methods you mentioned, I will give them a try. I have already run a demo of the clock drift algorithm, and I will continue to update the test results in this issue.

As for the copying process you mentioned, I directly passed the socket's file descriptor to C. At this point, using write in C to write the encoded data to the file descriptor seems to avoid the overhead of data transfer between C and Java. It only requires Java to pass information to C, and C can independently handle encoding and streaming.

Regarding ADB forward authentication, I didn't quite understand your point. What I meant is that when the device is connected remotely (i.e., when running adb devices), the device is listed with its IP address.

In this case, if a socket is created on the device with ip+port, and the scrcpy client starts the scrcpy-server to directly fetch the video stream through this socket, would it reduce latency? I will also test this later.

As for testing metrics, I feel that taking the average value over a period of time is a good approach. I use a millisecond-level stopwatch and speed up the camera shutter. However, this can only measure latency roughly and cannot measure the frame rate.

It is possible that using the NDK could allow the client to receive more frames, but this is just a guess.

I have written a client using the scrcpy-server. Once I finish some final touches, you can try it out. It essentially ports scrcpy's functionality to Android but uses the Android API for decoding.

I have also tried SDL2 + FFmpeg to port the complete scrcpy source code to Android, and it runs normally. However, the efficiency is relatively low because it uses software decoding.