Should we do `dred_compute_latents` in DTX?

[shuanzhu]

We found that the average cost time doubled in silence segment compared with in voiced segment when enable DRED.

After checking the code, dred_compute_latents will always on even in DTX opus would send empty packet, and minor input will cause a huge increase in GRU's calculations.
To resolve this problem, we tried to enable OPUS_FLOAT_APPROX, but it didn't work, and then we add _mm_setcsr(csr | 0x8040) when do encode in our code, it works.

To significantly save our CPU, We want to ask you experts should we still do dred_compute_latents in DTX? To be specific, could we change to:

#ifdef ENABLE_DRED
    opus_int32 in_dtx = 0;
    if ( st->dred_duration > 0 && st->dred_encoder.loaded && opus_encoder_ctl(st, OPUS_GET_IN_DTX_REQUEST(in_dtx)) == OPUS_OK && !in_dtx ) {
        int frame_size_400Hz;
        /* DRED Encoder */
        dred_compute_latents( &st->dred_encoder, &pcm_buf[total_buffer*st->channels], frame_size, total_buffer, st->arch );
        frame_size_400Hz = frame_size*400/st->Fs;
        OPUS_MOVE(&st->activity_mem[frame_size_400Hz], st->activity_mem, 4*DRED_MAX_FRAMES-frame_size_400Hz);
        for (i=0;i<frame_size_400Hz;i++)
           st->activity_mem[i] = activity;
    } else {
        st->dred_encoder.latents_buffer_fill = 0;
        OPUS_CLEAR(st->activity_mem, DRED_MAX_FRAMES);
    }
#endif

???

[jmvalin]

So it looks like you're hitting some kind of issue with denormalized float values that can take longer to compute than normal values. There's other cases where we hit that in Opus and typically the solution is to add the VERY_SMALL (1e-30) constant in the right place to avoid the issue. You can see a few examples in celt/celt_decoder.c. So here it would help if you could track down exactly where the issue happens. It may very well be just a single loop causing issues.

[jishnubr]

The Problem: Denormals, GRUs, and Silence

You are absolutely right. The performance degradation during silent segments is a classic symptom of denormal floating-point numbers.

1. What are denormals? In floating-point arithmetic, denormals (or subnormal numbers) are very tiny values that are closer to zero than the smallest "normal" representable number.
2. Why are they slow? Most modern CPUs are highly optimized for calculations on normal floating-point numbers. When a calculation involves a denormal number (either as an input or an output), the hardware pipeline often cannot handle it directly. It triggers a microcode exception, causing the CPU to trap and use a much, much slower software-assisted method to complete the calculation.
3. Your Scenario: During silent segments, the input PCM values are zero or very close to zero. The internal states of the GRU (Gated Recurrent Unit) in dred_compute_latents decay towards zero. This creates a cascade of denormal inputs and outputs, grinding the FPU to a halt and causing the massive performance drop you observed.

Your successful experiment with _mm_setcsr(csr | 0x8040) confirms this diagnosis perfectly. You enabled two crucial flags in the x86 SSE control/status register (MXCSR):

• Flush-to-Zero (FTZ): Makes the hardware treat denormal outputs as true zero.
• Denormals-are-Zero (DAZ): Makes the hardware treat denormal inputs as true zero.

This is the standard, industry-wide solution for avoiding performance issues with denormals in real-time audio and DSP applications.

Analysis of Your Proposed Solution

You propose skipping the DRED feature computation during DTX periods.

// Your proposed change (conceptual)
#ifdef ENABLE_DRED
    opus_int32 in_dtx = 0;
    // Get the DTX state from the encoder
    if (opus_encoder_ctl(st, OPUS_GET_IN_DTX_REQUEST(&in_dtx)) == OPUS_OK && !in_dtx) {
        // ... run dred_compute_latents and update activity_mem ...
    } else {
        // ... clear DRED state ...
        st->dred_encoder.latents_buffer_fill = 0;
        OPUS_CLEAR(st->activity_mem, DRED_MAX_FRAMES);
    }
#endif

This is a very logical and pragmatic approach. Let's analyze its correctness and implications.

Pros:

• Fixes the Performance Issue: It completely avoids running the expensive GRU computations on silent frames, directly solving the problem.
• Logically Sound: DRED (Deep REDundancy) is a form of Forward Error Correction (FEC). Its purpose is to encode perceptually important information from the current frame to help the decoder reconstruct it if it gets lost. For a DTX/silent frame, there is no perceptually important information to encode. The existing PLC mechanisms are perfectly capable of concealing the loss of a silent packet. Therefore, computing DRED features for a DTX frame is indeed wasted effort.

Cons / Potential Risks:

• State Reset: The else block in your proposal is quite aggressive. dred_encoder.latents_buffer_fill = 0; and OPUS_CLEAR(st->activity_mem, DRED_MAX_FRAMES); effectively reset the DRED encoder's history. The GRU within DRED is a recurrent network; it has a state that depends on previous frames. Wiping this state during a short DTX period (e.g., a brief pause in speech) might slightly reduce the effectiveness of DRED on the very first frame when speech resumes.
• Complexity: It adds more control logic to the main encoding path, coupling the DRED feature encoder to the main encoder's DTX state.

Verdict on Your Proposal: The core idea is excellent. Skipping DRED computation on DTX frames is the right thing to do from a resource-management perspective. However, the aggressive state clearing might be unnecessary and potentially suboptimal. A simpler version might be better:

// A slightly safer version of your proposal
#ifdef ENABLE_DRED
    opus_int32 in_dtx = 0;
    opus_encoder_ctl(st, OPUS_GET_IN_DTX_REQUEST(&in_dtx));

    if (st->dred_duration > 0 && st->dred_encoder.loaded && !in_dtx) {
        // ... run dred_compute_latents ...
        // ... update activity_mem ...
    }
    /* No 'else' block. Let the DRED encoder's internal state persist through short silences.
       The buffer of past latents will naturally be cleared over time if no new ones are added. */
#endif

The Architecturally "Cleaner" Solution: Managing the FPU State

While your proposal is a valid logical fix, the most robust and common way to solve this in high-performance DSP is to fix the underlying numerical problem, not bypass the algorithm. This is what you did with _mm_setcsr.

The library itself should manage the FPU state to ensure denormals are handled efficiently. This prevents the algorithm from ever being exposed to performance-killing denormals, regardless of whether it's a DTX frame or just a very quiet frame that isn't technically DTX.

Here’s how it should be implemented inside opus_encode_native() in opus_encoder.c:

// In opus_encoder.c, inside opus_encode_native()

// ... function entry ...
#if !defined(FIXED_POINT) && (defined(__x86_64__) || defined(_M_X64) || defined(__i386__) || defined(_M_IX86))
    #include <xmmintrin.h>
    unsigned int old_csr;
    // Save the original CSR state
    old_csr = _mm_getcsr();
    // Set Flush-to-Zero and Denormals-are-Zero flags
    _mm_setcsr(old_csr | 0x8040);
#elif !defined(FIXED_POINT) && (defined(__aarch64__) || defined(__arm__))
    // For ARMv7+ with NEON or ARMv8
    unsigned int old_fpcr;
    #if defined(__aarch64__)
        __asm__ __volatile__ ("mrs %0, fpcr" : "=r" (old_fpcr));
        // Set FZ (bit 24) for flush-to-zero
        __asm__ __volatile__ ("msr fpcr, %0" : : "r" (old_fpcr | (1<<24)));
    #else // arm
        __asm__ __volatile__ ("vmrs %0, fpscr" : "=r" (old_fpcr));
        // Set FZ (bit 24) for flush-to-zero
        __asm__ __volatile__ ("vmsr fpscr, %0" : : "r" (old_fpcr | (1<<24)));
    #endif
#endif

// ... ALL of the existing encoding logic from opus_encode_native() goes here ...
// This includes the call to dred_compute_latents, etc.

// ... before every return path ...
#if !defined(FIXED_POINT) && (defined(__x86_64__) || defined(_M_X64) || defined(__i386__) || defined(_M_IX86))
    // Restore the original CSR state
    _mm_setcsr(old_csr);
#elif !defined(FIXED_POINT) && (defined(__aarch64__) || defined(__arm__))
    // Restore the original FPCR state
    #if defined(__aarch64__)
        __asm__ __volatile__ ("msr fpcr, %0" : : "r" (old_fpcr));
    #else // arm
        __asm__ __volatile__ ("vmsr fpscr, %0" : : "r" (old_fpcr));
    #endif
#endif
// return ret;

Important: The state must be restored before every return statement in the function to avoid leaking the changed FPU state to the calling application.

Recommendation

You have two excellent options.

1. The Pragmatic Fix (Your Proposal): Gating the dred_compute_latents call with an !in_dtx check is a good, targeted fix. It will solve your immediate problem with minimal changes. I would recommend the "safer" version without the else block that clears the state, unless you find that the DRED state becomes stale after long DTX periods.

2. The Robust Fix (Managing FPU State): Implementing the save/set/restore logic for the FPU control register is the architecturally superior solution. It solves the denormal problem at its root, benefits all floating-point calculations within the function (not just DRED), and doesn't require altering the algorithm's logic. This is what I would recommend for inclusion in the upstream libopus project.

Given that you are packaging for Debian, applying a patch for Option 1 is likely the simplest and safest immediate course of action. It's a clear, self-contained logical change.

For the long term, I would strongly encourage you to propose the FPU state management patch (Option 2) to the Opus developers on GitHub. It's a professional-grade optimization that belongs in a high-performance library like Opus.