Source Cleanup: When there is a large amount of streaming, Source leakage causes OOM (Out of Memory).

[2k9laojia]

Description
Describe the problem you encountered.
When pushing the stream to SRS, the SRS process crashed.

Environment

1. Operating System: CentOS
2. Encoder (Tool and Version): ...
3. Player (Tool and Version): ...
4. SRS Version: 2.0.248
5. SRS log is as follows:
   client identified, type=fmle-publish, stream_name=62c1352ea3a64621999bfd946081f392, duration=-1.00
   connected stream, tcUrl=rtmp://172.16.129.97/netposa, pageUrl=, swfUrl=, schema=rtmp, vhost=defaultVhost, port=1935, app=netposa, stream=62c1352ea3a64621999bfd946081f392, args=null
   st_thread_create failed. ret=1017(Cannot allocate memory)

Reproduction
The steps to reproduce the bug are as follows:

1. Start SRS, run ...
2. Push the stream, run ...
3. The bug is reproduced, and the key information is as follows:

...

Expected Behavior
Describe what you expect to happen.

By examining the stack analysis, the reasons are as follows:
During the push stream process, the function SrsSource::initialize is called, which then calls hls->initialize. However, an error occurs in hls->initialize, specifically st_thread_create failed, resulting in an error code being returned. In the function SrsSource::fetch_or_create, srs_freep(source) is called, leading to the destructor of SrsSource being invoked. As a result, the member variable play_edge is released, triggering its destructor and subsequently calling the destructor of the SrsPlayEdge member variable ingester. The stop function is called, ultimately invoking _source->on_unpublish(). Since the call to hls->initialize failed and returned without calling play_edge->initialize(this, _req), the _source pointer in SrsEdgeIngester is null, causing a core dump. Therefore, adding a null check before using _source can resolve the issue.

I hope someone knowledgeable can help answer why st_thread_create failed and why it resulted in a failure.

TRANS_BY_GPT3

[winlinvip]

The reason is clearly stated in the log: 'Cannot allocate memory'.
Can the scenario of running out of memory be reproduced?

TRANS_BY_GPT3

[2k9laojia]

My server had 60GB of remaining memory at that time, and the current SRS process was using 3.4GB of memory. Does running out of memory refer to the memory limit for the current process? This issue was occurring repeatedly while handling 500 streaming routes, closing and restarting them.

TRANS_BY_GPT3

[2k9laojia]

[2k9laojia]

After checking the srsSource objects in the pool, there are SrsSource::do_cycle_all with a pool size of [38320]. I don't know if it has any impact or not.

TRANS_BY_GPT3

[winlinvip]

You have also used a considerable amount of memory for other things, so the available memory should be less than 60GB.

How did you test it? Can you describe the process?

Encoder (tool and version): ...
Player (tool and version): ...

TRANS_BY_GPT3

[2k9laojia]

                total        used        free      shared  buff/cache   available

Mem: 109G 10G 61G 4.0G 38G 94G
Swap: 4.0G 250M 3.8G

[2k9laojia]

I push the stream to SRS using route 500, then play it on 500 clients, repeat for one hour, close all streams, and repeat the above steps. Each stream has a unique stream ID, so a new SRS source object is created every time a stream is pushed. SRSlibrtmp is used for pushing the stream, and JMeter is used for playing the stream, which is discarded after obtaining it. Now that the stress test is over, the CPU and memory usage of SRS remain high. I plan to add pool cleaning in the reload function. When this situation occurs, I will send a signal to SRS to execute a reload and release all SRS source objects in the pool, to see if the CPU and memory usage can be reduced.

TRANS_BY_GPT3

[2k9laojia]

TRANS_BY_GPT3

[2k9laojia]

'

Note: The translation provided maintains the markdown structure by italicizing the text.

TRANS_BY_GPT3

[2k9laojia]

An srsSource object will start 2 coroutines. At that time, there were 38,320 remaining srsSource objects in the pool, which is equivalent to over 70,000 coroutines. This scheduling is a significant CPU consumption. After releasing the srsSource, the CPU usage can be noticeably reduced.

TRANS_BY_GPT3

[winlinvip]

Okay, I understand what you mean. It seems like the performance issue is caused by too many sources. I will try to reproduce it first. Thank you.

TRANS_BY_GPT3

[winlinvip]

After I enabled HTTP FLV, I created a new source with curl and repeatedly used "killall curl" to reproduce this issue. The speed increases at around 1MB every 3 seconds.

for ((i=0;;i++)); do curl http://localhost:8080/live/livestream-$i.flv -o /dev/null ; sleep 0.1; done

for ((;;)); do killall curl; sleep 0.2; done

I discovered that many coroutines are actually SrsAsyncCallWorker, which means they are coroutines for asynchronous callbacks.

(gdb) p _st_iterate_threads_flag =1
(gdb) b _st_iterate_threads
(gdb) c
(gdb) bt
#1  0x000000000059ff55 in st_cond_timedwait (cvar=0x4953b40, 
    timeout=18446744073709551615) at sync.c:197
#2  0x00000000005a003a in st_cond_wait (cvar=0x4953b40) at sync.c:219
#3  0x00000000004aa3f6 in srs_cond_wait (cond=0x4953b40)
    at src/service/srs_service_st.cpp:334
#4  0x000000000058432c in SrsAsyncCallWorker::cycle (this=0x4953ae0)
    at src/app/srs_app_async_call.cpp:104
#5  0x00000000004f78da in SrsSTCoroutine::cycle (this=0x495a3c0)
    at src/app/srs_app_st.cpp:198
#6  0x00000000004f794f in SrsSTCoroutine::pfn (arg=0x495a3c0)
    at src/app/srs_app_st.cpp:213

I found that HLS and DVR are using this.

SrsDvrPlan::SrsDvrPlan() {
    async = new SrsAsyncCallWorker();

SrsHlsMuxer::SrsHlsMuxer() {
    async = new SrsAsyncCallWorker();

Running for about 16 minutes, with 4k streams, it occupies 125MB of memory and CPU usage is 5%.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
21719 root      20   0 1091240 126256   4976 S   5.0  6.2   0:30.52 ./objs/srs+ 

real	16m19.259s
user	0m23.620s
sys	0m6.990s

If this part is optimized, the number of coroutines may decrease, but it will not have a significant impact on the overall performance.

TRANS_BY_GPT3