Web console: Severe performance problems viewing large log files

[spadgett]

I tried to load the log for my heapster pod, which is at least 20K+ lines. It took maybe 15 minutes, consumed 1GB of RAM, and pegged my CPU at over 100% when I finally stopped it. The only thing I've done in the browser tab since refreshing is view the log.

Looping `ng-repeat` over each log line is likely the problem. We're also passing each line to `filter : logSearch`, which I didn't think was used. Still working on profiling it.

I also noticed that my Vagrant VM running OpenShift was using ~125% CPU.

/cc @jwforres @benjaminapetersen

[spadgett]

CPU Usage:

[smarterclayton]

Verify you have at least master in the last day - just to eliminate the potential for the CPU bug jordan fixed.

[liggitt]

How does a large log get chunked? One message per line? 20k messages would be a lot

[liggitt]

Possible leads for large list performance: http://tech.small-improvements.com/2013/09/10/angularjs-performance-with-large-lists/

[jwforres]

Some thoughts at 2am...

We knew we were going to hit some issues here, which is why we requested the api enhancements for partial log retrieval. Did any of those happen to get done early?

Is there any way for us to do a preemptive HEAD request to know how big the log would be? For very large logs we could show a message like github does with large files/diffs and recommend using the cli... guessing our raw file download would still cause problems since it does things in memory

[spadgett]

Verify you have at least master in the last day - just to eliminate the potential for the CPU bug jordan fixed.

Yeah, I did a clean build of the latest yesterday evening before seeing this.

[spadgett]

How does a large log get chunked? One message per line?

Yes, one message for each line.

[spadgett]

If I remove the ng-repeat rendering of the log, CPU usage drops to 10-15%, so the problem seems to be client side.

[jwforres]

So the only reason to have the lines broken out into individual divs, vs
just using a

 was to add the line numbers to the front

On Thu, Oct 29, 2015 at 9:37 AM, Sam Padgett [email protected]

wrote:

If I remove the ng-repeat rendering of the log, CPU usage drops to

10-15%, so the problem seems to be client side.
—

Reply to this email directly or view it on GitHub

#5474 (comment).

[spadgett]

If I use direct DOM manipulation instead of ng-repeat, performance is quite a bit better although the log still loads somewhat slowly. We might be gated by how fast the data comes over the web socket.

[jwforres]

Not surprising, angular falls over after a certain number of data bindings
:)

On Thu, Oct 29, 2015 at 9:48 AM, Sam Padgett [email protected]
wrote:

If I use direct DOM manipulation instead of ng-repeat, performance is
quite a bit better although the log still loads somewhat slowly. We might
be gated by how fast the data comes over the web socket.

—
Reply to this email directly or view it on GitHub
#5474 (comment).

[spadgett]

So the only reason to have the lines broken out into individual divs, vs just using a <pre> was to add the line numbers to the front

We can still add line numbers, although this is not very "Angular."

        var lineNumber = 1;
        streamer.onMessage(function(msg) {
          $('<div class="log-line"/>').text(lineNumber + ": " + msg).appendTo('#logContent');
          lineNumber++;
        });

[smarterclayton]

All of the things we discussed are in.

limitBytes - total number of bytes returned in a chunk
tail - last N lines
sinceTimestamp = time or sinceSeconds = relative time in seconds

time + tail + limitBytes should work correctly.

On Thu, Oct 29, 2015 at 2:31 AM, Jessica Forrester <[email protected]

wrote:

Some thoughts at 2am...

We knew we were going to hit some issues here, which is why we requested
the api enhancements for partial log retrieval. Did any of those happen to
get done early?

Is there any way for us to do a preemptive HEAD request to know how big
the log would be? For very large logs we could show a message like github
does with large files/diffs and recommend using the cli... guessing our raw
file download would still cause problems since it does things in memory

—
Reply to this email directly or view it on GitHub
#5474 (comment).