Brakeman on Apple Mac Silicon makes more than 10 minutes to find files, even in fast mode

[anthonyamar]

Background

Brakeman version: 7.0.0
Rails version: 7.2.2.1
Ruby version: 3.4.1

Repo size: ~110k LOC
Device: MacBook Pro M3 Pro 36 GB

Hanging or Slowness

Launching brakeman -I to manage ignored warning take something between 10 and 20 minutes, stuck on the Finding files... step. Trying to launch it in faster mode doesn't help, neither the --debug flag output any error.

I also tried to search for any root within my Mac config (Gatekeeper, Spotlight, Rosetta), but nothing helped.

Also: tried with --no-prism, and passing the a flag to parse only the file where the false positive is, it's not faster.

Note that, my colleagues on WSL and Linux, on the same codebase, with less powerful computers, runs the same command in ~40 secondes.

Any idea?

Please run Brakeman with --debug to see which file may be causing the issue.

brakeman -I --debug --faster
[Notice] Using Prism parser
Loading scanner...
Processing application in /Users/anthonyamar/Code/tracks-web
Processing gems...
[Notice] Using Prism parser
Parsing Gemfile
[Notice] Detected Rails 7 application
(Processing gems) Duration: 0.004797 seconds
Processing configuration...
[Notice] Using Prism parser
Parsing config/environment.rb
[Notice] Using Prism parser
Parsing config/application.rb
[Notice] Using Prism parser
Parsing config/environments/production.rb
[Notice] Escaping HTML by default
[Notice] Skipping config setting: active_storage.queues.analysis
[Notice] Skipping config setting: active_storage.queues.purge
(Processing configuration) Duration: 0.003049 seconds
Finding files...

[presidentbeef]

You tried --only-files with a small directory or single file and still had a problem? 🤔

[anthonyamar]

You tried --only-files with a small directory or single file and still had a problem? 🤔

Both. A single file, and a folder with only one controller in it. Same problem.

Any idea?

[thedanbob]

I just ran into this myself. Weird thing is, it works in another project which is not significantly different (both are small rails 8 apps, both running brakeman 7.0.2).

[thedanbob]

I think I know why this is happening: Macs are extremely slow at enumerating files. On my linux machine, I can run find . -type f | wc -l in a large directory and get the result of 2M+ files in under 3 seconds, whereas the same command on my Mac takes around 6 seconds for only 90k files. The project where brakeman was slow had a populated storage directory (those 90k files). When I moved that directory somewhere else temporarily, brakeman ran in about the same amount of time as the other project.

Unfortunately, --skip-files and --only-files can't help here because brakeman gathers all files first, then filters based on those options.

[anthonyamar]

Thanks, @thedanbob!

I moved my files in /tmp and /storage, and yes, it worked! I also tried to cd in app and run brakeman --force into it, and it worked as well. So this is definitely a matter of files! It should be a great thing to have a real exclude files/folders, that prevent brakeman to parse it, instead of filtering afterward.

[thedanbob]

I went spelunking in the Rubocop code base to try to figure out why it is still fast on macOS and found this: target_finder.rb#L62

# Search for files recursively starting at the given base directory using the given flags that
# determine how the match is made. Excluded files will be removed later by the caller, but as an
# optimization find_files removes the top level directories that are excluded in configuration
# in the normal way (dir/**/*).
def find_files(base_dir, flags)
  # get all wanted directories first to improve speed of finding all files
  exclude_pattern = combined_exclude_glob_patterns(base_dir)
  dir_flags = flags | File::FNM_PATHNAME | File::FNM_EXTGLOB
  patterns = wanted_dir_patterns(base_dir, exclude_pattern, dir_flags)
  patterns.map! { |dir| File.join(dir, '*') }
  # We need this special case to avoid creating the pattern
  # /**/* which searches the entire file system.
  patterns = [File.join(base_dir, '**/*')] if patterns.empty?

  Dir.glob(patterns, flags).select { |path| FileTest.file?(path) }
end

And the default top-level excludes are:

# rubocop gem
- 'node_modules/**/*'
- 'tmp/**/*'
- 'vendor/**/*'
- '.git/**/*'

# rubocop-rails gem
- app/assets/**/*
- log/**/*
- public/**/*
- storage/**/*

Perhaps something similar could be done in Brakeman.