Error: <class 'TypeError'> Can't write on file xxx

[Kuroihitsuji]

Hello, when I m running the commands in the examples you gave, I am getting following errors when the program is trying to write data to file
Error: <class 'urllib.error.HTTPError'>
Can't write on file about-searching-on-github
Error: <class 'TypeError'>
Can't write on file marketplace?category=security&type=actions

The command I tried was: python3 torcrawl.py -v -u http://www.github.com/ -c -d 2 -p 5 -e
and if it let it finish enumerating the list, I eventually get an out of index error too:
IndexError: list index out of range

With python3 torcrawl.py -w -u http://www.github.com -o github.htm
I am getting the same error:
Error: <class 'TypeError'>
Can't write on file www.github.com/github.htm

I made the changes you mentioned in another issue post

Any pointers on how the issue can be fixed?

[MikeMeliz]

Hey @Kuroihitsuji, thanks for reporting this!

I guess unit tests would be useful after the migration of the script to python3, as some bugs appeared. No worries though, it's on top of the list.

For the Error: <class 'TypeError'>, opening the output file in binary mode wb will work, but I have some concerns if this is the best practice here. Let me know if you'll encounter any issues after changing this line extractor.py:56 to f = open(outputFile, 'wb') and line extractor.py:31 to f = open(outpath + "/" + outputFile, 'wb').

For IndexError: list index out of range, that's a known error reported here crawler.py:12. If a None keyword is included in the links.txt file -as a link- the script can't process it, and I haven't really find a way to exclude it from the list in the first place. A cheap solution would be to replace lines 20-27 of extractor.py with:

		# Generate name for every file
		try:
			pagename = line.rsplit('/', 1)
			clpagename = str(pagename[1])
			clpagename = clpagename[:-1]
			if len(clpagename) == 0:
				outputFile = "index.htm"
			else:
				outputFile = clpagename
		except IndexError as e:
			print("Error: %s" % e)
			continue

I'll change this bits of codes on the next commit, but still, this bug needs a proper solution to not mess up the crawler's output.

[MikeMeliz]

Hello, @Kuroihitsuji,

I've pushed the above changes to the repository, as another user kindly confirmed that fixed the problem, and I'll keep this issue as a reference for this bug in crawler.py.

I'll close this issue for now, but feel free to re-open it or submit another issue if you'll experience the same error.

Cheers ;)