Extract archive callback.

[MiyamuraMiyako]

Is your feature request related to a problem? Please describe.
I need complute extract progress and process every file when one file extracted.

Describe the solution you'd like
Add callback param in extractall function.

Describe alternatives you've considered
None

Additional context
None

[miurahr]

SevenZipFile.extract() calls Worker class to extract target. The core logic inside py7zr is as follows:

1. walk through file lists and detect directories and symbolic links, and then memory target output filepath into worker class.

py7zr/py7zr/py7zr.py

Line 723 in 98ac851

for f in self.files:

py7zr/py7zr/py7zr.py

Line 763 in 98ac851

self.worker.register_filelike(f.id, outfilename)

2. making directories

py7zr/py7zr/py7zr.py

Lines 765 to 767 in 98ac851

	for target_dir in sorted(target_dirs):
	try:
	target_dir.mkdir()

3. calling worker extractor

py7zr/py7zr/py7zr.py

Line 776 in 98ac851

self.worker.extract(self.fp, parallel=(not self.password_protected))

4. making symbolic links

py7zr/py7zr/py7zr.py

Lines 778 to 785 in 98ac851

	# create symbolic links on target path as a working directory.
	# if path is None, work on current working directory.
	for t in target_sym:
	sym_dst = t.resolve()
	with sym_dst.open('rb') as b:
	sym_src = b.read().decode(encoding='utf-8') # symlink target name stored in utf-8
	sym_dst.unlink() # unlink after close().
	sym_dst.symlink_to(pathlib.Path(sym_src))

We should add a callback to step3 worker extractor.

Here is an extractor main part.

py7zr/py7zr/compression.py

Lines 201 to 207 in 98ac851

	for i in range(numfolders):
	r, w = multiprocessing.Pipe(duplex=False)
	p = ctx.Process(target=self.extract_single,
	args=(filename, folders[i].files,
	self.src_start + positions[i], self.src_start + positions[i + 1], w))
	p.start()
	extract_processes.append((p, r))

Processes are run in spawned another process concurrently. We need to add some shared memory or pipe to communicate extract worker process and main process.

Here is a core part of worker extractor

py7zr/py7zr/compression.py

Lines 229 to 235 in 98ac851

	fp.seek(src_start)
	for f in files:
	fileish = self.target_filepath.get(f.id, None)
	if fileish is not None:
	with fileish.open(mode='wb') as ofp:
	if not f.emptystream:
	self.decompress(fp, f.folder, ofp, f.uncompressed[-1], f.compressed, src_end)

Here line 231 get a output file path for target compressed chunk which register in step1.

We can add a code to report progress to main process here.

[MiyamuraMiyako]

Yes, I think just report IsDirectory, FileName and PercentDone.

[miurahr]

py7zr extract files concurrently using threading. This may cause report become such as
[(IsDirectory, FileName, DoneItems, SubTotalItems), (IsDirectory, FileName, DoneItems, SubTotalItems).... ]

For example, test data 'tests/data/mblock_1.7z' has 3 blocks, when running '7z l mblock_1.7z' we can see

Listing archive: ./mblock_1.7z

--
Path = ./mblock_1.7z
Type = 7z
Physical Size = 631690
Headers Size = 2305
Method = LZMA2:1536k BCJ
Solid = +
Blocks = 3

So py7zr will run with 4 threads, first one extract directories at first,
second one extract following files;

2015-11-18 18:04:24 .R..A         5263        94014  C/7z.h
2015-11-09 18:41:08 .R..A         1548               C/7zAlloc.c
2015-03-26 01:07:58 .R..A          403               C/7zAlloc.h
.etc...

and third one extract concurrently with second thread following files;

2014-12-03 23:35:44 ....A         5522        25204  DOC/7zC.txt
2010-09-16 21:57:16 ....A         7573               DOC/7zFormat.txt
2016-09-27 19:42:33 ....A         3031               DOC/Methods.txt
etc...

and last extract concurrently

2016-10-05 00:13:34 ....A        35328       510167  bin/7zS2.sfx
2016-10-05 00:13:32 ....A        35328               bin/7zS2con.sfx
2016-10-05 00:12:31 ....A       113152               bin/7zSD.sfx
...etc

[MiyamuraMiyako]

All extract thread concurrently submit maybe cause issue, I think need add synchronize lock, but maybe cause extract efficiency.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[miurahr]

@MiyamuraMiyako now #130 implement a primitive progress callback mechanism.
Could you try it? You can see how to use it on test_basic.py:446 test_extract_callback test case.