Open
Description
I would like to use forum-dl to generate a list of links from a given forum that I could then send to SingleFile to generate HTML pages of all posts in the thread.
I am using this command:
forum-dl -g --no-boards --no-files https://forum.com/forums/showthread.php?12345-title-of-the-post/page19
When I run this command for vbulletin, it does not generate a list of all 19 pages in the thread as I would expect to happen -- just the one page that I entered. Like so:
https://forum.com/forums/showthread.php
https://forum.com/forums/showthread.php?12345-title-of-the-post/page19
https://forum.com/
This happens no matter which page in the forum I pass into forum-dl.
When I add -v to the above command, I get the following output:
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php {} {}
https://forum.com/forums/showthread.php
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): forum.com:443
DEBUG:urllib3.connectionpool:https://forum.com:443 "GET /forums/showthread.php HTTP/1.1" 200 None
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php?12345-title-of-the-post/page19 {} {}
https://forum.com/forums/showthread.php?12345-title-of-the-post/page19
DEBUG:urllib3.connectionpool:https://forum.com:443 "GET /forums/showthread.php?12345-title-of-the-post/page19 HTTP/1.1" 200 None
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php {} {}
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php {} {}
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php?12345-title-of-the-post/page19 {} {}
DEBUG:root:Attempting GET https://forum.com/forums/ {} {}
https://forum.com/forums/
DEBUG:urllib3.connectionpool:https://forum.com:443 "GET /forums/ HTTP/1.1" 200 None
DEBUG:root:Attempting GET https://forum.com/forums/showthread.php?12345-title-of-the-post/page19 {} {}
DEBUG:root:Attempting GET https://forum.com/forums/ {} {}
I tried running the command to output the files to a directory:
forum-dl --files-output="test/" https://forum.com/forums/showthread.php?12345-title-of-the-post/page19
I got the following error:
INFO:root:GET https://forum.com/forums/showthread.php {} {}
INFO:root:GET https://forum.com/forums/showthread.php?12345-title-of-the-post/page19 {} {}
INFO:root:GET https://forum.com/ {} {}
Traceback (most recent call last):
File "/home/user/.pyenv/versions/3.10.11/bin/forum-dl", line 8, in <module>
sys.exit(main())
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/__init__.py", line 34, in main
forumdl.download(
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/forumdl.py", line 24, in download
self.download_url(
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/forumdl.py", line 48, in download_url
writer.write(url)
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/writers/common.py", line 78, in write
self.write_board(base_node)
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/writers/common.py", line 103, in write_board
self._write_board_object(board)
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/writers/common.py", line 235, in _write_board_object
sys.stdout.write(f"{self._serialize_entry(entry)}\n")
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/forum_dl/writers/jsonl.py", line 10, in _serialize_entry
return entry.json(models_as_dict=False)
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/typing_extensions.py", line 2562, in wrapper
return __arg(*args, **kwargs)
File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pydantic/main.py", line 950, in json
raise TypeError('The `models_as_dict` argument is no longer supported; use a model serializer instead.')
TypeError: The `models_as_dict` argument is no longer supported; use a model serializer instead.
--
Result of pip3 --version
pip 23.2.1 from /home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/pip (python 3.10)
Result of uname -a
Linux computername 5.19.0-46-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 13:30:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Result of cat /etc/os-release
PRETTY_NAME="Ubuntu 22.10"
NAME="Ubuntu"
VERSION_ID="22.10"
VERSION="22.10 (Kinetic Kudu)"
VERSION_CODENAME=kinetic
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=kinetic
LOGO=ubuntu-logo