Scrapy Plugins

All

33 repositories

scrapy-playwright
Public
🎭 Playwright integration for Scrapy
hacktoberfest chrome-headless python-asyncio headless-browser javascript-renderer firefox-headless playwright playwright-python webkit-headless python
Python
•
BSD 3-Clause "New" or "Revised" License
•141•1.2k•18•2•Updated Aug 3, 2025Aug 3, 2025
scrapy-zyte-api
Public
Zyte API integration for Scrapy
plugin crawler proxy scraping scrapy
Python
•
BSD 3-Clause "New" or "Revised" License
•22•38•14•3•Updated Jul 28, 2025Jul 28, 2025
scrapy-crawl-maps
Public
Scrapy plugin that allows defining the logic of a spider using a directed graph defined in JSON format.
BSD 3-Clause "New" or "Revised" License
•0•2•0•1•Updated May 21, 2025May 21, 2025
scrapy-spider-metadata
Public
Python
•
MIT License
•3•5•3•0•Updated Mar 24, 2025Mar 24, 2025
scrapy-zyte-smartproxy
Public
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
plugin crawler proxy scraping scrapy crawler-detection
Python
•
BSD 3-Clause "New" or "Revised" License
•90•364•7•1•Updated Mar 24, 2025Mar 24, 2025
scrapy-deltafetch
Public
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
hacktoberfest hacktoberfest2021
Python
•48•273•13•4•Updated Feb 26, 2025Feb 26, 2025
scrapy-feedexporter-sftp
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•4•6•0•0•Updated Feb 12, 2025Feb 12, 2025
scrapy-splash
Public
Scrapy+Splash for JavaScript integration
scrapy headless-browsers
Python
•
BSD 3-Clause "New" or "Revised" License
•459•3.2k•62•7•Updated Feb 11, 2025Feb 11, 2025
zyte-spidermon
Public
Python
•1•0•0•0•Updated Sep 5, 2024Sep 5, 2024
scrapy-settings-log
Public
An extension that allows a user to display all or some of their scrapy spider settings at runtime.
Python
•
GNU General Public License v3.0
•4•0•0•0•Updated Aug 20, 2024Aug 20, 2024
scrapy-feedexporter-dropbox
Public
Scrapy feed exporter for Dropbox
Python
•3•1•0•0•Updated Jul 11, 2024Jul 11, 2024
scrapy-crawlera-fetch
Public archive
Scrapy Downloader Middleware for Crawlera Fetch API
Python
•
BSD 3-Clause "New" or "Revised" License
•10•8•5•5•Updated Jun 5, 2024Jun 5, 2024
scrapy-feedexporter-azure-storage
Public
Python
•6•3•0•3•Updated May 18, 2024May 18, 2024
scrapy-feedexporter-onedrive
Public
Export to OneDrive
Python
•1•1•0•0•Updated Feb 23, 2024Feb 23, 2024
scrapy-djangoitem
Public
Scrapy extension to write scraped items using Django models
Python
•
BSD 3-Clause "New" or "Revised" License
•87•502•10•3•Updated Oct 15, 2023Oct 15, 2023
scrapy-feedexporter-google-sheets
Public
Python
•0•1•0•0•Updated Aug 15, 2023Aug 15, 2023
scrapy-incremental
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•1•1•0•0•Updated May 1, 2023May 1, 2023
scrapy-feedexporter-google-drive
Public
Python
•2•3•1•0•Updated Jan 4, 2023Jan 4, 2023
scrapy-statsd
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•4•6•1•3•Updated Oct 19, 2022Oct 19, 2022
scrapy-dotpersistence
Public
A scrapy extension to sync `.scrapy` folder to an S3 bucket
Python
•
BSD 3-Clause "New" or "Revised" License
•8•17•2•1•Updated Mar 28, 2022Mar 28, 2022
scrapy-magicfields
Public
Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
Python
•
BSD 3-Clause "New" or "Revised" License
•7•56•0•0•Updated Mar 16, 2022Mar 16, 2022
scrapy-pagestorage
Public
A scrapy extension to store requests and responses information in storage service
Python
•
BSD 3-Clause "New" or "Revised" License
•6•26•3•0•Updated Mar 11, 2022Mar 11, 2022
scrapy-snowflake-stage-exporter
Public
Snowflake database loading utility with Scrapy integration
snowflake scrapy etl
Python
•2•5•0•0•Updated Dec 3, 2021Dec 3, 2021
scrapy-headless
Public
Python
•
BSD 3-Clause "New" or "Revised" License
•8•29•2•2•Updated Apr 28, 2021Apr 28, 2021
scrapy-jsonschema
Public
Scrapy schema validation pipeline and Item builder using JSON Schema
Python
•
BSD 3-Clause "New" or "Revised" License
•12•44•1•0•Updated Mar 26, 2021Mar 26, 2021
scrapy-jsonrpc
Public
Scrapy extension to control spiders using JSON-RPC
Python
•71•300•7•2•Updated Aug 26, 2019Aug 26, 2019
scrapy-hcf
Public
Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs
Python
•
BSD 3-Clause "New" or "Revised" License
•6•4•1•1•Updated Aug 28, 2018Aug 28, 2018
scrapy-monkeylearn
Public
A Scrapy pipeline to categorize items using MonkeyLearn
Python
•13•37•0•0•Updated Apr 28, 2017Apr 28, 2017
scrapy-splitvariants
Public
Scrapy spider middleware to split an item into multiple items using a multi-valued key
Python
•
BSD 3-Clause "New" or "Revised" License
•1•20•0•0•Updated Feb 8, 2017Feb 8, 2017
scrapy-streaming
Public
Python
•5•18•2•6•Updated Oct 12, 2016Oct 12, 2016