14 implement yara keyword search #15
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 7 commits
March 17, 2022 23:36
- Adds new argument '-y' to accept yara switch. - Implements new 'check_yara' method in extractor module to check for keyword matches from .yar file. - Implements new 'text' method to extract only the text elements from the http content response for yara parsing. - Amends extractor methods to accept new 'yara' argument and utilise new 'check_yara' and 'text' methods. See Issue: MikeMeliz#14
- Refactors check_yara method to remove category checking. Future Feature - Update main application docstring. See Issue: MikeMeliz#14
- Removes commented out code. - Amends absolute module calls. See Issue: MikeMeliz#14
- Updates requirements.txt to include lxml. - Ammends crawl method in crawler.py to dismiss None values. See Issue: MikeMeliz#14
- Ammeds torcrawl.py to support -y argument accepting a value. - Adds conditional handling for unexpected arguements for option '-y'. - Refactors module/extractor.py to perform content parsing within check_yara method based on -y argument. - Updates README.md to provide instructions for '-y/--yara' argument use. - Updates res/keywords.yar to support README.md examples. See Issue: MikeMeliz#14
- Updates docstrings in modules/extractor.py See Issue: MikeMeliz#14
- Resolve gramatical mistake. See Issue: MikeMeliz#14
MikeMeliz
approved these changes
Mar 26, 2022
There was a problem hiding this comment.
Awesome work @the-siegfried throughout the whole implementation!
|
Thanks @MikeMeliz :) |
the-siegfried
pushed a commit
to the-siegfried/TorCrawl.py
that referenced
this pull request
Jul 3, 2022
…keyword-search 14 implement yara keyword search
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This branch extends the torcrawl.py applications available arguments to include the '-y/--yara' option and value, which is later used to determine whether keyword searching is enabled and if the scope of the search includes the entire http response or just the content text.
Motivation and Context
To extend the capability of the application to support keyword searching across all extraction methods and allow for further extensibility. This feature will allow the applications users to make use of yara rule based text matching for basic keyword searching. However, by adapting this implementation we can later add support for site and page scoring/categorisation as well.
How Has This Been Tested?
Following the example in the updated README.md project documentation:
$python torcrawl.py -v -w -u https://github.com -c -d 1 -p 5 -e -y 0I was able to successfully run the application and test the newly implemented keyword searching capability. Please find the results attached.
Types of changes
Checklist:
proxy-git.cwkhome.fun.zip