Cannot get pylint to register custom checker, is there a missing step in the docs?

[daogilvie]

Question

Hi! First: Thanks for supporting and maintaining such a great ecosystem!

My question: I'm trying to make a simple checker around some custom formatting in log strings. I have followed every tutorial I can find from within the last few years, and naturally looked at the guide in the documentation.

However, I am finding that the register function, described as the way to get your checker actually loaded into pylint, simply is not being called. The module is being loaded, for sure — I can see breakpoints in the top level of the module being hit, but not in the register function itself. I've tried setting the --load-plugins argument, I've tried using the load_plugins setting in pylintrc.

Is there an additional step or constraint missing in the implementation of custom checkers from the docs? Or am I simply making a mistake?

Below is the code for my absolute bare-bones first attempt.

Stored in testchecker.py:

from astroid import nodes
from pylint.checkers import BaseChecker

breakpoint() # HIT!

class SimpleCheck(BaseChecker):
    """Simple"""
    name = "simple-test-check"
    msgs = {
            "W6900": ("Test", "test-check", "Just a test")
            }
    options = ( ("fake-option", {
                "default": False,
                "type": "yn", # EDIT: bool here was not allowed
                "metavar": "<y or n>",
                "help": "Some fake option"}
                ),
            )

    def visit_import(self, _: nodes.Import) -> None:
        breakpoint() # NOT HIT!

def register(linter):
    breakpoint() # NOT HIT!
    linter.register_checker(SimpleCheck(linter))

My file I am linting, lintme.py:

"""MODULE DOCSTRING"""
import logging

logger = logging.getLogger(__name__)

logger.info("hello world")

Both files are in the same directory, in which I execute:

pylint --load-plugins=testchecker lintme.py

$ pylint --version
pylint 2.14.5
astroid 2.11.7
Python 3.10.4 (main, May 19 2022, 08:51:27) [Clang 13.1.6 (clang-1316.0.21.2.5)]
$ uname -a
Darwin OVO4941L.ovoenergy.local 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:25 PDT 2022; root:xnu-8020.140.41~1/RELEASE_X86_64 x86_64

Thanks again!

Documentation for future user

I expect the guide to creating a custom checker to contain everything needed to do so, and to have a section highlighting common and easy mistakes (I'm assuming I have made such a mistake)

Additional context

I have searched stack overflow and pre-existing issues on this repo, and not found anyone reporting this.
There is https://stackoverflow.com/questions/71590677/cant-load-custom-pylint-module, but they report getting an error about not finding the module, then succeeding by placing the module in the pylint source. This is not what I am experiencing.

[Pierre-Sassoulas]

Thank you for opening the issue and the kind words. I don't see what you did wrong at first sight so we'll have to investigate, but I don't have the time right now. Here's a real world example in pylint_django to maybe unblock you without waiting for us: https://github.com/PyCQA/pylint-django/blob/master/pylint_django/checkers/__init__.py

[daogilvie]

Hi @Pierre-Sassoulas, thank you for such a quick response! It seems they use a different top-level function name (register_checkers, not just register), but this doesn't seem to work in my case. I was hoping I'd simply made a silly mistake, and maybe I still have! If it's not obvious though, then I will spend some time tomorrow digging into the pylint internals and see if I can't debug what's happening directly, hopefully save you the time.

[daogilvie]

Update: I added a breakpoint to the dynamic module loader in pylinter.py and... now it does load my checker? I have changed literally nothing else 🤔 Or, at least, I'm almost certain I've changed nothing else.

Update Update: It calls register, but actually there is some other error that results in a stack trace. Continuing to investigate.

pylint/config/argument.py", line 240, in __init__
    self.type = _TYPE_TRANSFORMERS[arg_type]
KeyError: 'bool'

I suspect I've supplied something wrong with the option spec.

Updated update update: Yep, I misunderstood the docs and assumed the type for options would mirror python types, but this is not the case. Have updated the example to accomodate. HOWEVER, adding a breakpoint seems to make it work, and taking the breakpoint away... makes it not work. 🤯 I've not encountered that before, so this is a fun new learning experience for me.

[daogilvie]

Ok, right. @Pierre-Sassoulas I don't know if this is a bug, or just a quirk that needs documenting.

NOTE: Have tweaked the body below after doing some further research after eating something 😄

TL;DR: pylint registers custom plugins specified on command line without the current directory included in sys.path, and even if an init-hook does this later it never retries registration if it can't import a module first time round.

Detailed Description

When supplying custom plugin module names to load via the command line, this list is sent to the importer code before the current directory is added to sys.path. This means that, as in my case, it fails to import the custom checker as it cannot find it in the path.
This first attempt at importing, crucially, is the only time that pylint tries to call register. It will never attempt to register a plugin module of the same name twice, adding their names to a set of ones it has tried, it seems. This means that, even though having the module in a pylintrc file would work on its own, specifying it once in the file and once in commandline will not work.

The reason the module is always being imported though, is that later on, pylint loops through the set of plugin names and attempts to load those modules again. This time, it looks for the load_configuration function. I don't define this in my case, so nothing happens, but by this time the current directory has been added to the sys.path list, by some init-hook code I have for an unrelated reason, so the module is imported without any trouble by astroid.

Putting a breakpoint in makes this work in 2 ways:

1. The act of putting a breakpoint in apparently adds the current working directory into the syspath, something I did not know.
2. It seems that the init-hook runs in time for the sys path to be updated for when I cont/step out of the breakpoint ¯_(ツ)_/¯

Steps to Reproduce

1. Create custom checker in current directory.
2. Run pylint, supplying --load_plugins on command line
3. Observe that your checker is imported but not registered.
4. Add a pylintrc file, where you add your plugin to the load_plugins configuration item, and an init-hook to add the current directory to sys.path.
5. Run pylint, with no command line plugin arguments
6. Observe that everything works as you expect

Happy to Help!

I'm happy to investigate further, and to contribute documentation, or code, to work around this, but I'd want to make sure I was doing something that was Pylint Approved™️ rather than waste your time with a PR that "solves" the "problem" in a way that the maintainers would never go for. I don't do enough open source, so here is as good a place to start as any 😄

[Pierre-Sassoulas]

Thank you for the very detailed investigation ! It seems it's indeed a bug with the configuration loading vs CLI argument that might be done differently.

[daogilvie]

Thank you for the very detailed investigation ! It seems it's indeed a bug with the configuration loading vs CLI argument that might be done differently.

Ok! If I can get the time over the next 3-4 days I'll try to open a PR as per the contribution guidelines, or I'll put a note here if that isn't going to happen.

[daogilvie]

EDIT: spoke too soon, please disregard this comment for the moment.

So it looks like the reason this is happening is that the pylint entrypoint attempts to process all provided command line arguments here in one go, (function defined config.utils#L230).
I'm not entirely sure what the right approach is, given the inter-relation between the arguments you can provide and what is in the file. It seems like quite an involved solution to retry CLI plugin loading if it failed and there is an init-hook defined in a config file. Perhaps the best balance is surfacing a more nuanced error if modules cannot be loaded from the CLI argument?
The right answer might simply be a documentation change to make people aware of this? I.e don't provide --load_plugins for a non-syspath location without also providing --init_hook for the path?