Three Dollar Quill

So, like I said, I filed the bug report, and then realized "Wait, I bet I can handle this." So here's the general idea: I've written a plugin for handling the concept of a dependent mapping, a mapping where the type of a value depends on the type of the corresponding key. I'm allowing for arbitrary relationships, which should require a concept along the lines of higher-kinded types, which Python's typing ecosystem mostly does not have. Now, a higher-kinded type is basically like, you have a type-level function between types, and you can write type-level functions that operate on type-level functions instead of plain types. So, you could have list[T] or T | None, and you'd like to be able to pass them around. This turns out to not really work; you can't pass around "a generic type with a hole in it".

However, we can instead take a type-level function, and turn it into the type of a function that transforms types the way we'd like it to.

Let's turn those examples above into what I'm talking about.

class ListOf(Protocol):
    def __call__(self, __k: T) -> list[T]:
        ...

class Maybe(Protocol):
    def __call__(self, __k: T) -> T | None:
        ...

Now, these callable protocols are entirely concrete, of course; just look at the class lines.

Now, these callable protocols are entirely generic, of course; just look at the def lines.

Absolutely not; why do you think we're in this mess?

Anyway, I could imagine various uses for this idiom; actually implementing these protocols could be useful for some kind of data processing pipeline, potentially. But I've instead been using them as the type arguments of a generic protocol, which is like Mapping, but is not Mapping. Most of the methods on the protocol actually typecheck perfectly fine, and to bridge the gap, I had to write a plugin that transforms a select few signature and method accesses, pulling up variables hidden inside the protocols.

This worked perfectly fine for a few versions, but with Mypy 1.7.0, some new type inference code became the default, and it gives those methods much less helpful default signatures. At first I thought "Well, I need the signatures to go back the way they were", but then I realized that most of the information is still intact, just not where I want it to be, and there is a finite set of methods to update, so I can hardcode rather a lot if I need to.

Let's have a look at what I've put together so far. Now, none of this is functional code changes; it doesn't really work to rush into writing a Mypy plugin; getting small details wrong can crash it outright. So, I'm currently focused on diagnostics. Print-based diagnostics. Just stick a print statement in the plugin code, and it shows up in the command-line output. EZ.

Here's what I've got currently:

CALLABLE_SLOTS = (
    "arg_types",
    "arg_kinds",
    "arg_names",
    "min_args",
    "ret_type",
    "name",
    "definition",
    "variables",
    "is_ellipsis_args",
    "implicit",
    "special_sig",
    "from_type_type",
    "bound_args",
    "def_extras",
    "type_guard",
    "from_concatenate",
    "imprecise_arg_kinds",
    "unpack_kwargs",
)


def pprint_callable_type(callable_type, when):
    print(when, "(")
    for slot in CALLABLE_SLOTS:
        print(slot, "=", getattr(callable_type, slot))
    print(")")


def debug_hook(hook):
    def new_hook(ctx):
        self_arg = ctx.type.args[0]
        print(self_arg.type["__call__"].type)
        before = ctx.default_signature
        pprint_callable_type(before, "before")
        after = hook(ctx)
        pprint_callable_type(after, "after")
        return after

    return new_hook

Most of the hooks in the plugin focus on rewriting the method signature, so I've written a diagnostic decorator that details the information relevant to the plugin's functionality, and how the hook transforms the information that it's supposed to transform. Because it's a decorator, I can put it on a hook, and take it off, without worrying about messing up the hook's body.

I've put together a small synthetic test file, and am comparing the debug output with and without the new type inference. So far, I'm seeing that the output is mostly the same; it's just that arg_types and ret_type have Never instead of anything useful.

And... I just discovered a crash that was latent in the plugin code all along. I'm going to have to figure out how to convert that into a proper call, or at least a reasonable error telling people "Hey, you can't use the plugin like that. Stop it."

Anyway, it's going to take some more work and a bunch of notes to put together a plan for how to handle this stuff. I'd better put this entry down and see what I'm up for the rest of the night.

Good night.