I ended up not being able to swap out the core logic of the port to use pointers, but I was able to convert everything I needed at the edges. This is both technically sufficient and deeply unsatisfying, but I've got work to do.
The biggest obvious hole in the serialization code, after I handled the pointer stuff, is the lack of serializing runtime objects. I'm still not excited to try to mix subclassing with the way I did this, so I've got a big Union of all of the subclass equivalents. It turns out that a fifteen-plus-variant union generates a lot of type errors.
I've been working my way through things, getting bits of code actually typed, but I've hit a snag that I'm too sleepy to handle right now: I've somehow broken Mypy. (Again.) I tried to do a simple refactor, and suddenly it's telling me to test against master and file a bug. I kind of need a reproduction first, and my attempts so far have not reproduced the issue.
Anyway, time to wrap up.