Syntax Highighting - Retrospective 2018-08-20

By Max Woerner Chase

In this post:

This week, having finally gotten tired of looking at the awful highlighting I was getting out of my custom color scheme, I decided to start learning the ropes of properly developing a syntax file.

I determined that the big gap in my knowledge was in writing and using syntax test files, since I'm already quite capable of tossing together yaml to try to get behavior I want. I also decided I wanted a better name for the family of syntax files I'm working on than "Edge".

My plan for the week was to write up some syntax requirements based on the Python language reference, and then write tests based on those requirements.

I was able to write up several subsections, but when it came to the subsections on identifiers, I instead wrote up over 400 words detailing my issues with how the syntax is now, and how I want to try to change it. Since I want to have the syntax solid in the less contentious aspects before I mess with it in a big way, I elected to skip specifying those tests for now. I was able to write tests for a few basic areas, though, and make fixes to the syntax file when some of those tests failed. For most areas, it seems to be pretty quick to write up a set of tests.

I never did come up with a better name. I'll have to keep on that on and off.

I wonder if it's possible to define syntax tests at a meta level, like in a pytest file, then use Hypothesis to try to generate edge cases. Probably not worth it.

I decided I will be plugging away at the language reference and several other sections of the documentation, getting solid test coverage, before I try to make the syntax behave like I want. What I discovered from specifying comments is that I can feel pretty confident in messing with code that I have tests for, and the thing about redoing the identifier stuff is, that would potentially touch most of the code. So it needs tests everywhere.

Next week, I haven't decided. There is one thing I have in mind that would mainly be research, with the heavy lifting, such as it is, to come later. Alternatively, I could put another week towards Structured Data and see what that gets me.