Three Dollar Quill

I was looking over my previous code for Crafting Interpreters, and I found an interesting comment.

-- Can this be switched to the iterator protocol?
-- Don't try yet. That kind of speculative refactoring can go... poorly.
function scanner_index:scan_tokens()

And I thought... It's a fresh codebase. Why not try it now?

To do this right, I need to get a handle on just what this would entail. Basically, I want to think of jlox's scanTokens() method instead as a top-level function that takes a source string and returns a Lua iterator of tokens. Now, the end goal (for now) is to accumulate this iterator state into a table, so I put together a little module that I think looks reasonable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

return function(func, iterator_function, state, initial_value, _closing_value)
  local accumulator = {}
  local control_variable = initial_value
  while control_variable ~= nil do
    local loop_variables = {iterator_function(state, control_variable)}
    control_variable = loop_variables[1]
    if control_variable == nil then
      break
    end
    table.insert(accumulator, (func(table.unpack(loop_variables))))
  end
  return accumulator
end

I'm not sure if I'll actually use this, but it all basically makes sense. It essentially handles the Lua iterator protocol, and uses a function to condense the possibly-multiple return values into a single result.

Anyway, the basic picture of what I'm going for with scan_tokens() is to make that into an iterator that uses the source code as the invariant state, and packs the start, current, and line values into a table in the control variable. Then the methods translate into functions that take arguments of source and the control variable.

Okay, that all went smoothly, except for right at the end: The scanToken() method reaches out to the static error() method on the core Lox class, and that... It makes me feel... I don't like doing that...

Last time around, I passed the whole interpreter to the Scanner class I wrote, so it could call the method. And this is vaguely annoying to do with the way I wrote this, because I guess I'd need to change the state variable...

Okay, updating the state variable wasn't too hard, and now there's just two things I want to do currently:

• Wrap the various bits where I construct a control variable in functions, so I know where to update if I add more fields to the control variable.
• Make the Lox class that the state variable needs an instance of... exist.

But I can't work on those right now, because it is late.

Good night.