[Oil-dev] Rewrite of the entire word evaluation pipeline

[Andy Chu]

In the last blog post and one of the last e-mails, I mentioned a globbing
problem exposed by gold-test.sh.  It reminded me that the word evaluation
code was messy -- it didn't quite express the "problem domain".

I spent the last 4 days or so rewriting it, and I'm very pleased with the
result.  Here is the commit:

https://github.com/oilshell/oil/commit/c5af69702bb784953d12f7d08e647859cfbd6392

The diff is large and not worth reading, but this tiny new ASDL schema is:

https://github.com/oilshell/oil/blob/master/core/runtime.asdl

Surprisingly (to me), I used ASDL in the runtime as well.  I had known that
algebraic data types are a good model for representing ASTs, but here's
some first hand experience that they're good for specifying the runtime
semantics of a language as well.

There used to be a hand-written Python Value type, but now there are the
concepts of part_value, fragment, arg_value, and (word) value.  Word
evaluation is one of the hairiest parts of shell, if not THE hairiest, and
these types really clarify what's happening.

Previous test stats: 532 total / 311 osh pass / 163 osh fail
Stats after this comment: 556 total / 329 osh pass / 169 osh fail

-----

Now there's a solid skeleton, so there's one obvious place to fix every
bug, and bug fixes don't cause new bugs.  Evidence of that is these
subsequent quick bug fixes:

https://github.com/oilshell/oil/commit/6e640ddd2ee13a3a54e3c006d9bcf3320d3a084c

https://github.com/oilshell/oil/commit/e607cda09313096a8a11a141d88473797c17c4ce

https://github.com/oilshell/oil/commit/e38b5bb91d75e4404718a5c735526ac670767dc5

https://github.com/oilshell/oil/commit/751345c9d6d571c8011eb5882e448fc550fbe003

Stats After bug fixes: 558 total / 340 osh pass / 160 osh fail

This new architecture could use a blog post, but I'm not sure if I'll have
time for it.  I do plan to blog about keeping track of progress with the
spec test stats... I find it to be pretty motivating.  Word splitting all
works now:

http://www.oilshell.org/git-branch/master/751345c9/andy-home/spec/word-split.html

Full tests:

http://www.oilshell.org/git-branch/master/751345c9/andy-home/spec/RESULTS.html

Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.oilshell.org/pipermail/oil-dev-oilshell.org/attachments/20170320/faf0fecd/attachment.htm>