[Oil-dev] Running Oil tests

[Andy Chu]

OK, I just committed a bunch of test suite fixes/enhancements to the master
branch.  (Please git pull.)

Good news is that I made everything nicer, it doesn't depend on anything in
my ~/bin dir -- it's all checked in now!

On my machine "label andy-home" everything passes.  In other words, this
works:

https://github.com/oilshell/oil/wiki/Contributing

http://www.oilshell.org/git-branch/master/ab8c0876/andy-home/spec/RESULTS.html

Green or yellow means passing.  Version information is at the bottom of the
page, so this is Ubuntu 14.04.

But the big surprise was how shell-version-dependent the tests are.  I
guess I hit a lot of corner cases, so running it on a Linode with Debian
Jessie produces different results:

http://www.oilshell.org/git-branch/master/ab8c0876/andy-linode/spec/RESULTS.html

See all the red lines.  Only one difference is related to osh!  The rest
are because of different versions of bash, mksh, and zsh!  I thought those
shells were pretty stable, but apparently not.

For example, these test failures is because bash 4.3 implements local -n,
but bash 4.2 doesn't:

http://www.oilshell.org/git-branch/master/ab8c0876/andy-linode/spec/var-ref.html

I don't really care about local -n, named references, so maybe we can just
disable those.

This error seems to be because bash 4.3 exits error code 2 with "exit
invalid", but bash 4.2 exits error code 255.

http://www.oilshell.org/git-branch/master/ab8c0876/andy-linode/spec/builtins.html#details-2-bash

http://www.oilshell.org/git-branch/master/ab8c0876/andy-linode/spec/builtins.test.html#L20

This can be verified without the test framework:

$ bash -c 'exit invalid'; echo $?
bash: line 0: exit: invalid: numeric argument required
2

$ bash -c 'exit invalid'; echo $?
bash: line 0: exit: invalid: numeric argument required
255

I guess users probably don't care too much about these differences.  But I
was a little surprised that they came up.

But I do want to everyone else to have the ability to get an all
green/yellow test run on their machine.  If there is any red, it's
distracting.

Hm I'll have to think about what to do about this... it depends how much it
causes people a problem.

Andy





On Fri, Mar 3, 2017 at 12:42 AM, Andy Chu <andychup at gmail.com> wrote:

> +oil-dev so others can benefit
>
> On Thu, Mar 2, 2017 at 11:33 AM, Andrew Owen <yumaikas94 at gmail.com> wrote:
>
>>
>>
>> I do run into errors with the spec tests, which currently do not run on
>> OSX, complaining about an unbound BUSYBOX_ASH varaiable, and in wild.sh
>> complains about an illegal option to ln.
>>
>> I suspect some of those might be related to out of date and/or different
>> versions of bash on osx. If running on osx is a current goal of yours, I'll
>> open up issues for them, otherwise I'll just test it from an Ubunutu box as
>> I have time.
>>
>
>
> I made some changes to the test infrastructure, and documented it here:
>
> https://github.com/oilshell/oil/wiki/Contributing
>
> As mentioned I do want to get it working on OS X, but if you can try
> Ubuntu first that would help.  The main issue is all the other shells, and
> the old bash on OS X (bash 3 from 2007).
>
> Everything up to "./spec.sh smoke" should work -- let me know if it
> doesn't.  I have to fix that one tomorrow.  It depends on an "argv" script
> that is only on my machine!
>
> Andy
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.oilshell.org/private.cgi/oil-dev-oilshell.org/attachments/20170303/ab8b2cb7/attachment.htm>