[flashrom] RFC: Unit testing framework

[David Hendricks]

Hi,
Once upon a time, testing flashrom was simple: Read, erase, write, verify,
done.

Over the past few months, we've introduced some very important features to
Flashrom. Unfortunately, testing is no longer quite so simple. Partial
writes require that we test various patterns (and we do with a newly
introduced torture script), and we do that to some extend with the torture
test that we recently checked in. Embedded controllers introduce a whole
other world of pain. Long story short, a better testing strategy is needed.

To get the ball rolling, I have a work-in-progress patch for the
Chromium.org branch of Flashrom right now to introduce a unit testing
framework and would appreciate thoughts / comments:
http://codereview.chromium.org/5136001 . You can download the unified diff
from there (as to avoid duplication here). It's written in shell script
(tested on dash and bash) since I felt that's the lowest common denominator
and the lowest barrier-to-entry for folks wanting to add/modify the
tests. Feel free to post comments in this thread, since the code review will
be closed once the code is checked in.

The idea is simple: One master script does generic setup, backs up the
firmware image, runs unit tests, and unconditionally restores the firmware
image at the end. The user sets a few (hopefully no more than two or three)
environment variables and runs the master script, specifying desired unit
tests as arguments on the command line. Each unit test should be
independent, and should have its own documentation in case it needs special
environment variables or something. Due to the variety of tests which may be
performed, it's up to the user to decide which tests to run on their
hardware.

If you're curious to try the patch, here is a simple invocation that will
test partial writes for an x86 BIOS chip: ./do_tests.sh
partial_writes_x86_bios.sh

I've attached some sample console output for a test run that does partial
writes for x86 BIOS chip and some write protection commands. In this
example, I had to pass in some env variables and specify multiple tests as
arguments. The invocation: FLASHROM="../flashrom" FLASHROM_PARAM="-p
internal:bus=spi" sh do_tests.sh partial_writes_x86_bios.sh chip_size.sh
wp-toggle.sh wp-range.sh

If I were to test some EC stuff, I'd need to add additional parameters since
my EC test script requires a special layout file and an secondary EC
firmware image: LAYOUT_FILE=/tmp/ec_layout.txt
ALT_EC_IMAGE=/tmp/alt_ec_image.bin FLASHROM="../flashrom"
 FLASHROM_PARAM="-p internal:bus=lpc" ./do_tests.sh partial_writes_ec.sh

-- 
David Hendricks (dhendrix)
Systems Software Engineer, Google Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.flashrom.org/pipermail/flashrom/attachments/20101122/2242d445/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: console_output.log
Type: text/x-log
Size: 8785 bytes
Desc: not available
URL: <http://www.flashrom.org/pipermail/flashrom/attachments/20101122/2242d445/attachment.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: partial_writes_x86_bios.sh.log
Type: text/x-log
Size: 972 bytes
Desc: not available
URL: <http://www.flashrom.org/pipermail/flashrom/attachments/20101122/2242d445/attachment-0001.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wp-range.sh.log
Type: text/x-log
Size: 131 bytes
Desc: not available
URL: <http://www.flashrom.org/pipermail/flashrom/attachments/20101122/2242d445/attachment-0002.log>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wp-toggle.sh.log
Type: text/x-log
Size: 51 bytes
Desc: not available
URL: <http://www.flashrom.org/pipermail/flashrom/attachments/20101122/2242d445/attachment-0003.log>