Quality Assurance

Test as a Service

Nard offer a regression testing cloud service to fulfill product requirements. As part of your nightly builds the final SD-card image can be uploaded to our TaaS where an automated audit will take place. If all tests pass one can be pretty sure the binary image work as expected.

The audit is accomplished on real hardware. A Raspberry Compute Module is connected via custom built electronics to a dedicated server for thorough monitoring and control during the tests. Uploaded SD-card images fully replace the Raspberry flash content and thus the system can always be recovered should it need to. Don't worry if you blunder and brick the device – Nard TaaS will heal automatically. The service is fully integrated into the build system and issued with a simple command:

$ make skeleton
$ make test
Nard TaaS is online and unoccupied; uploading image to http://www ...
######################################################### 79.6%

--------- Unpacking ------------
MD5 of audit.tar.gz: 9e4b9ac5ae65ce839918c5d7856c3073
Unpacking finished

--------- Flashing ------------
Uploading image to Compute Module...
Success, resetting Compute Module into normal operation...

--------- Boot console (short) ------------
Uncompressing Linux... done, booting the kernel.
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 3.10.19 (rln@foo) (gcc version 4.4.5 ...
... snipp ...

--------- Tests summary ------------
Executing test-003-heartbeat-led.sh... OK
Executing test-005-ping.sh... OK
Executing test-009-node-name.sh... OK
Executing test-048-ssh-root-key.sh... OK
Executing test-051-ssh-host-key.sh... OK
Executing test-053-scp-file-transfer.sh... OK
Executing test-056-inetd-enable-many-ssh.sh... OK
Executing test-060-set-time-ntp.sh... OK
Executing test-065-syslog.sh... OK
Executing test-068-automount.sh... OK
Executing test-080-gdb-hello-world.sh... OK
Executing test-090-mount-usb-mem-stick.sh... OK
Executing test-170-sqlite.sh... OK
Executing test-190-openssl.sh... OK
Executing test-195-wget-https.sh... OK
Executing test-300-python.sh... OK
Executing test-310-i2c.sh... OK
Executing test-320-samba.sh... OK
Executing test-330-perl.sh... OK
Executing test-380-rrdtool.sh... OK
Executing test-800-sdcard-removal.sh... OK
Executing test-810-usb-hotplug.sh... OK
Audit success, all tests passed.

Had the audit failed make test would have exited with an error instead. This is perfect for Continuous Integration (CI) with Jenkins or even a daily scheduled cron job. It's a good habit to audit your upgrades before sending them away to any remotely located device.

Sequencing

Specifying what to test is done with a configuration that accompanies the product recipe. (Remember that recipes inherit from each other and "add up".) Here is a simplified example:

This will test that SSH, I²C and SD-card hotplugging works in your image. Any product building "on top" of the board raspi_x_revx will do those tests too. Here is complete list of what can be specified in taasSeqs:

heartbeat	The slowly flashing green LED, indicating watchdog is running
net	IPv4 and discoverable with make scan and broadcast ping
ssh	SSH and SCP login with the key generated during build
syslog	Sets clock with NTP and verify system log
automount	Mounts SD-card only when in use
sys	Misc basics, such as correct file ownerships
gdb	cross-compilig with flags for debugability of everything (incl. syslibs)
usb	Hotplug and mount a USB memorystick
sqlite	Creates a SQLite database
openssl	Creates an encrypted connection to a webserver
wget-https	Downloads from a webserver with TLS encryption
python2	Basic Python 2.7 language
i2c	I²C communication and related drivers
samba	Windows shares exporting and mounting
perl	Basic Perl language
minidlna	The MiniDLNA daemon (UPnP)
rrdtool	Data logging and statistics graphing
sdcard	Hotswap the SD-card

If you need widened tests for a custom product; contact the authors.