---
title: "Software Carpentry: SC Test; does your software do what you meant?"
description: "The SC Test competition invited proposals for better testing tools."
slug: sc-test
date: 2016-10-06T18:51:42+01:00
draft: false
type: post
tags:
- Software Carpentry
- Web archaeology
- Testing
- Technology
series: swc-archaeology
---

> "The single most important rule of testing is to **do it**."  
> --- *Brian Kernighan and Rob Pike, The Practice of Programming (quote taken from [SC Test page][SC Test]*

[SC Test]: https://web.archive.org/web/20071014042742/http://software-carpentry.com/sc_test/index.html

One of the trickiest aspects of developing software
is making sure that it actually does what it's supposed to.
Sometimes failures are obvious:
you get completely unreasonable output
or even (shock!) a comprehensible error message. 

But failures are often more subtle.
Would you notice if your result was out by a few percent,
or consistently ignored the first row of your input data?

The solution to this is testing:
take some simple example input with a *known* output,
run the code and compare the actual output with the expected one.
Implement a new feature, test and repeat.
Sounds easy, doesn't it?

But then you implement a new bit of code.
You test it and everything seems to work fine,
except that your new feature required changes to existing code
and those changes broke something else.
So in fact you need to test *everything*,
and do it *every time you make a change*.
Further than that,
you probably want to test
that all your separate bits of code work together properly (*integration testing*)
as well as testing the individual bits separately (*unit testing*).
In fact, splitting your tests up like that is a good way of holding on to your sanity. 

This is actually a lot less scary than it sounds,
because there are plenty of tools now to automate that testing:
you just type a simple `test` command and everything is verified.
There are even tools that enable you to have tests run automatically
when you check the code into version control,
and even automatically deploy code that passes the tests,
a process known as *continuous integration* or CI.

The big problems with testing are that
it's tedious, your code seems to work without it
and *no-one tells you off for not doing it*.

At the time when the Software Carpentry competition was being run,
the idea of testing wasn't new,
but the tools to help were in their infancy.

> "Existing tools are obscure, hard to use, expensive, don't actually provide much help, or all three."

The [SC Test category][SC Test] asked entrants
"to design a tool, or set of tools, which will help programmers
construct and maintain black box and glass box tests of software
components at all levels, including functions, modules, and classes,
and whole programs."

The SC Test category is interesting
in that the competition administrators clearly found it difficult
to specify what they wanted to see in an entry.
In fact,
the whole category was reopened
with a refined set of rules and expectations.

Ultimately, it's difficult to tell whether this category
made a significant difference.
Where the tools to write tests used to be very sparse and difficult to use
they are now many and several options exist for most programming languages.
With this proliferation,
several tried-and-tested methodologies have emerged
which are consistent across many different tools,
so while things still aren't perfect they are much better.

In recent years there has been a culture shift
in the wider software development community towards
both testing in general and test-first development,
where the tests for a new feature are written *first*,
and then the implementation is coded incrementally
until all tests pass.

The current challenge is to transfer this culture shift
to the academic research community!
