I have created an installation profile for most of my sites, which I use when creating new platforms on my Barracuda-Octopus-Aegir setup. When creating new platforms I only utilize the make-files, which pretty much leaves the installation profile untested - until a second developer joins the project.
Therefore I looked into a way of getting my changes tested. All my sites are on github.com, and thus I could utilize Travis CI-integration. Turned out that it was not to difficult. I simply enabled Travis CI for the project and added the following .travis.yml to the root of my Drupal 7-installation profile.
php: - 5.3 mysql: database: drupal username: root encoding: utf8 before_script: - "mysql -e 'create database drupal;'" - pear channel-discover pear.drush.org - pear install drush/drush - phpenv rehash - drush make vih_dk.build ../vih.dk - cd ../vih.dk - sudo chmod -R 777 sites/all/modules - drush si vih_dk --db-url=mysql://root:@127.0.0.1/drupal --account-name=admin --account-pass=VIH@2012 --firstname.lastname@example.org --site-name="vih.dk" --yes - drush cc all --yes script: - test -d profiles/vih_dk - test -d profiles/vih_dk/modules/contrib - test -d profiles/vih_dk/modules/vih - test -d profiles/vih_dk/themes
To see the results, see the build status on Travis, and you can even implement a little image on you website telling whether your installation profile builds correctly.
E.g. the build status for the installation profile for vih.dk:
More to test
There is probably more that you could test automatically. What are you including in your tests?