Skip to end of metadata
Go to start of metadata

Create and/or update QA test plans for the release

For each significant new unit of functionality that has been added to this release:

If any existing functionality has been changed in this release, in a way that materially affects customer-facing behavior, make sure to update the relevant QA test plans accordingly.

Set up a testing tasks page for the release

Create a testing tasks page for the release. This page links to all of the QA test plans relevant to that release.

The entire list of QA test plans can be found on the QA Test plans - Listing page. On that page, you can also find a more or less complete list of what to test for major and minor releases. (In general, more QA tests need to be performed for major releases than for minor releases.)

As representative examples, you might look at:

Confirm the end of development on the release

When development is scheduled to end on a release, the Project or Release Manager (who may be CollectionSpace's QA Lead or some other designated person) will confirm with the developers working on each of CollectionSpace's layers that they have:

  • Completed what was planned for the release; or
  • Otherwise arranged for any unimplemented functionality/remaining bugs to be pushed to the next release.

Create release branches

Once development on the release has been confirmed to be completed, the Project or Release Manager - or the developers for each layer, if specifically requested by one of these managers - creates a release branch in each of CollectionSpace's layers.  See Creating Release Branches for step-by-step instructions for branching for a release, including the naming conventions used for release branches.

These release branches are then frozen to any commits, except for:

  • Fixes to Blocker or Critical bugs identified during QA testing, and designated by the Project Manager as needing to be fixed during that release; or
  • As otherwise specifically permitted by the Project or Release Manager.

Note that each bug fixed will likely require some additional QA testing: first to verify that the bug has been fixed, and secondly, to ensure that no regression has occurred as a result.

The master (aka trunk) branch is still open to commits during this time. However, it is expected that many developers will be participating in QA testing and/or fixing bugs encountered during the QA testing period, rather than working on the master branch.

Update the UI landing page to use the lastest version number

As of release v4.4, the landing page included just one file (src/main/webapp/defaults/js/Footer.js) that included the version number.  Edit that file and replace the old version number with the new/next version number.

Set up the CollectionSpace server for QA testing

Continuous Integration setup using Bamboo

Introduction to Bamboo

CollectionSpace uses Atlassian Bamboo, a server-based continuous integration application, to automate the build process on the project's QA server. This simplifies the tasks of setting up, maintaining and updating the QA server during the entire QA process. For more details, see Continuous Integration Service.

The initial build for QA requires each layer to have branched to the current Git branch. This can be verified by looking on each layer's github account for that branch version. For instance, the v4.4 branch for the UI layer can be found on github here: https://github.com/collectionspace/ui/tree/v4.4

Bamboo uses build plans to organize and configure how a complex project should be built, and the Bamboo instance for the QA server is broken into several build plans that span across all the layers. Each build plan represents a different aspect of the build process and all of them must be "pointed" to the same version branch in the CollectionSpace source code repository (depending on which layer is involved in that build plan, of course).

Here is a list of the current build plans (https://qa.collectionspace.org/bamboo/browse/NIGHTLY)

  • 01  - Service Layer - Build (build the entire service layer)
  • 02  - Tomcat - Stop Servers (Stops the Tomcat server. Necessary for deploying and reinitializing the database. No git access needed)
  • 03  - Service Layer - Deploy (Deploys the built war files)
  • 04  - Service Layer - Database (Reinitializes the database. This wipes out all saved data, recreates all tables (based on schema and Nuxeo), and restores all permissions to default values)
  • 05. - (deprecated)
  • 06  - Tomcat - Start Servers (Starts up the Tomcat servers. No git access needed)
  • 07  - Service Layer - Test (Runs through the Service layer's unit tests)
  • 07b - Tomcat - Stop Servers (Stops the Tomcat server.  Necessary to build and deploy the App layer)
  • 08a - App Layer - Build and Deploy (Builds and deploys the Application layer)
  • 08aa- Tomcat - Start Servers (Start Tomcat server so we can run the App layer tests)
  • 08b - App Layer - Test (Runs through the Application layer's unit tests.)
  • 08c - UI Layer (default) (Builds and deploys the UI layer with minification)
  • 08d - Initialize Authorities - As of v4.4, not necessary. (Populates the CollectionSpace instance with default vocabularies)
  • 08e - Create Reports (Populates the CollectionSpace instance with the default reports.)
  • 09 - Populate Data - Off by default (Populates the CollectionSpace instance with sample data.  This is useful for stepping through the scripted test plans.)
  • 10. - Tarball and Distribute (Creates a tarball for post-QA testing. This may not be current anymore.

Bamboo lists the build plans in alphabetical order.  Therefore, we've prefixed the plan names with numbers to show the order in which they are executed.  Build plan "08b - App Layer - Test" should be configured to have either "08c - UI Layer (default)".  Each build plan can have a dependency and a dependent -which determines the order in which the plans are executed.

Edit the release branch name

To change the name of the release branch from which the QA server is built (for instance, to change it from a v4.0 to a v4.1 branch, when entering QA testing for v4.1):

  • Log into Bamboo at http://qa.collectionspace.org/bamboo
  • Click the Administration tab at top
  • Click Global Variables in the sidebar at left
  • Change the value of currentReleaseBranch to the name of the release branch

Note that, by default, all of the build plans use the value of Bamboo's global variable, currentReleaseBranch, to identify the branch used for the build

If, during a QA test period or otherwise, you'd like to change the branch used to build CollectionSpace's Application, Services, or UI layers - individually or across two or three layers - you don't need to visit the configuration for every build plan and make changes in each. Rather, you can change the branch to be built from globally for any layer, via https://qa.collectionspace.org/bamboo/admin/configureGlobalVariables!default.action

Rebuild the QA server

The QA server should be rebuilt:

  • Just before QA testing starts.
  • Whenever a fix is made to a Blocker or Critical bug, that requires additional QA testing.
  • Under rare circumstances, if the server must be rebuilt in order to fix critical problems that interfere with QA testing.

The QA server is typically rebuilt outside testing hours. (Since testers may be working in a variety of different time zones, and may even be international, these hours should be arranged and announced in advance.)  Under exigent circumstances, and ideally with at least some advance notification, it can be rebuilt during testing hours.

To rebuild the QA server:

Some plans should remain off/deactivated until needed. For instance, plan #10 doesn't need to be run until the QA process for the entire release is complete. Some plans don't need to be run more than once. This varies depending on circumstances, but generally steps 08(d-e) only need to be run once at the very beginning of QA. In this case build plans 8(d-e) could be temporarily disabled after the QA server is built for a QA period.

Build plan disabling/enabling

Throughout the QA process it may be necessary to update one or more of the layers depending on what has been recently committed. It is possible to run the build plans out of order without starting from plan #1. Remember, the build plans are linked and will start the next successive build plan once one completes successfully. It is possible to temporarily deactivate a build plan in order to prevent it from running because its previous build plan completed. For example, say you want to rebuild ONLY the Services layer.  You could do this by disabling plan 07b -which prevents plans 07b - 08e from also running once you kick off a build of 01 to rebuild the Services layer.

A build plan can be disabled/re-enabled in the following steps

  • Click the pencil icon on the far right of the build plan you wish to disable
  • Click the Actions pop-up menu near the top on the far right
  • Click the Disable/Enable menu option listed

QA Testing

Once the server is ready, an announcement is sent to the project's "Talk" and "Announce" email lists saying that QA testing can begin. The email should include a link to the testing tasks page and the server to test on (usually http://qa.collectionspace.org/). It is also a good idea to mention what functionality is new/affect, and ask testers to prioritize these tasks (eg. for in version v4.x, testing of  Roles and Permissions and Vocabulary related things is priority)

Testers can then begin by assigning themselves one or more of the testing tasks set up for the release.

If any blockers are found during QA that should be fixed. The fix for the blocker should be committed to release branch and potentially to the master branch as well. Once this is done, the testers should be notified of downtime of the QA server, and the steps described above should be carried out to update the server (only the sources of the layers that were involved in the fix need to be updated). Once the QA server is back up and running, the testers should be notified and testing can start again. Any area potentially affected by the fix should be at least superficially, and preferably thoroughly, tested again.

Automated testing

In the QA testing phase, the automated tests should be run too. It's a good idea to do this early in the testing phase, since some tests might be outdated/broken and requires fixes. Optionally, during the development phase, the tests can be run against nightly to check whether they are broken so they can be fixed before QA starts. In any case, the automated tests should also be run during QA - instructions on how to do so can be found here: Running Automated QA Tests

Tag the release

Once testing is done and all blockers are fixed, the layers are ready to be tagged. The Project or Release Manager - or the developers for each layer, if specifically requested by one of these managers - will create a tag in each layer.

See Tagging Releases in Git for step-by-step instructions for tagging a release, including naming conventions used.

Create the tarball and publish on FTP server

As soon as all the layers have tagged the release, we are ready to create a "tarball" of a CollectionSpace server folder for that release. Use the CollectionSpace Atlassian Bamboo build plan on the QA (https://qa.collectionspace.org:8443) server -see https://qa.collectionspace.org:8443/browse/FB-TR1CDT.  This creates a tar file on the qa.collectionspace.org server here: /var/www/html/builds.  You need to copy this tar file to the CollectionSpace FTP server on nightly.collectionspace.org.  For example, after you've logged into the qa.collectionspace.org server, use these commands (used for creating the v4.4 release) as an example/guide:

Test the tarball and update the install instructions for Linux distributions

After the tarballs have been created and made available on FTP (placed at /srv/ftp/pub/collectionspace/releases on nightly.collectionspace.org), they need to be tested in our supported environments.

<TBD>

Update the wiki and more with the new release

The implementer documentation, and a release page for the new release should be set up on the wiki. The procedure for this can be found here: End of Release Check List, towards the bottom of the page. "Documentation rotation" is the process of cycling out the the unreleased documentation into the current release documentation.

If this is a major release:

Update the demo server
  1. If possible, do a fresh install of OS on the slice
  2. Set up a "Temporarily Down for Upgrade" page on the server
  3. Follow the install instructions to install the release
  4. Check that the install is successful
  5. Replace the temporarily down page with a redirect to collectionspace
Release is announced to list

See Also

Documentation rotation

End of Release Check List