A Framework for Testing Concurrent Programs

http://www.cs.rice.edu/~mgricken/research/concutest/intro.shtml

Incremental, test-driven development is sweeping the software industry, supplanting more monolithic approaches to software development. Test-driven approaches to software development like Extreme Programming elevate testing from an ancillary activity to an integral part of the programming process. Tests for a program unit are written before the code, driving the design of the program and catching bugs early in its development. These tests form an integral part of the program code base, facilitating program reliability, continual integration, and refactoring.

In our recent experience developing production programs in Java, unit testing has proven effective in assuring the reliability of code with a single thread of control. Open source testing frameworks like JUnit work well in this context. Unfortunately, we have found unit testing is much less effective at assuring the reliabilty of code with multiple threads of control, often simply because the the testing framework silently ignores failures in auxiliary threads. Since thread scheduling is non-deterministic, a unit test can succeed on one run and fail on the next. Moreover, the scheduling characteristics of the test platform can skew test results, hiding synchronization bugs until the program is run on a different platform.

Java libraries and user programs furthermore frequently make assumptions about the threading context in which they execute, but these assumptions are rarely enforced by the actual code and typically only appear in program documentation or white papers, if at all.

To make development of concurrent programs easier, we are developing:

1. ConcJUnit: An extension of the JUnit framework, actively supporting the developer by treating tests that could silently ignore failures in auxiliary threads as test errors;
2. Thread Checker: A lightweight annotation language, which can be used to specify and check the threading invariants of both existing and new code;
3. Schedule-Based Execution: A testing framework that can execute unit tests according to a specified set of recorded or generated schedules, elevating the unit testing of concurrent programs to a rigorous, deterministic process.

There is more information available about the components on their respective pages:

• ConcJUnit
• Thread Checker
• Schedule-Based Execution

Supplemental material

• A Framework for Testing Concurrent Programs (PhD Thesis, view embedded):
  MathiasRicken's PhD thesis, April 19, 2011.
• A Framework for Testing Concurrent Programs (PhD Thesis Defense Presentation, PowerPoint, PDF, view embedded):
  MathiasRicken's presentation from his PhD thesis defense, January 10, 2011.
• A Framework for Testing Concurrent Programs (PhD Thesis Proposal Presentation, PowerPoint, PDF, view embedded):
  MathiasRicken's presentation from his PhD thesis proposal, December 2, 2010.
• Testing Concurrent Programs, A 7-Minute Jargon-Free Introduction (Presentation, PowerPoint, view embedded):
  Mathias Ricken's presentation to the Thesis Writing Seminar of the School of Engineering at Rice University, February 25, 2010.
• Test-First Java Concurrency for the Classroom (Presentation, PowerPoint, view embedded):
  Mathias Ricken's presentation at SIGCSE 2010, March 12, 2010.).
• Test-First Java Concurrency for the Classroom (Paper, PDF, view embedded, assignment):
  Paper to be published in the proceedings of the 41st Technical Symposium on Computer Science Education (SIGCSE 2010).
• ConcJUnit: Unit Testing for Concurrent Programs (Presentation, PowerPoint, view embedded):
  Mathias Ricken's presentation at the 7th International Conference on the Principles and Practice of Programming in Java (PPPJ 2009), August 24, 2009.
• ConcJUnit: Unit Testing for Concurrent Programs (Presentation, PowerPoint, view embedded):
  Mathias Ricken's presentation for the COMP 600 seminar, August 24, 2009.
• ConcJUnit: Unit Testing for Concurrent Programs (Paper, PDF, view embedded):
  Paper published in the proceedings of the 7th International Conference on the Principles and Practice of Programming in Java (PPPJ 2009).
• Practical Tools for Testing Concurrent Programs (Poster, PDF, view embedded):
  created for 2006 Corporate Affiliates Meeting at the Computer Science Department of Rice University to summarise our efforts.
• Unit Testing for Concurrent Programs (Poster, PDF, view embedded):
  created for 2005 Corporate Affiliates Meeting at the Computer Science Department of Rice University to summarise our efforts.
• Unit Testing for Concurrent Programs (Poster, PDF, view embedded): older version created for 2004 Corporate Affiliates Meeting at the Computer Science Department of Rice University to summarise our efforts.
• Unit Testing for Concurrent Programs (Poster Abstract, PDF, view embedded):
  abstract for 2004 Corporate Affiliates Meeting at the Computer Science Department of Rice University to summarise our efforts.
• A Framework for Testing Concurrent Programs (MS Defense Presentation, PowerPoint, view embedded):
  MathiasRicken's presentation from his Master of Science defense, June 14, 2007.
• A Framework for Testing Concurrent Programs (MS Thesis, PDF, view embedded):
  Mathias Ricken's Master of Science thesis
• A Framework for Testing Concurrent Programs (Presentation, PowerPoint, view embedded):
  MathiasRicken's presentation for the COMP 600 seminar, August 27, 2007.
• Testing Concurrent Programs (Presentation, PowerPoint, view embedded):
  Mathias Ricken's presentation for the Rice Computer Science Club, October 4, 2007.
  • Programming Examples from the presentation.
• Java Annotations for Invariant Specification (Presentation, PowerPoint):
  Mathias Ricken's presentation for the COMP 617 seminar, September 22, 2008.
• Invariant Specification and Multi-Staging using Java Annotations (Poster, PDF, view embedded):
  created for 2008 Corporate Affiliates Meeting at the Computer Science Department of Rice University.
• Direct access to the download directory