py.test/pylib 1.3.2: API and reporting refinements, many fixes

The pylib/py.test 1.3.2 release brings many bug fixes and some new features. It was refined for and tested against the recently released Python2.7 and remains compatibile to the usual armada of interpreters (Python2.4 through to Python3.1.2, Jython and PyPy). Note that for using distributed testing features you'll need to upgrade to the jointly released pytest-xdist-1.4 because of some internal refactorings.

See http://pytest.org for general documentation and below for a detailed CHANGELOG.

cheers & particular thanks to Benjamin Peterson, Ronny Pfannschmidt and all issue and patch contributors,

holger krekel

Changes between 1.3.1 and 1.3.2

New features

  • fix issue103: introduce py.test.raises as context manager, examples:

    with py.test.raises(ZeroDivisionError):
        x = 0
        1 / x
    
    with py.test.raises(RuntimeError) as excinfo:
        call_something()
    
    # you may do extra checks on excinfo.value|type|traceback here
    

    (thanks Ronny Pfannschmidt)

  • Funcarg factories can now dynamically apply a marker to a test invocation. This is for example useful if a factory provides parameters to a test which are expected-to-fail:

    def pytest_funcarg__arg(request):
        request.applymarker(py.test.mark.xfail(reason="flaky config"))
        ...
    
    def test_function(arg):
        ...
    
  • improved error reporting on collection and import errors. This makes use of a more general mechanism, namely that for custom test item/collect nodes node.repr_failure(excinfo) is now uniformly called so that you can override it to return a string error representation of your choice which is going to be reported as a (red) string.

  • introduce '--junitprefix=STR' option to prepend a prefix to all reports in the junitxml file.

Bug fixes / Maintenance

  • make tests and the pytest_recwarn plugin in particular fully compatible to Python2.7 (if you use the recwarn funcarg warnings will be enabled so that you can properly check for their existence in a cross-python manner).
  • refine --pdb: ignore xfailed tests, unify its TB-reporting and don't display failures again at the end.
  • fix assertion interpretation with the ** operator (thanks Benjamin Peterson)
  • fix issue105 assignment on the same line as a failing assertion (thanks Benjamin Peterson)
  • fix issue104 proper escaping for test names in junitxml plugin (thanks anonymous)
  • fix issue57 -f|--looponfail to work with xpassing tests (thanks Ronny)
  • fix issue92 collectonly reporter and --pastebin (thanks Benjamin Peterson)
  • fix py.code.compile(source) to generate unique filenames
  • fix assertion re-interp problems on PyPy, by defering code compilation to the (overridable) Frame.eval class. (thanks Amaury Forgeot)
  • fix py.path.local.pyimport() to work with directories
  • streamline py.path.local.mkdtemp implementation and usage
  • don't print empty lines when showing junitxml-filename
  • add optional boolean ignore_errors parameter to py.path.local.remove
  • fix terminal writing on win32/python2.4
  • py.process.cmdexec() now tries harder to return properly encoded unicode objects on all python versions
  • install plain py.test/py.which scripts also for Jython, this helps to get canonical script paths in virtualenv situations
  • make path.bestrelpath(path) return ".", note that when calling X.bestrelpath the assumption is that X is a directory.
  • make initial conftest discovery ignore "--" prefixed arguments
  • fix resultlog plugin when used in an multicpu/multihost xdist situation (thanks Jakub Gustak)
  • perform distributed testing related reporting in the xdist-plugin rather than having dist-related code in the generic py.test distribution
  • fix homedir detection on Windows
  • ship distribute_setup.py version 0.6.13

Changes between 1.3.0 and 1.3.1

New features

  • issue91: introduce new py.test.xfail(reason) helper to imperatively mark a test as expected to fail. Can be used from within setup and test functions. This is useful especially for parametrized tests when certain configurations are expected-to-fail. In this case the declarative approach with the @py.test.mark.xfail cannot be used as it would mark all configurations as xfail.

  • issue102: introduce new --maxfail=NUM option to stop test runs after NUM failures. This is a generalization of the '-x' or '--exitfirst' option which is now equivalent to '--maxfail=1'. Both '-x' and '--maxfail' will now also print a line near the end indicating the Interruption.

  • issue89: allow py.test.mark decorators to be used on classes (class decorators were introduced with python2.6) and also allow to have multiple markers applied at class/module level by specifying a list.

  • improve and refine letter reporting in the progress bar: . pass f failed test s skipped tests (reminder: use for dependency/platform mismatch only) x xfailed test (test that was expected to fail) X xpassed test (test that was expected to fail but passed)

    You can use any combination of 'fsxX' with the '-r' extended reporting option. The xfail/xpass results will show up as skipped tests in the junitxml output - which also fixes issue99.

  • make py.test.cmdline.main() return the exitstatus instead of raising SystemExit and also allow it to be called multiple times. This of course requires that your application and tests are properly teared down and don't have global state.

Fixes / Maintenance

  • improved traceback presentation: - improved and unified reporting for "--tb=short" option - Errors during test module imports are much shorter, (using --tb=short style) - raises shows shorter more relevant tracebacks - --fulltrace now more systematically makes traces longer / inhibits cutting
  • improve support for raises and other dynamically compiled code by manipulating python's linecache.cache instead of the previous rather hacky way of creating custom code objects. This makes it seemlessly work on Jython and PyPy where it previously didn't.
  • fix issue96: make capturing more resilient against Control-C interruptions (involved somewhat substantial refactoring to the underlying capturing functionality to avoid race conditions).
  • fix chaining of conditional skipif/xfail decorators - so it works now as expected to use multiple @py.test.mark.skipif(condition) decorators, including specific reporting which of the conditions lead to skipping.
  • fix issue95: late-import zlib so that it's not required for general py.test startup.
  • fix issue94: make reporting more robust against bogus source code (and internally be more careful when presenting unexpected byte sequences)

Changes between 1.2.1 and 1.3.0

Changes between 1.2.1 and 1.2.0

Changes between 1.2 and 1.1.1

Changes between 1.1.1 and 1.1.0

Changes between 1.1.0 and 1.0.2

Changes between 1.0.1 and 1.0.2

Changes between 1.0.0 and 1.0.1

Changes between 1.0.0b9 and 1.0.0

Changes between 1.0.0b8 and 1.0.0b9

Changes between 1.0.0b7 and 1.0.0b8

Changes between 1.0.0b3 and 1.0.0b7

Changes between 1.0.0b1 and 1.0.0b3

Changes between 0.9.2 and 1.0.0b1

XXX lots of things missing here XXX

Changes between 0.9.1 and 0.9.2

Changes between 0.9.0 and 0.9.1

This is a fairly complete list of changes between 0.9 and 0.9.1, which can serve as a reference for developers.