Awesome Python Testing¶
https://en.wikipedia.org/wiki/Software_testing
https://en.wikipedia.org/wiki/Python_(programming_language)
Contents
Background¶
Software Testing¶
Workflow Evolution¶
1. edit, edit, commit
2. edit, commit
3. todo, edit, commit
4. todo, edit, test, commit
5. todo, test, edit, test, commit
6. todo, test, edit, test, commit, tag
7. todo, branch, test, edit, test, commit, { tag, push, send patch }
8. todo, qnew, test, edit, test, commit, finish, { tag, push, send patch }
Python¶
https://pypi.python.org/pypi/unittest2 (python < 2.7)
https://pypi.python.org/pypi/mock (python < 3.3)
https://pypi.python.org/pypi/pathlib (python < 3.4)
IPython¶
def test_true(): assert True == True
%logstart log_input_to_here.py
%logstart -o log_input_and_output_to_here.py
%ed log_input_and_output_to_here.py
!nosetests ./log_input_and_output_to_here.py
!nosetests --help
!nosetests --ipdb # pip install ipdbplugin # nose-progressive
# Run a file in IPython's namespace, print timing, skip sys.exit
%run -i -t -e ./log_input_and_output_to_here.py
# Run a file in IPython's namespace, print timing, skip sys.exit, and pdb
%run -i -t -e -d ./log_input_and_output_to_here.py
# Add >>> and ... prompts (useful for recording doctests)
%doctest_mode
# List defined aliases
%alias
# Get help for an alias
%logstart?
%alias?
# List available aliases and commands
%<TAB>
# Run a statement through the Python line profiler
%prun test_true()
%prun (2**2)*2)
# Time a number of loops
%timeit test_true()
%timeit (lambda x: (x**2)*2)(10)
# Run tests (see ipython_nose)
%nose
Logging¶
Python Logging Primer¶
A primer on Python logging (and testing):
#### Logging examples
# test setup (fixtures)
x = 2
data = {'x': x}
# asserts get compiled out with python -O2
assert x == 2
# these always run
class TestTwo(unittest.TestCase):
def test_001_x_equals_2(self):
self.assertEqual(self, x, 2)
unittest.main()
if x != 2:
### Exceptions
err = ('...', x)
raise Exception(err)
raise ValueError(err)
### Logging
import logging
log = logging # global logger
log = __import__('logging') # global logger
log = logging.getLogger(__name__) # local logger (<__name__>.py)
log.debug(err)
log.info(err)
log.error(err)
log.exception(err)
log.critical(err)
log.fatal(err)
# see: supervisor.loggers.LevelsByName and .LevelsByDescription
TRAC = LevelsByName.TRAC = 5
BLAT = LevelsByName.BLAT = 3
log.log(err, TRAC)
logging.addLevelName(TRAC, 'TRAC')
logging.addLevelName(BLAT, 'BLAT')
log.setLevel(TRAC)
log.log(err, BLAT) # not printed because msg.level < log.level
log.log((err, 'string'), BLAT)
# run unittest.main() when run like:
# python ./filename.py
# but not when imported like:
# import filename
if __name__ == "__main__":
import sys
sys.exit(unittest.main())
See also:
logbook¶
Logbook is a drop-in replacement for the Python logging module with a number of simplifications and lots of additional handlers.
MultiProcessingHandler https://logbook.readthedocs.io/en/stable/api/queues.html#logbook.queues.MultiProcessingHandler
ThreadedWrapperHandler https://logbook.readthedocs.io/en/stable/api/queues.html#logbook.queues.ThreadedWrapperHandler
structlog w/ logbook https://github.com/hynek/structlog/issues/30#issuecomment-67754813
Logging Strategies¶
syslog¶
logging.handlers.SyslogHandler(logging.Handler) blocks
https://docs.python.org/2/library/logging.handlers.html#sysloghandler
http://www.somethinkodd.com/oddthinking/2010/04/29/how-python-logging-bit-me-part-1/
logbook ThreadedWrapperHandler, MultiProcessingHandler
rsyslog¶
“20.7 Interaction of rsyslog and journal” [journald] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-interaction_of_rsyslog_and_journal.htmlq
syslog-ng¶
sentry¶
graylog¶
Elasticsearch, MongoDB, Java 8+
logstash¶
ELK¶
ElasticSearch: storage, indexing, search
Logstash: logging adapters
Kibana: ElasticSearch Search UI
Logstash¶
Logstash GELF Plugin https://www.loggly.com/docs/rsyslog-tls-configuration/
Kibana¶
Test Runners¶
Nose¶
nosetests --help
nosetests --with-xunit
# python -m pdb --help
nosetests --pdb
nosetests --pdb-failures
nosetests --pdb-errors
# pip install nose_ipdb
nosetests --ipdb --ipdb-failure
# pip install nose-progressive
nosetests --with-progressive
nose-ipdb¶
nose-ipdb is a Nose plugin for running pdb debugger
with IPython (so tab completion, obj?
and obj??
all work.
nose-parameterized¶
nose-parameterized is a Nose plugin for separating test data from test code.
See also:
nose-progressive¶
nose-progressive is a Nose plugin which adds a progress bar, better/different configurable tracebacks, and editor shortcuts
ipython_nose¶
ipython_nose is a Nose plugin for running test_*
functions
within the current Jupyter Notebook.
!pip install -e git+https://github.com/taavi/ipython_nose#egg=ipython_nose
%load_ext ipython_nose
def test_func():
assert sum([1, 1]) == 2
%nose # discover and run "test_*" functions
py.test¶
py.test --pdb # --full-trace
pytest-sugar¶
pytest-sugar adds a progressbar
See also nose-progressive
Continuous Integration (CI) and Continuous Delivery (CD)¶
Buildbot¶
xUnit XML¶
Version Control¶
Patches, Tags, Branches, Merging¶
Debugging¶
Debugging: Console, CLI, Terminal, REPL, IPython¶
ipdb¶
import ipdb; ipdb.set_trace() # ipdb
nosetests --ipdb --ipdb-failure # --with-progressive # nose-progressive
pdbpp¶
pdb++ (pdbpp) is a “drop-in replacement for pdb” with additional debugging commands, tab completion, syntax highlighting, sticky mode,
When pdb++ is installed (in
sys.path
) pdb++ overloads pdb:
pip install pdbpp funcsigs # installs pdb.py
python -m pdb ./code.py
nosetests --pdb ./code.py
py.test --pdb ./code.py
Vim¶
Performance Instrumentation and Profiling¶
Web Frameworks¶
https://en.wikipedia.org/wiki/Web_application_framework
Plone¶
Django¶
Flask¶
Bottle¶
Bottle is a one file WSGI Python web framework.
pgs is written with bottle.py:
Web Applications¶
WebTest¶
WebTest is a tool for testing WSGI applications without running a webserver (e.g. Pyramid, Flask, Django, Bottle)
Splinter¶
multi webdrivers (chrome webdriver, firefox webdriver, phantomjs webdriver, zopetestbrowser, remote webdriver)
css and xpath selectors *
ghost.py¶
Webkit based scriptable web browser for python.
PyQT or PySide