Difference between revisions of "Testing Gramps"

From Gramps
Jump to: navigation, search
(unit tests in the main tree)
(test/RunAllTests.py)
(33 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
== testing of reports ==
 
== testing of reports ==
 
=== test/runtest.sh ===
 
=== test/runtest.sh ===
Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
+
 
 +
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/runtest.sh test/runtest.sh]''' - Report test for GRAMPS: Generate every report in every format. Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
  
 
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-runtest.sh found-by-runtest.sh]
 
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-runtest.sh found-by-runtest.sh]
Line 10: Line 11:
  
 
==test/impex.sh==
 
==test/impex.sh==
Import/export test for GRAMPS.
+
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/impex.sh test/impex.sh]''' - Import/export test for Gramps.
  
 
From the file header:
 
From the file header:
Line 25: Line 26:
 
== Unit testing ==
 
== Unit testing ==
 
=== test/RunAllTests.py ===
 
=== test/RunAllTests.py ===
Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
+
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/RunAllTests.py test/RunAllTests.py]''' - Testing framework for performing a variety of unttests for GRAMPS. Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
 +
 
 +
Starting with gramps40 branch, these tests include non-automated unit tests only. The automated unit tests are all under gramps/.
  
 
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-RunAllTests.py found-by-RunAllTests.py]
 
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-RunAllTests.py found-by-RunAllTests.py]
  
Two tests in GtkHandler testing code pop up the GRAMPS error dialog, but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...
+
GtkHandler testing code pops up the Gramps error dialog, but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...
  
One test currently fails, see {{bug|6940}}.
+
=== unit tests in the main tree ===
 +
python setup.py test
  
=== unit tests in the main tree ===
+
See [[Unit Test Quickstart]] for detailed running instructions.
See [[Unit Test Quickstart]] for running instructions.
 
  
 
{| {{prettytable}}
 
{| {{prettytable}}
Line 44: Line 47:
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
 
|3 tests
 
|3 tests
 +
|-
 +
|gramps/cli/test/user_test.py
 +
|bgcolor="#80ff80"| OK
 +
|8 tests (6 require mocking to run)
 
|-
 
|-
 
|gramps/gen/test/config_test.py
 
|gramps/gen/test/config_test.py
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
 
|1 test
 
|1 test
 +
|-
 +
|gramps/gen/test/constfunc_test.py
 +
|bgcolor="#80ff80"| OK
 +
|1 test (linux only, skipped elsewhere)
 +
|-
 +
|gramps/gen/test/user_test.py
 +
|bgcolor="#80ff80"| OK
 +
|1 test. See #{{bug|7013}} for context
 +
|-
 +
|gramps/gen/datehandler/test/datehandler_test.py
 +
|bgcolor="#80ff80"| OK
 +
|9 tests
 
|-
 
|-
 
|gramps/gen/db/test/cursor_test.py
 
|gramps/gen/db/test/cursor_test.py
Line 63: Line 82:
 
|gramps/gen/lib/test/date_test.py
 
|gramps/gen/lib/test/date_test.py
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
|8 tests
+
|8 tests (locale-based, OK for en, fails for some other locales)
 
|-
 
|-
 
|gramps/gen/lib/test/grampstype_test.py
 
|gramps/gen/lib/test/grampstype_test.py
Line 70: Line 89:
 
|-
 
|-
 
|gramps/gen/lib/test/merge_test.py
 
|gramps/gen/lib/test/merge_test.py
|bgcolor="#80ff80"| OK
+
|bgcolor="#ff8080"| FAIL
|201 tests
+
|201 tests, 2 failed skipped ({{bug|7027}}- Bug shows resolved)
 
|-
 
|-
 
|gramps/gen/merge/test/merge_ref_test.py
 
|gramps/gen/merge/test/merge_ref_test.py
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
|53 tests, 2 failures (bug #{{bug|6960}})
+
|53 tests
 
|-
 
|-
 
|gramps/gen/utils/test/callback_test.py
 
|gramps/gen/utils/test/callback_test.py
Line 84: Line 103:
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
 
|4 tests
 
|4 tests
 +
|-
 +
|gramps/gen/utils/test/place_test.py
 +
|bgcolor="#ff8080"| FAIL
 +
|28 tests, 4 failures (bug #{{bug|7044}} - Bug shows resolved)
 
|-
 
|-
 
|gramps/gui/logger/test/rotate_handler_test.py
 
|gramps/gui/logger/test/rotate_handler_test.py
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
 
|2 tests
 
|2 tests
 +
|-
 +
|gramps/gui/test/user_test.py
 +
|bgcolor="#80ff80"| OK
 +
|1 test (requires mocking to run)
 
|-
 
|-
 
|gramps/plugins/export/test/exportvcard_test.py
 
|gramps/plugins/export/test/exportvcard_test.py
Line 96: Line 123:
 
|bgcolor="#80ff80"| OK
 
|bgcolor="#80ff80"| OK
 
|66 tests
 
|66 tests
 +
|-
 +
|gramps/plugins/importer/test/importxml_test.py
 +
|bgcolor="#80ff80"| OK
 +
|4 tests
 +
|-
 +
|gramps/plugins/importer/test/importgeneweb_test.py
 +
|bgcolor="#80ff80"| OK
 +
|2 tests
 
|-
 
|-
 
|gramps/test/test/gedread_util_test.py
 
|gramps/test/test/gedread_util_test.py
Line 112: Line 147:
  
 
There is also semi-interactive testing via __main__ in some code:
 
There is also semi-interactive testing via __main__ in some code:
* Relationship calculator testing
+
 
* more!..
+
{| {{prettytable}}
 +
! File
 +
! Status
 +
! Comments
 +
|-
 +
|gramps/gen/relationship.py
 +
|bgcolor="#ff8080"| To Do
 +
|Relationship calculator
 +
|-
 +
|gramps/gen/db/bsddbtxn.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/txn.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/undoredo.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/write.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/txn.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/lib/styledtext.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/utils/place.py
 +
|bgcolor="#80ff80"| OK
 +
|Done
 +
|-
 +
|gramps/gen/utils/docgen/odstab.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/utils/docgen/csvtab.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gui/ddtargets.py
 +
|bgcolor="#ff8080"| To Do
 +
|Not worth running?
 +
|-
 +
|gramps/gui/widgets/undoablebuffer.py
 +
|bgcolor="#ff8080"| To Do
 +
|Not worth running?
 +
|-
 +
|gramps/gui/dialog.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/validatedmaskedentry.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/progressdialog.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/statusbar.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/plugins/lib/libsubstkeyword.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/plugins/rel/rel_*.py
 +
|bgcolor="#ff8080"| To Do
 +
|Relationship calculator plugins
 +
|-
 +
|gramps/plugins/tool/phpgedviewconnector.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|}
  
 
== Manual test plan ==
 
== Manual test plan ==
See TestPlan.txt in gramps toplevel. I believe this is only done at a major release (like 4.0.0).
+
See [http://sourceforge.net/p/gramps/source/ci/master/tree/TestPlan.txt TestPlan.txt] in gramps toplevel. (I believe this is only done at a major release (like 4.0.0)).
  
 
=Test status=
 
=Test status=
Line 122: Line 238:
  
 
=Possible improvements=
 
=Possible improvements=
* revive the in-tree tests
+
* Unify running all the tests, in- or out- tree
* unify running all the tests, in- or out- tree
+
** Try switching the feature tests and non-automated unit tests in test/ from our runner scripts to [http://docs.python.org/2/library/unittest.html#test-discovery python native unittest discovery mechanism].  
** Try switching from our runners to [http://docs.python.org/2/library/unittest.html#test-discovery python native unittest discovery mechanism].  
+
** We have a stale gramps/test/regrtest.py runner, notable for logging init. - should we revive that, or maybe integrate into setup.py test runner?
** Integrate with "python setup.py"? (need to split interactive vs non-interactive first to allow fully automated runs)
+
* Coverage analysis
* coverage analysis
+
* Continuous test status report, coverage, automatic deployment into win/mac/linux VMs (needs server capacity to be hosted online)(I can dream, can't I?)
* (needs server capacity to be hosted online) continous test status report, coverage, automatic deployment into win/mac/linux VMs (I can dream, can't I?)
+
* Automated regression tests for our GUI. The following links look interesting:
 +
** http://unpythonic.blogspot.co.il/2007/03/unit-testing-pygtk.html
 +
** http://ldtp.freedesktop.org/wiki/
  
 
=See also=
 
=See also=

Revision as of 07:47, 30 June 2014

Currently used tests and frameworks

testing of reports

test/runtest.sh

test/runtest.sh - Report test for GRAMPS: Generate every report in every format. Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.

Bugs tagged as found-by-runtest.sh

other report testing

See more specialized scripts in test/, status unknown.

test/impex.sh

test/impex.sh - Import/export test for Gramps.

From the file header:

  • Import example XML data and create internal Gramps DB
  • Open produced Gramps DB, then
    • check data for integrity
    • output in all formats
  • Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
  • Import every exported file produced if the format is also supported for import, and run a text summary report.
  • Diff each report with the summary of the produced example DB.

Bugs tagged as found-by-impex.sh

Unit testing

test/RunAllTests.py

test/RunAllTests.py - Testing framework for performing a variety of unttests for GRAMPS. Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.

Starting with gramps40 branch, these tests include non-automated unit tests only. The automated unit tests are all under gramps/.

Bugs tagged as found-by-RunAllTests.py

GtkHandler testing code pops up the Gramps error dialog, but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...

unit tests in the main tree

python setup.py test

See Unit Test Quickstart for detailed running instructions.

File Status Comments
gramps/cli/test/cli_test.py OK 3 tests
gramps/cli/test/user_test.py OK 8 tests (6 require mocking to run)
gramps/gen/test/config_test.py OK 1 test
gramps/gen/test/constfunc_test.py OK 1 test (linux only, skipped elsewhere)
gramps/gen/test/user_test.py OK 1 test. See #7013 for context
gramps/gen/datehandler/test/datehandler_test.py OK 9 tests
gramps/gen/db/test/cursor_test.py OK 4 tests, 2 skipped
gramps/gen/db/test/db_test.py OK 6 tests
gramps/gen/db/test/reference_map_test.py OK 5 tests
gramps/gen/lib/test/date_test.py OK 8 tests (locale-based, OK for en, fails for some other locales)
gramps/gen/lib/test/grampstype_test.py OK 3 tests
gramps/gen/lib/test/merge_test.py FAIL 201 tests, 2 failed skipped (7027- Bug shows resolved)
gramps/gen/merge/test/merge_ref_test.py OK 53 tests
gramps/gen/utils/test/callback_test.py OK 10 tests
gramps/gen/utils/test/keyword_test.py OK 4 tests
gramps/gen/utils/test/place_test.py FAIL 28 tests, 4 failures (bug #7044 - Bug shows resolved)
gramps/gui/logger/test/rotate_handler_test.py OK 2 tests
gramps/gui/test/user_test.py OK 1 test (requires mocking to run)
gramps/plugins/export/test/exportvcard_test.py OK 28 tests
gramps/plugins/importer/test/importvcard_test.py OK 66 tests
gramps/plugins/importer/test/importxml_test.py OK 4 tests
gramps/plugins/importer/test/importgeneweb_test.py OK 2 tests
gramps/test/test/gedread_util_test.py No longer required?
gramps/test/test/test_util_test.py No longer required?
gramps/webapp/grampsdb/view/png.py

There is also semi-interactive testing via __main__ in some code:

File Status Comments
gramps/gen/relationship.py To Do Relationship calculator
gramps/gen/db/bsddbtxn.py To Do
gramps/gen/db/txn.py To Do
gramps/gen/db/undoredo.py To Do
gramps/gen/db/write.py To Do
gramps/gen/db/txn.py To Do
gramps/gen/lib/styledtext.py To Do
gramps/gen/utils/place.py OK Done
gramps/gen/utils/docgen/odstab.py To Do
gramps/gen/utils/docgen/csvtab.py To Do
gramps/gui/ddtargets.py To Do Not worth running?
gramps/gui/widgets/undoablebuffer.py To Do Not worth running?
gramps/gui/dialog.py To Do Interactive
gramps/gui/widgets/validatedmaskedentry.py To Do Interactive
gramps/gui/widgets/progressdialog.py To Do Interactive
gramps/gui/widgets/statusbar.py To Do Interactive
gramps/plugins/lib/libsubstkeyword.py To Do
gramps/plugins/rel/rel_*.py To Do Relationship calculator plugins
gramps/plugins/tool/phpgedviewconnector.py To Do

Manual test plan

See TestPlan.txt in gramps toplevel. (I believe this is only done at a major release (like 4.0.0)).

Test status

We currently don't have a record of tests executed, the platforms and environments they were run upon, and what code they covered. The only indirect evidence is available in open bugs, when people care to fill in these details. :-(

Possible improvements

See also