Difference between revisions of "Testing Gramps"

From Gramps
Jump to: navigation, search
(Possible improvements: incorporate Nick's & John's suggestions for further improving the way the tests are run)
(test/impex.sh)
Line 2: Line 2:
  
 
=Currently used tests and frameworks=
 
=Currently used tests and frameworks=
== testing of reports==
+
== testing of reports ==
=== runtest.sh ===
+
=== test/runtest.sh ===
 
Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
 
Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
  
Line 10: Line 10:
 
== other report testing ==
 
== other report testing ==
 
See more specialized scripts in test/, status unknown.
 
See more specialized scripts in test/, status unknown.
 +
 +
==test/impex.sh==
 +
Import/export test for GRAMPS.
 +
 +
From the file header:
 +
* Import example XML data and create GRDB
 +
* Open produced GRDB, then
 +
** check data for integrity
 +
** output in all formats
 +
* Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
 +
* Import ever file produced and run summary on it.
 +
 +
Currently broken, see {{bug|6951}}
  
 
== Unit testing ==
 
== Unit testing ==
=== RunAllTests.py ===
+
=== test/RunAllTests.py ===
 
Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
 
Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
  

Revision as of 02:28, 28 July 2013

Gramps-notes.png
This may be an outdated section. Please help keep Gramps Wiki a useful resource by updating and expanding this section.

Currently used tests and frameworks

testing of reports

test/runtest.sh

Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.

See the "attached issues" at bugs tagged as found by runtest.sh

other report testing

See more specialized scripts in test/, status unknown.

test/impex.sh

Import/export test for GRAMPS.

From the file header:

* Import example XML data and create GRDB
* Open produced GRDB, then
** check data for integrity
** output in all formats
* Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
* Import ever file produced and run summary on it.

Currently broken, see 6951

Unit testing

test/RunAllTests.py

Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.

See the "attached issues" at Bugs tagged as found by RunAllTests.py

Two tests in GtkHandler testing code pop up the GRAMPS error dialog, but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...

One test currently fails, see 6940.

unit tests in the main tree

File Status Comments
gramps/gen/db/test/db_test.py Doesn't use unittest module
gramps/gen/lib/test/date_test.py
gramps/gen/lib/test/grampstype_test.py
gramps/gen/lib/test/merge_test.py
gramps/gen/merge/test/merge_ref_test.py
gramps/gen/utils/test/callback_test.py
gramps/plugins/export/test/exportvcard_test.py
gramps/plugins/importer/test/importvcard_test.py
gramps/test/config_test.py
gramps/test/gramps_cli_test.py
gramps/test/utils_test.py
gramps/test/test/gedread_util_test.py
gramps/test/test/test_util_test.py
gramps/webapp/grampsdb/view/png.py

There used to be a way to run unit tests from within the main tree (src/ before gramps40, now gramps/). These seem currently broken; they're described at Unit Test Quickstart.

There is also semi-interactive testing via __main__ in some code:

  • Relationship calculator testing
  • more!..

Manual test plan

See TestPlan.txt in gramps toplevel. I believe this is only done at a major release (like 4.0.0).

Test status

We currently don't have a record of tests executed, the platforms and environments they were run upon, and what code they covered. The only indirect evidence is available in open bugs, when people care to fill in these details. :-(

Possible improvements

  • revive the in-tree tests
  • unify running all the tests, in- or out- tree
    • Try switching from our runners to python-based unittest discovery mechanism.
    • Integrate with "python setup.py"? (need to split interactive vs non-interactive first to allow fully automated runs)
  • coverage analysis
  • (needs server capacity to be hosted online) continous test status report, coverage, automatic deployment into win/mac/linux VMs (I can dream, can't I?)

See also

  • Unit Test Quickstart (obsolete? need to revive the in-tree unit tests, now that runtest.sh and RunAllTests.py run again!)