Difference between revisions of "Testing Gramps"

From Gramps
Jump to: navigation, search
(test/impex.sh)
(add wikilink)
(90 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{stub}}
+
=Test status=
 +
As of Jan 2015 we don't have a record of tests executed, the platforms and environments they were run upon, and what code they covered. The only indirect evidence is available in open bugs, when people care to fill in these details. :-(
 +
 
 +
==Possible improvements==
 +
* Unify running all the tests, in- or out- tree
 +
** Try switching the feature tests and non-automated unit tests in test/ from our runner scripts to [https://docs.python.org/library/unittest.html#test-discovery python native unittest discovery mechanism].
 +
** We have a stale [https://github.com/gramps-project/gramps/blob/master/gramps/test/regrtest.py gramps/test/regrtest.py] runner, notable for logging init. - should we revive that, or maybe integrate into setup.py test runner?
 +
* [https://wiki.python.org/moin/CodeCoverage Coverage analysis]
 +
* [http://docs.python-guide.org/en/latest/scenarios/ci/ Continuous] test status report, coverage, automatic deployment into win/mac/linux VMs (needs server capacity to be hosted online)(I can dream, can't I?){{bug|8294}}
 +
* Automated regression tests for our GUI. The following links look interesting:
 +
** [https://wiki.python.org/moin/PythonTestingToolsTaxonomy#GUI_Testing_Tools Python Wiki:GUI Testing Tools]
 +
** [http://unpythonic.blogspot.co.il/2007/03/unit-testing-pygtk.html Unit Testing PyGTK]
 +
** [http://ldtp.freedesktop.org/wiki/ Cross Platform GUI Test Automation tool]
 +
 
 +
=Tests and frameworks used=
 +
== Manual test plan ==
 +
See [http://sourceforge.net/p/gramps/source/ci/master/tree/TestPlan.txt TestPlan.txt] in gramps toplevel. (I believe this is only done at a major release (like Gramps version 4.0.0)).
 +
 
 +
== Specialized scripts for testing ==
 +
See more specialized scripts in [http://sourceforge.net/p/gramps/source/ci/master/tree/test/ test/], status unknown.
 +
 
 +
Contents of [http://sourceforge.net/p/gramps/source/ci/master/tree/test/ test/] :
 +
* GrampsLogger/
 +
** ErrorReportAssistant_Test.py
 +
** GtkHandler_Test.py
 +
* LosHawlos_bsddbtest.py
 +
* LosHawlos_dbtest.py
 +
* RunAllTests.py
 +
* dates.sh
 +
* det_ancestor_report.sh
 +
* det_descendant_report.sh
 +
* impex.sh
 +
* runtest.sh
 +
* tools.sh
  
=Currently used tests and frameworks=
+
=== testing of reports ===
== testing of reports ==
+
==== test/runtest.sh ====
=== test/runtest.sh ===
 
Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
 
  
See the "attached issues" at [http://www.gramps-project.org/bugs/tag_view_page.php?tag_id=77 bugs tagged as found by runtest.sh]
+
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/runtest.sh test/runtest.sh]''' - Report test for Gramps: Generate every report in every format. Runs all possible reports using the report cli interface, based on the [[example.gramps]] database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.
  
== other report testing ==
+
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-runtest.sh found-by-runtest.sh]
See more specialized scripts in test/, status unknown.
 
  
==test/impex.sh==
+
===test/impex.sh===
Import/export test for GRAMPS.
+
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/impex.sh test/impex.sh]''' - Import/export test for Gramps.
  
 
From the file header:
 
From the file header:
* Import example XML data and create GRDB
+
* Import example XML data and create internal Gramps DB
* Open produced GRDB, then
+
* Open produced Gramps DB, then
** check data for integrity
+
** check data for integrity
** output in all formats
+
** output in all formats
* Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
+
* Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
* Import ever file produced and run summary on it.
+
* Import every exported file produced if the format is also supported for import, and run a text summary report.
 +
* Diff each report with the summary of the produced example DB.
  
Currently broken, see {{bug|6951}}
+
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-impex.sh  found-by-impex.sh]
  
 
== Unit testing ==
 
== Unit testing ==
 
=== test/RunAllTests.py ===
 
=== test/RunAllTests.py ===
Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
+
'''[http://sourceforge.net/p/gramps/source/ci/master/tree/test/RunAllTests.py test/RunAllTests.py]''' - Testing framework for performing a variety of unittests for Gramps. Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.
  
See the "attached issues" at [http://www.gramps-project.org/bugs/tag_view_page.php?tag_id=80 Bugs tagged as found by RunAllTests.py]
+
{{man note|Starting with gramps4.x branch|these tests include non-automated unit tests only. The automated unit tests are all under gramps/.}}
  
Two tests in GtkHandler testing code pop up the GRAMPS error dialog, but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...
+
Bugs tagged as [http://www.gramps-project.org/bugs/search.php?tag_string=found-by-RunAllTests.py found-by-RunAllTests.py]
  
One test currently fails, see {{bug|6940}}.
+
{{man warn|GtkHandler testing code pops up the Gramps error dialog|but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...}}
  
 
=== unit tests in the main tree ===
 
=== unit tests in the main tree ===
{| {{prettytable}}
+
 
 +
See [[Unit Test Quickstart]] for detailed running instructions.
 +
 
 +
==== using - python setup.py test ====
 +
python setup.py test
 +
 
 +
{| class="wikitable sortable"
 
! File
 
! File
 +
! Test
 
! Status
 
! Status
 
! Comments
 
! Comments
 +
|-
 +
|gramps/cli/test/cli_test.py
 +
|3
 +
|bgcolor="#80ff80"| OK
 +
|
 +
|-
 +
|gramps/cli/test/user_test.py
 +
|8
 +
|bgcolor="#80ff80"| OK
 +
|(6 test require [[Unit_Test_Quickstart#Mocking_external_dependencies_with_unittest.mock| mocking]] to run)
 +
|-
 +
|gramps/gen/test/config_test.py
 +
|1
 +
|bgcolor="#80ff80"| OK
 +
|
 +
|-
 +
|gramps/gen/test/constfunc_test.py
 +
|1
 +
|bgcolor="#80ff80"| OK
 +
|(linux only, skipped elsewhere)
 +
|-
 +
|gramps/gen/test/user_test.py
 +
|1
 +
|bgcolor="#80ff80"| OK
 +
|See #{{bug|7013}} for context
 +
|-
 +
|gramps/gen/datehandler/test/datehandler_test.py
 +
|9
 +
|bgcolor="#80ff80"| OK
 +
|
 +
|-
 +
|gramps/gen/db/test/cursor_test.py
 +
|4
 +
|bgcolor="#80ff80"| OK
 +
|2 skipped
 
|-
 
|-
 
|gramps/gen/db/test/db_test.py
 
|gramps/gen/db/test/db_test.py
 +
|6
 +
|bgcolor="#80ff80"| OK
 +
|
 +
|-
 +
|gramps/gen/db/test/reference_map_test.py
 +
|5
 +
|bgcolor="#80ff80"| OK
 
|
 
|
|Doesn't use unittest module
 
 
|-
 
|-
 
|gramps/gen/lib/test/date_test.py
 
|gramps/gen/lib/test/date_test.py
|
+
|8
|
+
|bgcolor="#80ff80"| OK
 +
|(locale-based, OK for en, fails for some other locales)
 
|-
 
|-
 
|gramps/gen/lib/test/grampstype_test.py
 
|gramps/gen/lib/test/grampstype_test.py
|
+
|3
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 
|-
 
|-
 
|gramps/gen/lib/test/merge_test.py
 
|gramps/gen/lib/test/merge_test.py
|
+
|201
|
+
|bgcolor="#ff8080"| FAIL
 +
|2 failed skipped ({{bug|7027}}- Bug shows resolved)
 
|-
 
|-
 
|gramps/gen/merge/test/merge_ref_test.py
 
|gramps/gen/merge/test/merge_ref_test.py
 +
|53
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 +
|-
 +
|gramps/gen/utils/test/callback_test.py
 +
|10
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 
|-
 
|-
|gramps/gen/utils/test/callback_test.py
+
|gramps/gen/utils/test/keyword_test.py
 +
|4
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 +
|-
 +
|gramps/gen/utils/test/place_test.py
 +
|28
 +
|bgcolor="#ff8080"| FAIL
 +
|4 failures (bug #{{bug|7044}} - Bug shows resolved)
 +
|-
 +
|gramps/gui/logger/test/rotate_handler_test.py
 +
|2
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 +
|-
 +
|gramps/gui/test/user_test.py
 +
|1
 +
|bgcolor="#80ff80"| OK
 +
|(requires mocking to run)
 
|-
 
|-
 
|gramps/plugins/export/test/exportvcard_test.py
 
|gramps/plugins/export/test/exportvcard_test.py
|
+
|28
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 
|-
 
|-
 
|gramps/plugins/importer/test/importvcard_test.py
 
|gramps/plugins/importer/test/importvcard_test.py
|
+
|66
|
+
|bgcolor="#80ff80"| OK
|-
 
|gramps/test/config_test.py
 
|
 
 
|
 
|
 
|-
 
|-
|gramps/test/gramps_cli_test.py
+
|gramps/plugins/importer/test/importxml_test.py
|
+
|4
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 
|-
 
|-
|gramps/test/utils_test.py
+
|gramps/plugins/importer/test/importgeneweb_test.py
|
+
|2
 +
|bgcolor="#80ff80"| OK
 
|
 
|
 
|-
 
|-
Line 87: Line 191:
 
|
 
|
 
|
 
|
 +
|No longer required?
 
|-
 
|-
 
|gramps/test/test/test_util_test.py
 
|gramps/test/test/test_util_test.py
 
|
 
|
 
|
 
|
 +
|No longer required?
 
|-
 
|-
 
|gramps/webapp/grampsdb/view/png.py
 
|gramps/webapp/grampsdb/view/png.py
 +
|
 
|
 
|
 
|
 
|
Line 98: Line 205:
 
|}
 
|}
  
There used to be a way to run unit tests from within the main tree (src/ before gramps40, now gramps/). These seem currently broken; they're described at [[Unit Test Quickstart]].
+
{{man note|Note from Nick Hall|Some of the merge tests are slow because they actually run Gramps from the command line to import and export files.<br><br>Extra packages and add-ons are required to run some of the tests. From memory, you will need the CliMerge and ExportRaw add-ons, and the libxml2, libxslt and mocking python packages. Some of these may not be available for python3 yet.}}
 +
 
 +
==== semi-interactive ====
  
 
There is also semi-interactive testing via __main__ in some code:
 
There is also semi-interactive testing via __main__ in some code:
* Relationship calculator testing
 
* more!..
 
  
== Manual test plan ==
+
{| class="wikitable sortable"
See TestPlan.txt in gramps toplevel. I believe this is only done at a major release (like 4.0.0).
+
! File
 
+
! Status
=Test status=
+
! Comments
We currently don't have a record of tests executed, the platforms and environments they were run upon, and what code they covered. The only indirect evidence is available in open bugs, when people care to fill in these details. :-(
+
|-
 
+
|[http://sourceforge.net/p/gramps/source/ci/master/tree/gramps/gen/relationship.py#l1889 gramps/gen/relationship.py]
=Possible improvements=
+
|bgcolor="#ff8080"| To Do
* revive the in-tree tests
+
|Relationship calculator
* unify running all the tests, in- or out- tree
+
|-
** Try switching from our runners to python-based unittest discovery mechanism.  
+
|gramps/gen/db/bsddbtxn.py
** Integrate with "python setup.py"? (need to split interactive vs non-interactive first to allow fully automated runs)
+
|bgcolor="#ff8080"| To Do
* coverage analysis
+
|
* (needs server capacity to be hosted online) continous test status report, coverage, automatic deployment into win/mac/linux VMs (I can dream, can't I?)
+
|-
 +
|gramps/gen/db/txn.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/undoredo.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/write.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/db/txn.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/lib/styledtext.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/utils/place.py
 +
|bgcolor="#80ff80"| OK
 +
|Done
 +
|-
 +
|gramps/gen/utils/docgen/odstab.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gen/utils/docgen/csvtab.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/gui/ddtargets.py
 +
|bgcolor="#ff8080"| To Do
 +
|Not worth running?
 +
|-
 +
|gramps/gui/widgets/undoablebuffer.py
 +
|bgcolor="#ff8080"| To Do
 +
|Not worth running?
 +
|-
 +
|gramps/gui/dialog.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/validatedmaskedentry.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/progressdialog.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/gui/widgets/statusbar.py
 +
|bgcolor="#ff8080"| To Do
 +
|Interactive
 +
|-
 +
|gramps/plugins/lib/libsubstkeyword.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|gramps/plugins/rel/rel_*.py
 +
|bgcolor="#ff8080"| To Do
 +
|Relationship calculator plugins
 +
|-
 +
|gramps/plugins/tool/phpgedviewconnector.py
 +
|bgcolor="#ff8080"| To Do
 +
|
 +
|-
 +
|}
  
 
=See also=
 
=See also=
* [[Unit Test Quickstart]] (obsolete? need to revive the in-tree unit tests, now that runtest.sh and RunAllTests.py run again!)
+
* [[Unit Test Quickstart]]
 +
* [[Test_date_handlers#TO_CHECK]]
 +
* [[Gramps_Performance#The_Test_Results]]
  
 
[[Category:Developers/Reference]]
 
[[Category:Developers/Reference]]
 
[[Category:Developers/Quality Assurance]]
 
[[Category:Developers/Quality Assurance]]

Revision as of 00:35, 3 February 2016

Test status

As of Jan 2015 we don't have a record of tests executed, the platforms and environments they were run upon, and what code they covered. The only indirect evidence is available in open bugs, when people care to fill in these details. :-(

Possible improvements

Tests and frameworks used

Manual test plan

See TestPlan.txt in gramps toplevel. (I believe this is only done at a major release (like Gramps version 4.0.0)).

Specialized scripts for testing

See more specialized scripts in test/, status unknown.

Contents of test/ :

  • GrampsLogger/
    • ErrorReportAssistant_Test.py
    • GtkHandler_Test.py
  • LosHawlos_bsddbtest.py
  • LosHawlos_dbtest.py
  • RunAllTests.py
  • dates.sh
  • det_ancestor_report.sh
  • det_descendant_report.sh
  • impex.sh
  • runtest.sh
  • tools.sh

testing of reports

test/runtest.sh

test/runtest.sh - Report test for Gramps: Generate every report in every format. Runs all possible reports using the report cli interface, based on the example.gramps database. This test is not fully self-contained, in that it depends on various environment settings, such as your locale, your preferred name display formats, and your report options. Last, but not the least, the verification of the resulting reports is entirely manual.

Bugs tagged as found-by-runtest.sh

test/impex.sh

test/impex.sh - Import/export test for Gramps.

From the file header:

  • Import example XML data and create internal Gramps DB
  • Open produced Gramps DB, then
    • check data for integrity
    • output in all formats
  • Check resulting XML for well-formedness and validate it against DTD and RelaxNG schema.
  • Import every exported file produced if the format is also supported for import, and run a text summary report.
  • Diff each report with the summary of the produced example DB.

Bugs tagged as found-by-impex.sh

Unit testing

test/RunAllTests.py

test/RunAllTests.py - Testing framework for performing a variety of unittests for Gramps. Runs out-of-tree (not in gramps/) testing code, by looking for any test/*_Test.py files and executing the test suites therein. See the current code in test/*_Test.py for example and python standard unittest docs.

Gramps-notes.png
Starting with gramps4.x branch

these tests include non-automated unit tests only. The automated unit tests are all under gramps/.

Bugs tagged as found-by-RunAllTests.py

Gnome-important.png
GtkHandler testing code pops up the Gramps error dialog

but this is actually for testing the error reporting itself. Don't be scared by the dialog, it's expected. Your manual work is required to close the dialogs with the "Cancel" button. The relevant tests still pass (unless there's another bug there)...

unit tests in the main tree

See Unit Test Quickstart for detailed running instructions.

using - python setup.py test

python setup.py test
File Test Status Comments
gramps/cli/test/cli_test.py 3 OK
gramps/cli/test/user_test.py 8 OK (6 test require mocking to run)
gramps/gen/test/config_test.py 1 OK
gramps/gen/test/constfunc_test.py 1 OK (linux only, skipped elsewhere)
gramps/gen/test/user_test.py 1 OK See #7013 for context
gramps/gen/datehandler/test/datehandler_test.py 9 OK
gramps/gen/db/test/cursor_test.py 4 OK 2 skipped
gramps/gen/db/test/db_test.py 6 OK
gramps/gen/db/test/reference_map_test.py 5 OK
gramps/gen/lib/test/date_test.py 8 OK (locale-based, OK for en, fails for some other locales)
gramps/gen/lib/test/grampstype_test.py 3 OK
gramps/gen/lib/test/merge_test.py 201 FAIL 2 failed skipped (7027- Bug shows resolved)
gramps/gen/merge/test/merge_ref_test.py 53 OK
gramps/gen/utils/test/callback_test.py 10 OK
gramps/gen/utils/test/keyword_test.py 4 OK
gramps/gen/utils/test/place_test.py 28 FAIL 4 failures (bug #7044 - Bug shows resolved)
gramps/gui/logger/test/rotate_handler_test.py 2 OK
gramps/gui/test/user_test.py 1 OK (requires mocking to run)
gramps/plugins/export/test/exportvcard_test.py 28 OK
gramps/plugins/importer/test/importvcard_test.py 66 OK
gramps/plugins/importer/test/importxml_test.py 4 OK
gramps/plugins/importer/test/importgeneweb_test.py 2 OK
gramps/test/test/gedread_util_test.py No longer required?
gramps/test/test/test_util_test.py No longer required?
gramps/webapp/grampsdb/view/png.py
Gramps-notes.png
Note from Nick Hall

Some of the merge tests are slow because they actually run Gramps from the command line to import and export files.

Extra packages and add-ons are required to run some of the tests. From memory, you will need the CliMerge and ExportRaw add-ons, and the libxml2, libxslt and mocking python packages. Some of these may not be available for python3 yet.

semi-interactive

There is also semi-interactive testing via __main__ in some code:

File Status Comments
gramps/gen/relationship.py To Do Relationship calculator
gramps/gen/db/bsddbtxn.py To Do
gramps/gen/db/txn.py To Do
gramps/gen/db/undoredo.py To Do
gramps/gen/db/write.py To Do
gramps/gen/db/txn.py To Do
gramps/gen/lib/styledtext.py To Do
gramps/gen/utils/place.py OK Done
gramps/gen/utils/docgen/odstab.py To Do
gramps/gen/utils/docgen/csvtab.py To Do
gramps/gui/ddtargets.py To Do Not worth running?
gramps/gui/widgets/undoablebuffer.py To Do Not worth running?
gramps/gui/dialog.py To Do Interactive
gramps/gui/widgets/validatedmaskedentry.py To Do Interactive
gramps/gui/widgets/progressdialog.py To Do Interactive
gramps/gui/widgets/statusbar.py To Do Interactive
gramps/plugins/lib/libsubstkeyword.py To Do
gramps/plugins/rel/rel_*.py To Do Relationship calculator plugins
gramps/plugins/tool/phpgedviewconnector.py To Do

See also