Difference between revisions of "GEPS 023: Storing data from large sources"

From Gramps
Jump to: navigation, search
(Links)
m (Finished Gramps 3.4)
(48 intermediate revisions by 5 users not shown)
Line 1: Line 1:
Proposed changes for enhancing GRAMPS by enhancing the mechanism for storing data from ‘large’ sources
+
Proposed changes for enhancing GRAMPS by enhancing the mechanism for storing data from ‘large’ sources.
 +
 
 +
See [http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/ SVN] and
 +
[http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/?view=tar tarball].
 +
 
 +
Current code is now in trunk [http://gramps.1791082.n4.nabble.com/GEPS023-merged-into-trunk-td4157519.html Dec 05, 2011]
 +
 
 +
Finished: Gramps 3.4
  
 
== User story (Problem that needs to be solved) ==
 
== User story (Problem that needs to be solved) ==
Line 50: Line 57:
 
When it comes to adding the scan, the only option I really have is to add it to the source itself, despite the fact that the scan only relates to one page.
 
When it comes to adding the scan, the only option I really have is to add it to the source itself, despite the fact that the scan only relates to one page.
  
An alternative way to record the media is that suggested in the tutorial “Recording UK Census data”. Here the media object has a source reference which contains the details of the Date; Volume/page and Confidence that the media is associated with. This has the advantage that the details are unambiguously associated with the media. However, the relevant media cannot be found simply from the events.  
+
An alternative way to record the media is that suggested in the tutorial “[[Recording UK Census data]]”. Here the media object has a source reference which contains the details of the Date; Volume/page and Confidence that the media is associated with. This has the advantage that the details are unambiguously associated with the media. However, the relevant media cannot be found simply from the events.  
  
 
[[File:Separate source ref media.jpg|300px]]
 
[[File:Separate source ref media.jpg|300px]]
Line 74: Line 81:
 
A new object which we call here a Citation is introduced.
 
A new object which we call here a Citation is introduced.
  
A CitationRef would replace the current SourceRef. The CitationRef would not have any associated fields (just the reference to the Citation).
+
A CitationBase would replace the current SourceRef. The CitationRef would not have any associated fields (just the reference to the Citation). The CitationBase plus the Citation would be equivalent to the Sourceef.
  
 
The Source object is unchanged.
 
The Source object is unchanged.
Line 94: Line 101:
 
[[File:Source citation changes.jpg|500px]]
 
[[File:Source citation changes.jpg|500px]]
  
Note that there is no need for a CitationRef editor, because this does not have any associated properties.
+
Note that there is no need for a CitationBase editor, because this does not have any associated properties.
  
 
Recording the information about the people would proceed exactly as at present. The first source to be recorded would display an empty Citation editor, which would be completed in the normal way. Subsequent data would be linked to the same Citation object.
 
Recording the information about the people would proceed exactly as at present. The first source to be recorded would display an empty Citation editor, which would be completed in the normal way. Subsequent data would be linked to the same Citation object.
Line 199: Line 206:
 
== Design ==
 
== Design ==
  
The design is based on the first of the 'Other changes' not being carried out at this time, because the approach to providing the information for GEPS 018: Evidence style sources has not yet been decided.
+
The design is based on the 'Other changes' not being carried out at this time.
 +
 
 +
The first is not applied because the approach to providing the information for GEPS 018: Evidence style sources has not yet been decided.
 +
 
 +
The second is not applied because it would be better as the subject of a separate GEPS.
  
The design does apply the second of the 'Other changes' (i.e. deduction content). However, perhaps this should be a separate change.
+
The design is intended to have minimal change from the existing user interface. Aunt Martha should be able to continue to use Gramps just as at present. Only if a user wants to take advantage of the ability to share Citations should the user ned to be aware of the changes.
  
 
=== Database changes ===
 
=== Database changes ===
  
A Global Confidence field is added to the Source object:
+
The Source PrimaryObject is unchanged.
 
 
Source
 
  ...no changes except add...
 
  1 Global Confidence
 
  
A new Citation object has the following content:
+
A new Citation PrimaryObject has the following content:
  
 
  Citation
 
  Citation
   1 SourceRef --> Source
+
   1 RefBase --> Source
 
   1 Gramps Id
 
   1 Gramps Id
 
   1 Confidence (5 values)
 
   1 Confidence (5 values)
Line 224: Line 231:
 
   1 Private
 
   1 Private
  
This should be a full primary object so that it has all the properties of a primary object.  
+
A Citation object '''always''' refers to one and only one Source object. Therefore when creating a new Citation object, one first chooses a Source object to refer to. When deleting a Source object, all Citations that refer to that Source object must first be deleted, and references to those Citations object must be deleted before the Citation object.
 
 
A new CitationRef object replaces existing SourceRef objects. The object has the following content:
 
  
CitationRef
+
A new CitationBase object is simply a list of references to Citations, with no attributes. This object is analogous to the NoteBase and TagBase objects.
  1 Type: Transcript or Deduction
 
  1 Deduction Confidence (5 values)
 
  1 Argument (one line string)
 
  n Note
 
  
 
The proxies will also need to be updated.
 
The proxies will also need to be updated.
Line 238: Line 239:
 
=== Upgrade ===
 
=== Upgrade ===
  
When upgrading from an old database version all objects that have a SourceRef need to be changed to the new format.  The primary objects Person, Family, Event, Media and Place contain SourceRef objects.  These also contain secondary objects which have SourceRef objects.
+
When upgrading from an old database version all objects that have a SourceRef need to be changed to the new format.  The primary objects Person, Family, Event, Media and Place contain SourceRef objects.  These Primary objects also contain secondary objects which have SourceRef objects.
  
 
  Person
 
  Person
Line 268: Line 269:
 
   Attribute
 
   Attribute
  
Each old SourceRef object should be used to create a new Citation record.  The old SourceRef will be replaced by a new CitationRef.
+
Repository (Repositories themselves do not have SourceRefs)
 +
  Address
 +
 
 +
Each old SourceRef object should be used to create a new Citation record.  The old SourceRef will be replaced by a new CitationBase.
  
 
Because upgrading an old database version is automatic, the program should not prejudge the user's intention for similar SourceRefs. Therefore, SourceRefs should only be merged if they have the same Volume/Page, Date, Confidence and source and all Notes refer to shared copies of the same Notes (this will be the case where a SourceRef had been created by dragging and dropping on the Clipboard). It would be convenient if there were a separate Gramplet to automatically merge Citation objects on less stringent and probably configurable criteria. Such a Gramplet would need to merge Notes into the merged Citation.
 
Because upgrading an old database version is automatic, the program should not prejudge the user's intention for similar SourceRefs. Therefore, SourceRefs should only be merged if they have the same Volume/Page, Date, Confidence and source and all Notes refer to shared copies of the same Notes (this will be the case where a SourceRef had been created by dragging and dropping on the Clipboard). It would be convenient if there were a separate Gramplet to automatically merge Citation objects on less stringent and probably configurable criteria. Such a Gramplet would need to merge Notes into the merged Citation.
Line 378: Line 382:
 
|
 
|
 
     +1 PAGE <WHERE_WITHIN_SOURCE> {0:1} p.64
 
     +1 PAGE <WHERE_WITHIN_SOURCE> {0:1} p.64
|| Citation:Volume/page
+
|| SourceRef:Volume/Page becomes Citation:Volume/page
 
|-
 
|-
 
|
 
|
 
     +1 EVEN <EVENT_TYPE_CITED_FROM> {0:1} p.49
 
     +1 EVEN <EVENT_TYPE_CITED_FROM> {0:1} p.49
 
       +2 ROLE <ROLE_IN_EVENT> {0:1} p.61
 
       +2 ROLE <ROLE_IN_EVENT> {0:1} p.61
|| Not supported. See feature request [http://www.gramps-project.org/bugs/view.php?id=2918 2918] and [http://www.gramps-project.org/bugs/view.php?id=2924 2924], (which are mostly duplicates of each other)
+
|| Not supported. See feature request [http://www.gramps-project.org/bugs/view.php?id=2918 2918] and [http://www.gramps-project.org/bugs/view.php?id=2924 2924], (which are mostly duplicates of each other). No change. Any change should be implemented as a separate enhancement, since it is not directly related to the objectives of this GEPS. The data should be stored in the data of the Citation.
 
|-
 
|-
 
|
 
|
 
     +1 DATA {0:1}
 
     +1 DATA {0:1}
 
       +2 DATE <ENTRY_RECORDING_DATE> {0:1} p.48
 
       +2 DATE <ENTRY_RECORDING_DATE> {0:1} p.48
|| Citation:Log Date
+
|| SourceRef:Log Date becomes Citation:Log Date
 
|-
 
|-
 
|
 
|
 
       +2 TEXT <TEXT_FROM_SOURCE> {0:M} p.63
 
       +2 TEXT <TEXT_FROM_SOURCE> {0:M} p.63
 
         +3 [CONC|CONT] <TEXT_FROM_SOURCE> {0:M}
 
         +3 [CONC|CONT] <TEXT_FROM_SOURCE> {0:M}
|| Not directly supported. Note that in Gramps, one would store the text from source in a note. On import, the 'text from source' should probably be stored as the contents of a Citation:NoteId. On export this GEDCOM field would not be output.
+
|| On import, this is stored as a note with Note Type 'Source text'. On export, the first note of type 'Source text' is output as the 'text from source'. No change.
 
|-
 
|-
 
|
 
|
 
     +1 <<MULTIMEDIA_LINK>> {0:M} p.37, 26
 
     +1 <<MULTIMEDIA_LINK>> {0:M} p.37, 26
|| Contents stored as the contents of a Citation:MediaRef
+
|| Not supported at present. Contents stored as the contents of a Citation:MediaRef
 
|-
 
|-
 
|
 
|
 
     +1 <<NOTE_STRUCTURE>> {0:M} p.37
 
     +1 <<NOTE_STRUCTURE>> {0:M} p.37
|| Contents stored as the contents of a Citation:NoteId. On export all the Data:Value pairs should probably be concatenated with comma separators into another separate note. On export, the fields in the CitationRef should probably be output as notes.
+
|| Contents stored as a note, referenced from the SourceRef, which becomes referenced from the Citation.
 +
 
 +
A possible future enhancement (not part of this GEPS, as it is not directly related to the objectives of this GEPS) would be to support the Data:Value pairs in the SourceRef (which becomes the Citation). On export all the Data:Value pairs should probably be concatenated with comma separators into another separate note.
 
|-
 
|-
 
|
 
|
 
     +1 QUAY <CERTAINTY_ASSESSMENT> {0:1} p.43
 
     +1 QUAY <CERTAINTY_ASSESSMENT> {0:1} p.43
|| Citation:Confidence
+
|| SourceRef:Confidence becomes Citation:Confidence
 
|-
 
|-
 
|
 
|
 
|}
 
|}
  
 +
=== User Interface changes ===
 +
 +
==== Source model and view ====
  
=== User Interface changes ===
+
The existing source model and the source view in the navigator are retained unchanged. This minimises the user interface changes. Source objects can be created, edited and deleted as at present.
  
==== Source model ====
+
==== Citation model ====
  
Change the existing source model to introduce a SourceTreeModel. Change the model so that it encompasses both Source objects and Citation objects. The tree model should have lines for Sources with a disclosure triangle, and subsidiary lines for Citations. Add new fields Title/Page (actually Source:Title or Citation:Volume/Page) and Citation:Volume/Page, retaining Source:Title.  Id is changed to "Source:id or Citation:Id". Source only fields are blank for Citations, and Citation only fields are blank for Sources. Title/Page displays either Title or Volume/Page (alternative of displaying Title:Volume/Page rejected). Change ID to be either Source Id or Citation Id depending on the record. Change the model so that either a Source or a Citation can be selected, and the returned value indicates which.
+
New CitationTreeModel and CitationListModel are introduced. The models encompass both Source objects and Citation objects. The tree model should have lines for Sources with a disclosure triangle, and subsidiary lines for Citations. Fields for Source should include Title, Author, ID, Abbreviation, Publication information and date last changed. Fields for Citation should include Volume/Page, ID, Date, Confidence and date last changed. The models should either a Source or a Citation can be selected, and the returned value indicates which.
  
 
This arrangement would allow sorting by Volume/Page (in case this were useful for some users - e.g. if Title is Birth certificate or Marriage certificate, and Volume/Page is the name of the individual).
 
This arrangement would allow sorting by Volume/Page (in case this were useful for some users - e.g. if Title is Birth certificate or Marriage certificate, and Volume/Page is the name of the individual).
  
Should the view show all Sources and on separate lines all Citations? Or should some views just show one or the other. Suggest that all views should show all Sources and on separate lines all Citations.
+
==== Citation Selector ====
 
 
==== Source Selector ====
 
  
This uses the modified source model.
+
This uses the new CitationTreeModel.
  
 
[[File:select source.jpg|200px]]
 
[[File:select source.jpg|200px]]
  
The display for the source selector comprises Title/Page and Id, using the SourceTreeModel.
+
The display for the source selector comprises Title/Page and Id.
  
==== Source View ====
+
==== Citation View ====
  
This uses the modified source model.
+
This uses the new citation models.
  
Add buttons to select Source view or Source Tree view. The default display for either source view comprises Title/Page, Id, Author, Publication information.  
+
Add buttons to select Source view or Source Tree view. The default display for either source view comprises Title/Page, Id, Date and Confidence.  
  
 
There are several possible objectives to add/edit:
 
There are several possible objectives to add/edit:
# Edit a Citation (select Citation, click Edit, - allows Citation and Source to be changed),
+
# Edit a Citation (Citation view: select Citation, click Edit, - allows Citation and Source to be changed),
# Edit a Source (select Source, click Edit - only allows Source to be changed),
+
# Edit a Source (Source view: select Source, click Edit - only allows Source to be changed),
# Add a new Source (selection is irrelevant, click Add - allows Citation and Source to be added - just input a source),
+
# Add a new Source (Source view: click Add - allows Source to be added),
# Add a new Citation to an existing Source (select the source, click Add new Citation - allows new Citation, and allows the Source to be changed).  
+
# Add a new Citation to an existing Source (Citation view: click Add; select the source - allows new Citation, and allows the Source to be changed).  
 
+
# Add a new citation to a new source (Source view: add the Source; then Citation view: add the Citation)
Add button for "Add a new citation" (Suggest using GTK_STOCK_INDENT)
 
  
If the "Add a new source" button is clicked: Bring up the Source-Citation editor. On clicking OK, if just Source information is entered, then just store a new Source object. Otherwise store both a Citation and a Source object.
+
In the Citation view:
If the "Add a new Citation" button is clicked: Bring up the Source-Citation editor with the specified Source populated. On clicking OK, if just Source information is entered, then just update the Source object. Otherwise store the new Citation and if changed, update the Source object.
+
*If the "Add a new Citation" button is clicked: Bring up the Source selector. When a Source is chosen, bring up the Source-Citation editor with the specified Source populated. On clicking OK, store the new Citation and if changed, update the Source object.
 
+
*If the "Edit" button is clicked, and a Citation is highlighted, the Source-Citation editor should allow both the Source and Citation to be changed.
Would it be better to avoid adding an extra button (with extra complication for the user) and just add a Citation if a Source row is highlighted and add a Source and Citation if not?
+
*If the "Edit" button is clicked, and a Source is highlighted: Do nothing
 
+
*If the "Remove" button is clicked, then the highlighted object and all objects that reference it should be removed.
If the "Edit" button is clicked, and a Source is highlighted: Bring up the current Source editor. (i.e. this does not allow a Citation to be input or amended).
 
If the "Edit" button is clicked, and a Citation is highlighted, the editor should allow both the Source and Citation to be changed.
 
 
 
If the "Remove" button is clicked, then the highlighted object and all objects that reference it should be removed.
 
  
 
==== Editor Source tabs ====
 
==== Editor Source tabs ====
  
Should these be hierarchical or flat? Given the likely relatively small number of CitationRefs in a given editor, this should probably remain flat.
+
CitationEmbedList replaces all occurrences of SourceEmbedList. The default fields displayed and the buttons etc. remain unchanged, except that the ID field contains the Citation ID rather than the Source ID.
 
 
The possible scenarios are different here, because "add a new Source" does not apply, because a source is always linked to the current primary or secondary object through a Citation, so a Citation is always created. There is therefore no need for an additional button.
 
 
 
Because the CitationRef has data, the editor will need three areas, as follows:
 
  
[[File:Combined source editor.jpg|center|200px]]
+
The same Source-Citation editor is used as for editing a Citation.
  
If the "Create and add a new source" button is clicked, then allow the creation of both Source and Citation objects. This is consistent with the current model, where "Create and add a new source" adds a new Source object and create the current sourceref. It is distinct from the "Add an existing source" which allows either an existing Source or an existing Citation and associated Source to be selected. Bring up the Source-Citation editor with all fields empty. On clicking OK, save both a new Source and a new Citation. Link the CitationRef to the new Citation.
+
If the "Create and add a new citation" button is clicked, then allow the creation of both Source and Citation objects. This is consistent with the current model, where "Create and add a new source" adds a new Source object and creates the current sourceref. It is distinct from the "Add an existing source" which allows either an existing Source or an existing Citation and associated Source to be selected. Bring up the Source-Citation editor with all fields empty. On clicking OK, save both a new Source and a new Citation. Error if the source is not entered. Link the CitationBase to the new Citation.
  
If the "Add an existing source" (Share) button is clicked, the user will either have selected a Source or a Citation. Pre-populate the editor according to what has been selected. On clicking OK, save either a new or an updated Citation; save the Source if it was updated (actually, the Source seems to be re-saved even if it was not changed, which may affect the last changed date incorrectly). Link the CitationRef to the new Citation.
+
If the "Add an existing source" (Share) button is clicked, the citation selector is dispalyed. The user will either select a Source or a Citation. Pre-populate the editor according to what has been selected. On clicking OK, save either a new or an updated Citation; save the Source if it was updated (actually, the Source seems to be re-saved even if it was not changed, which may affect the last changed date incorrectly). Link the CitationBase to the new Citation.
 
   
 
   
If the "Remove" button is clicked, then the highlighted Citation should be removed.
+
If the "Remove" button is clicked, then the highlighted Citation should be removed, together with the links to the citation. The Source is not removed.
  
 
==== Editors ====
 
==== Editors ====
  
There are three separate 'source' editors:
+
There are two separate 'citation/source' editors:
  
 
{|cellspacing="0" border="1" align="center" width="95%"
 
{|cellspacing="0" border="1" align="center" width="95%"
Line 477: Line 475:
 
|style="padding: 0.3em;" |[[Image:Add_source_1.png‎|center|thumb|200px]] editsource
 
|style="padding: 0.3em;" |[[Image:Add_source_1.png‎|center|thumb|200px]] editsource
 
|style="padding: 0.3em;" |[[Image:Source citation changes.jpg|center|thumb|200px]] editcitation. As used with the Citation reference having data, then this would not have the warning signs in the top half.
 
|style="padding: 0.3em;" |[[Image:Source citation changes.jpg|center|thumb|200px]] editcitation. As used with the Citation reference having data, then this would not have the warning signs in the top half.
|style="padding: 0.3em;" |[[Image:Combined source editor.jpg|center|thumb|200px]] editcitationref
 
 
|}
 
|}
  
editsource is used from the Source view when selecting a source and clicking the "Edit" button. Existing editsource is unchanged (adding a new source is not done through this interface, so the code that deals with that case can be removed).
+
editsource is used from
 +
* Source view when selecting a source and clicking the "Edit" button,
 +
* Source view when clicking the "Add" button.
 +
Existing editsource is unchanged.
  
 
editcitation is used from
 
editcitation is used from
* Source view when selecting a Citation and clicking the "Edit" button,
+
* Citation view when selecting a Citation and clicking the "Edit" button,
* Source view when clicking the "Add a new source" button (irrespective of the selection),
+
* Citation view when clicking the "Add a new citation" button (following the display of the source selector, and selection of a source),
* Source view when selecting a source and clicking the "Add a new Citation".
+
* Editor Source tabs (CitationEmbedList) when clicking the 'Create and add a new citation' button.
 +
* Editor Source tabs (CitationEmbedList) when clicking the 'Edit the selected citation' button.
 +
* Editor Source tabs (CitationEmbedList) when clicking the 'Add an existing citation' button.  
  
 
This is similar to the current editsourceref. On clicking OK,
 
This is similar to the current editsourceref. On clicking OK,
Line 491: Line 493:
 
* if nothing was passed in, if the Citation is blank store the new Source else store a new Citation and Source,
 
* if nothing was passed in, if the Citation is blank store the new Source else store a new Citation and Source,
 
* if a Source is passed in, add a new Citation and update the Source.
 
* if a Source is passed in, add a new Citation and update the Source.
 
editcitationref is used from the editor source tabs when clicking the "Add an existing source" or the "Create and add a new source".
 
 
On clicking OK,
 
* if a Citation was passed in, update the source and Citation and link the CitationRef to the Citation,
 
* if a Source is passed in, store a new Citation (even if the fields are empty), update the Source and link the CitationRef to the Citation,
 
* if nothing is passed in, if the Source is blank, error, else, store a new Citation (even if the fields are empty), store a new Source and link the CitationRef to the Citation
 
  
 
==== Rules ====
 
==== Rules ====
Line 508: Line 503:
  
 
Some changes will be needed in the Narrative Web and Simple Database Access functionality.
 
Some changes will be needed in the Narrative Web and Simple Database Access functionality.
 +
 +
== Known Issues ==
 +
 +
I believe that all the known issues below can be deferred till after merging GEPS023 into trunk.
 +
 +
==== Issues ====
 +
 +
* Count at bottom right corner of window in citation tree view is incorrect.  ''done'' fixed by change to allow filtering of secondary objects [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18706]
 +
* Add ID code for citations to the preferences dialog.  ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18714]
 +
* Checks and warnings on 'Save' and 'Cancel' in EditCitation are not quite right.  ''done '' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18549]
 +
 +
==== Deferred ====
 +
 +
* '''src/plugins/import/ImportCsv.py'''      Needs to be updated for citations ''done '' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18539]
 +
* '''src/plugins/export/ExportCsv.py'''      Needs to be updated for citations ''done '' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18539]
 +
* '''src/plugins/import/ImportGeneWeb.py'''  Needs changing for citations, but I can't do this as I don't have access to the application : ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18526]
 +
* '''src/plugins/export/ExportGeneWeb.py'''  Needs changing for citations, but I can't do this as I don't have access to the application : ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18526]
 +
* '''src/plugins/import/ImportProGen.py'''    Needs changing for citations, but I can't do this as I don't have access to the application
 +
* '''src/plugins/tool/TestCaseGenerator.py''' Needs to be updated for citations  ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18713]
 +
* '''src/plugins/tool/Check.py''' This has been changed so that it runs, but it needs further enhancement to specifically check citation data  ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18713]
 +
* '''src/webapp/grampsdb/models.py'''        ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18580]
 +
* '''src/webapp/grampsdb/libdjango.py'''      ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18580]
 +
* '''src/webapp/grampsdb/utils.py'''          moved to src/webapp/utils.py?  ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18581]
 +
* '''SimpleAccess, SimpleDoc and SimpleTable''' Needs to be changed for citations, but I am not yet sure how to test these.
 +
* '''po/POTFILES.in and  POTFILES.skip''' New files that need translation to be added [http://www.gramps-project.org/bugs/view.php?id=5395#c21977]
 +
* Improve source citation support (import/export) ? see [[Gramps_and_GEDCOM#Information_in_GEDCOM_5.5_not_in_Gramps_.28lost_on_import.29|Gramps and GEDCOM]]
 +
 +
==== Queries ====
 +
 +
* The icon for citation might need to be changed to more closely meet the Tango guidelines. [http://www.gramps-project.org/wiki/index.php?title=GRAMPS_icon_set][http://gramps-project.org/2008/01/30-artwork/][http://gramps.svn.sourceforge.net/viewvc/gramps/branches/maintenance/gramps22/src/images/sources.svg]  ''done'' [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18761]
 +
* Should the default views be changed (e.g. remove the separate source and citation views and provide flat and tree presentations as separate presentations within a single view. Also, should there be a flat sources view).
 +
* In the citationtreeview you can't filter on citation criteria (e.g. confidence) - This is because of the way the views work, the 'Source view' (or citationtreeview) is based on Sources only, so sorting, searching and filtering are just based on sources. If you want to filter the citations, then you have to use the citations view, and the citations filter in that view.  ''done'' Improve treebasemodel to allow filtering of secondary object nodes.  [http://gramps.svn.sourceforge.net/viewvc/gramps?view=revision&revision=18706]
 +
* '''src/gen/proxy/refernced.py'''            I have updated this for citations, but it does not appear to be used, so I have not tested it
 +
* '''src/plugins/export/ExportFtree.py'''    Not changed for citations because it does not appear to be used
 +
* '''src/plugins/import/ImportGrdb.py'''      Don't plan to change this as it appears to be outdated - should it be removed?
 +
* '''src/test/GrampsDb/GrampsDbTestBase.py''' ?? Is this actually used? could be done after merge into citations
 +
* '''src/lib/test/merge_test.py'''            I don't plan on upgrading this because it does not seem to be used
 +
* '''src/Merge/merge_ref_test.py'''          I don't plan on upgrading this because it does not seem to be used
 +
 +
The test modules have not been updated because of the following comment
 +
[http://gramps.1791082.n4.nabble.com/quot-We-can-always-use-more-developers-quot-td3760985.html#a3762016 The unit tests seem pretty skimpy]
 +
<blockquote>
 +
Yes, somebody started it and it got at the sidetrack. We hardly have time to develop, so unit tests is not something that is in the DNA of our community. Personally I believe that for GUI it is too much work to maintain for a small devel community.
 +
For the library stuff, I would love to see something easy. I however never run unit tests, never heard how to actually run those that are present. The only thing I do for certain parts is add tests at the bottom of the python file that run when the script is executed stand alone (see eg PlaceUtils.py).
 +
During my time at Gramps only those tests have actually catched anything, I never heard of the existing unit tests actually catching any bugs.
 +
Don't understand me wrong, I take testing of my own code very seriously, but I didn't yet see anybody proposing a unit test system for the Gramps codebase that is sufficiently developer friendly so that anybody actually runs them.
 +
</blockquote>
 +
 +
== Completion of the change ==
 +
 +
Although GEPS 023 'only' introduced one new object, it was a major change because it had ramifications throughout all of the Gramps code.
 +
 +
Here are some figures:
 +
 +
* The gep-023-large-sources branch was created on 1 Jul 2011.
 +
* The branch was merged back into trunk on 4 Dec 2011.
 +
* So it took almost 5 months to implement the change.
 +
* 159 files were changed by the time the branch was merged back.
 +
* There were 13,485 lines of u-diff.
 +
 +
I would like to thank Nick Hall for the enormous help and 
 +
encouragement I have received from him, especially in implementing the 
 +
tree model for two different primary objects which is largely his 
 +
code. Without his help, this enhancement would not have been possible.
 +
 +
Also thanks for help, testing and support from Michiel Nauta (advice 
 +
on upgrade transactions), John Ralls (fixes to some make files), 
 +
Jérôme Rapinat (fixes to translation files etc.), Benny Malengier 
 +
(advice on upgrade), Brian Matherly (svn admin), Rob Healey, Gary 
 +
Burton, Doug Blank and Cedric Scott and everyone else whose names I 
 +
may have missed.
 +
 +
== Changes needed ==
 +
 +
This section provides a guide to the changes that are need to other code (for example add-ons) to interface with citations.
 +
 +
The user interface has been designed on the principle that Aunt Martha 
 +
should not need to know or even be aware that citations have changed. 
 +
Hence she will see a Source navigation with similar information to the 
 +
current view. There will be a disclosure triangle, but she can ignore 
 +
that. When she wants to add a source to a person or event, she clicks 
 +
on the add or add existing button, and gets a similar choice of source 
 +
as at present. (Again she can ignore the disclosure triangles). The 
 +
editor that appears is then very similar to the existing editor. Only 
 +
if a user particularly wants to share citations between objects do 
 +
they need to be aware of the change. The change is also intended to be 
 +
entirely compatible with GEDCOM.
 +
 +
Simple access to, and updating of, citations or sources does not 
 +
involve very extensive changes to existing code. The database changes, 
 +
the various editors, filters and proxies etc. have involved more 
 +
extensive changes.
 +
 +
The Citation primary object corresponds pretty closely to the 
 +
SourceRef secondary object. Data in the SourceRef is mapped to the 
 +
Citation. A Citation has the additional possibility of having media 
 +
objects attached (which is not possible with SourceRefs). A Citation 
 +
points to exactly one Source (the code may assume that here is always 
 +
a source object present).
 +
 +
The 'shape' of existing objects is unchanged. The change is that 
 +
SourceBase is replaced by CitationBase.
 +
 +
For creating data, where you previously created a SourceRef, you now 
 +
create a Citation object, and set_reference_handle() to point to the 
 +
Source object. There is a difference that you then have to do a 
 +
db.add_citation(citation, transaction). You then do an 
 +
obj.add_citation(citation_handle) instead of an 
 +
obj.add_source_reference(sourcerefobj) to add the SourceRef/Citation 
 +
to the object.
 +
 +
An example of the changes is:
 +
http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/lib/libgedcom.py?r1=18405&r2=18415
 +
 +
 +
As far as data access is concerned, you do obj.get_citation_list() 
 +
instead of obj.get_source_references() to get a list of Citation 
 +
handles/SourceRefs respectively. With the citation, you then need to 
 +
do a db.get_citation_from_handle(citation_handle). Having got the 
 +
object, you do get_reference_handle() to get the source object handle.
 +
 +
An example of the changes is:
 +
http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/export/ExportGedcom.py?r1=18405&r2=18408
 +
 +
 +
Note 1. The names of the functions and the return values are mostly 
 +
chosen to match the way notes are attached to objects, so 
 +
obj.add_citation(citation_handle) corresponds with add_note - both 
 +
taking handles as parameters, and obj.get_citation_list() corresponds 
 +
with get_note_list() - both returning handles.
 +
 +
Note 2. I am aware of PEP 8 and pylint and have followed them for new 
 +
code. However, where I have just been modifying existing code, or have 
 +
largely copied some existing code, I have retained the style of the 
 +
existing code, for example in terms of code layout, naming conventions 
 +
and the number of parameters or methods in a class. This may result in 
 +
a lowered pylint score.
 +
 +
[N.B. This is only a guide, no guarantee!]
 +
  
  
Line 521: Line 656:
 
=== Bug tracker ===
 
=== Bug tracker ===
  
[http://www.gramps-project.org/bugs/view.php?id=1022 1022: Sources dialog hangs for 2 minutes after opening]
+
*{{bug|1022}}: Sources dialog hangs for 2 minutes after opening - This was closed unable to reproduce 2 Feb 2011
 +
*{{bug|2918}}: Add Gedcom Source Citation Fields
 +
*{{bug|2924}}: Implement GEDCOM EVENT_TYPE_CITED_FROM and ROLE_IN_EVENT - This was closed as a duplicate of 2918 on 22 Jan 2011
 +
*{{bug|4491}}: (Feature request): Matching source and quality level - This was closed resolved fixed 28 Jan 2011
 +
*{{bug|4913}}: (Feature request): Additional Event Filters
 +
 
 +
=== Others interfaces ===
 +
 
 +
*[http://www.ancestralquestonline.com/desbk/aq_sources.htm Source edition into Ancestral Quest]
 +
*[http://west-penwith.org.uk/misc/ftmsour.htm Source Citations in FTW], now [http://www.geneamusings.com/2011/01/peeking-at-family-tree-maker-2011.html Source citation into Family Tree Maker]
 +
*[http://www.geneamusings.com/2011/01/peeking-at-legacy-family-tree-75-source.html Source citation into Legacy Family Tree]
 +
 
 +
=== Templates ===
 +
 
 +
*[http://sourcetemplates.org/ Source templates project]
 +
**[http://sourcetemplates.org/details.php details]
 +
 
 +
=== Other ===
 +
 
 +
*[http://bettergedcom.wikispaces.com/Sources+and+Citations Sources and Citations]
 +
*[[GEPS_015:_Repository_Research_Support|GEPS 015:Repository Research Support]]
 +
*[[GEPS_018:_Evidence_style_sources|GEPS 018:Evidence style sources]]
 +
*[[GEPS_024:_Certificates|GEPS 024:Certificates]]
  
 
[[Category:GEPS|S]]
 
[[Category:GEPS|S]]

Revision as of 00:15, 27 December 2012

Proposed changes for enhancing GRAMPS by enhancing the mechanism for storing data from ‘large’ sources.

See SVN and tarball.

Current code is now in trunk Dec 05, 2011

Finished: Gramps 3.4

User story (Problem that needs to be solved)

I have a book that details, on page 7:

“In the 1870s B moved to the town of BT. It was here that I's father K was born in 1860. By the time he was 30 he had married. His first child M was born there. Shortly afterwards his wife died and two years later he married G. M was 12 before her brother I appeared.”

So I wish to record B, K, and the fact that K was born in 1860, and married around 1890. K's children were M and I, M was born around 1890 and I was born around 1900. [Actually, from other sources he was born on 5 Dec 1902.] I need to record page 7 of this book as the source for all these pieces of information.

Some time later I decide I should record a transcript of the source text.

Some time later, I decide to scan that page of the book, and need to store the scan as the source.

Later still, I discover that page 212 of the same book details that I married W in 1946.

Now I wish to record W, and the marriage of W and I in 1946. The source for all this is page 212 of the book, and this time I record the scan against the source.

List various solutions

  • Record each page as a source reference with the book as a single source.
  • Record each page as a separate source (i.e. page 7 as one source and page 212 as a second source).
  • Modify Gramps to introduce Source Content that can be shared and can have media attachments and record each page as a Source Content.

Record each page as a source reference

The page number and the text from the page are stored in the Source Reference, while the Title and author of the book are stored in the shared Source.

This may be considered the natural approach, given the fact that the Source Reference headings include the page number, while the shared Source includes the Title and the Author etc.

This approach may be illustrated as follows (only some of the people and facts are shown here):

Separate source ref.gif

The problems with this approach are

  • The Source Reference does not allow the Media scan to be stored.
  • The Source Reference is not shared, there is a separate instance for each place where it occurs (e.g. each event).

The media scan can be stored with the source, as shown in the figure above. However, when one looks at the source, it will not be clear which media object in the gallery relate to which source reference, except by some naming convention (the applicable media file may be obvious in this case, but not in others). Also, in the source reference editor, one cannot immediately see which scan relates to this particular page.

The fact that the source reference is not shared (and is not a separate object) means that any updates to the source reference will have to be done for each occurrence, rather than once. For example, when I want to add the transcript of the paragraph to the source reference, I need to find each source reference and update it individually. It is also difficult to find all the source references, because they are not listed in a separate tab.

Note that there is an argument that separate source references for the different events is preferable, because the exact text that relates to that particular event can be attached. For example, for the birth event for person K, one could attach: “…the town of BT. It was here that I's father K was born in 1860…”. There are two objections to this:

  • It is difficult to identify exactly which parts of the text are relevant to each event. Should I’s father be included in the source for K’s birth?
  • It is far too tedious and laborious to devise separate source texts for each event. Given that the original paragraph giving family history information (this is a genuine example) is quite short, it is much quicker and easier to include the whole paragraph in each reference.

When it comes to adding the scan, the only option I really have is to add it to the source itself, despite the fact that the scan only relates to one page.

An alternative way to record the media is that suggested in the tutorial “Recording UK Census data”. Here the media object has a source reference which contains the details of the Date; Volume/page and Confidence that the media is associated with. This has the advantage that the details are unambiguously associated with the media. However, the relevant media cannot be found simply from the events.

Separate source ref media.jpg

Record each page as a source

The page number and the text from the page, the book Title and the author of the book are all stored in the Source. The Source Reference does not hold any particular information.

This arrangement can be illustrated as here (again only some of the people and facts are shown here):

Separate source.gif

This arrangement has the advantage that the transcript and media scan can be directly attached to the relevant page.

However it has a number of problems:

  • the Volume/Page field in the source reference is not used for its standard purpose
  • the information about the book itself is duplicated many times in each source object
  • the approach rapidly becomes unmanageable if there are a large number of pages to be recorded for a single source (i.e. if the source is 'large')

Modify Gramps to introduce a Citation object that can be shared and can have media attachments and record each page as a Citation

A new object which we call here a Citation is introduced.

A CitationBase would replace the current SourceRef. The CitationRef would not have any associated fields (just the reference to the Citation). The CitationBase plus the Citation would be equivalent to the Sourceef.

The Source object is unchanged.

Newproposal.jpg

The existing icon options in the Source tabs would be unchanged, but add existing source would bring up a treeview Source-Citation.

Add source citation.jpg

The treeview would be as follows:

Select source.jpg

The user would be able to select a Source as shown by the highlighting. In this case, the subsequent Citation dialogue would not be populated in the Citation area, only in the Source area. On the other hand, if a Citation were to be selected, then the subsequent Citation dialogue would be opened already populated with the shared Citation data.

The source reference editor would be changed to a Source-Citation editor as follows:

Source citation changes.jpg

Note that there is no need for a CitationBase editor, because this does not have any associated properties.

Recording the information about the people would proceed exactly as at present. The first source to be recorded would display an empty Citation editor, which would be completed in the normal way. Subsequent data would be linked to the same Citation object.

When it comes to record the transcript of the source text, there would only be a single Citation object to which the transcript would be added as a Note.

The scan of the page could be added as a Gallery item in the Citation object.

A new display category would be needed for the Citation, and it would be important to implement the merge facility for this category so that existing separate Citations could be merged.

There is no particular need to provide a means to make an existing Citation object refer to a different Source object. At present, it is not possible to change a source reference so as to preserve the information that has been input (such as date, Volume/Page, confidence or the links to notes) but to make it point to a different source. The source reference must be deleted and a new one created. Similarly, if one wanted to change a Citation to refer to a different source, the Citation would have to be deleted.

Discussion

Either of the two ways of using the current Gramps features has difficulties. This is shown not only by the points made in the descriptions above, but also by the fact that there have been many discussions in the lists about how to use the features.

The new approach make it simple to implement the given user story. The story focusses on recording information from a book, because that is a scenario which everyone can understand and relate to. However, it also applies equally to other scenarios, like recording census data.

The new approach is a very simple extension to the current features from the user's point of view. Gramps can be used exactly as at present with no additional inputs required from the user (the workflow is completely unchanged). If a particular Citation is to be shared, then again there is no need for any additional inputs, just use of the feature to add an existing Citation, instead of creating a new one. Apart from the Citation category display there are no additional screens.

The new approach is somewhat similar to the subsource approach in http://gramps.1791082.n4.nabble.com/sources-subsources-and-sourceref-td1794804.html. Subsource corresponds with Citation. However, in order to avoid any extra complexity in the approach, it is proposed that the notes are kept to the 'subsource'/Citation only, and not to the reference to them.

Using subsources can remove the above problems, as a subsource would be a nucleus information set: one line in a census, one marriage act, ...., and that connected to a source.

Why not use the sourceref?

The sourceref cannot be used for this! The text of an entire source, or an image of an entire source is something you need to share between objects, so it must be a source, not a sourceref. The note in the sourceref should only be used to explain how the information of the source was used for the object.

However, it is true that a subsource can have a date/page/volume/number within the larger source. Note that in the census example this is repeated on *all* sourceref objects. In the case of a subsource, it would be entered once in the subsource, and then not be repeated in the sourceref. I don't care to much for this as in the case where subsource is usefull, there is no problem of mentioning this in the note, .... section.

Is sourceref useless then?

I don't think this. For sources of which little information is learnt, or real books (bibliography, diary, ...) the sourceref is ideal. However, for sources which contain many unrelated short subparts that each can lead to large amount of changes throughout your genealogical database, the sourceref is less usefull, and a subsource is in order.

Note that adding a hierarchy of subsource objects would introduce much greater complexity, which is not warranted by the problems outlined.

Also, http://gramps.1791082.n4.nabble.com/Source-references-names-and-notes-td1813805.html#a1813818

The problem is still there when you discover a mistake in the source ref (page number), or want to add a note to these source references (eg transcript of the original latin text). Then you need to track down all source ref that where copies, and change them all. This is why in my workflow I keep most notes in the source object, where the media files (scans) also live. This is why I think about a subsource implementation, it does not hurt the people who want to keep working as they do now. The main thing holding such a thing back is GEDCOM though. The more we deviate of that in GRAMPS, the more difficult to map our internal data to something that can eg be uploaded to websites logically as you want it. It is really hard to keep having to drag an old and dead standard with us :-(

The proposed approach is directly compatible with GEDCOM; variants like hierarchical sources would not be.

References

There have been a large number of posts about this topic on the Gramps mailing lists. A selection is shown below.

  1. database issues sourceref references (2005) among other things suggests "We want to allow in the future that the sourceref can have a media coupled to it"
  2. sources subsources and sourceref (2007) proposes subsources between sources and objects
  3. GEDCOM and sourceref (2007) too much information in sourceref/citation or problem summarised as need to manually edit every independent source reference to change a page number
  4. Sources and sourceref (2007) second proposal is for "sourceref is made a primary object that an object can share, but is unique to a source source 1 -----> n sourceref n <----> n object (person, attribute, event, ...)"
  5. Media and attributes in data model for Gramps 3.0 (2007)
  6. local gallery tab in source reference (2007)
  7. nested sources - sourceref - one big source (2007)
  8. sources vs. repositories (2007) "Some genealogy apps allow to subdivide a source in pieces, eg divide a book in chapters. GRAMPS does _not_ have this at the moment"
  9. Sources, media and galleries: how to tie it all up "as intended" (2008)
  10. Medias, sources and sourcerefs (2009) concerned with relationship between media and sourceref
  11. Source references names and notes (2009) "have to stop using my sourceref approach and just put everything as a source, which I find clumsy and lacking in elegance. Since sourcerefs can't be shared, only copied, it is extremely difficult to keep track of them."
  12. Storing data from large sources (2010)
  13. Sharing sources (2010) actually he is talking about sharing source references (i.e. similar to the current proposal but with different terminology)

Other changes

Some other related changes have been suggested here:

  1. Use the Data key-value pairs field to store Publication data in the Source, and split the Volume/Page in the Citation, and add a couple of extra fields. This seems to be related to changing the data stored in a Citation as part of GEPS 018: Evidence style sources. It may be better to wait till GEPS 018 is resolved.
  2. Adding deduction content to the CitationRef, namely a type, confidence argument and set of notes, and a global confidence field to the Source. This is related to the BetterGEDCOM and methodology proposals.


Design

The design is based on the 'Other changes' not being carried out at this time.

The first is not applied because the approach to providing the information for GEPS 018: Evidence style sources has not yet been decided.

The second is not applied because it would be better as the subject of a separate GEPS.

The design is intended to have minimal change from the existing user interface. Aunt Martha should be able to continue to use Gramps just as at present. Only if a user wants to take advantage of the ability to share Citations should the user ned to be aware of the changes.

Database changes

The Source PrimaryObject is unchanged.

A new Citation PrimaryObject has the following content:

Citation
  1 RefBase  --> Source
  1 Gramps Id
  1 Confidence (5 values)
  1 Volume/Page
  1 Log Date (The date that this data was entered into the original source document)
  n Information (key-value pairs, current Data)
  n NoteIds
  n MediaRef (Region, Src, attr, notes)  --> Media
  1 Private

A Citation object always refers to one and only one Source object. Therefore when creating a new Citation object, one first chooses a Source object to refer to. When deleting a Source object, all Citations that refer to that Source object must first be deleted, and references to those Citations object must be deleted before the Citation object.

A new CitationBase object is simply a list of references to Citations, with no attributes. This object is analogous to the NoteBase and TagBase objects.

The proxies will also need to be updated.

Upgrade

When upgrading from an old database version all objects that have a SourceRef need to be changed to the new format. The primary objects Person, Family, Event, Media and Place contain SourceRef objects. These Primary objects also contain secondary objects which have SourceRef objects.

Person
 Name
 Address
 Attribute
 PersonRef
 MediaRef
  Attribute
 LdsOrd
Family
 Attribute
 ChildRef
 MediaRef
  Attribute
 LdsOrd
Event
 Attribute
 MediaRef
  Attribute
MediaObject
 Attribute
Place
 MediaRef
  Attribute
Repository (Repositories themselves do not have SourceRefs)
 Address

Each old SourceRef object should be used to create a new Citation record. The old SourceRef will be replaced by a new CitationBase.

Because upgrading an old database version is automatic, the program should not prejudge the user's intention for similar SourceRefs. Therefore, SourceRefs should only be merged if they have the same Volume/Page, Date, Confidence and source and all Notes refer to shared copies of the same Notes (this will be the case where a SourceRef had been created by dragging and dropping on the Clipboard). It would be convenient if there were a separate Gramplet to automatically merge Citation objects on less stringent and probably configurable criteria. Such a Gramplet would need to merge Notes into the merged Citation.

   Upgrade needs to process every SourceRef in primary objects and secondary objects.
   for each SourceRef:
       assemble Volume/Page, Date, Confidence and SourceId and all NoteIds
           for each Citation:
               if Volume/Page, Date, Confidence and SourceId and all NoteIdsfor are the same:
                   use the existing Citation
       if no match, then create a new Citation

Should the criteria for matching Citations be weaker, for example just the Volume/Page, Date, Confidence and source matching, with Notes from each SourceRef being added into the Citation?

Import/Export

The following formats will need to be updated: Gramps XML, GEDCOM, CSV, GeneWeb.

Gramps XML

This will need a new <Citations> section with <Citation> entries.

Gramps should be able to import both old and new versions of the Gramps XML. If a <sourceref> tag appears outside of a <Citation> entry then it will indicate an old version.

GEDCOM

GEDCOM Gramps
SOURCE_RECORD:=
  n @<XREF:SOUR>@ SOUR {1:1}                              
    +1 DATA {0:1}                                         
      +2 EVEN <EVENTS_RECORDED> {0:M}                    
        +3 DATE <DATE_PERIOD> {0:1}                     
        +3 PLAC <SOURCE_JURISDICTION_PLACE> {0:1}       
      +2 AGNC <RESPONSIBLE_AGENCY> {0:1}                
      +2 <<NOTE_STRUCTURE>> {0:M}                       
Not supported. See feature request 1371. Discussion in Role and event tags...
    +1 AUTH <SOURCE_ORIGINATOR> {0:1}
      +2 [CONC|CONT] <SOURCE_ORIGINATOR> {0:M}
Source:Author
    +1 TITL <SOURCE_DESCRIPTIVE_TITLE> {0:1}
      +2 [CONC|CONT] <SOURCE_DESCRIPTIVE_TITLE> {0:M}
Source:Title
    +1 ABBR <SOURCE_FILED_BY_ENTRY> {0:1}
Source:Abbreviation
    +1 PUBL <SOURCE_PUBLICATION_FACTS> {0:1}
      +2 [CONC|CONT] <SOURCE_PUBLICATION_FACTS> {0:M}
If the publication data is changed to key-value pairs, then on import, this GEDCOM field should be stored as a predefined key (e.g. PUBL) , and on export all the value should probably be concatenated with comma separators.
    +1 TEXT <TEXT_FROM_SOURCE> {0:1}
      +2 [CONC|CONT] <TEXT_FROM_SOURCE> {0:M}
Not directly supported. Note that in Gramps, one would store the text from source in a note. On import, the 'text from source' should probably be stored as the contents of a Source:NoteId. On export this GEDCOM field would not be output.
    +1 <<SOURCE_REPOSITORY_CITATION>> {0:M}
Contents stored as the content of a Source:RepoRef
    +1 REFN <USER_REFERENCE_NUMBER> {0:M}
      +2 TYPE <USER_REFERENCE_TYPE> {0:1}
Not supported for data interchange (discussion in REFN strategy)
    +1 RIN <AUTOMATED_RECORD_ID> {0:1}
Not supported for data interchange. The Source:gramps_id can be considered to be the AUTOMATED_RECORD_ID
    +1 <<CHANGE_DATE>> {0:1}
Source:change (automatically maintained by Gramps)
    +1 <<NOTE_STRUCTURE>> {0:M}                 ||
Contents stored as the content of a Source:NoteId
    +1 <<MULTIMEDIA_LINK>> {0:M}                 ||
Contents stored as the contents of a Source:MediaRef
Not supported Source:Global Confidence


GEDCOM Gramps
SOURCE_CITATION:=
  n SOUR @<XREF:SOUR>@ {1:1} p.27
    +1 PAGE <WHERE_WITHIN_SOURCE> {0:1} p.64
SourceRef:Volume/Page becomes Citation:Volume/page
    +1 EVEN <EVENT_TYPE_CITED_FROM> {0:1} p.49
      +2 ROLE <ROLE_IN_EVENT> {0:1} p.61
Not supported. See feature request 2918 and 2924, (which are mostly duplicates of each other). No change. Any change should be implemented as a separate enhancement, since it is not directly related to the objectives of this GEPS. The data should be stored in the data of the Citation.
    +1 DATA {0:1}
      +2 DATE <ENTRY_RECORDING_DATE> {0:1} p.48
SourceRef:Log Date becomes Citation:Log Date
      +2 TEXT <TEXT_FROM_SOURCE> {0:M} p.63
        +3 [CONC|CONT] <TEXT_FROM_SOURCE> {0:M}
On import, this is stored as a note with Note Type 'Source text'. On export, the first note of type 'Source text' is output as the 'text from source'. No change.
    +1 <<MULTIMEDIA_LINK>> {0:M} p.37, 26
Not supported at present. Contents stored as the contents of a Citation:MediaRef
    +1 <<NOTE_STRUCTURE>> {0:M} p.37
Contents stored as a note, referenced from the SourceRef, which becomes referenced from the Citation.

A possible future enhancement (not part of this GEPS, as it is not directly related to the objectives of this GEPS) would be to support the Data:Value pairs in the SourceRef (which becomes the Citation). On export all the Data:Value pairs should probably be concatenated with comma separators into another separate note.

    +1 QUAY <CERTAINTY_ASSESSMENT> {0:1} p.43
SourceRef:Confidence becomes Citation:Confidence

User Interface changes

Source model and view

The existing source model and the source view in the navigator are retained unchanged. This minimises the user interface changes. Source objects can be created, edited and deleted as at present.

Citation model

New CitationTreeModel and CitationListModel are introduced. The models encompass both Source objects and Citation objects. The tree model should have lines for Sources with a disclosure triangle, and subsidiary lines for Citations. Fields for Source should include Title, Author, ID, Abbreviation, Publication information and date last changed. Fields for Citation should include Volume/Page, ID, Date, Confidence and date last changed. The models should either a Source or a Citation can be selected, and the returned value indicates which.

This arrangement would allow sorting by Volume/Page (in case this were useful for some users - e.g. if Title is Birth certificate or Marriage certificate, and Volume/Page is the name of the individual).

Citation Selector

This uses the new CitationTreeModel.

Select source.jpg

The display for the source selector comprises Title/Page and Id.

Citation View

This uses the new citation models.

Add buttons to select Source view or Source Tree view. The default display for either source view comprises Title/Page, Id, Date and Confidence.

There are several possible objectives to add/edit:

  1. Edit a Citation (Citation view: select Citation, click Edit, - allows Citation and Source to be changed),
  2. Edit a Source (Source view: select Source, click Edit - only allows Source to be changed),
  3. Add a new Source (Source view: click Add - allows Source to be added),
  4. Add a new Citation to an existing Source (Citation view: click Add; select the source - allows new Citation, and allows the Source to be changed).
  5. Add a new citation to a new source (Source view: add the Source; then Citation view: add the Citation)

In the Citation view:

  • If the "Add a new Citation" button is clicked: Bring up the Source selector. When a Source is chosen, bring up the Source-Citation editor with the specified Source populated. On clicking OK, store the new Citation and if changed, update the Source object.
  • If the "Edit" button is clicked, and a Citation is highlighted, the Source-Citation editor should allow both the Source and Citation to be changed.
  • If the "Edit" button is clicked, and a Source is highlighted: Do nothing
  • If the "Remove" button is clicked, then the highlighted object and all objects that reference it should be removed.

Editor Source tabs

CitationEmbedList replaces all occurrences of SourceEmbedList. The default fields displayed and the buttons etc. remain unchanged, except that the ID field contains the Citation ID rather than the Source ID.

The same Source-Citation editor is used as for editing a Citation.

If the "Create and add a new citation" button is clicked, then allow the creation of both Source and Citation objects. This is consistent with the current model, where "Create and add a new source" adds a new Source object and creates the current sourceref. It is distinct from the "Add an existing source" which allows either an existing Source or an existing Citation and associated Source to be selected. Bring up the Source-Citation editor with all fields empty. On clicking OK, save both a new Source and a new Citation. Error if the source is not entered. Link the CitationBase to the new Citation.

If the "Add an existing source" (Share) button is clicked, the citation selector is dispalyed. The user will either select a Source or a Citation. Pre-populate the editor according to what has been selected. On clicking OK, save either a new or an updated Citation; save the Source if it was updated (actually, the Source seems to be re-saved even if it was not changed, which may affect the last changed date incorrectly). Link the CitationBase to the new Citation.

If the "Remove" button is clicked, then the highlighted Citation should be removed, together with the links to the citation. The Source is not removed.

Editors

There are two separate 'citation/source' editors:

Add source 1.png
editsource
Source citation changes.jpg
editcitation. As used with the Citation reference having data, then this would not have the warning signs in the top half.

editsource is used from

  • Source view when selecting a source and clicking the "Edit" button,
  • Source view when clicking the "Add" button.

Existing editsource is unchanged.

editcitation is used from

  • Citation view when selecting a Citation and clicking the "Edit" button,
  • Citation view when clicking the "Add a new citation" button (following the display of the source selector, and selection of a source),
  • Editor Source tabs (CitationEmbedList) when clicking the 'Create and add a new citation' button.
  • Editor Source tabs (CitationEmbedList) when clicking the 'Edit the selected citation' button.
  • Editor Source tabs (CitationEmbedList) when clicking the 'Add an existing citation' button.

This is similar to the current editsourceref. On clicking OK,

  • if a Citation was passed in, update the Citation and the linked Source,
  • if nothing was passed in, if the Citation is blank store the new Source else store a new Citation and Source,
  • if a Source is passed in, add a new Citation and update the Source.

Rules

Do we need extra rules to match a Citation?

Reports

Reports access Source References through the Bibliography and Endnotes functionality. This allows the changes to be made in a single place.

Some changes will be needed in the Narrative Web and Simple Database Access functionality.

Known Issues

I believe that all the known issues below can be deferred till after merging GEPS023 into trunk.

Issues

  • Count at bottom right corner of window in citation tree view is incorrect. done fixed by change to allow filtering of secondary objects [1]
  • Add ID code for citations to the preferences dialog. done [2]
  • Checks and warnings on 'Save' and 'Cancel' in EditCitation are not quite right. done [3]

Deferred

  • src/plugins/import/ImportCsv.py Needs to be updated for citations done [4]
  • src/plugins/export/ExportCsv.py Needs to be updated for citations done [5]
  • src/plugins/import/ImportGeneWeb.py Needs changing for citations, but I can't do this as I don't have access to the application : done [6]
  • src/plugins/export/ExportGeneWeb.py Needs changing for citations, but I can't do this as I don't have access to the application : done [7]
  • src/plugins/import/ImportProGen.py Needs changing for citations, but I can't do this as I don't have access to the application
  • src/plugins/tool/TestCaseGenerator.py Needs to be updated for citations done [8]
  • src/plugins/tool/Check.py This has been changed so that it runs, but it needs further enhancement to specifically check citation data done [9]
  • src/webapp/grampsdb/models.py done [10]
  • src/webapp/grampsdb/libdjango.py done [11]
  • src/webapp/grampsdb/utils.py moved to src/webapp/utils.py? done [12]
  • SimpleAccess, SimpleDoc and SimpleTable Needs to be changed for citations, but I am not yet sure how to test these.
  • po/POTFILES.in and POTFILES.skip New files that need translation to be added [13]
  • Improve source citation support (import/export) ? see Gramps and GEDCOM

Queries

  • The icon for citation might need to be changed to more closely meet the Tango guidelines. [14][15][16] done [17]
  • Should the default views be changed (e.g. remove the separate source and citation views and provide flat and tree presentations as separate presentations within a single view. Also, should there be a flat sources view).
  • In the citationtreeview you can't filter on citation criteria (e.g. confidence) - This is because of the way the views work, the 'Source view' (or citationtreeview) is based on Sources only, so sorting, searching and filtering are just based on sources. If you want to filter the citations, then you have to use the citations view, and the citations filter in that view. done Improve treebasemodel to allow filtering of secondary object nodes. [18]
  • src/gen/proxy/refernced.py I have updated this for citations, but it does not appear to be used, so I have not tested it
  • src/plugins/export/ExportFtree.py Not changed for citations because it does not appear to be used
  • src/plugins/import/ImportGrdb.py Don't plan to change this as it appears to be outdated - should it be removed?
  • src/test/GrampsDb/GrampsDbTestBase.py ?? Is this actually used? could be done after merge into citations
  • src/lib/test/merge_test.py I don't plan on upgrading this because it does not seem to be used
  • src/Merge/merge_ref_test.py I don't plan on upgrading this because it does not seem to be used

The test modules have not been updated because of the following comment The unit tests seem pretty skimpy

Yes, somebody started it and it got at the sidetrack. We hardly have time to develop, so unit tests is not something that is in the DNA of our community. Personally I believe that for GUI it is too much work to maintain for a small devel community. For the library stuff, I would love to see something easy. I however never run unit tests, never heard how to actually run those that are present. The only thing I do for certain parts is add tests at the bottom of the python file that run when the script is executed stand alone (see eg PlaceUtils.py). During my time at Gramps only those tests have actually catched anything, I never heard of the existing unit tests actually catching any bugs. Don't understand me wrong, I take testing of my own code very seriously, but I didn't yet see anybody proposing a unit test system for the Gramps codebase that is sufficiently developer friendly so that anybody actually runs them.

Completion of the change

Although GEPS 023 'only' introduced one new object, it was a major change because it had ramifications throughout all of the Gramps code.

Here are some figures:

  • The gep-023-large-sources branch was created on 1 Jul 2011.
  • The branch was merged back into trunk on 4 Dec 2011.
  • So it took almost 5 months to implement the change.
  • 159 files were changed by the time the branch was merged back.
  • There were 13,485 lines of u-diff.

I would like to thank Nick Hall for the enormous help and encouragement I have received from him, especially in implementing the tree model for two different primary objects which is largely his code. Without his help, this enhancement would not have been possible.

Also thanks for help, testing and support from Michiel Nauta (advice on upgrade transactions), John Ralls (fixes to some make files), Jérôme Rapinat (fixes to translation files etc.), Benny Malengier (advice on upgrade), Brian Matherly (svn admin), Rob Healey, Gary Burton, Doug Blank and Cedric Scott and everyone else whose names I may have missed.

Changes needed

This section provides a guide to the changes that are need to other code (for example add-ons) to interface with citations.

The user interface has been designed on the principle that Aunt Martha should not need to know or even be aware that citations have changed. Hence she will see a Source navigation with similar information to the current view. There will be a disclosure triangle, but she can ignore that. When she wants to add a source to a person or event, she clicks on the add or add existing button, and gets a similar choice of source as at present. (Again she can ignore the disclosure triangles). The editor that appears is then very similar to the existing editor. Only if a user particularly wants to share citations between objects do they need to be aware of the change. The change is also intended to be entirely compatible with GEDCOM.

Simple access to, and updating of, citations or sources does not involve very extensive changes to existing code. The database changes, the various editors, filters and proxies etc. have involved more extensive changes.

The Citation primary object corresponds pretty closely to the SourceRef secondary object. Data in the SourceRef is mapped to the Citation. A Citation has the additional possibility of having media objects attached (which is not possible with SourceRefs). A Citation points to exactly one Source (the code may assume that here is always a source object present).

The 'shape' of existing objects is unchanged. The change is that SourceBase is replaced by CitationBase.

For creating data, where you previously created a SourceRef, you now create a Citation object, and set_reference_handle() to point to the Source object. There is a difference that you then have to do a db.add_citation(citation, transaction). You then do an obj.add_citation(citation_handle) instead of an obj.add_source_reference(sourcerefobj) to add the SourceRef/Citation to the object.

An example of the changes is: http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/lib/libgedcom.py?r1=18405&r2=18415


As far as data access is concerned, you do obj.get_citation_list() instead of obj.get_source_references() to get a list of Citation handles/SourceRefs respectively. With the citation, you then need to do a db.get_citation_from_handle(citation_handle). Having got the object, you do get_reference_handle() to get the source object handle.

An example of the changes is: http://gramps.svn.sourceforge.net/viewvc/gramps/branches/geps/gep-023-large-sources/src/plugins/export/ExportGedcom.py?r1=18405&r2=18408


Note 1. The names of the functions and the return values are mostly chosen to match the way notes are attached to objects, so obj.add_citation(citation_handle) corresponds with add_note - both taking handles as parameters, and obj.get_citation_list() corresponds with get_note_list() - both returning handles.

Note 2. I am aware of PEP 8 and pylint and have followed them for new code. However, where I have just been modifying existing code, or have largely copied some existing code, I have retained the style of the existing code, for example in terms of code layout, naming conventions and the number of parameters or methods in a class. This may result in a lowered pylint score.

[N.B. This is only a guide, no guarantee!]


Links

Background

Bug tracker

  • 1022: Sources dialog hangs for 2 minutes after opening - This was closed unable to reproduce 2 Feb 2011
  • 2918: Add Gedcom Source Citation Fields
  • 2924: Implement GEDCOM EVENT_TYPE_CITED_FROM and ROLE_IN_EVENT - This was closed as a duplicate of 2918 on 22 Jan 2011
  • 4491: (Feature request): Matching source and quality level - This was closed resolved fixed 28 Jan 2011
  • 4913: (Feature request): Additional Event Filters

Others interfaces

Templates

Other