Difference between revisions of "Addon:Place completion tool"

From Gramps
Jump to: navigation, search
m (Download)
(Download resources)
(81 intermediate revisions by 9 users not shown)
Line 1: Line 1:
[[Category:Proposed_Tool_Specifications]][[Category:Plugins]]
+
{{Third-party plugin}}
[[Category:Developers/General]]
 
 
 
A tool to bring the places in your GRAMPS database in accordance with the GRAMPS requirements: batch add country, county; look-up latitude-longitude; set description (title); ...
 
  
There is a version for 3.1.x, as well as an old version that works with GRAMPS version 3.0.x or 2.2.5+ ! [[Place_completion_tool#Download|Download]].
+
[[File:Place completion tool example1.png|right|300px|Place completion tool - Example Results]]
  
[[Category:Plugins]]
+
A tool to bring the places in your Gramps database in accordance with the Gramps requirements: batch add country, county; look-up latitude-longitude; set description (title); ...
  
== Place Completion tool ==
 
 
This tool helps you fill in the place attributes like county, country, ..., by allowing you to select the places you work on, and do changes on all these places with one button click.
 
This tool helps you fill in the place attributes like county, country, ..., by allowing you to select the places you work on, and do changes on all these places with one button click.
  
 
The general aims are:
 
The general aims are:
*Place/Location is a newer concept in GRAMPS. Many older databases only have a Place title field which is a descriptive text containing city, state, country. This should be parsed to insert the values in the correct attribute fields.
+
*Place/Location is a newer concept in Gramps. Many older databases only have a Place title field which is a descriptive text containing city, state, country. This should be parsed to insert the values in the correct attribute fields.
 
*Latitude and longitude are important to show data on a map. However, doing a look-up of this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.
 
*Latitude and longitude are important to show data on a map. However, doing a look-up of this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.
*Setting of an attribute of a set of places in one go. Eg you give a
+
*Setting of an attribute of a set of places in one go.
*Conversion of latitude and longitude to a fixed data format. On import one might obtain latitude and longitude in several different formats. A conversion tool to store them all in the same format is usefull.
+
*Conversion of latitude and longitude to a fixed data format. On import one might obtain latitude and longitude in several different formats. A conversion tool to store them all in the same format is useful.
*Construction of a uniform title/description field, from the data in the place object
+
*Construction of a uniform title/description field, from the data in the place object.
  
== Design Specification ==
+
Follow the installation details in the [[Place_completion_tool#Download|Download]] section for your version of Gramp.
See [[Place completion tool specification]]
 
Place/Location is a newer concept in GRAMPS. Many older databases only have a Place title field which is a descriptive text containing city, state, country. To distribute these values into the correct attribute fields, see Parse Title Details below
 
  
== Manual ==
+
== Usage Instructions ==
The place completion tool gives a lot of functionality. This manual should help you to understand how it works.
+
The place completion tool provides a lot of functionality. These usage instruction should help you to understand how it works.
  
 
=== Download resources ===
 
=== Download resources ===
The place completion tool can look up for you latitude/longitude, add county information (USA), ... . For some of this functionality, you must download datafiles of the countries you are interested in. Right now you have three options:
+
{{stub}}<!--page needs to be reworked and maybe split-->
 +
The place completion tool can look up for you latitude/longitude, add county information (USA), ... . For some of this functionality, you must download data files of the countries you are interested in. Right now you have three options:
  
#Download geonames country files. You can do this [http://download.geonames.org/export/dump/ here freely]. Geonames parses fastest, so is the advised format to use
+
#Download geonames country files. You can do this [http://download.geonames.org/export/dump/ here freely]. {{man menu|Geonames parses fastest, so is the advised format to use.}}
 
#Download geonames USA state files. You can do this [http://geonames.usgs.gov/domestic/download_data.htm here freely]. This is advised for USA searches, as the data in the USA country file contains many doubles, which can be avoided by searching state per state. State info also contains county information.
 
#Download geonames USA state files. You can do this [http://geonames.usgs.gov/domestic/download_data.htm here freely]. This is advised for USA searches, as the data in the USA country file contains many doubles, which can be avoided by searching state per state. State info also contains county information.
#download GNS Geonet country files (not available for usa). You can do this [ftp://ftp.nga.mil/pub/gns_data here freely with ftp].  
+
#Download GNS Geonet country files (not available for usa). You can do this [ftp://ftp.nga.mil/pub2/gns_data here freely with ftp].  
  
Watch out, some of these downloads are '''VERY''' large, especially USA data. Only download what you need!
+
Watch out, some of these downloads are '''VERY''' large, especially USA data. Only download what you need! If the download is a compressed zip file, you will need to extract the data file before you can use it.
  
<small>'''Note''': The geonames data of popular places is in English, so eg municipalities in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name (see below)</small>
+
{{man note|Note:|The geonames data of popular places is in English, so e.g. municipalities in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name (see below)}}
  
 
{{man warn|Warning:|DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP.}}
 
{{man warn|Warning:|DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP.}}
  
 
=== Starting the tool ===
 
=== Starting the tool ===
You will find the plugin under 'Tools > Utilities > Interactive place completion'
+
You will find the plugin under {{man menu|Tools > Utilities > PlaceCompletion...}}
  
 
=== The dialog explained ===
 
=== The dialog explained ===
[[Image:place_completion_tool_empty.png|500px|The Place Completion Tool]]
+
[[File:Place completion tool empty.png|600px|The Place Completion Tool]]
  
 
The Dialog consists of 4 parts:
 
The Dialog consists of 4 parts:
====Part 1: selection of places====
+
====Part 1. Selection of places====
 
First you need to choose with which places you want to work. You can use several methods to define your places:
 
First you need to choose with which places you want to work. You can use several methods to define your places:
 
# Use a place filter. You can use two preset filters: ''All places'', which returns all places, and ''No Latitude/Longitude given'', which returns all places of which the latitude or the longitude is not set. You can also created a custom place filter in the place view, test it with the filter sidebar, and then use it in this tool. All custom filters you made will be available
 
# Use a place filter. You can use two preset filters: ''All places'', which returns all places, and ''No Latitude/Longitude given'', which returns all places of which the latitude or the longitude is not set. You can also created a custom place filter in the place view, test it with the filter sidebar, and then use it in this tool. All custom filters you made will be available
Line 51: Line 46:
 
# Use a latitude, longitude rectangle. Eg, suppose you have the latitude and longitude of all places in the UK, and now want to add in the state attribute ''Wales'', for all places in Wales. You can look on a map, note down the centre of Wales in latitude and longitude, as well as roughly the width and height of this rectangle. This will allow you to obtain all places in Wales (and some in England), allowing to much faster set the state information.
 
# Use a latitude, longitude rectangle. Eg, suppose you have the latitude and longitude of all places in the UK, and now want to add in the state attribute ''Wales'', for all places in Wales. You can look on a map, note down the centre of Wales in latitude and longitude, as well as roughly the width and height of this rectangle. This will allow you to obtain all places in Wales (and some in England), allowing to much faster set the state information.
  
====Part 2. Completion of Places ====
+
====Part 2. Completion of places ====
 
#The first possibility is to look up in a datafile the latitude and longitude of your places. For this you must have downloaded the necessary resources, see section above. You can select with a file dialog the file you want to search, and set how this data must be parsed. The following parsing options are available:
 
#The first possibility is to look up in a datafile the latitude and longitude of your places. For this you must have downloaded the necessary resources, see section above. You can select with a file dialog the file you want to search, and set how this data must be parsed. The following parsing options are available:
 
##''GeoNames country file, city search'': use the city attribute to look for lat/lon in a GeoNames country file. This is the fastest search.
 
##''GeoNames country file, city search'': use the city attribute to look for lat/lon in a GeoNames country file. This is the fastest search.
Line 67: Line 62:
 
###All in degree notation: use the classical degree notation with degree, minutes and seconds.
 
###All in degree notation: use the classical degree notation with degree, minutes and seconds.
 
###All in decimal notation: use the decimal system to denote lat/lon.
 
###All in decimal notation: use the decimal system to denote lat/lon.
###Correct -50° in 50°S: a much seen error is to use - for the classical degree notation, which is wrong, and which GRAMPS will not be able to interpret. With this option this error is looked for and corrected.
+
###Correct -50° in 50°S: a much seen error is to use - for the classical degree notation, which is wrong, and which Gramps will not be able to interpret. With this option this error is looked for and corrected.
# A third option is to set attributes of all selected places. You can set the country, state, county, parish, zip/postal code and city attributes of all places in one sweep.
 
  
 
====Part 3. Overview of the results ====  
 
====Part 3. Overview of the results ====  
After having entered all data in Part 1 and 2, you click find for GRAMPS to search all changes that will occur. This part of the dialog shows all changes that will occur.  
+
After having entered all data in Part 1 and 2, you click find for Gramps to search all changes that will occur. This part of the dialog shows all changes that will occur.  
  
 
[[Image:place_completion_tool_results.png]]
 
[[Image:place_completion_tool_results.png]]
Line 79: Line 73:
 
If the change will '''overwrite''' an existing entry, the subentry is '''shown in orange'''.  
 
If the change will '''overwrite''' an existing entry, the subentry is '''shown in orange'''.  
  
 
+
{{man warn|Warning|TO AVOID PROBLEMS, GO OVER ALL CHANGES QUICKLY, AND CHECK ALL ENTRIES IN ORANGE!}}
'''TO AVOID PROBLEMS, GO OVER ALL CHANGES QUICKLY, AND CHECK ALL ENTRIES IN ORANGE!'''
 
 
 
  
 
The following actions are possible in the result screen:
 
The following actions are possible in the result screen:
Line 89: Line 81:
  
 
====Part 4. Actions ====  
 
====Part 4. Actions ====  
After you have checked the changes in Part 3, you can now apply them all with one button click, by clicking the ''Apply'' button.  
+
After you have checked the changes in '''Part 3''', you can apply them by clicking the {{man button|Apply}} button.  
  
 
+
Selecting {{man button|Help}} will bring you to this page, clicking {{man button|Close}} will close the ''Place completion tool'' window and clicking {{man button|Google Maps} when an entry is selected in the results field has the same effect as pressing tab on an entry (see above).
Clicking ''Help'' will bring you to this page, clicking ''Close'' will close the window and clicking ''Google Maps'' when an entry is selected in the results field has the same effect as pressing tab on an entry (see above).
 
  
 
== Example ==  
 
== Example ==  
  
Open the example file from the examples where latitude and longitude are empty: [http://gramps.svn.sourceforge.net/viewvc/*checkout*/gramps/trunk/gramps2/example/gramps/example.gramps?revision=7186 example.gramps].  
+
Open the example file from the examples where latitude and longitude are empty: [http://svn.code.sf.net/p/gramps/code/branches/maintenance/gramps34/example/gramps/example.gramps example.gramps].  
  
We will now show how the places in this file can be completed. The best thing to do is open a new family tree (.grdb), give it a name, and import the example.gramps file. This file has 852 places, which would mean a lot of manual edits if you do not use this tool!
+
We will now show how the places in this file can be completed. The best thing to do is to create a new Family Tree, give it a name, and import the example.gramps file. This file has 852 places, which would mean a lot of manual edits if you do not use this tool!
  
 
Now, open the place view. You will see all places are of the form:
 
Now, open the place view. You will see all places are of the form:
Line 108: Line 99:
  
 
We open the ''Place completion tool'':
 
We open the ''Place completion tool'':
[[Image:place_completion_tool_example1.png|500px|Parse the Place Name Field]]
+
[[File:Place completion tool example1.png|500px|left|Parse the Place Name Field]]
Here we have selected ''All Places'', and we parse the title as ''City [,|.] State''. Click on ''Find'', quickly scan the data if all looks ok, and then click on ''Apply''. You are notified that 851 places are updated. This is one less that the number of places. Indeed, one place does have a different type of title: ''Puerto Rico'' has no state information.
+
Here we have selected {{man label|All Places}}, and we '''Change the title into''' as {{man label|City [,State]}}. Click on {{man button|Find}}, quickly scan the data if all looks ok, and then click on {{man button|Apply}}. You are notified that 443 place records were modified. This is one less that the number of places. Indeed, one place does have a different type of title: ''Puerto Rico'' has no state information.
 +
 
 +
{{-}}
  
 
=== Step 2: Look-up latitude and longitude ===
 
=== Step 2: Look-up latitude and longitude ===
 
We have downloaded the GeoNames datafiles for the USA states, and will now use that to complete the latitude and longitude of the data. At the same time, this will fill up the county field.
 
We have downloaded the GeoNames datafiles for the USA states, and will now use that to complete the latitude and longitude of the data. At the same time, this will fill up the county field.
  
[[Image:place_completion_tool_example2.png|500px|Look up lat/lon for Alaska]]
+
[[Image:place_completion_tool_example2.png|500px|left|Look up lat/lon for Alaska]]
 
 
In the above screenshot, you see we have selected All Places with State=AK. In the second part of the window we give that we want to search in the AK_DECI.txt file downloaded from GeoNames, using the parsing method: ''GeoNames USA state file, city search''.
 
 
 
Note that if you want to change AK into Alaska, this would be possble. Just set state=Alaska in the set attributes section of the window.
 
  
 +
In the screenshot, you see we have selected All Places with State=AK. In the second part of the window we give that we want to search in the AK_DECI.txt file downloaded from GeoNames, using the parsing method: ''GeoNames USA state file, city search''.
 +
{{-}}
 +
Note that if you want to change AK into Alaska, this would be possible. Just set state=Alaska in the set attributes section of the window.
  
 
Do this now for all the states. Always check for doubles. Eg, for state ''AL'', going over the changes, we encounter:
 
Do this now for all the states. Always check for doubles. Eg, for state ''AL'', going over the changes, we encounter:
Line 133: Line 125:
 
In case google maps did not allow you to determine which is the correct city, you can double click on the city to open the Place Dialog ('''Warning: this will preenter the data of the Place Completion tool. So hit cancel here if you want to exit without these changes done'''). In this dialog the references tab allows you to navigate to all events coupled to this place. This will give you extra information you might use to decide which of the two found places is the correct place.
 
In case google maps did not allow you to determine which is the correct city, you can double click on the city to open the Place Dialog ('''Warning: this will preenter the data of the Place Completion tool. So hit cancel here if you want to exit without these changes done'''). In this dialog the references tab allows you to navigate to all events coupled to this place. This will give you extra information you might use to decide which of the two found places is the correct place.
  
=== Step 3: Problem entries===
+
=== Step 3: Problem entries ===
 
While updating all places in step 2, you will have noticed some errors in the state information: Some places have a dubious state: eg OH-AL
 
While updating all places in step 2, you will have noticed some errors in the state information: Some places have a dubious state: eg OH-AL
  
You can obtain these states by choosing ''All Places'' en setting the state search box to '''-'''. Clicking Find will give you all these problem places. You can use google maps or the place dialog to sort them out. You can also use the USA country GeoNames file to search these places in the entire USA. You will need sufficient memory for this, or you will obtain a MemoryError (see below)!
+
You can obtain these states by choosing ''All Places'' en setting the state search box to '''-'''. Clicking Find will give you all these problem places. You can use google maps or the place dialog to sort them out. You can also use the USA country GeoNames file to search these places in the entire USA.  
 +
 
 +
{{man note|Note:|You will need sufficient memory for this, or you will obtain a MemoryError (see [[Place_completion_tool#Memory_Error|below]])!}}
  
 
=== Step 4: Lat/Lon not found ===
 
=== Step 4: Lat/Lon not found ===
Line 144: Line 138:
 
It will be clear that many of those can be quickly corrected: abbreviations, eg the city field contains ''St.George'', which should be ''Saint George''; double names, eg Waterloo-Cedar Falls, IA means Waterloo near Cedar Falls, changing the city to Waterloo and redoing the search using Google Maps will allow to quickly find which coordinates for Waterloo are needed.
 
It will be clear that many of those can be quickly corrected: abbreviations, eg the city field contains ''St.George'', which should be ''Saint George''; double names, eg Waterloo-Cedar Falls, IA means Waterloo near Cedar Falls, changing the city to Waterloo and redoing the search using Google Maps will allow to quickly find which coordinates for Waterloo are needed.
  
== Advanced Usage ==
+
== Advanced usage ==
This is for advanced users only knowing regular expressions.  
+
 
 +
 
 +
This is for advanced users who know how to use [http://en.wikipedia.org/wiki/Regular_expression regular expressions].  
  
 
The parsing fields have entry fields allowing you to give your own parsing. Parsing uses regular expressions. You can use this to parse your title, and to parse a lat/lon file in your own way. For reference, here an overview of the parsing codes used for the predefined parses:
 
The parsing fields have entry fields allowing you to give your own parsing. Parsing uses regular expressions. You can use this to parse your title, and to parse a lat/lon file in your own way. For reference, here an overview of the parsing codes used for the predefined parses:
 +
 
=== Parse title details ===
 
=== Parse title details ===
 +
 +
 
In 'Parsing and Conversion of existing title or position', 'Parse title' and 'Change title into' provide some pre-defined options for extracting location values from a Place Title. Otherwise regex parsing is needed.
 
In 'Parsing and Conversion of existing title or position', 'Parse title' and 'Change title into' provide some pre-defined options for extracting location values from a Place Title. Otherwise regex parsing is needed.
  
 
Write your regex in 'Parse title:'.Click on 'Find', which shows the proposed changes. Then click 'Apply'.
 
Write your regex in 'Parse title:'.Click on 'Find', which shows the proposed changes. Then click 'Apply'.
  
In many 16th and 17th century English IGI records the Place Title takes the form of 3 strings (Town, County, Country) or 4 strings (Parish, District, County, Country) for example:
+
====Example 1====
 +
 
 +
For France, some [http://www.geneawiki.com/index.php/Informatique_-_saisie_des_lieux practical rules] could be useful for seizing place. It needs :
 +
 +
* the city name + [http://en.wikipedia.org/wiki/INSEE_code#Geographical_codes INSEE code] (at option). This code is unique and can identify with certainty a common (with the county, district, township and municipality). It can identify with a common insurance even if it has changed its name. This code is used in Archives. ''Using postal code is not advisable ...''
 +
 +
* a subdivision: identifies a parish or a place called within a municipality
 +
 +
* the state (at option) ''or county but is already on [http://en.wikipedia.org/wiki/INSEE_code#Geographical_codes INSEE code]''
 +
 +
* the country (at option). Ideally it should still take the country. It is understandable that this is tedious. ''Maybe do not enter the country if the genealogy is mostly of one country and seize enter the country for events outside the country's main. Everyone will appreciate.''
 +
 
 +
e.g. Avignon,84000,Vaucluse,Provence-Cote-d'Azur,FRANCE,
 +
 
 +
where some fields may be missing:
 +
 
 +
e.g. Woerth,,,Alsace,FRANCE,
 +
 
 +
the regex:
 +
 
 +
<code>\s*(?P<city>[^,]+)[,]\s*(?P<zip>\d*)[,](?P<county>[^,]*)[,](?P<state>[^,]*)[,](?P<country>[^,]*)[,]*$</code>
 +
 
 +
treating the comma character as an end-of-string delimiter, will distribute
 +
Avignon to City, 84000 to ZIP, Vaucluse to County, Province-Cote-d'Azur to State and France to Country
 +
 
 +
and in the case of missing fields, as long as the first is not empty, will distribute Woerth to City, Alsace to State and FRANCE to Country.
 +
 
 +
It allows initial whitespace and an optional comma after the Country
 +
 +
====Example 2====
 +
 
 +
In many 16th and 17th century English IGI records the situation is more complicated.
 +
 
 +
The Place Title takes the form of 3 strings (Town, County, Country) or 4 strings (Parish, District, County, Country) for example:
  
 
(a) Chester le Street, Durham, England - 3-string
 
(a) Chester le Street, Durham, England - 3-string
Line 168: Line 200:
 
Regex A:
 
Regex A:
  
\s*(Of[,]*\s*)*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$
+
<code>\s*(Of[,]*\s*)*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$</code>
  
  
Line 175: Line 207:
 
Regex B:
 
Regex B:
  
\s*(Of[,]*\s*)*(?P<parish>[^,]+?)[,]\s*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$
+
<code>\s*(Of[,]*\s*)*(?P<parish>[^,]+?)[,]\s*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$</code>
  
  
Line 186: Line 218:
 
To avoid this, when using Regex B click on 'Find' to display the records proposed for change and delete all the 3-string 'Of,' records before clicking 'Apply'.
 
To avoid this, when using Regex B click on 'Find' to display the records proposed for change and delete all the 3-string 'Of,' records before clicking 'Apply'.
  
 
+
{{man note|take care when pasting a Regex (for example, after testing in a regex editor) into the Parse title details field. Pasting adds redundant spaces before or after the Regex that will prevent it working properly in the Place Completion Tool}}
  
 
The predefined regex expressions are as follows, where for brevity we use some variables defined lower.
 
The predefined regex expressions are as follows, where for brevity we use some variables defined lower.
  
Note: For those new to Python and Regex please review the HOWTO here:
+
{{man note|Regex Help|For those new to Python and Regex please review the HOWTO here:<br /> http://docs.python.org/dev/howto/regex.html}}
 
 
      http://docs.python.org/dev/howto/regex.html
 
 
   
 
   
 
+
#"City [,|.] State" is parsed by : <code>r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+state_translated +r'>.+?)\s*$'</code>
#<nowiki>"City [,|.] State" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+state_translated +r'>.+?)\s*$'</nowiki>
+
#"City [,|.] Country" is parsed by : <code>r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+country_translated +r'>.+?)\s*$'</code>
#<nowiki>"City [,|.] Country" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+country_translated +r'>.+?)\s*$'</nowiki>
+
#"City (Country)" is parsed by : <code>r'\s*(?P<'+city_translated +r'>.*?)\s*\(\s*(?P<'+country_translated +r'>[^\)]+)\s*\)\s*$'</code>
#<nowiki>"City (Country)" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*\(\s*(?P<'+country_translated +r'>[^\)]+)\s*\)\s*$'</nowiki>
+
#"City" is parsed by : <code>r'\s*(?P<'+city_translated +r'>.*?)\s*$'</code>
#<nowiki>"City" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*$'</nowiki>
 
  
 
Here the variables used are:  
 
Here the variables used are:  
 +
<pre>
 
lat_translated = _('lat')
 
lat_translated = _('lat')
 
lon_translated = _('lon')
 
lon_translated = _('lon')
Line 207: Line 237:
 
state_translated = _('state')
 
state_translated = _('state')
 
country_translated = _('country')
 
country_translated = _('country')
 +
</pre>
  
 
You can use one of these variables as a group, and the tool will recognise them, and use as values for the corresponding place attributes.
 
You can use one of these variables as a group, and the tool will recognise them, and use as values for the corresponding place attributes.
Line 213: Line 244:
 
For the regex of lat/lon lookup, you need to indicate which data must be replaced with existing place attributes for the search, as well as indicate which regex groups must be extracted.  
 
For the regex of lat/lon lookup, you need to indicate which data must be replaced with existing place attributes for the search, as well as indicate which regex groups must be extracted.  
  
#<nowiki>"GeoNames country file, city search" is parsed with: r'\t'+CITY_transl +r'\t[^\t]*\t[^\t]*\t' +latgr +  r'[\d+-][^\t]*)\t' +  longr + r'[\d+-][^\t]*)\tP'</nowiki>
+
#"GeoNames country file, city search" is parsed with: <code>r'\t'+CITY_transl +r'\t[^\t]*\t[^\t]*\t' +latgr +  r'[\d+-][^\t]*)\t' +  longr + r'[\d+-][^\t]*)\tP'</code>
#<nowiki>"GeoNames country file, city localized variants search" is parsed with: r'[\t,]'+CITY_transl+r'[,\t][^\t\d]*\t?' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\tP'</nowiki>
+
#"GeoNames country file, city localized variants search" is parsed with: <code>r'[\t,]'+CITY_transl+r'[,\t][^\t\d]*\t?' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\tP'</code>
#<nowiki>"GeoNames country file, title begin, general search" is parsed with: r'\t'+TITLEBEGIN_transl +r'\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\t[PSTV]'</nowiki>
+
#"GeoNames country file, title begin, general search" is parsed with: <code>r'\t'+TITLEBEGIN_transl +r'\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\t[PSTV]'</code>
#<nowiki>"GeoNames USA state file, city search" is parsed with: r'\t'+CITY_transl+r'\tPopulated Place\t[^\t]*\t[^\t]*\t' + countygr + r'[^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)'</nowiki>
+
#"GeoNames USA state file, city search" is parsed with: <code>r'\t'+CITY_transl+r'\tPopulated Place\t[^\t]*\t[^\t]*\t' + countygr + r'[^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)'</code>
#<nowiki>"GNS Geonet country file, city search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\tP\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+CITY_transl+r'\t[^\t]*\t[^\t\n]+$'</nowiki>
+
#"GNS Geonet country file, city search" is parsed with: <code>r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\tP\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+CITY_transl+r'\t[^\t]*\t[^\t\n]+$'</code>
#<nowiki>"GNS Geonet country file, title begin search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[PLSTV]\t[^\t]*\t[^\t]*'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+TITLEBEGIN_transl+r'\t[^\t]*\t[^\t\n]+$'</nowiki>
+
#"GNS Geonet country file, title begin search" is parsed with: <code>r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[PLSTV]\t[^\t]*\t[^\t]*'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+TITLEBEGIN_transl+r'\t[^\t]*\t[^\t\n]+$'</code>
#Read of mediawiki CSV dump. This reads the files on [http://tools.wikimedia.de/~kolossos/wp-world/pub_CSV_test3.csv.gz] (for more information, see http://meta.wikimedia.org/wiki/WikiProjects_Geographical_coordinates) (''Contribution by nomeata'')
+
#<i>Read of mediawiki CSV dump. This reads the files on [http://tools.wikimedia.de/~kolossos/wp-world/pub_CSV_test3.csv.gz](link gone) (for more information, see http://meta.wikimedia.org/wiki/WikiProjects_Geographical_coordinates) (''Contribution by nomeata'')</i>
  
For extraction of data you can use the same groupnames as in title parsing, so eg latgr in above should read: <nowiki>r'(?P<'+lat_translated +r'>'</nowiki> .
+
For extraction of data you can use the same groupnames as in title parsing, so eg latgr in above should read: <code>r'(?P<'+lat_translated +r'>'</code> .
  
 
The syntax for the values that need to be used for searching in the file, eg CITY_transl, is given by : _('CITY'). You can use as substitution values:
 
The syntax for the values that need to be used for searching in the file, eg CITY_transl, is given by : _('CITY'). You can use as substitution values:
Line 228: Line 259:
 
The tool will read in the given regex, replace the substitution strings by the values in the place object, do the search, and extract the regex groups given from the result.
 
The tool will read in the given regex, replace the substitution strings by the values in the place object, do the search, and extract the regex groups given from the result.
  
Resource: [http://de.wikipedia.org/wiki/Wikipedia:GEOnet_Names_Server GEOnet]
+
Resource: [https://en.wikipedia.org/wiki/GEOnet_Names_Server GEOnet Names Server]
  
 
== Troubleshooting ==
 
== Troubleshooting ==
Line 234: Line 265:
 
The place completion tool expects the input files for location lookup to be in unicode (utf-8). On the occasion this is not the case, you will get the error:
 
The place completion tool expects the input files for location lookup to be in unicode (utf-8). On the occasion this is not the case, you will get the error:
 
    
 
    
<code><nowiki>File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletion.py", line 851, in load_latlon_file
+
<pre>File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletion.py", line 851, in load_latlon_file
 
     self.latlonfile_datastr = infile.read()
 
     self.latlonfile_datastr = infile.read()
 
   File "/usr/lib/python2.4/codecs.py", line 481, in read
 
   File "/usr/lib/python2.4/codecs.py", line 481, in read
Line 240: Line 271:
 
   File "/usr/lib/python2.4/codecs.py", line 293, in read
 
   File "/usr/lib/python2.4/codecs.py", line 293, in read
 
     newchars, decodedbytes = self.decode(data, self.errors)
 
     newchars, decodedbytes = self.decode(data, self.errors)
  UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1610092-1610094: invalid data</nowiki></code>  
+
  UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1610092-1610094: invalid data</pre>
  
 
Note that the Place Completion tool catches these errors and shows you an information box. After this, the tool will attempt to read the file with utf-8 (unicode), ignoring errors. This might give good results, but will of course fail to produce results on non-unicode encoded files.  
 
Note that the Place Completion tool catches these errors and shows you an information box. After this, the tool will attempt to read the file with utf-8 (unicode), ignoring errors. This might give good results, but will of course fail to produce results on non-unicode encoded files.  
Line 258: Line 289:
 
  MemoryError</nowiki>
 
  MemoryError</nowiki>
  
The tool has to load the datafile for latitude/longitude searching into memory. For large files like USA.txt, this might be impossible if you have limited memory. You can try to close as many programs running at together with GRAMPS, and try the tool again.
+
The tool has to load the datafile for latitude/longitude searching into memory. For large files like USA.txt, this might be impossible if you have limited memory. You can try to close as many programs running at together with Gramps, and try the tool again.
 
 
  
== Download ==
+
==See also==
  
If you use Gramps 3.2 , then you will need [http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/branches/gramps32/download/PlaceCompletion.addon.tgz this addon].
+
*[[GeoCodes]]
  
If you use Gramps 3.1.x, then you will need version 1.2 of the Place Completion Tool. You can find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_2.tar.gz placecompletion_1_2.tar.gz].
+
=== Design specification ===
 +
*See [[Place completion tool specification]]
  
If you use Gramps 3.0.x, then you will need version 1.1 of the Place Completion Tool. You find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_1.tar.gz placecompletion_1_1.tar.gz].
+
Place/Location is a newer concept in Gramps. Many older databases only have a Place title field which is a descriptive text containing city, state, country. To distribute these values into the correct attribute fields, see Parse Title Details below
  
If you use Gramps 2.2.5+, then you will need version 1.0 of the Place Completion Tool. You find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_0.tar.gz placecompletion_1_0.tar.gz].
+
== Download ==
 
 
Extract the two files that are in the download. Put the .glade and .py file both in the plugins directory. For linux:
 
  
* local install: place in <code>~/.gramps/plugins</code>
+
If you use Gramps 4.2 , then use the [[4.2_Addons|automatic installation]].
* global install: <code>place_to_gramps_install/src/plugins</code>
 
  
{{man warn|Warning:|DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP, THEN RUN THE TOOL}}
+
If you use Gramps 4.1 , then use the [[4.1_Addons|automatic installation]].
  
== Patches ==
+
If you use Gramps 3.4 , then use the [[3.4_Addons|automatic installation]].
  
 +
If you use Gramps 3.3 , then use the [[3.3_Addons|automatic installation]].
  
===Non-english locale ===
+
If you use Gramps 3.2 , then use the [[3.2_Addons|automatic installation]].
PlaceCompletion 1.2 does not work correctly if your environment locale is not english (problem identified with locale set to fr_FR.UTF-8).
 
To solve this issue, you may apply the following patch:
 
  
<pre>
+
=== Manual install ===
--- ../../Documents/genealogie/plugins/placecompletion_1_2/PlaceCompletion.py 2009-02-11 23:04:27.000000000 +0100
 
+++ ./PlaceCompletion.py 2009-11-22 13:50:35.000000000 +0100
 
@@ -56,10 +56,18 @@
 
 
from Filters.Rules.Place import *
 
import Utils
 
-from QuestionDialog import OkDialog, WarningDialog
 
+from QuestionDialog import OkDialog, WarningDialog, ErrorDialog
 
import PlaceUtils
 
import Errors
 
 
+if hasattr(Utils, "ProgressMeter"):
 
+    ProgressMeter = Utils.ProgressMeter
 
+else:
 
+    import gui.utils
 
+    if hasattr(gui.utils, "ProgressMeter"):
 
+        ProgressMeter = gui.utils.ProgressMeter
 
+    else:
 
+        raise ImportError("can't find ProgressMeter")
 
#------------------------------------------------------------------------
 
#
 
#
 
@@ -177,8 +185,8 @@
 
        self.placechecklist = [self.find_latlon, self.match_regex_title,
 
                    self.construct_title, self.convert_latlon, self.set_data
 
                    ]
 
-        self.regextitlegroups = [_('street'), _('city'), _('parish'),
 
-                    _('county'), _('state'), _('country'), _('zip'),_('title')]
 
+        self.regextitlegroups = [('street'), ('city'), ('parish'),
 
+                    ('county'), ('state'), ('country'), ('zip'), ('title')]
 
 
        self.glade.signal_autoconnect({
 
            "destroy_passed_object" : self.close,
 
@@ -198,29 +206,29 @@
 
        ''' Get the value corresponding to a group
 
            The special groupname #latlon is allowed, returning lat/lon
 
        '''
 
-        if group == _('title') :
 
+        if group == ('title') :
 
            return place.get_title()
 
-        elif group == _('latitude') :
 
+        elif group == ('latitude') :
 
            return place.get_latitude()
 
-        elif group == _('longitude') :
 
+        elif group == ('longitude') :
 
            return place.get_longitude()
 
        elif group == '#latlon' :
 
            return (place.get_latitude() , place.get_longitude())
 
        else :
 
            loc = place.get_main_location()
 
-            if group == _('city') :
 
+            if group == ('city') :
 
                return loc.get_city()
 
-            elif group == _('country') :
 
+            elif group == ('country') :
 
                return loc.get_country()
 
-            elif group == _('county') :
 
+            elif group == ('county') :
 
                return loc.get_county()
 
-            elif group == _('state') :
 
+            elif group == ('state') :
 
                return loc.get_state()
 
-            elif group == _('street') :
 
+            elif group == ('street') :
 
                return loc.get_street()
 
-            elif group == _('parish') :
 
+            elif group == ('parish') :
 
                return loc.get_parish()
 
-            elif group == _('zip') :
 
+            elif group == ('zip') :
 
                return loc.get_postal_code()
 
            else :
 
                ErrorDialog(_("Error in PlaceCompletion.py"),
 
@@ -234,11 +242,11 @@
 
        '''
 
        if not group :
 
            return place
 
-        if group == _('title') :
 
+        if group == ('title') :
 
            place.set_title(val)
 
-        elif group == _('latitude') :
 
+        elif group == ('latitude') :
 
            place.set_latitude(val)
 
-        elif group == _('longitude') :
 
+        elif group == ('longitude') :
 
            place.set_longitude(val)
 
        elif group == '#latlon' :
 
            place.set_latitude(val[0])
 
@@ -247,20 +255,23 @@
 
            loc = place.get_main_location()
 
            if loc == None :
 
                loc = Location()
 
-            if group == _('city') :
 
+            if group == ('city') :
 
                loc.set_city(val)
 
-            elif group == _('country') :
 
+            elif group == ('country') :
 
                loc.set_country(val)
 
-            elif group == _('county') :
 
+            elif group == ('county') :
 
                loc.set_county(val)
 
-            elif group == _('state') :
 
+            elif group == ('state') :
 
                loc.set_state(val)
 
-            elif group == _('street') :
 
+            elif group == ('street') :
 
                loc.set_street(val)
 
-            elif group == _('parish') :
 
+            elif group == ('parish') :
 
                loc.set_parish(val)
 
-            elif group == _('zip') :
 
+            elif group == ('zip') :
 
                loc.set_postal_code(val)
 
+            elif group == ('title') :
 
+                loc.set_title(val)
 
+
 
            else :
 
                ErrorDialog(_("Error in PlaceCompletion.py"),
 
                        _("Non existing group used in set"))
 
@@ -421,12 +432,12 @@
 
        # Compile Regex file search partially
 
        self.matchlatlon = None
 
        self.extrafindgroup = []  #find always finds lat/lon, what extra groups?
 
-        possibleextrafindgroups=[_('county'), _('state')]
 
+        possibleextrafindgroups=[('county'), ('state')]
 
        if findregex :
 
            try:
 
                #compile regex aware of locale and unicode
 
                self.matchlatlon = re.compile(findregex,re.U|re.L|re.M)
 
-                latlongroup = [_('lat'),_('lon')]
 
+                latlongroup = [('lat'),('lon')]
 
                for group in latlongroup :
 
                    if findregex.find(r'(?P<'+group+r'>') == -1 :
 
                        WarningDialog(_('Missing regex groups in match lat/lon'),
 
@@ -498,8 +509,8 @@
 
       
 
        self.nrplaces_in_tree = len(ind_list)
 
        # Populating might take a while, add a progress bar
 
-        progress = Utils.ProgressMeter(
 
-                                _("Finding Places and appropriate changes"),'')
 
+        progress = ProgressMeter(
 
+            _("Finding Places and appropriate changes"),'')
 
        #apply the filter
 
        progress.set_pass(_('Filtering'),1)
 
        progress.step()
 
@@ -559,7 +570,7 @@
 
            # a line of text, do nothing
 
            newval = 'no change'
 
        if groupname == '#latlon' :
 
-            self.model.set(id, 0, _('lat')+r'/'+_('lon') + ' :' \
 
+            self.model.set(id, 0, ('lat')+r'/'+('lon') + ' :' \
 
                                + olddata
 
                                + ' -> ' + newval[0]+r'/'+newval[1])
 
        else :
 
@@ -704,7 +715,7 @@
 
 
        self.trans = self.db.transaction_begin("",batch=True)
 
        self.db.disable_signals()
 
-        progress = Utils.ProgressMeter(_('Doing Place changes'),'')
 
+        progress = ProgressMeter(_('Doing Place changes'),'')
 
        #we do not know how many places in the treeview, and counting would
 
        # mean transversing the tree. Set the progress to the possible maximum
 
        progress.set_pass('',self.db.get_number_of_places())
 
@@ -822,26 +833,26 @@
 
        if len(filename) == 0:
 
            return True
 
        elif os.path.isdir(filename):
 
-            QuestionDialog.ErrorDialog(
 
+            ErrorDialog(
 
                _('Cannot open file'),
 
                _('The selected file is a directory, not '
 
                  'a file.'))
 
            return True
 
        elif os.path.exists(filename):
 
            if not os.access(filename, os.R_OK):
 
-                QuestionDialog.ErrorDialog(
 
+                ErrorDialog(
 
                    _('Cannot open file'),
 
                    _('You do not have read access to the selected '
 
                      'file.'))
 
                return True
 
            elif not stat.S_ISREG(os.stat(filename)[stat.ST_MODE]):
 
-                QuestionDialog.ErrorDialog(
 
+                ErrorDialog(
 
                    _('Cannot open file'),
 
                    _('The file you want to access is not a regular file.'))
 
                return True
 
        else :
 
            # file does not exist
 
-            QuestionDialog.ErrorDialog(
 
+            ErrorDialog(
 
                    _('Cannot open file'),
 
                    _('The file does not exist.'))
 
            return True
 
@@ -892,36 +903,36 @@
 
        # we need to lookup the latitude and longitude, construct regex:
 
        pattern = self.matchlatlon.pattern
 
        loc = place.get_main_location()
 
-        if re.search(_('CITY'),pattern) :
 
+        if re.search(('CITY'),pattern) :
 
            if loc.get_city().strip() == '' :
 
                return valoud, valnew, valaction, place
 
-            pattern = re.sub(_('CITY'), loc.get_city().strip(), pattern)
 
-        if re.search(_('TITLEBEGIN'),pattern) :
 
+            pattern = re.sub(('CITY'), loc.get_city().strip(), pattern)
 
+        if re.search(('TITLEBEGIN'),pattern) :
 
            tit = place.get_title().strip()
 
            titb= tit.split(',')[0].strip()
 
            if titb == '' :
 
                return valoud, valnew, valaction, place
 
-            pattern = re.sub(_('TITLEBEGIN'), titb, pattern)
 
-        if re.search(_('TITLE'),pattern) :
 
+            pattern = re.sub(('TITLEBEGIN'), titb, pattern)
 
+        if re.search(('TITLE'),pattern) :
 
            if place.get_title().strip() == '' :
 
                return valoud, valnew, valaction, place
 
-            pattern = re.sub(_('TITLE'), loc.get_title().strip(), pattern)
 
-        if re.search(_('STATE'),pattern) :
 
+            pattern = re.sub(('TITLE'), loc.get_title().strip(), pattern)
 
+        if re.search(('STATE'),pattern) :
 
            if place.get_state().strip() == '' :
 
                return valoud, valnew, valaction, place
 
-            pattern = re.sub(_('STATE'), loc.get_state().strip(), pattern)
 
-        if re.search(_('PARISH'),pattern) :
 
+            pattern = re.sub(('STATE'), loc.get_state().strip(), pattern)
 
+        if re.search(('PARISH'),pattern) :
 
            if loc.get_parish().strip() == '' :
 
                return valoud, valnew, valaction, place
 
-            pattern = re.sub(_('PARISH'), loc.get_parish().strip(), pattern)
 
+            pattern = re.sub(('PARISH'), loc.get_parish().strip(), pattern)
 
        print 'DEBUG info: pattern for search is ' , pattern
 
        regexll = re.compile(pattern,re.U|re.L|re.M)
 
-        latlongroup = [_('lat'),_('lon')]
 
+        latlongroup = [('lat'),('lon')]
 
        #find all occurences in the data file
 
        iterator = regexll.finditer(self.latlonfile_datastr)
 
        for result in iterator :
 
-            lato =self.group_get(place, _('latitude'))
 
-            lono =self.group_get(place, _('longitude'))
 
+            lato =self.group_get(place, ('latitude'))
 
+            lono =self.group_get(place, ('longitude'))
 
                       
 
            lat = result.group(latlongroup[0])
 
            lon = result.group(latlongroup[1])
 
@@ -976,26 +987,26 @@
 
        if latnew != None and latnew != lat :
 
            valoud.append(lat)
 
            valnew.append(latnew)
 
-            valaction.append([_('latitude'),latnew])
 
+            valaction.append([('latitude'),latnew])
 
            #do the action in memory
 
-            place = self.group_set(place, _('latitude'),latnew)
 
+            place = self.group_set(place, ('latitude'),latnew)
 
        elif lat != '' and latnew == None :
 
            valoud.append(_('invalid lat or lon value, %(lat)s, %(lon)s')
 
                        % {'lat' : lat, 'lon' : lon})
 
            valnew.append(None)
 
            #do nothing
 
-            valaction.append([_('latitude'), None])
 
+            valaction.append([('latitude'), None])
 
        if lonnew != None and lonnew != lon :
 
            valoud.append(lon)
 
            valnew.append(lonnew)
 
-            valaction.append([_('longitude'),lonnew])
 
+            valaction.append([('longitude'),lonnew])
 
            #do the action in memory
 
-            place = self.group_set(place, _('longitude'),lonnew)
 
+            place = self.group_set(place, ('longitude'),lonnew)
 
        elif lon != '' and lonnew == None :
 
            valoud.append(_('invalid lat or lon value, %(lat)s, %(lon)s')
 
                        % {'lat' : lat, 'lon' : lon})
 
            valnew.append(None)
 
-            valaction.append([_('longitude'), None])
 
+            valaction.append([('longitude'), None])
 
        return valoud, valnew, valaction, place
 
   
 
    def construct_title(self, place) :
 
@@ -1011,9 +1022,9 @@
 
            if place.get_main_location().get_state() :
 
                new += ', ' + place.get_main_location().get_state()
 
            valnew.append(new)
 
-            valaction.append([_('title'),new])
 
+            valaction.append([('title'),new])
 
            #do the action in memory
 
-            place = self.group_set(place, _('title'),new)
 
+            place = self.group_set(place, ('title'),new)
 
        elif type == "T1CS" :
 
            old = place.get_title()
 
            valoud.append(old)
 
@@ -1024,9 +1035,9 @@
 
            if place.get_main_location().get_state() :
 
                new += ', ' + place.get_main_location().get_state()
 
            valnew.append(new)
 
-            valaction.append([_('title'),new])
 
+            valaction.append([('title'),new])
 
            #do the action in memory
 
-            place = self.group_set(place, _('title'),new)
 
+            place = self.group_set(place, ('title'),new)
 
           
 
        return valoud, valnew, valaction, place
 
       
 
@@ -1035,12 +1046,12 @@
 
        valnew = []
 
        valaction = []
 
       
 
-        for newval in [(self.options.handler.options_dict['countryset'],_('country')),
 
-                (self.options.handler.options_dict['stateset'],_('state')),
 
-                (self.options.handler.options_dict['countyset'],_('county')),
 
-                (self.options.handler.options_dict['cityset'],_('city')),
 
-                (self.options.handler.options_dict['parishset'],_('parish')),
 
-                (self.options.handler.options_dict['zipset'],_('zip'))] :
 
+        for newval in [(self.options.handler.options_dict['countryset'],('country')),
 
+                (self.options.handler.options_dict['stateset'],('state')),
 
+                (self.options.handler.options_dict['countyset'],('county')),
 
+                (self.options.handler.options_dict['cityset'],('city')),
 
+                (self.options.handler.options_dict['parishset'],('parish')),
 
+                (self.options.handler.options_dict['zipset'],('zip'))] :
 
            #we allow ' ' to mean store '':
 
            if newval[0] :
 
                nv = newval[0].strip()
 
@@ -1076,23 +1087,23 @@
 
   
 
    # geonames : http://download.geonames.org/export/dump/
 
    # geonet : ftp://ftp.nga.mil/pub/gns_data
 
-    lat_translated = _('lat')
 
-    lon_translated = _('lon')
 
-    city_translated = _('city')
 
-    county_translated = _('county')
 
-    state_translated = _('state')
 
-    country_translated = _('country')
 
-    CITY_transl = _('CITY')
 
-    TITLE_transl = _('TITLE')
 
-    TITLEBEGIN_transl = _('TITLEBEGIN')
 
-    STATE_transl = _('STATE')
 
-    PARISH_transl = _('PARISH')
 
-    latgr = r'(?P<'+lat_translated +r'>'
 
-    longr = r'(?P<'+lon_translated +r'>'
 
-    citygr = r'(?P<'+city_translated +r'>'
 
-    countygr = r'(?P<'+county_translated +r'>'
 
-    countrygr = r'(?P<'+country_translated +r'>'
 
-    stategr = r'(?P<'+state_translated +r'>'
 
+#    lat_translated = _('lat')
 
+#    lon_translated = _('lon')
 
+#    city_translated = _('city')
 
+#    county_translated = _('county')
 
+#    state_translated = _('state')
 
+#    country_translated = _('country')
 
+    CITY_transl = ('CITY')
 
+    TITLE_transl = ('TITLE')
 
+    TITLEBEGIN_transl = ('TITLEBEGIN')
 
+    STATE_transl = ('STATE')
 
+    PARISH_transl = ('PARISH')
 
+    latgr = r'(?P<'+'lat' +r'>'
 
+    longr = r'(?P<'+'lon' +r'>'
 
+    citygr = r'(?P<'+'city' +r'>'
 
+    countygr = r'(?P<'+'county' +r'>'
 
+    countrygr = r'(?P<'+'country' +r'>'
 
+    stategr = r'(?P<'+'state' +r'>'
 
    titleregex = (
 
        ("citystate", "City [,|.] State", _("City [,|.] State")
 
            , r'\s*'+citygr +r'.+?)\s*[.,]\s*'+stategr+r'.+?)\s*$'),
 
</pre>
 
  
'''TODO''' : to commit this patch on [http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/contrib/PlaceCompletion/PlaceCompletion.py?view=log gramps addons], see [[Addons_Development|Gramps addons]].
+
For the following older versions of Gramps you will need to do a Manual install of the files.
  
===Parsing place title===
+
Extract the three files that are in the download. Put the .glade and .py files in the plugins directory. For linux:
 
For France, some [http://www.geneawiki.com/index.php/Informatique_-_saisie_des_lieux practical rules] could be useful for seizing place. We need :
 
* the city name + INSEE code (at option). This code is unique and can identify with certainty a common (with the county, district, township and municipality). It can identify with a common insurance even if it has changed its name. This code is used in Archives. ''Using postal code is not advisable ...''
 
* a subdivision: identifies a parish or a place called within a municipality
 
* the state (at option) ''or county but is already on INSEE code''
 
* the country (at option). Ideally it should still take the country. It is understandable that this is tedious. ''Maybe do not enter the country if the genealogy is mostly of one country and seize enter the country for events outside the country's main. Everyone will appreciate.''
 
  
If you have a place name like :
+
* local install: place in <code>~/.gramps/plugins</code>
'''City, code, state, country''', you could try this patch on current ''Place completion tool'' for parsing place title on location fields :
 
 
1080a1081
 
+    code_translated = _('zip')
 
1091a1093
 
+    codegr = r'(?P<'+code_translated +r'>'
 
1100a1103,1106
 
+        ("citycodestatecountry", "City[,|.] Code[,|.] State[,|.] Country",
 
+            _("City[,|.] Code[,|.] State[,|.] Country")
 
+            , r'\s*'+citygr+r'.*?)\s*[.,]\s*'+codegr+r'.*?)\s*[.,]\s*'
 
+            +stategr+r'.*?)\s*[.,]\s*'+countrygr+r'.*?)\s*$'),
 
  
Note: this patch does not work with locale set to french (e.g. LANGUAGE=fr_FR.UTF-8). Before getting this fixed, users should temporarily move to english locale to use it.
+
If you still use Gramps 3.1.x, then you will need version 1.2 of the Place Completion Tool. You can find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_2.tar.gz placecompletion_1_2.tar.gz].  
  
===For GRAMPS 3.2===
+
If you still use Gramps 3.0.x, then you will need version 1.1 of the Place Completion Tool. You find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_1.tar.gz placecompletion_1_1.tar.gz].
  
To allow PlaceCompletion 1.2 to work with trunk, you'll need to make a couple of changes:
+
If you still use Gramps 2.2.5+, then you will need version 1.0 of the Place Completion Tool. You find it at [http://cage.ugent.be/~bm/varia/placecompletion_1_0.tar.gz placecompletion_1_0.tar.gz].
  
<pre>
+
[[Category:Plugins]]
--- .gramps/PlaceCompletion.py 2009-02-11 17:04:27.000000000 -0500
+
[[Category:Developers/General]]
+++ .gramps/plugins/PlaceCompletion.py 2009-09-06 12:06:49.000000000 -0400
+
[[Category:Tools]]
@@ -60,6 +60,14 @@
 
import PlaceUtils
 
import Errors
 
 
+if hasattr(Utils, "ProgressMeter"):
 
+    ProgressMeter = Utils.ProgressMeter
 
+else:
 
+    import gui.utils
 
+    if hasattr(gui.utils, "ProgressMeter"):
 
+        ProgressMeter = gui.utils.ProgressMeter
 
+    else:
 
+        raise ImportError("can't find ProgressMeter")
 
#------------------------------------------------------------------------
 
#
 
#
 
@@ -498,8 +506,8 @@
 
       
 
        self.nrplaces_in_tree = len(ind_list)
 
        # Populating might take a while, add a progress bar
 
-        progress = Utils.ProgressMeter(
 
-                                _("Finding Places and appropriate changes"),'')
 
+        progress = ProgressMeter(
 
+            _("Finding Places and appropriate changes"),'')
 
        #apply the filter
 
        progress.set_pass(_('Filtering'),1)
 
        progress.step()
 
@@ -704,7 +712,7 @@
 
 
        self.trans = self.db.transaction_begin("",batch=True)
 
        self.db.disable_signals()
 
-        progress = Utils.ProgressMeter(_('Doing Place changes'),'')
 
+        progress = ProgressMeter(_('Doing Place changes'),'')
 
        #we do not know how many places in the treeview, and counting would
 
        # mean transversing the tree. Set the progress to the possible maximum
 
        progress.set_pass('',self.db.get_number_of_places())
 
</pre>
 

Revision as of 07:52, 5 January 2017

Gramps-notes.png

Please use carefully on data that is backed up, and help make it better by reporting any comments or problems to the author, or issues to the bug tracker
Unless otherwise stated on this page, you can download this addon by following these instructions.
Please note that some Addons have prerequisites that need to be installed before they can be used.
This Addon/Plugin system is controlled by the Plugin Manager.


Place completion tool - Example Results

A tool to bring the places in your Gramps database in accordance with the Gramps requirements: batch add country, county; look-up latitude-longitude; set description (title); ...

This tool helps you fill in the place attributes like county, country, ..., by allowing you to select the places you work on, and do changes on all these places with one button click.

The general aims are:

  • Place/Location is a newer concept in Gramps. Many older databases only have a Place title field which is a descriptive text containing city, state, country. This should be parsed to insert the values in the correct attribute fields.
  • Latitude and longitude are important to show data on a map. However, doing a look-up of this data on the internet is slow and time consuming. The tool allows to search in the free resources on the net.
  • Setting of an attribute of a set of places in one go.
  • Conversion of latitude and longitude to a fixed data format. On import one might obtain latitude and longitude in several different formats. A conversion tool to store them all in the same format is useful.
  • Construction of a uniform title/description field, from the data in the place object.

Follow the installation details in the Download section for your version of Gramp.

Usage Instructions

The place completion tool provides a lot of functionality. These usage instruction should help you to understand how it works.

Download resources

Gramps-notes.png

This article's content is incomplete or a placeholder stub.
Please update or expand this section.


The place completion tool can look up for you latitude/longitude, add county information (USA), ... . For some of this functionality, you must download data files of the countries you are interested in. Right now you have three options:

  1. Download geonames country files. You can do this here freely. Geonames parses fastest, so is the advised format to use.
  2. Download geonames USA state files. You can do this here freely. This is advised for USA searches, as the data in the USA country file contains many doubles, which can be avoided by searching state per state. State info also contains county information.
  3. Download GNS Geonet country files (not available for usa). You can do this here freely with ftp.

Watch out, some of these downloads are VERY large, especially USA data. Only download what you need! If the download is a compressed zip file, you will need to extract the data file before you can use it.

Gramps-notes.png
Note:

The geonames data of popular places is in English, so e.g. municipalities in Italy will be found, but Roma not, as this is Rome in English. To find data with these you need to search in the localised variants of the name (see below)

Gnome-important.png
Warning:

DO NOT BETA TEST WITH YOUR RESEARCH DATA. EXPORT DATA FIRST TO HAVE A BACKUP.

Starting the tool

You will find the plugin under Tools > Utilities > PlaceCompletion...

The dialog explained

The Place Completion Tool

The Dialog consists of 4 parts:

Part 1. Selection of places

First you need to choose with which places you want to work. You can use several methods to define your places:

  1. Use a place filter. You can use two preset filters: All places, which returns all places, and No Latitude/Longitude given, which returns all places of which the latitude or the longitude is not set. You can also created a custom place filter in the place view, test it with the filter sidebar, and then use it in this tool. All custom filters you made will be available
  2. To prevent the need to make a filter for every city, ... in your data, you can set country,state,county,city or parish of the places you want to search on. This works just like in the filter sideview in the places view.
  3. Use a latitude, longitude rectangle. Eg, suppose you have the latitude and longitude of all places in the UK, and now want to add in the state attribute Wales, for all places in Wales. You can look on a map, note down the centre of Wales in latitude and longitude, as well as roughly the width and height of this rectangle. This will allow you to obtain all places in Wales (and some in England), allowing to much faster set the state information.

Part 2. Completion of places

  1. The first possibility is to look up in a datafile the latitude and longitude of your places. For this you must have downloaded the necessary resources, see section above. You can select with a file dialog the file you want to search, and set how this data must be parsed. The following parsing options are available:
    1. GeoNames country file, city search: use the city attribute to look for lat/lon in a GeoNames country file. This is the fastest search.
    2. GeoNames country file, city localized variants search: use the city attribute to look for lat/lon in a GeoNames country file using the localised (non-English) known names in the GeoNames file. Eg, Roma will be found with this option (as Roma is the Italian local variant of the English name Rome)
    3. GeoNames country file, title begin, general search: Use the start of the title field to search in a GeoNames file. With start it is meant everything before a comma:, . This allows to find landmarks, squares, ... . Eg, if the title of your place is: Piazza Navona, Rome, using this search will find you the latitude and longitude of this famous square in Rome.
    4. GeoNames USA state file, city search: Looking for places in the USA file is almost worthless: it takes a long time and every name exists several times. Hence, it is worthwhile to use state by state. If a USA state file is selected for doing a search, you must select this option. The city attribute is used for the search.
    5. GNS Geonet country file, city search: use the city attribute to search in a GNS file (slower than GeoNames search!).
    6. GNS Geonet country file, title begin search: use the start of the title of a place to search in a GNS file. With start everything appearing before the first comma is meant.
  2. A second option is to parse some existing data in your places.
    1. You can parse the title attribute to extract information from it. Eg a title like Albany, NY can be used to set the city attribute to Albany and the state attribute to NY.
    2. You can set the title of all the selected places to a uniform way. This is interesting if due to imports you have different styles for the title field, which can be annoying in reports. At the moment there are two options:
      1. Set title field to City[, State]: This means the title of your places will contain the city, and if the state field is present, the state will be appended with a comma.
      2. Set title field to Titlestart[, City][, State]: This means the present start of your title will be kept. If this start is not the city, then the city will be appended. If state is present, also state will be appended. An example: suppose your title is Piazza Navona, Italy, the city is Rome and the State is Lazio. Using this option to set the title would change the title attribute into Piazza Navona, Rome, Lazio.
    3. Convert latitude and longitude to a uniform way. Again due to import, copy/paste, you might have latitude and longitude entered in different formats. This is annoying on reports. This options allows you to set for all selected places the lat/lon to one form. The options are:
      1. All in degree notation: use the classical degree notation with degree, minutes and seconds.
      2. All in decimal notation: use the decimal system to denote lat/lon.
      3. Correct -50° in 50°S: a much seen error is to use - for the classical degree notation, which is wrong, and which Gramps will not be able to interpret. With this option this error is looked for and corrected.

Part 3. Overview of the results

After having entered all data in Part 1 and 2, you click find for Gramps to search all changes that will occur. This part of the dialog shows all changes that will occur.

Place completion tool results.png

All selected places are shown. If changes will be done all changes are listed as subentries of the place. Every change will be a subentry.

If the change will overwrite an existing entry, the subentry is shown in orange.

Gnome-important.png
Warning

TO AVOID PROBLEMS, GO OVER ALL CHANGES QUICKLY, AND CHECK ALL ENTRIES IN ORANGE!

The following actions are possible in the result screen:

  1. press delete to delete the entry, making sure that this change will not occur. You can delete the entry to delete all changes, or select one subentry, to only delete that specific chagne
  2. double-click on an entry to open the place dialog. If you double-click on the entry, all changes will be preentered. If you double-click on a subentry, only this specific change will be preentered in the place dialog.
  3. press tab to open in a browser window google maps. Pressing tab on a subentry showing a new lat/lon entry will open google maps on this new lat/lon position. Pressing tab on the top place entry will give open google maps with the old lat/lon position, or if that is not known the title/city field is used for the search.

Part 4. Actions

After you have checked the changes in Part 3, you can apply them by clicking the Apply button.

Selecting Help will bring you to this page, clicking Close will close the Place completion tool window and clicking {{man button|Google Maps} when an entry is selected in the results field has the same effect as pressing tab on an entry (see above).

Example

Open the example file from the examples where latitude and longitude are empty: example.gramps.

We will now show how the places in this file can be completed. The best thing to do is to create a new Family Tree, give it a name, and import the example.gramps file. This file has 852 places, which would mean a lot of manual edits if you do not use this tool!

Now, open the place view. You will see all places are of the form:

Aberdeen, WA

This value is the Place Name attribute (the title or description of the place).

Step 1: City and State data

Our first step will be to split this field into a City value (here Aberdeen), and a State value (here WA).

We open the Place completion tool:

Parse the Place Name Field

Here we have selected All Places, and we Change the title into as City [,State]. Click on Find, quickly scan the data if all looks ok, and then click on Apply. You are notified that 443 place records were modified. This is one less that the number of places. Indeed, one place does have a different type of title: Puerto Rico has no state information.


Step 2: Look-up latitude and longitude

We have downloaded the GeoNames datafiles for the USA states, and will now use that to complete the latitude and longitude of the data. At the same time, this will fill up the county field.

Look up lat/lon for Alaska

In the screenshot, you see we have selected All Places with State=AK. In the second part of the window we give that we want to search in the AK_DECI.txt file downloaded from GeoNames, using the parsing method: GeoNames USA state file, city search.
Note that if you want to change AK into Alaska, this would be possible. Just set state=Alaska in the set attributes section of the window.

Do this now for all the states. Always check for doubles. Eg, for state AL, going over the changes, we encounter:


Double in lat/lon, city Enterprise exists in two counties


We see that the first time 'Enterprise' if found, it is in county Coffee in lat/lon:31.31/-85.85. The second hit is for county Chilton with lat/lon:32.73/-86.62.

You can now use the Google Maps button (or press TAB key) while the lat/lon subentry is selected to see where this city is in both cases. From this it will be clear for example that one is a hamlet, not really a city, while the first is a real city. So now, select the second lat/lon entry, and delete it by pressing the DEL key. Do the same for the second county entry.

In case google maps did not allow you to determine which is the correct city, you can double click on the city to open the Place Dialog (Warning: this will preenter the data of the Place Completion tool. So hit cancel here if you want to exit without these changes done). In this dialog the references tab allows you to navigate to all events coupled to this place. This will give you extra information you might use to decide which of the two found places is the correct place.

Step 3: Problem entries

While updating all places in step 2, you will have noticed some errors in the state information: Some places have a dubious state: eg OH-AL

You can obtain these states by choosing All Places en setting the state search box to -. Clicking Find will give you all these problem places. You can use google maps or the place dialog to sort them out. You can also use the USA country GeoNames file to search these places in the entire USA.

Gramps-notes.png
Note:

You will need sufficient memory for this, or you will obtain a MemoryError (see below)!

Step 4: Lat/Lon not found

After the above, still some 45 places have no latitude/longitude found. You can now select these places by setting the Place filter to 'No Latitude/Longitude', which will find you all places with no coordinates.

It will be clear that many of those can be quickly corrected: abbreviations, eg the city field contains St.George, which should be Saint George; double names, eg Waterloo-Cedar Falls, IA means Waterloo near Cedar Falls, changing the city to Waterloo and redoing the search using Google Maps will allow to quickly find which coordinates for Waterloo are needed.

Advanced usage

This is for advanced users who know how to use regular expressions.

The parsing fields have entry fields allowing you to give your own parsing. Parsing uses regular expressions. You can use this to parse your title, and to parse a lat/lon file in your own way. For reference, here an overview of the parsing codes used for the predefined parses:

Parse title details

In 'Parsing and Conversion of existing title or position', 'Parse title' and 'Change title into' provide some pre-defined options for extracting location values from a Place Title. Otherwise regex parsing is needed.

Write your regex in 'Parse title:'.Click on 'Find', which shows the proposed changes. Then click 'Apply'.

Example 1

For France, some practical rules could be useful for seizing place. It needs :

  • the city name + INSEE code (at option). This code is unique and can identify with certainty a common (with the county, district, township and municipality). It can identify with a common insurance even if it has changed its name. This code is used in Archives. Using postal code is not advisable ...
  • a subdivision: identifies a parish or a place called within a municipality
  • the state (at option) or county but is already on INSEE code
  • the country (at option). Ideally it should still take the country. It is understandable that this is tedious. Maybe do not enter the country if the genealogy is mostly of one country and seize enter the country for events outside the country's main. Everyone will appreciate.
e.g. Avignon,84000,Vaucluse,Provence-Cote-d'Azur,FRANCE,

where some fields may be missing:

e.g. Woerth,,,Alsace,FRANCE,

the regex:

\s*(?P<city>[^,]+)[,]\s*(?P<zip>\d*)[,](?P<county>[^,]*)[,](?P<state>[^,]*)[,](?P<country>[^,]*)[,]*$

treating the comma character as an end-of-string delimiter, will distribute Avignon to City, 84000 to ZIP, Vaucluse to County, Province-Cote-d'Azur to State and France to Country

and in the case of missing fields, as long as the first is not empty, will distribute Woerth to City, Alsace to State and FRANCE to Country.

It allows initial whitespace and an optional comma after the Country

Example 2

In many 16th and 17th century English IGI records the situation is more complicated.

The Place Title takes the form of 3 strings (Town, County, Country) or 4 strings (Parish, District, County, Country) for example:

(a) Chester le Street, Durham, England - 3-string

(b) Of Middleton-in-Teesdale, Durham, England - 3-string

(c) Bishoply,Stanhope, Durham, England - 4-string

(d) Of St. Margaret's, Stanhope, Durham, England - 4-string


For the 3-string record the following regex, treating the comma character as an end-of-string delimiter, will distribute the 3 strings correctly to City, County, Country locations, leaving 4-string records untouched.

Regex A:

\s*(Of[,]*\s*)*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$


For 4-string records the following regex will distribute the 4 strings correctly to Parish, City, County, Country locations, leaving 3-string records untouched.

Regex B:

\s*(Of[,]*\s*)*(?P<parish>[^,]+?)[,]\s*(?P<city>[^,]+?)[,]\s*(?P<county>[^,]+?)[,]\s*((?P<country>[^,]+?)){1,1}$


"Bogus" 4-string records: a not uncommon error in these old records is for 'Of' to be followed by a comma, e.g.

(e) Of, Houghton-le-Spring, Durham, England

Regex A will parse 3-string records correctly and Regex B 4-string records. However Regex B will attempt to treat a 3-string record with 'Of,' as if it were a 4-string record. This would give the 3-string record a non-existent Parish called "Of" !

To avoid this, when using Regex B click on 'Find' to display the records proposed for change and delete all the 3-string 'Of,' records before clicking 'Apply'.

Gramps-notes.png
take care when pasting a Regex (for example, after testing in a regex editor) into the Parse title details field. Pasting adds redundant spaces before or after the Regex that will prevent it working properly in the Place Completion Tool

The predefined regex expressions are as follows, where for brevity we use some variables defined lower.

Gramps-notes.png
Regex Help

For those new to Python and Regex please review the HOWTO here:
http://docs.python.org/dev/howto/regex.html

  1. "City [,|.] State" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+state_translated +r'>.+?)\s*$'
  2. "City [,|.] Country" is parsed by : r'\s*(?P<'+city_translated +r'>.+?)\s*[.,]\s*(?P<'+country_translated +r'>.+?)\s*$'
  3. "City (Country)" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*\(\s*(?P<'+country_translated +r'>[^\)]+)\s*\)\s*$'
  4. "City" is parsed by : r'\s*(?P<'+city_translated +r'>.*?)\s*$'

Here the variables used are:

lat_translated = _('lat')
lon_translated = _('lon')
city_translated = _('city')
county_translated = _('county')
state_translated = _('state')
country_translated = _('country')

You can use one of these variables as a group, and the tool will recognise them, and use as values for the corresponding place attributes.

Lat/Lon lookup parsing

For the regex of lat/lon lookup, you need to indicate which data must be replaced with existing place attributes for the search, as well as indicate which regex groups must be extracted.

  1. "GeoNames country file, city search" is parsed with: r'\t'+CITY_transl +r'\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\tP'
  2. "GeoNames country file, city localized variants search" is parsed with: r'[\t,]'+CITY_transl+r'[,\t][^\t\d]*\t?' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\tP'
  3. "GeoNames country file, title begin, general search" is parsed with: r'\t'+TITLEBEGIN_transl +r'\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)\t[PSTV]'
  4. "GeoNames USA state file, city search" is parsed with: r'\t'+CITY_transl+r'\tPopulated Place\t[^\t]*\t[^\t]*\t' + countygr + r'[^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t' +latgr + r'[\d+-][^\t]*)\t' + longr + r'[\d+-][^\t]*)'
  5. "GNS Geonet country file, city search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\tP\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+CITY_transl+r'\t[^\t]*\t[^\t\n]+$'
  6. "GNS Geonet country file, title begin search" is parsed with: r'\t'+latgr+r'[\d+-][^\t]*)\t'+longr+r'[\d+-][^\t]*)'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[PLSTV]\t[^\t]*\t[^\t]*'+ r'\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t[^\t]*\t[^\t]*\t[^\t]*' + r'\t'+TITLEBEGIN_transl+r'\t[^\t]*\t[^\t\n]+$'
  7. Read of mediawiki CSV dump. This reads the files on [1](link gone) (for more information, see http://meta.wikimedia.org/wiki/WikiProjects_Geographical_coordinates) (Contribution by nomeata)

For extraction of data you can use the same groupnames as in title parsing, so eg latgr in above should read: r'(?P<'+lat_translated +r'>' .

The syntax for the values that need to be used for searching in the file, eg CITY_transl, is given by : _('CITY'). You can use as substitution values: _('CITY'), _('TITLE'), _('TITLEBEGIN'), _('STATE'), _('PARISH').

The tool will read in the given regex, replace the substitution strings by the values in the place object, do the search, and extract the regex groups given from the result.

Resource: GEOnet Names Server

Troubleshooting

Non UTF-8 latitude/longitude file

The place completion tool expects the input files for location lookup to be in unicode (utf-8). On the occasion this is not the case, you will get the error:

File "/home/benny/programms/gramps/gramps2/src/plugins/PlaceCompletion.py", line 851, in load_latlon_file
    self.latlonfile_datastr = infile.read()
  File "/usr/lib/python2.4/codecs.py", line 481, in read
    return self.reader.read(size)
  File "/usr/lib/python2.4/codecs.py", line 293, in read
    newchars, decodedbytes = self.decode(data, self.errors)
 UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1610092-1610094: invalid data

Note that the Place Completion tool catches these errors and shows you an information box. After this, the tool will attempt to read the file with utf-8 (unicode), ignoring errors. This might give good results, but will of course fail to produce results on non-unicode encoded files.

In the above example it is clear the problem is in two bytes, so you can correct this manually: open the file with eg KHexEdit Binary Editor, go to the specified position (offset 1610092), and change the two bytes with a space.

In the case the file is completely non-unicode, you will have to convert it to unicode with a tool, before using it in the placecompletion tool.

Memory Error

The tool might fail with the error:

self.latlonfile_datastr = infile.read()
  File "/usr/lib/python2.4/codecs.py", line 481, in read
    return self.reader.read(size)
  File "/usr/lib/python2.4/codecs.py", line 293, in read
    newchars, decodedbytes = self.decode(data, self.errors)
 MemoryError

The tool has to load the datafile for latitude/longitude searching into memory. For large files like USA.txt, this might be impossible if you have limited memory. You can try to close as many programs running at together with Gramps, and try the tool again.

See also

Design specification

Place/Location is a newer concept in Gramps. Many older databases only have a Place title field which is a descriptive text containing city, state, country. To distribute these values into the correct attribute fields, see Parse Title Details below

Download

If you use Gramps 4.2 , then use the automatic installation.

If you use Gramps 4.1 , then use the automatic installation.

If you use Gramps 3.4 , then use the automatic installation.

If you use Gramps 3.3 , then use the automatic installation.

If you use Gramps 3.2 , then use the automatic installation.

Manual install

For the following older versions of Gramps you will need to do a Manual install of the files.

Extract the three files that are in the download. Put the .glade and .py files in the plugins directory. For linux:

  • local install: place in ~/.gramps/plugins

If you still use Gramps 3.1.x, then you will need version 1.2 of the Place Completion Tool. You can find it at placecompletion_1_2.tar.gz.

If you still use Gramps 3.0.x, then you will need version 1.1 of the Place Completion Tool. You find it at placecompletion_1_1.tar.gz.

If you still use Gramps 2.2.5+, then you will need version 1.0 of the Place Completion Tool. You find it at placecompletion_1_0.tar.gz.