4,604
edits
Changes
m
== Overview ==Starting with [[Database_Formats#Gramps_5.0|Gramps 5.0]], there is a new manner to select data from the database. The goal of this API is to make it easy to select data by different criteria, and make selection as fast as possible.
Starting with Gramps 5This page documents the interface for developers.0, there is a new method on the database object called "select" that works as follows:
db.select(TABLE-NAME, SELECT-LIST, where=WHERE-LIST, order_byQuerySet =ORDER-BY-LIST)
As an example, consider selecting Each database backend now supports the gramps_id from all people who have a surname of "Smith" and whose name begins with a "J", ordered by the gramps_idfollowing QuerySet objects:
Building on thator even: >> person.get_field("primary_name.surname_list.surname") ["Johnson", "Johansen", a db"Johnston"] In the last example, the "surname" field was applied to each of the surname_list items.select method has been added so that you Each primary object can get all also have a list of those extra secondary fields from all People, and sort and filter on all of /indexes. These are specified in the regular primary object's method "get_extra_secondary_fields(string, int) fields. This works independently " which returns a list of SQL. However, if you re-implement that method for DB-API, and have the appropriate sql-fields, and sql-indexes, then you have an large speeddotted-up for large datapath strings.
So, our old system required a scan of all data, unpickling, creating objects, and sorting for any use. If you have 100k records, that required processing all of them. With the new DBThe dotted-API implementation, you can do that same query in a fraction of that time. Views (written appropriately) will appear very quickly (milliseconds) regardless of the size of the databasefield path strings are mapped to SQL names by replacing dots with two underscores.
To interactively test it out, you can try these methods with a regular BSDDB database, but you won't see any speed enhancements. To see it work fast, you'll need a DBAPI database built from the latest addons-source/DBAPIBackend. The new webapp takes search text and turns it into a SQL statement directly for seeing the speed and flexibility of the functionality.== Speed Tests ==
In this stringified Using he dotted-field path string field-based Select API, we would can write code as follows. Consider that we want to select the handle of all people whose surname is "Smith", given name starts with a "J", and ordered by gramps_id. We would write:
Currently DB-API is automatically creating SQL fields and indexes for all regular (non-list, standard Python types) primary-object attributes (like gramps_id, privacy, etc.). This takes time and space. We may want to manage this a bit more carefully.[[Category:GEPS|D]]
no edit summary
* db.Person* db.Family* db.Note* db.Source* db.Citation* db.Place* db.Repository* db.Event* db.Tag Each of these QuerySets allows you to chain together a series of QuerySet method calls. QuerySet supports the following methods: * .select([fields...]) - returns generator* .order(fieldname, ...) - order-by fields given as strings; use "-name" for descending* .filter(obj, args...) - applies a FilterObject or Python callable* .map(f) - applies a function to all selected objects* .proxy(name, args...) - applies a named-proxy ("living", "private", or "referenced")* .limit() - set start or limit or selection* .count() - returns the number of matches* .tag(tag_name) - puts a tag on all selected items, if not tagged with tag_name already These methods can be lined up to create a series of operations: db.Person.filter(lambda person: person.private=True).select("gramps_id") == Database.QuerySet.select() == db.Person", .select() [ db.Person.select("gramps_id"], "handle") where=[= Database.QuerySet.order() == db.Person.order("ANDgramps_id", [).select() Examples: * db.Person.order("primary_namegramps_id").select()* db.surname_listPerson.0order("primary_name.surnamefirst_name", "-gramps_id").select() The first-listed field name is the primary sort, followed by secondary sorts. In the last example above, the data would be sorted first by first_name ascending, and inside that, by gramps_id descending, such as: Avery, I0003 Avery, I0002 Avery, I0001 Barry, I0013 Barry, I0012 Barry, I0011 == Database.QuerySet.filter() == == Database.QuerySet.map() == == Database.QuerySet.proxy() == == Database.QuerySet.limit() == == Database.QuerySet.count() == == Database.QuerySet.tag() ==" == Joins == The database.select method can also do joins across primary objects in each of the WHERE, ORDER, or select fields. For example consider this request: >>> db.Family.select("Smithmother_handle.gramps_id") This will return the gramps_id's of the mothers of each family, if there is a mother. You can join across multiple tables with more complex queries, such as: >>> list(db.Family.select("primary_namemother_handle.event_ref_list.ref.first_namegramps_id")) That returns all of the Event gramps_ids for all events for the mother (Person) of all families, looking something like: [{"mother_handle.event_ref_list.ref.gramps_id": ["LIKEE0001", "J%E0002")]]}, {"mother_handle.event_ref_list.ref.gramps_id": "E0003"}, order_by= {"mother_handle.event_ref_list.ref.gramps_id": [("gramps_idE0003", "ASCE0001")])}, ... ] Note that these joins are done per record and are therefore not yet optimized---each requires another database access. In the future, these could be optimized via a SQL JOIN.
== Implementation ==
There are now two database backends: Berkeley DB (BSDDB), and [https://www.python.org/dev/peps/pep-0249/ Python's DB-API]. BSDDB is a data store with much of the database code written in Python, and DB-API is a common interface to the popular SQL engines. We have used BSDDB in Gramps for many years, but are now transitioning to DB-API.
With BSDDB, Gramps has a pipeline design when it comes to accessing the data. For example, consider getting the People for the flat view. First we get a cursor that iterates over the data. Then we sort it, on whatever criteria we have requested. Finally, we filter the data. The select method will always perform a linear search on fully expanded data.
In order to make this the select operation faster for DB-API, we need to know the filter information, and sort order when we ask for the data. With SQL we can simply add WHERE clauses and ORDER BY clauses to the basic SELECT statement. But these are only useful if we can have indexes on the relevant data.
This is made more difficult because Gramps uses a hierarchical representation of data. For example, we might wish to have the People data sorted by "surname, given" of the primary_name. But that information is actually in:
* person.primary_name.first_name
respectively. We could can make special fields for these, and special indexes. But it would be much more flexible if we could create a variety of ad hoc queries on the flyGramps 5. The BSDDB datastore doesn't have any schema, which means that it has no idea of 0 creates "gramps_idsecondary" fields and indexes in SQL for every str, int, or "primary_name" or any fieldbool data on a primary object. An idea of These secondary fields are known from the primary object's schema has been developed over the last few years. This makes possible the Database Differences Report, without having to write any field-specific code: the data knows its own structure.
The schema idea has been augmented with additional methods based on the idea of "fields". Now, you can ask a person object:
"Johnson"
list(db.select("Person", ["handle", "gramps_id"], where=["AND", [.filter("primary_name.surname_list.0.surname", "primary_name__surname_list__0__surname=", "Smith"), ("primary_name.first_name", "LIKE", primary_name__first_name__LIKE="J%")]], order_by=[.order("gramps_id", "ASC")].select())
This code works on BSDDB as well as DB-API. Let's see the difference in timing on databases that have 187,294 people (created from GenFan, this is 20 full generations).
So, where we can access the data via SQL, we can get a speedup, the biggest will always be in the filter as it makes it so we don't have to load into Python many objects. We have linear code in many places that could benefit from using db.select().
