Open main menu

Gramps β

Difference between revisions of "Database Query API"

(ORDER-BY-LIST)
m
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Starting with Gramps 5.0, there is a new method on the database object called "select." The goal of select is to make it possible to select most any data, based on most any criteria, sorted in most any order. In addition, where possible, the selection would be as fast as possible.
+
Starting with [[Database_Formats#Gramps_5.0|Gramps 5.0]], there is a new manner to select data from the database. The goal of this API is to make it easy to select data by different criteria, and make selection as fast as possible.
  
The database select method works as follows:
+
This page documents the interface for developers.
  
db.select(TABLE-NAME,
+
= QuerySet =  
          SELECT-LIST,
 
          where=WHERE-EXPRESSION,
 
          order_by=ORDER-BY-LIST,
 
          start=START-ROW,
 
          limit=LIMIT-ROW-COUNT)
 
  
The following are required:
+
Each database backend now supports the following QuerySet objects:
  
* TABLE-NAME - the name of the table. That would be "Person", Family", "Media", "Repository", "Place", "Note", "Source", "Citation", or "Tag"
+
* db.Person
* SELECT-LIST - a list of dot-separated field path strings from this object (eg, "gramps_id", "primary_name.first_name", etc)
+
* db.Family
 +
* db.Note
 +
* db.Source
 +
* db.Citation
 +
* db.Place
 +
* db.Repository
 +
* db.Event
 +
* db.Tag
  
Optional arguments:
+
Each of these QuerySets allows you to chain together a series of QuerySet method calls. QuerySet supports the following methods:
  
* WHERE-EXPRESSION - a matching expression, such as ("gramps_id", "=", "I0001"). These can be nested (see below)
+
* .select([fields...]) - returns generator
* ORDER-BY-LIST - a list of dot-separated field path strings, each paired with a sorting direction, for example [("gramps_id", "ASC")]
+
* .order(fieldname, ...) - order-by fields given as strings; use "-name" for descending
* START-ROW - the row number on which to start. Default is 0, meaning start at beginning
+
* .filter(obj, args...) - applies a FilterObject or Python callable
* LIMIT-ROW-COUNT - the limit of how many rows to return. Default is -1, meaning no limit
+
* .map(f) - applies a function to all selected objects
 +
* .proxy(name, args...) - applies a named-proxy ("living", "private", or "referenced")
 +
* .limit() - set start or limit or selection
 +
* .count() - returns the number of matches
 +
* .tag(tag_name) - puts a tag on all selected items, if not tagged with tag_name already
  
As an example, consider selecting the gramps_id from all people who have a surname of "Smith" and whose name begins with a "J", ordered by the gramps_id:
+
These methods can be lined up to create a series of operations:
  
  db.select("Person",
+
  db.Person.filter(lambda person: person.private=True).select("gramps_id")
          ["gramps_id"],
 
          where=["AND", [("primary_name.surname_list.0.surname", "=", "Smith"),
 
                          ("primary_name.first_name", "LIKE", "J%")]],
 
          order_by=[("gramps_id", "ASC")])
 
  
The parameters "start" and "limit" are used for paged selects. The result will also return the total of the selection as if start or limit had not been given (see Result below).
+
== Database.QuerySet.select() ==
  
=== WHERE-EXPRESSION ===
+
db.Person.select()
 +
db.Person.select("gramps_id", "handle")
  
The where expression must be in one of these four forms (tuples or lists allowed):
+
== Database.QuerySet.order() ==
  
* None - no filter applied to data
+
db.Person.order("gramps_id").select()
* (dot-separated field path string, COMPARISON-OPERATOR, value)
 
* ["AND" | "OR", [WHERE-EXPRESSION, WHERE-EXPRESSION, ...]]
 
* ["NOT", WHERE-EXPRESSION]
 
  
COMPARISON-OPERATOR is one of:
+
Examples:
  
* "LIKE" - use with "%" wildcard
+
* db.Person.order("gramps_id").select()
* "="
+
* db.Person.order("primary_name.first_name", "-gramps_id").select()
* "!=", or "<>"
 
* "<
 
* "<="
 
* ">"
 
* ">="
 
* "IS"
 
* "IS NOT"
 
* "IN"
 
  
Currently, value is limited to be a non-database value, such as None, a str, int, or bool. Fields are not currently allowed on the right-hand side of the operator.
+
The first-listed field name is the primary sort, followed by secondary sorts. In the last example above, the data would be sorted first by first_name ascending, and inside that, by gramps_id descending, such as:
  
Examples:
+
Avery, I0003
 +
Avery, I0002
 +
Avery, I0001
 +
Barry, I0013
 +
Barry, I0012
 +
Barry, I0011
  
* ("primary_name.first_name", "=", "Mary")
+
== Database.QuerySet.filter() ==
* ["OR", [("primary_name.first_name", "=", "Mary"), ("primary_name.first_name", "LIKE", "Eliza%")]]
 
* ["NOT", ("primary_name.first_name", "=", "Mary")]
 
  
Note that where expressions may be recursively nested:
+
== Database.QuerySet.map() ==
  
* ["NOT", ["AND", ("primary_name.first_name", "=", "Mary"), ("gramps_id", "=", "I0003")]]
+
== Database.QuerySet.proxy() ==
  
=== ORDER-BY-LIST ===
+
== Database.QuerySet.limit() ==
  
The ORDER-BY-LIST is either None or is a list of dotted-field path strings paired with "ASC" or "DESC".
+
== Database.QuerySet.count() ==
  
Examples:
+
== Database.QuerySet.tag() ==
  
* [("gramps_id", "DESC")]
+
== Joins ==
* [("primary_name.first_name", "ASC"), ("gramps_id", "DESC")]
 
  
The first-listed field name is the primary sort, followed by secondary sorts. In the last example above, the data would be sorted first by first_name ascending, and inside that, by gramps_id descending, such as:
+
The database.select method can also do joins across primary objects in each of the WHERE, ORDER, or select fields. For example consider this request:
  
  Avery, I0003
+
  >>> db.Family.select("mother_handle.gramps_id")
Avery, I0002
 
Avery, I0001
 
Barry, I0013
 
Barry, I0012
 
Barry, I0011
 
 
 
== Result ==
 
  
The database.select() method will always return a Result. A result is a collection of all of the data (ie, it is not a generator). You can find out how many records are returned with len(result).
+
This will return the gramps_id's of the mothers of each family, if there is a mother.
  
Results are a subclass of the Python list object, with additional properties:
+
You can join across multiple tables with more complex queries, such as:
  
* result.total - total number of records (in the case of start or limit is given. In that case len(result) != result.total)
+
>>> list(db.Family.select("mother_handle.event_ref_list.ref.gramps_id"))
* result.time - the time in seconds it too to collect the data
 
* result.expanded - whether the data needed to be expanded (unpickled and primary objects created). BSDDB selects alway are expanded
 
* result.query - the actual SQL query, if one
 
  
When writing selects, the goal is to always have a query, and to always have expanded be False. Those will be the fastest queries. However, of course, that is not always possible. With BSDDB databases, there will never be a query, and the data will always be expanded.
+
That returns all of the Event gramps_ids for all events for the mother (Person) of all families, looking something like:
  
Each element in the Result list is a dictionary of dotted-field path strings and their data (if any). If a value is None, it either means that the value is None, or that there is no such field for this primary object. For example:
+
[{"mother_handle.event_ref_list.ref.gramps_id": ["E0001", "E0002"]},
 +
  {"mother_handle.event_ref_list.ref.gramps_id": "E0003"},
 +
  {"mother_handle.event_ref_list.ref.gramps_id": ["E0003", "E0001"]},
 +
  ...
 +
]
  
[{"gramps_id": "I0000"},
+
Note that these joins are done per record and are therefore not yet optimized---each requires another database access. In the future, these could be optimized via a SQL JOIN.
  {"gramps_id": "I0001"},
 
  {"gramps_id": "I0002"},
 
  {"gramps_id": "I0003"},
 
...]
 
  
 
== Implementation ==
 
== Implementation ==
Line 145: Line 129:
 
Using he dotted-field path string field-based Select API, we can write code as follows. Consider that we want to select the handle of all people whose surname is "Smith", given name starts with a "J", and ordered by gramps_id:
 
Using he dotted-field path string field-based Select API, we can write code as follows. Consider that we want to select the handle of all people whose surname is "Smith", given name starts with a "J", and ordered by gramps_id:
  
  db.select("Person",
+
  list(db.Person.filter(primary_name__surname_list__0__surname="Smith",  
          ["handle", "gramps_id"],
+
                      primary_name__first_name__LIKE="J%").order("gramps_id").select())
          where=["AND", [("primary_name.surname_list.0.surname", "=", "Smith"),  
 
                          ("primary_name.first_name", "LIKE", "J%")]],
 
          order_by=[("gramps_id", "ASC")])
 
 
   
 
   
 
This code works on BSDDB as well as DB-API. Let's see the difference in timing on databases that have 187,294 people (created from GenFan, this is 20 full generations).  
 
This code works on BSDDB as well as DB-API. Let's see the difference in timing on databases that have 187,294 people (created from GenFan, this is 20 full generations).  
Line 161: Line 142:
  
 
So, where we can access the data via SQL, we can get a speedup, the biggest will always be in the filter as it makes it so we don't have to load into Python many objects. We have linear code in many places that could benefit from using db.select().
 
So, where we can access the data via SQL, we can get a speedup, the biggest will always be in the filter as it makes it so we don't have to load into Python many objects. We have linear code in many places that could benefit from using db.select().
 +
 +
[[Category:GEPS|D]]

Latest revision as of 04:48, 23 December 2020

Starting with Gramps 5.0, there is a new manner to select data from the database. The goal of this API is to make it easy to select data by different criteria, and make selection as fast as possible.

This page documents the interface for developers.

Contents

QuerySet

Each database backend now supports the following QuerySet objects:

  • db.Person
  • db.Family
  • db.Note
  • db.Source
  • db.Citation
  • db.Place
  • db.Repository
  • db.Event
  • db.Tag

Each of these QuerySets allows you to chain together a series of QuerySet method calls. QuerySet supports the following methods:

  • .select([fields...]) - returns generator
  • .order(fieldname, ...) - order-by fields given as strings; use "-name" for descending
  • .filter(obj, args...) - applies a FilterObject or Python callable
  • .map(f) - applies a function to all selected objects
  • .proxy(name, args...) - applies a named-proxy ("living", "private", or "referenced")
  • .limit() - set start or limit or selection
  • .count() - returns the number of matches
  • .tag(tag_name) - puts a tag on all selected items, if not tagged with tag_name already

These methods can be lined up to create a series of operations:

db.Person.filter(lambda person: person.private=True).select("gramps_id")

Database.QuerySet.select()

db.Person.select()
db.Person.select("gramps_id", "handle")

Database.QuerySet.order()

db.Person.order("gramps_id").select()

Examples:

  • db.Person.order("gramps_id").select()
  • db.Person.order("primary_name.first_name", "-gramps_id").select()

The first-listed field name is the primary sort, followed by secondary sorts. In the last example above, the data would be sorted first by first_name ascending, and inside that, by gramps_id descending, such as:

Avery, I0003
Avery, I0002
Avery, I0001
Barry, I0013
Barry, I0012
Barry, I0011

Database.QuerySet.filter()

Database.QuerySet.map()

Database.QuerySet.proxy()

Database.QuerySet.limit()

Database.QuerySet.count()

Database.QuerySet.tag()

Joins

The database.select method can also do joins across primary objects in each of the WHERE, ORDER, or select fields. For example consider this request:

>>> db.Family.select("mother_handle.gramps_id")

This will return the gramps_id's of the mothers of each family, if there is a mother.

You can join across multiple tables with more complex queries, such as:

>>> list(db.Family.select("mother_handle.event_ref_list.ref.gramps_id"))

That returns all of the Event gramps_ids for all events for the mother (Person) of all families, looking something like:

[{"mother_handle.event_ref_list.ref.gramps_id": ["E0001", "E0002"]},
 {"mother_handle.event_ref_list.ref.gramps_id": "E0003"},
 {"mother_handle.event_ref_list.ref.gramps_id": ["E0003", "E0001"]},
 ...
]

Note that these joins are done per record and are therefore not yet optimized---each requires another database access. In the future, these could be optimized via a SQL JOIN.

Implementation

There are now two database backends: Berkeley DB (BSDDB), and Python's DB-API. BSDDB is a data store with much of the database code written in Python, and DB-API is a common interface to the popular SQL engines. We have used BSDDB in Gramps for many years, but are now transitioning to DB-API.

With BSDDB, Gramps has a pipeline design when it comes to accessing the data. For example, consider getting the People for the flat view. First we get a cursor that iterates over the data. Then we sort it, on whatever criteria we have requested. Finally, we filter the data. The select method will always perform a linear search on fully expanded data.

In order to make the select operation faster for DB-API, we need to know the filter information, and sort order when we ask for the data. With SQL we can simply add WHERE clauses and ORDER BY clauses to the basic SELECT statement. But these are only useful if we can have indexes on the relevant data.

This is made more difficult because Gramps uses a hierarchical representation of data. For example, we might wish to have the People data sorted by "surname, given" of the primary_name. But that information is actually in:

  • person.primary_name.surname_list[0].surname
  • person.primary_name.first_name

respectively. We can make special fields for these, and special indexes. Gramps 5.0 creates "secondary" fields and indexes in SQL for every str, int, or bool data on a primary object. These secondary fields are known from the primary object's schema.

The schema idea has been augmented with additional methods based on the idea of "fields". Now, you can ask a person object:

>> person.get_field("primary_name.first_name")
"Sarah"

and with some additional syntax:

>> person.get_field("primary_name.surname_list.0.surname")
"Johnson"

or even:

>> person.get_field("primary_name.surname_list.surname")
["Johnson", "Johansen", "Johnston"]

In the last example, the "surname" field was applied to each of the surname_list items.

Each primary object can also have a list of extra secondary fields/indexes. These are specified in the primary object's method "get_extra_secondary_fields()" which returns a list of dotted-path strings.

The dotted-field path strings are mapped to SQL names by replacing dots with two underscores.

Speed Tests

Using he dotted-field path string field-based Select API, we can write code as follows. Consider that we want to select the handle of all people whose surname is "Smith", given name starts with a "J", and ordered by gramps_id:

list(db.Person.filter(primary_name__surname_list__0__surname="Smith", 
                      primary_name__first_name__LIKE="J%").order("gramps_id").select())

This code works on BSDDB as well as DB-API. Let's see the difference in timing on databases that have 187,294 people (created from GenFan, this is 20 full generations).

Here is a summary:

       | Filter |  Select All   | Sort All
-------|--------|---------------|----------
BSDDB  | 20.6s  |       9.2s    | 18.1s
DB-API |   .3s  |        .5s    |   .6s

So, where we can access the data via SQL, we can get a speedup, the biggest will always be in the filter as it makes it so we don't have to load into Python many objects. We have linear code in many places that could benefit from using db.select().