Database Query API
Starting with Gramps 5.0, there is a new method on the database object called "select" that works as follows:
db.select(TABLE-NAME, SELECT-LIST, where=WHERE-LIST, order_by=ORDER-BY-LIST)
As an example, consider selecting the gramps_id from all people who have a surname of "Smith" and whose name begins with a "J", ordered by the gramps_id:
db.select("Person", ["gramps_id"], where=["AND", [("primary_name.surname_list.0.surname", "=", "Smith"), ("primary_name.first_name", "LIKE", "J%")]], order_by=[("gramps_id", "ASC")])
There are now two database backends: Berkeley DB (BSDDB), and Python's DB-API. BSDDB is a data store with much of the database code written in Python, and DB-API is a common interface to the popular SQL engines. We have used BSDDB in Gramps for many years, but are now transitioning to DB-API.
In order to make this operation faster for , we need to know the filter information, and sort order when we ask for the data. With SQL we can simply add WHERE clauses and ORDER BY clauses to the basic SELECT statement. But these are only useful if we can have indexes on the relevant data.
respectively. We could make special fields for these, and special indexes. But it would be much more flexible if we could create a variety of ad hoc queries on the fly.
The BSDDB datastore doesn't have any schema, which means that it has no idea of "gramps_id" or "primary_name" or any field. An idea of schema has been developed over the last few years. This makes possible the Database Differences Report, without having to write any field-specific code: the data knows its own structure.
The schema idea has been augmented with additional methods based on the idea of "fields". Now, you can ask a person object:
>> person.get_field("primary_name.first_name") "Sarah"
and with some additional syntax:
>> person.get_field("primary_name.surname_list.0.surname") "Johnson"
Building on that, a db.select method has been added so that you can get all of those fields from all People, and sort and filter on all of the regular (string, int) fields. This works independently of SQL. However, if you re-implement that method for DB-API, and have the appropriate sql-fields, and sql-indexes, then you have an large speed-up for large data.
So, our old system required a scan of all data, unpickling, creating objects, and sorting for any use. If you have 100k records, that required processing all of them. With the new DB-API implementation, you can do that same query in a fraction of that time. Views (written appropriately) will appear very quickly (milliseconds) regardless of the size of the database.
db.select("Person", ["handle", "gramps_id"], where=["AND", [("primary_name.surname_list.0.surname", "=", "Smith"), ("primary_name.first_name", "LIKE", "J%")]], order_by=[("gramps_id", "ASC")])
This code works on BSDDB as well as DB-API. Let's see the difference in timing on databases that have 187,294 people (created from GenFan, this is 20 full generations).
Here is a summary:
| Filter | Select All | Sort All -------|--------|---------------|---------- BSDDB | 20.6s | 9.2s | 18.1s DB-API | .3s | .5s | .6s
So, where we can access the data via SQL, we can get a speedup, the biggest will always be in the filter as it makes it so we don't have to load into Python many objects. We have linear code in many places that could benefit from using db.select().
Currently DB-API is automatically creating SQL fields and indexes for all regular (non-list, standard Python types) primary-object attributes (like gramps_id, privacy, etc.). This takes time and space. We may want to manage this a bit more carefully.