GEPS 013: Gramps Webapp

From Gramps
Revision as of 15:35, 2 August 2009 by Dsblank (talk | contribs) (Interactive Client)
Jump to: navigation, search

Proposition of a server mode for GRAMPS. Now that the main graphical user interface (GUI) has been separated from the command-line interface (CLI), a server-mode would be the next logical step.

Motivation

Web developers are in need of a method of accessing and creating functionality with their GRAMPS data on the web. Having a GRAMPS server-mode would allow a GRAMPS-based web project.

Sample Usage

A server mode for GRAMPS is suggested to work as follows. A server program is started, and run in the background. It listens for requests from other processes (on that machine, or others), executes commands, and serves back data.

The server mode is very similar to the CLI mode of GRAMPS. However, rather than just running a report, the program doesn't exit, but waits for requests. Clients need not keep track of any state. The server would host a single database, and allow multiple people to work on it simultaneously.

Functionality

A GRAMPS server should allow clients to do most of what a user would want to do with the GRAMPS gtk application:

  1. browse data, including all of the primary objects
  2. have links that would take the user directly to view a particular object's data
  3. a method of adding and editing existing data

Communication

In order for such a client/server architecture to work, there needs to be a method of communication between them. There needs to be a format for making requests from clients to the server, and a for returning results from the server to the clients.

There are plenty of options for communication formats, including XML. There is also a library called Python Remote Objects for handing this kind of data. However, one possibility is to use Python's native pickle format, as that can be easily sent between connections. Likewise, all of the primary data's raw data can be pickled. Using pickled data requires that no database objects be pickled---only data. One would have to be careful about how you write web applications so as not to involve objects such as generators, and databases.

Prototype

A prototype of a GRAMPS server mode has been checked into gramps/trunk (targeting version 3.2). You may have to run "./configure" and "make" after doing an "svn update". It works as follows.

Server

You start the server by selecting the database and listening port:

python src/gramps.py --server=50000 -O "My Family Tree"

(If you don't know your tree names, use "python src/gramps.py -L").

This will start a socket listener on port 50000. If the database is locked, then, like usually, you can supply the --force-unlock flag:

python src/gramps.py --server=50000 -O "My Family Tree" --force-unlock

In fact, you can do any of the things you normally do with the CLI, before beginning serving.

After starting, the server produces output like:

$ python src/gramps.py --server=50000 -O "MyRelate"
Opened successfully!
GRAMPS server listening on port 50000.
Use CONTROL+C to exit...
--------------------------------------------------
Connection opened from 127.0.0.1:50114
Connection closed from 127.0.0.1:50114
Connection opened from 127.0.0.1:50115
    Request: self.dbstate.db.surname_list.__getslice__(0, 2147483647)
Connection closed from 127.0.0.1:50115

The server currently lists each connection start/end, and each request. The server is written in a fashion such that it can take many requests at once. It spawns off threads that handle the details of the request. TODO: does memory get reclaimed appropriately?

The server receives messages that it evaluates, pickles, and returns.

Interactive Client

To make programming as natural as possible on the client side, a RemoteObject has been written which hides the gory details of the socket communication. Here is a sample client:

from cli.client import RemoteObject
self = RemoteObject("localhost", 50000)

Here, you can use "self" very similarly to the use of self in the server. For example:

$ python
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26) 
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from cli.client import RemoteObject
>>> self = RemoteObject("localhost", 50000)
>>> self.dbstate
<DbState.DbState object at 0x98e088c>
>>> self.dbstate.db
<gen.db.dbdir.GrampsDBDir object at 0x9dc4c0c>

Under the hood, the names "db" and "dbstate" are collected, sent over the socket to the server as the string "self.dbstate.db" which is evaluated, and the representation is sent back. Notice that the server cannot return an entire dbstate or db object to the client, but it merely returns the repr string.

You can interactively explore the remote objects, too:

>>> self.dbstate.db.dir()
['_Callback__BLOCK_ALL_SIGNALS', '_Callback__LOG_ALL', '_Callback__block_instance_signals',
...
'source_prefix', 'surname_list', 'surnames', 'transaction_begin', 'transaction_commit', 
'translist', 'txn', 'undo', 'undo_available', 'undo_callback', 'undo_data', 
'undo_history_callback', 'undo_history_timestamp', 'undo_reference', 'undodb', 
'undoindex', 'undolog', 'update_empty', 'update_real', 'update_reference_map', 'url_types', 
'version_supported', 'write_version']

Notice that you can't wrap a "dir()" around a property, but you can tack a ".dir()" on the end to provide the same functionality. You can't wrap a dir() around the self.dbstate.db because that would get applied on the client side, and by the time self.dbstate.db is evaluated, it is just the repr string.

But, you can also get back some full GRAMPS objects:

>>> self.dbstate.db.get_default_person()
<gen.lib.person.Person object at 0xb7d7ac6c>

This isn't just the repr string returned this time, but a real object. You can test that by:

>>> p = self.dbstate.db.get_default_person()
>>> p.get_primary_name().get_surname()
u'Blank'

Because you can't get back generators nor database objects, sometimes you need to do some processing on the server. To do this, you can use self.remote:

>>> self.remote("name = self.sdb.name(self.dbstate.db.get_default_person())")
>>> self.remote("name")
u'Blank, Living'

self.remote takes a string and sends it to the server to be evaluated (or executed). self contains:

>>> self.dir()
['__doc__', '__init__', '__module__', 'arghandler', 'climanager', 'dbstate', 'env', 'eval', 'reset', 'sdb']
  • arghandler - the object that handles the command-line operations
  • climanager - low-level CLI management
    • db_loader
    • dbstate
    • do_load_plugins
    • file_loaded
    • open_activate
  • clidbmanager - Functions related to database management
    • active
    • break_lock
    • create_new_db_cli
    • current_names
    • dbstate
    • empty
    • family_tree_list
    • family_tree_summary
    • get_dbdir_summary
    • get_family_tree_path
    • icon_values
    • import_new_db
    • is_locked
    • msg
    • needs_recovery
  • dbstate - holds the state of the database
    • dbstate.db - the database
  • sdb - simple database access

Web Client

Finally, here is an example for listing out the surnames of a family tree on the web:

#!/usr/bin/python

import sys, os
sys.path.append("/home/dblank/gramps/trunk/src")
os.environ["HOME"] = "/home/dblank/html/gramps"

from cli.client import *

self = RemoteObject("localhost", 50000)

print "Content-type: text/html"
print
print "<html>"
print "<body>"
print "First Demo of GRAMPS --server"

surnames = self.dbstate.db.surname_list[:]

for name in surnames:
    print "<li><a href=\"?surname=%s\">%s</a></li>" % (name, name)

print "<hr>"
print "</body>"
print "</html>"

Save the above in a file ending with .cgi. I used index.cgi. I then configured apache to allow CGI, added index.cgi, turned off SELiux, and started up the httpd service.

The resulting screen shot:

GRAMPS-server.png

Discussion

  • I had to make some other changes in GRAMPS that I haven't committed yet that allows GRAMPS to run without having X or a display. But you won't notice this issue when you run the code if you have a display---it only shows up when running without one.
  • It isn't clear yet from the prototype if this interface will be able to do everything. I have changed the default person in the dbstate, and I have changed a name. Can you get all (and edit all) of the data by just using handles?
  • The interface allows for creating variables on the server, but that could get overwritten by multiple clients. Do we need variables, and how can we keep them separate by client?
  • A new interface may need to be created, similar to SimpleAccess, but for this type of communication.
  • How could security be implemented to protect one's data from viewing or edits?
  • Note that this proposal does not limit the webapp language to be Python. Any language that implements the Python pickle format can be used. In reality, the system uses a subset of pickle. In addition, there are raw_data only methods, so one restrict further the required subset of functionality of the pickle.

The next step would be to build a web app. The organisation of the web app could also be discussed here, although that would quickly become a bigger topic. In any event, some notes on this aspect of the proposal are below.

Web App Architecture

  • There is a rewritten GRAMPS html formatting library ./src/plugins/lib/libhtml.py which should be used
  • Consider the report system to generate the web app client
  • Use CGI at first
  • Use the new CSS tags and organization
  • Use as much as possible from Narrated and Webcal reports
  • This could be a new plugin type, webapp