Proposition of Many Gramps users would like to collaborate or share their genealogy data on the web. This GEP describes a webapp, a web-based application that runs in your browser, and requires a server mode for GRAMPS. Now that A prototype is on-line at http://gramps-connect.org/ which is running trunk on a sample database. You can log into the main graphical site, as a:*superuser (id=admin, password=gramps) or a *regular user interface (GUIid=admin1, password=gramps) has been separated from the commandor just view as an anonymous user. There are two additional pages on this project: * [[Gramps-Connect]] - getting started* [[Gramps-line interface (CLI), a serverConnect: Developer Introduction]] -mode would be the next logical step.introduction for developers
== Motivation ==
Web developers are in need The main focus of a method of accessing and creating functionality with Gramps-based webapp is collaboration. The Gramps webapp will allow users to easily move their GRAMPS genealogy data on to the web. Having to be seen, and edited with proper login and permissions, in a GRAMPS server-mode would allow a GRAMPS-based web projectlive, collaborative environment.
Here is a small list of goals:
# Create an interface that would allow very easy a fullscale Gramps web deployment of a GRAMPS-based system, including CGI.framework # Use a client/server architecture that would allow at least one person to use the GRAMPS database on Allow multiple users via the standard web. This should be able to be expanded to multiple users (need to do some multi-user tests).browser## Users will log in and have various levels of permissions# Restrict use to ports Build on same machine. Restrict processing to databaseGramps codebase and wealth of resources## Reports## Tools## Visualizations## Date and calendar functions## Translations## Manual and documentation# Use standards and well-only operations. Initial use will require all users on machine to be trusted. At the worstknown, a user on the machine could view, edit or delete your genealogy data. That might sound bad, but is not as bad as deleting your files, or send spam from your account. (Looking at some method to restrict port or authenticate users...)well-tested frameworks where possible## WSGI protocol for running code# Requests to server will be a string repr of Python function calls, or a simple XML representation of a function call.# Django framework# Returned data is in a simple XML format. Currently, this isn't our GRAMPS XML, but could be.# Underlying powerful database engines
== Sample Usage = FAQ ===
A server mode for GRAMPS is suggested to work as follows1. A server program is started''Aren't there already many fine, and run in web-based genealogy programs? Why don't you just use one of those? Aren't you re-inventing the background. It listens for requests from other processes (on that machine, or others), executes commands, and serves back data. wheel?''
The server mode is very similar to the CLI mode of GRAMPSThere are indeed many fine, web-based genealogy programs, and some are even free software/open source. However, rather than just running there are a report, the program doesn't exit, but waits for requests. Clients need not keep track of any state. The server would host few good reasons to develop a single database, and allow multiple people to work on it simultaneously.Gramps-based webapp:
== Functionality ==# Gramps has hundreds of thousands of lines of code, some of which could be re-used directly in a webapp. For example, the reports could be run and downloaded directly from the webapp.# Gramps has a very well-defined set of tables and relationships that could be re-implemented for on-line use. # Users have grown to appreciate the design of Gramps, and we want to continue to build on this design.# Many users want to collaborate. Currently, they would either have to move their data in and out of Gramps, or give up Gramps completely.# We want to keep the developers and users that we have, and so not splinter our groups. By building the webapp on top of core gramps code, we continue to refine and make better our current code, and keep our current developers working on the parts that they know and love.
A GRAMPS server should allow clients to 2. ''Why do most of what you need a user would want to do with web framework like Django? Can't you just use the GRAMPS gtk application:same Python code, and same database that Gramps already uses?''
# browse dataWe can't use the same database (what is called a "backend") directly. Currently Gramps uses BSDDB, and it is not configured for use in a multiuser, including all of client/server environment. But even if we could use the primary objects# have links that same backend, we would take still want some type of web development framework. Django is one of the user directly best in any language, and it just happens to view a particular object's data# a method of adding and editing existing databe in Python.
== Communication ==3. ''How easy will this be for me to use on my website?''
In order for such a client/server architecture We have designed it to workbe as easy as it can be, there needs to be a method of communication between themgiven that we are using Python. There needs to be a format for making requests from clients to the serverMany web sites allow Python programs, and a for returning results from Django allows many different variations in running. We picked the server to protocol with the clientsmost availability (WSGI). Don't worry if you haven't heard of it. Your webserver can probably run it.
There are plenty of options for communication formats, including XML4. There ''When is also a library called [http://pyro.sourceforge.net/ Python Remote Objects] for handing this kind of data. However, some possibilities:going to be available?''
* use Python's native pickle format, as that can be easily sent between connectionsWe are hoping to have a fully functioning webapp ready for testing July 2010. * use a simple XML schema* use the GRAMPS XML schema
Whichever method is chosen, all of the primary data5. ''s raw data How can be encoded. Using encoded data requires that no database objects be encoded---only data. One would have to be careful about how you write web applications so as not to involve objects such as generators, and databases.I help?''
== Prototype ==You can start by reading the rest of this page and sending ideas and comments to the Gramps-developers mailing list, and running the code if you can.
A prototype of a GRAMPS server mode has been checked into gramps/trunk (targeting version 3.2). You may have to run "./configure" and "make" after doing an "svn update". It works as follows.== Overview ==
=== Server ===The Gramps webapp is written in Django. Django is a sophisticated, modern web development framework. Django is written in Python, albeit in a very different style from Gramps. However, part of the motivation of using Django is that it breaks up web development into very clearly defined parts following the [http://en.wikipedia.org/wiki/Model_view_controller Model-View-Controller] paradigm. Two of these parts require no special programming knowledge, and thus will allow more people to be able to possibly customize and participate in the Gramps project.
You start the server by selecting the database The Gramps webapp (and listening portDjango in general) is broken into three well-defined parts:
python src# models/gramps.py --server=50000 -O "My Family Tree"views# templates # CSS (Cascading Style Sheets)
The models define the tables and relationships, but this is done in Python (If you don't know your tree namesnot SQL). The models also define the API to read/writing/editing the data. The views are also written in Python, and are closely tied to the models. The templates are written in HTML and a template language that is very easy for non-programmers to use "python src/grampsand understand. Finally, CSS is just Cascading Style Sheets, where all of the graphical definitions are made.py The webapp uses pre-Lexisting CSS created for the ")Narrated Web" report of Gramps which was used for created static web pages. Let's take a look at specific examples of each of these parts.
This will start a socket listener on port 50000. If the database is locked, then, like usually, you can supply the --force-unlock flag:=== Models/Views ===
python srcHere is the model that defines the Person table from [{{Code Base}}gramps/webapp/grampsdb/models.py gramps/webapp/grampsdb/models.py --server=50000 -O "My Family Tree" --force-unlock]:
In fact<pre>class Person(PrimaryObject): gender_type = models.ForeignKey('GenderType') families = models.ManyToManyField('Family', you can do any of the things you normally do with the CLIblank=True, before beginning servingnull=True) parent_families = models.ManyToManyField('Family', related_name="parent_families", blank=True, null=True) references = generic.GenericRelation('PersonRef', related_name="refs", content_type_field="object_type", object_id_field="object_id")</pre>
After startingHere, you can see that Person only has 4 parts: gender_type, families, parent_families, and references. There are additional properties, but they are defined in the server produces output likePrimaryObject class which is shared with other tables. Here is PrimaryObject:
<pre>
$ python src/grampsclass PrimaryObject(models.py --serverModel): class Meta: abstract =50000 -O "MyRelate"TrueOpened successfully! id = models.AutoField(primary_key=True)GRAMPS server listening on port 50000 handle = models.CharField(max_length=19, unique=True)Use CONTROL+C to exit.. gramps_id = models.CharField('gramps id', max_length=25, blank=True)--------------------------------------------------Connection opened from 127 last_saved = models.0.0.1:50114DateTimeField('last changed', auto_now=True) Connection closed from 127 last_changed = models.0.0.1:50114DateTimeField('last changed', null=True,Connection opened from 127.0.0.1:50115 blank=True) # user edits Request: self.dbstate.dbprivate = models.surname_list.__getslice__BooleanField(0, 2147483647'private')Connection closed from 127 marker_type = models.0.0.1:50115ForeignKey('MarkerType')
</pre>
The server currently lists each connection start/endbig difference here between typical Python programming is that the Person class defines the Person table, and each requestthe interface to it. The server is written in Most Python code would probably have Person be an instance of a fashion such that it can take class, but Django uses classes to represent many requests at oncethings. It spawns off threads that handle Here are three examples using the details of the request. TODOPerson class: does memory get reclaimed appropriately?
The server receives messages that it evaluates<pre> % cd trunk % PYTHONPATH=src DJANGO_SETTINGS_MODEL=webapp.settings python >>> from webapp.grampsdb.models import Person >>> Person.objects.all() [<Person>, <Person>, encodes it ...] >>> Person.objects.get(pickle or XMLid=1), and returns <Person> >>> Person.objects. get(handle='gh71234dhf3746347734') <Person></pre>
=== Interactive Client ===The first retrieves all of the rows for the Person table; the second retrieves just the one record that has the unique, primary key 1, and the third retrieves the single record that has the unique handle of 'gh71234dhf3746347734'. Note that we never connected onto a database... Django is (currently) define to connect on to one database, and it does it on import. The database is set in src/webapp/settings.py.
To make programming as natural as possible on the client side, a RemoteObject has been written which hides the gory details An alternative method of interactively talking to the socket communicationdatabase is to use ''manage. Here is a sample clientpy'':
<pre>
from cli % cd trunk/src/webapp % PYTHONPATH=..client import RemoteObject/../src python manage.py shellself = RemoteObject("localhost", 50000) >>>
</pre>
HereThat will give you an ipython shell, if you have it installed. Very nice environment! You can also use "self" very similarly the Person interface to the use select a subset of self in the server. For examplepeople:
<pre>
$ pythonPython 2.6 (r26:66714, Jun 8 2009, 16:07:26) [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2Type "help", "copyright", "credits" or "license" for more information. >>> from cliwebapp.client grampsdb.models import RemoteObject* >>> self Person.objects.filter(gender_type= RemoteObject("localhost", 500001) [<Person>>> self.dbstate, <DbState.DbState object at 0x98e088cPerson>>>> self, .dbstate.db<gen.db.dbdir.GrampsDBDir object at 0x9dc4c0c>]
</pre>
Under the hood, the names "db" and "dbstate" are collected, sent over the socket to the server as the string "self.dbstate.db" which is evaluated, and the representation is sent back. Notice that the server cannot return an entire dbstate or db object to the client, but it merely returns the repr string. You can interactively explore the remote objects, tooeven more clearly:
<pre>
>>> selfPerson.dbstateobjects.db.dirfilter(gender_type__name="Male") ['_Callback__BLOCK_ALL_SIGNALS'<Person>, '_Callback__LOG_ALL', '_Callback__block_instance_signals'<Person>,...'source_prefix', 'surname_list', 'surnames', 'transaction_begin', 'transaction_commit', 'translist', 'txn', 'undo', 'undo_available', 'undo_callback', 'undo_data', 'undo_history_callback', 'undo_history_timestamp', 'undo_reference', 'undodb', 'undoindex', 'undolog', 'update_empty', 'update_real', 'update_reference_map', 'url_types', 'version_supported', 'write_version']
</pre>
Notice that you can't wrap a The double-underscore in the keyword "dir()gender_type__name" around a property, but you can tack of the filter method is a "Django convention.dir()It means " on replace wth the end to provide the same functionalitycorrect syntax". You canIf Python allowed it, it would be written as 't wrap a dir''Person.objects.filter(gender_type.name="Male") around the self.dbstate.db because ''' but that would get applied on the client side, and by the time self.dbstate.db is evaluated, it is just the repr stringnot legal syntax.
But, you can also get back some full GRAMPS objects==== Model overview ====Here is an overview of all of the models and how they are related:
<pre>[[Image:all-tables.gif]] {{out of date}}>>> self{{man tip| 1=To update this (Gramps 3.dbstatex and earlier) |2=To see more graphical representations of the data, run "make docs" in the src/webapp/ directory, and then look in src/webapp/docs/.db}} * [https://gramps-project.get_default_person()<org/docs/gen/gen_db.lib.person.Person object at 0xb7d7ac6c>html#dbdjango Gramps DbDjango] </pre>=== Templates ===
This isnTemplates are used to describe 't just the repr string returned this time, but 'what'' to display. Here is a real objecttemplate from [{{Code Base}}data/templates/main_page.html data/templates/main_page. You can test that byhtml]:
<pre>
>>> p = self{% extends "gramps-base.dbstate.db.get_default_person()html" %}>>> p.get_primary_name().get_surname()u'Blank'{% block title %}GRAMPS Connect - main page {% endblock %}</pre>{% block heading %}GRAMPS - main page {% endblock %}
Because you can't get back generators nor database objects, sometimes you need to do some processing on the server. To do this, you can use self.remote:{% block content %}
>>> self.remote(<p id="name = self.sdb.name(self.dbstate.db.get_default_person())description") >>> selfWelcome to GRAMPS Connect, a new web-based collaboration tool.remote("name") u'Blank, Living'
self{% if user.remote takes is_authenticated %} You are now logged in as <a string and sends it to the server to be evaluated (or executed)href="/user/{{user. self contains:username}}">{{user.username}}</a>.{% endif %}</p>
<prep id="description">Database information:<ul>{% for view in views %} <li><a href="/{{view|lower}}">{{view}}</a></li>{% endfor %}</ul></p> self.dir()['__doc__', '__init__', '__module__', 'arghandler', 'climanager', 'dbstate', 'env', 'eval', 'reset', 'sdb']{% endblock %}
</pre>
* arghandler - the object that handles the command-line operations** dbman - Functions related to database management*** active*** break_lock*** create_new_db_cli*** current_names*** dbstate*** empty*** family_tree_list*** family_tree_summary*** get_dbdir_summary*** get_family_tree_path*** icon_values*** import_new_db*** is_locked*** msg*** needs_recovery* climanager - low-level CLI management** db_loader** dbstate** do_load_plugins** file_loaded** open_activate* dbstate - holds the state of the database** dbstate.db - the database* sdb - simple database access=== CSS ===
=== Web Client ===Finally, here is a screen shot of the main_page.html (above) showing some initial testing of Gramps in Django using the Mainz CSS from the NarrWeb report:
Finally, here is an example for listing on the web the properties and surnames of a family tree[[Image:Gramps_in_django.gif]]
<pre>=== Getting Started with Gramps in Django === A prototype of a Gramps Django webapp is now in trunk and gramps32. To run it, do the following: # Download Django version 1.3 or greater## On yum-based systems, try "yum install Django"## On apt-based systems, try "sudo apt-get install python-django"##!Other systems: get the sources from http://www.djangoproject.com/download/usr# clone the Git repository and checkout either the gramps32 or master branch # cd src/binweb/python# Build the database, and load with default data:## make clean## make## This will ask for an id, email, and password for a superuser. You can add one later if you don't do it now.# Run the test webserver:## make run# Point your webbrowser to:## http://127.0.0.1:8000/ At this point, you can now export your Gramps data to Django (and back). In another terminal window:
import sys, os# Start up gramps:sys## cd .path.append("/home/dblank..## python src/gramps.py# Download the Django Import/trunk/src")osExport Addon from [[3.environ["HOME"3_Addons]] = "/home/dblank/html/gramps"# Run the Django Exporter## Select Family Tree -> Export## Select Django
from cliThis will export your regular Gramps BSDDB data into whatever Django database you have defined in settings.py above. You now have your data in a sqlite SQL database, and can access it via the webbrowser.client import *
self = RemoteObject("localhost", 50001)To import data back from Django's SQL table back into Gramps from the website:
print # Create a file named "Content-type: text/htmlimport.django" somewhere (just needs to end in ".django").print# Start up this version of Grampsprint "<html>"## python src/gramps.py# Run the Django Importerprint "<body## Select Family Tree ->"Import [[Image:DjangoImportExport.jpg|thumb|right|150px]]print ## Select the "<h1>First Demo of GRAMPS --server</h1>import.django"(from above) as the file to import
dbfile = self.dbstate.db.full_namepeople, dbversion = self.arghandler.dbman.get_dbdir_summaryTo add a superuser (dbfileafter the initialization)summary_list = self.arghandler.dbman.family_tree_summary()summary_dict = {}:
for sdict in summary_list:# cd src/web if sdict["Path"] # PYTHONPATH== dbfile: summary_dict = sdict../../src python manage.py createsuperuser
print "<h2>Properties:</h2>"for prop in summary_dict: print "<b>%s</b>For more on Django, try their tutorial: %s <br/>" % (prop, summary_dict[prop])
print "<h2>Surnames* Tutorial: http:</h2>"surnames = self/docs.dbstatedjangoproject.db.surname_list[:10]for name in surnames: if len(name) == 0: name = "[Missing Surname]" print "<li><a href=\"?surname=%s\">%s<com/en/dev/intro/a><tutorial01/li>" % (name, name)#intro-tutorial01
print "<hr>"print "</body>"print "</html>"</pre>=== Webapp Files ===
Save the above in a file ending with .cgi. I used index.cgi. I then configured apache There are two subdirectories and two files of interest to allow CGI, added index.cgi, turned off SELinux, and started up the httpd service.Gramps webapp:
The resulting screen shot# {{Code Base}}data/templates/ - HTML templates# {{Code Base}}gramps/webapp/ - Webapp main directory## {{Code Base}}gramps/webapp/libdjango.py - library interface## {{Code Base}}gramps/webapp/grampsdb - gramps table models# http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ExportDjango.py?view=markup - Exporter# http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ImportDjango.py?view=markup - Importer
[[Image:Screenshot-server.gif]]== Roadmap ==
== Discussion == Phase 1: get the basic Django skeleton in place, including the core HTML templates, models, views, and templatetags. Should be able to browse the 8 primary tables. Get translations in place. Goal for version 0.1 to be announced with Gramps 3.2 in March 2010.
* I had to make some other changes in GRAMPS that I haven't committed yet that allows GRAMPS Phase 2: Be able to run without having X or a display. But you won't notice this issue when you run all of the code if you have a display---it only shows up when running without one.* It isn't clear yet reports directly from the prototype if this web with an option interface will be . Be able to do everything. I have changed import/export from the default person in the dbstate, and I have changed a nameweb. Can you get all (and edit all) of the data by just using handles?* The interface allows for creating variables This will largely depend on the server, but that could get overwritten by multiple clientsa gen/db/dbdjango library. Do we need variables, and how can we keep them separate by client?* A new interface may need to be created, similar to SimpleAccess, but Goal for this type of communication.* How could security be implemented to protect one's data from viewing or edits?* Note that this proposal does not limit the webapp language to be Pythonversion 0. Any language that implements the encoding format can be used. In reality, the system uses very few types. In addition, there are raw_data only methods on the DB5beta, so one restrict further the required subset of functionality of the encoderMay 2010.
Richard Taylor notedPhase 3:add and edit data from the web. This would complete the functionality of the web interface. Goal July 2010.
* I think that you need to think very carefully about the use of the Threads in the server model. I don't think that database backend if thread safe Phase 4: Refine and strange things might happenpolish. (I have not looked at the database for about Release with Gramps 3 years, so things might have changed). You might think about a async server approach using something like twisted.* The use of eval to execute the client instructions is very unsafe from a security point of very. If you are going to use this model you should restrict connections to those from localhost or use only unix domain sockets or think about strong authentication3.
Brian Matherly notedIf you would like to work on an area, please note it here:
* I think the RPC strategy should be XML based. In particular, it makes sense # Kathy - edits and adding new data# Doug - Integration with gramps core; browsing data# - Translation system# - Proxy interface to me for the Gramps "server" to be Web Services basedshow Private data# - concurrent edits# - date widget# - running reports interface# - media files. We should look at something standard like SOAP. That will make it most friendly for clients to access and opens a whole world of unimagined possibilities in the future for mashups.where do they go?* If the Gramps project is eventually expanded to include a "web server"/ "web app"/"web services" aspect# - options interface, I strongly believe that it should be done as a separate application. In fact, the end goal should probably be for a multiediting options to run report# -part repository of code that includes: A GTK based desktop application, a CLI only application with much fewer dependencies, a import GEDCOM from web server application that provides a web service, a web server application that serves actual web pages (real time NavWeb), and one or more core libraries to support these applications in a well thought# -out, abstract and maintainable manorfull djangodb.py to replicate all functions of bsddb* In order to archive the goals I listed in #2- user support (email, we first need agreement from the developers that we want to go that direction. If we achieve thatmailing lists, we need an architectural plan that can bring us there. If notpermissions, we should branch this idea off as a separate project on SF.etc)
Benny Malengier noted:== Issues ==
* BSDDB has a multiuser flag (which is not switched on), to allow for things like this. * Goal 1 should be a definition of how a server can work on bsddb using present src/gen, how request from a client can be done, and how replies should be structured (I would hope our own xml schema). Then some framework to do the client itself, so as not to reinvent the wheel.* Goal 2 a barebones client: a listview of all primary objects (shown per 25/50/100 entries ), and html editor for the main info of the primary objects.* Thoughts about breaking GRAMPS into parts* For a server part I would restrict to a linux server strictly now.=== Concurrent Edits ===
The next step would be to build a web appConcurrent access for write and read imply several problems when people by accident change the same objects at the same time. The organization of the web app could also be discussed here, although that would quickly become a bigger topicGramps itself has an elaborate signal handling for cases when dialogs are open with no longer current information. In any eventa web environment, some notes on this aspect of the proposal are belowbecomes more difficult however. This is not built into Django.
== Web App Architecture ==For discussion on this issue in Django, see:
* There is a rewritten GRAMPS html formatting library [http://groups.google.com/group/srcdjango-users/pluginsbrowse_thread/libthread/libhtml.py which should be usedc138ec11c6ad282e?hl=en# Django User Question] * Consider the report system to generate the web app client* Use CGI at first* Use the new CSS tags and organization* Use as much as possible from Narrated and Webcal reports* This could be a new plugin report type, webapp[http://groups.google.com/group/django-developers/browse_thread/thread/fd5d45fc6cd6a760 Developer discussion on topic]
== Notes Example GMS Web Sites ==
Sometimes you may need to add We now have a new function example gramps webapp on the server. The original db.find_backlink_handles returned a generator, but we can't use that. Here it is as a listweb:
$ python -i src* http:/cli/clientgramps-connect.py localhost 50002 GRAMPS Remote interface; use 'self' to access GRAMPS >>> db = self.dbstate.db >>> default_person = db.get_default_person() >>> db.find_backlink_handles_list(default_person.handle)org/
Note that a remote reference (not a function call) such as '''self.dbstate.db.full_name''' returns a reference to a temporary object that will repeatedly contact Genealogy Management Systems on the server when referenced. web:
>>> full_name = self* http://www.dbstatedertinger.dbde/Dertinger_database/en/en_index.full_namehtm (Oxy-gen)* http://www.admiraal.org (PhpGedView): Note here: the intro page is a collection of gadgets/controls, which then link into the real data.* http://webtrees.net/demo/next (webtrees)* http://beck.org.il/humogen/ (HuMogen)* http://genealogies.geneamania.net/servin/ (Généamania)* http://www.geneotree.com/geneotree/index.php (Geneotree)* http://ancestorsnow.com/ancestors* http://www.phpmyfamily.net/demo/* http://www.frog.za.net/family/surname-list.php ([[Other_genealogy_tools#Gramps-php-exporter|gramps-php-exporter]])
No server access.Collaborative database (user/wizard/password):
>>> full_name* http://roglo.eu/roglo?lang=en (GeneWeb) '* http://gennus.org ([http://beta.gennus.org/homeen/dblankpage/about.grampshtml beta][http:/grampsdb/4a79400a'beta.gennus.org/en/page/releasenotes.html]) >>> full_name '* http://homebrozer.fr (alpha[http:/dblank/www.grampsinnovup.com/grampsdbevenement/4a79400a'124/89-actualites-agenda.htm])
Accesses the server twice. To make these types of references actual values on the client side, use something likeSource oriented:
>>> full_name = self* http://solumslekt.dbstateorg/forays/yggdrasil.db.full_namephp [http:] >>> full_name '/home/dblank/code.google.grampscom/grampsdbp/4a79400a' >>> full_name 'yggdrasil-genealogy/home][http:/dblank/solumslekt.grampsorg/grampsdbblog/4a79400a']
Only one server access (the assignment). Afterward it is local.==See also==*[[Gramps-Connect: Introduction|gramps-connect]]
[[Category:GEPS|SG]]
