Difference between revisions of "GEPS 013: Gramps Webapp"

From Gramps
Jump to: navigation, search
m (Web App Architecture)
(Model overview)
(111 intermediate revisions by 11 users not shown)
Line 1: Line 1:
Proposition of a server mode for GRAMPS. Now that the main graphical user interface (GUI) has been separated from the command-line interface (CLI), a server-mode would be the next logical step.
+
Many Gramps users would like to collaborate or share their genealogy data on the web. This GEP describes a webapp, a web-based application that runs in your browser, and requires a server.  
 +
 
 +
A prototype is on-line at http://gramps-connect.org/ which is running trunk on a sample database. You can log into the site, as a:
 +
*superuser (id=admin, password=gramps) or a
 +
*regular user (id=admin1, password=gramps)
 +
or just view as an anonymous user.
 +
 
 +
There are two additional pages on this project:
 +
 
 +
* [[Gramps-Connect]] - getting started
 +
* [[Gramps-Connect: Developer Introduction]] - introduction for developers
  
 
== Motivation ==
 
== Motivation ==
  
Web developers are in need of a method of accessing and creating functionality with their GRAMPS data on the web. Having a GRAMPS server-mode would allow a GRAMPS-based web project.
+
The main focus of a Gramps-based webapp is collaboration. The Gramps webapp will allow users to easily move their genealogy data to the web to be seen, and edited with proper login and permissions, in a live, collaborative environment.
 +
 
 +
Here is a small list of goals:
 +
 
 +
# Create a fullscale Gramps web framework
 +
# Allow multiple users via the standard web browser
 +
## Users will log in and have various levels of permissions
 +
# Build on Gramps codebase and wealth of resources
 +
## Reports
 +
## Tools
 +
## Visualizations
 +
## Date and calendar functions
 +
## Translations
 +
## Manual and documentation
 +
# Use standards and well-known, well-tested frameworks where possible
 +
## WSGI protocol for running code
 +
## Django framework
 +
## Underlying powerful database engines
  
== Sample Usage ==
+
=== FAQ ===
  
A server mode for GRAMPS is suggested to work as follows. A server program is started, and run in the background. It listens for requests from other processes (on that machine, or others), executes commands, and serves back data.
+
1. ''Aren't there already many fine, web-based genealogy programs? Why don't you just use one of those? Aren't you re-inventing the wheel?''
  
The server mode is very similar to the CLI mode of GRAMPS. However, rather than just running a report, the program doesn't exit, but waits for requests. Clients need not keep track of any state. The server would host a single database, and allow multiple people to work on it simultaneously.
+
There are indeed many fine, web-based genealogy programs, and some are even free software/open source. However, there are a few good reasons to develop a Gramps-based webapp:
  
== Functionality ==
+
# Gramps has hundreds of thousands of lines of code, some of which could be re-used directly in a webapp. For example, the reports could be run and downloaded directly from the webapp.
 +
# Gramps has a very well-defined set of tables and relationships that could be re-implemented for on-line use.
 +
# Users have grown to appreciate the design of Gramps, and we want to continue to build on this design.
 +
# Many users want to collaborate. Currently, they would either have to move their data in and out of Gramps, or give up Gramps completely.
 +
# We want to keep the developers and users that we have, and so not splinter our groups. By building the webapp on top of core gramps code, we continue to refine and make better our current code, and keep our current developers working on the parts that they know and love.
  
A GRAMPS server should allow clients to do most of what a user would want to do with the GRAMPS gtk application:
+
2. ''Why do you need a web framework like Django? Can't you just use the same Python code, and same database that Gramps already uses?''
  
# browse data, including all of the primary objects
+
We can't use the same database (what is called a "backend") directly. Currently Gramps uses BSDDB, and it is not configured for use in a multiuser, client/server environment. But even if we could use the same backend, we would still want some type of web development framework. Django is one of the best in any language, and it just happens to be in Python.
# have links that would take the user directly to view a particular object's data
 
# a method of adding and editing existing data
 
  
== Communication ==
+
3. ''How easy will this be for me to use on my website?''
  
In order for such a client/server architecture to work, there needs to be a method of communication between them. There needs to be a format for making requests from clients to the server, and a for returning results from the server to the clients.
+
We have designed it to be as easy as it can be, given that we are using Python. Many web sites allow Python programs, and Django allows many different variations in running. We picked the protocol with the most availability (WSGI). Don't worry if you haven't heard of it. Your webserver can probably run it.
  
There are plenty of options for communication formats, including XML. There is also a library called [http://pyro.sourceforge.net/ Python Remote Objects] for handing this kind of data. However, one possibility is to use Python's native pickle format, as that can be easily sent between connections. Likewise, all of the primary data's raw data can be pickled. Using pickled data requires that no database objects be pickled---only data. One would have to be careful about how you write web applications so as not to involve objects such as generators, and databases.
+
4. ''When is this going to be available?''
  
== Prototype ==
+
We are hoping to have a fully functioning webapp ready for testing July 2010.
  
A prototype of a GRAMPS server mode has been checked into gramps/trunk (targeting version 3.2). You may have to run "./configure" and "make" after doing an "svn update". It works as follows.
+
5. ''How can I help?''
  
=== Server ===
+
You can start by reading the rest of this page and sending ideas and comments to the Gramps-developers mailing list, and running the code if you can.
  
You start the server by selecting the database and listening port:
+
== Overview ==
  
python src/gramps.py --server=50000 -O "My Family Tree"
+
The Gramps webapp is written in Django. Django is a sophisticated, modern web development framework. Django is written in Python, albeit in a very different style from Gramps. However, part of the motivation of using Django is that it breaks up web development into very clearly defined parts following the [http://en.wikipedia.org/wiki/Model_view_controller Model-View-Controller] paradigm. Two of these parts require no special programming knowledge, and thus will allow more people to be able to possibly customize and participate in the Gramps project.
  
(If you don't know your tree names, use "python src/gramps.py -L").
+
The Gramps webapp (and Django in general) is broken into three well-defined parts:
  
This will start a socket listener on port 50000. If the database is locked, then, like usually, you can supply the --force-unlock flag:
+
# models/views
 +
# templates
 +
# CSS (Cascading Style Sheets)
  
python src/gramps.py --server=50000 -O "My Family Tree" --force-unlock
+
The models define the tables and relationships, but this is done in Python (not SQL). The models also define the API to read/writing/editing the data. The views are also written in Python, and are closely tied to the models. The templates are written in HTML and a template language that is very easy for non-programmers to use and understand. Finally, CSS is just Cascading Style Sheets, where all of the graphical definitions are made. The webapp uses pre-existing CSS created for the "Narrated Web" report of Gramps which was used for created static web pages. Let's take a look at specific examples of each of these parts.
  
In fact, you can do any of the things you normally do with the CLI, before beginning serving.
+
=== Models/Views ===
  
After starting, the server produces output like:
+
Here is the model that defines the Person table from [{{Code Base}}gramps/webapp/grampsdb/models.py gramps/webapp/grampsdb/models.py]:
  
 
<pre>
 
<pre>
$ python src/gramps.py --server=50000 -O "MyRelate"
+
class Person(PrimaryObject):
Opened successfully!
+
    gender_type = models.ForeignKey('GenderType')
GRAMPS server listening on port 50000.
+
    families = models.ManyToManyField('Family', blank=True, null=True)
Use CONTROL+C to exit...
+
    parent_families = models.ManyToManyField('Family',
--------------------------------------------------
+
                                            related_name="parent_families",
Connection opened from 127.0.0.1:50114
+
                                            blank=True, null=True)
Connection closed from 127.0.0.1:50114
+
     references = generic.GenericRelation('PersonRef', related_name="refs",
Connection opened from 127.0.0.1:50115
+
                                        content_type_field="object_type",
     Request: self.dbstate.db.surname_list.__getslice__(0, 2147483647)
+
                                        object_id_field="object_id")
Connection closed from 127.0.0.1:50115
 
 
</pre>
 
</pre>
  
The server currently lists each connection start/end, and each request. The server is written in a fashion such that it can take many requests at once. It spawns off threads that handle the details of the request. TODO: does memory get reclaimed appropriately?
+
Here, you can see that Person only has 4 parts: gender_type, families, parent_families, and references. There are additional properties, but they are defined in the PrimaryObject class which is shared with other tables. Here is PrimaryObject:
  
The server receives messages that it evaluates, pickles, and returns.  
+
<pre>
 +
class PrimaryObject(models.Model):
 +
    class Meta: abstract = True
 +
    id = models.AutoField(primary_key=True)
 +
    handle = models.CharField(max_length=19, unique=True)
 +
    gramps_id =  models.CharField('gramps id', max_length=25, blank=True)
 +
    last_saved = models.DateTimeField('last changed', auto_now=True)
 +
    last_changed = models.DateTimeField('last changed', null=True,
 +
                                        blank=True) # user edits
 +
    private = models.BooleanField('private')
 +
    marker_type = models.ForeignKey('MarkerType')
 +
</pre>
  
=== Interactive Client ===
+
The big difference here between typical Python programming is that the Person class defines the Person table, and the interface to it. Most Python code would probably have Person be an instance of a class, but Django uses classes to represent many things.
  
To make programming as natural as possible on the client side, a RemoteObject has been written which hides the gory details of the socket communication. Here is a sample client:
+
Here are three examples using the Person class:
  
 
<pre>
 
<pre>
from cli.client import RemoteObject
+
  % cd trunk
self = RemoteObject("localhost", 50000)
+
  % PYTHONPATH=src DJANGO_SETTINGS_MODEL=webapp.settings python
 +
  >>> from webapp.grampsdb.models import Person
 +
  >>> Person.objects.all()
 +
  [<Person>, <Person>, ...]
 +
  >>> Person.objects.get(id=1)
 +
  <Person>
 +
  >>> Person.objects.get(handle='gh71234dhf3746347734')
 +
  <Person>
 
</pre>
 
</pre>
  
Here, you can use "self" very similarly to the use of self in the server. For example:
+
The first retrieves all of the rows for the Person table; the second retrieves just the one record that has the unique, primary key 1, and the third retrieves the single record that has the unique handle of 'gh71234dhf3746347734'. Note that we never connected onto a database... Django is (currently) define to connect on to one database, and it does it on import. The database is set in gramps/webapp/settings.py.
 +
 
 +
An alternative method of interactively talking to the database is to use ''manage.py'':
  
 
<pre>
 
<pre>
$ python
+
  % cd master/gramps/webapp
Python 2.6 (r26:66714, Jun  8 2009, 16:07:26)
+
  % PYTHONPATH=../../gramps python manage.py shell
[GCC 4.4.0 20090506 (Red Hat 4.4.0-4)] on linux2
+
  >>>
Type "help", "copyright", "credits" or "license" for more information.
 
>>> from cli.client import RemoteObject
 
>>> self = RemoteObject("localhost", 50000)
 
>>> self.dbstate
 
<DbState.DbState object at 0x98e088c>
 
>>> self.dbstate.db
 
<gen.db.dbdir.GrampsDBDir object at 0x9dc4c0c>
 
 
</pre>
 
</pre>
  
Under the hood, the names "db" and "dbstate" are collected, sent over the socket to the server as the string "self.dbstate.db" which is evaluated, and the representation is sent back. Notice that the server cannot return an entire dbstate or db object to the client, but it merely returns the repr string.
+
That will give you an ipython shell, if you have it installed. Very nice environment!
  
You can interactively explore the remote objects, too:
+
You can also use the Person interface to select a subset of people:
  
 
<pre>
 
<pre>
>>> self.dbstate.db.dir()
+
  >>> from webapp.grampsdb.models import *
['_Callback__BLOCK_ALL_SIGNALS', '_Callback__LOG_ALL', '_Callback__block_instance_signals',
+
  >>> Person.objects.filter(gender_type=1)
...
+
  [<Person>, <Person>, ...]
'source_prefix', 'surname_list', 'surnames', 'transaction_begin', 'transaction_commit',
 
'translist', 'txn', 'undo', 'undo_available', 'undo_callback', 'undo_data',
 
'undo_history_callback', 'undo_history_timestamp', 'undo_reference', 'undodb',
 
'undoindex', 'undolog', 'update_empty', 'update_real', 'update_reference_map', 'url_types',
 
'version_supported', 'write_version']
 
 
</pre>
 
</pre>
  
Notice that you can't wrap a "dir()" around a property, but you can tack a ".dir()" on the end to provide the same functionality. You can't wrap a dir() around the self.dbstate.db because that would get applied on the client side, and by the time self.dbstate.db is evaluated, it is just the repr string.
+
or even more clearly:
 
 
But, you can also get back some full GRAMPS objects:
 
  
 
<pre>
 
<pre>
>>> self.dbstate.db.get_default_person()
+
  >>> Person.objects.filter(gender_type__name="Male")
<gen.lib.person.Person object at 0xb7d7ac6c>
+
  [<Person>, <Person>, ...]
 
</pre>
 
</pre>
  
This isn't just the repr string returned this time, but a real object. You can test that by:
+
The double-underscore in the keyword "gender_type__name" of the filter method is a Django convention. It means "replace with the correct syntax". If Python allowed it, it would be written as '''Person.objects.filter(gender_type.name="Male")''' but that is not legal syntax.
 +
 
 +
==== Model overview ====
 +
Here is an overview of all of the models and how they are related:
 +
 
 +
[[Image:all-tables.gif]]
 +
 
 +
{{stub}}
 +
{{man tip| 1=To update this (Gramps 3.x and earlier) |2=To see more graphical representations of the data, run "make docs" in the  src/webapp/ directory, and then look in src/webapp/docs/.}}
 +
 
 +
* [https://gramps-project.org/docs/gen/gen_db.html#dbdjango Gramps DbDjango]
 +
 
 +
=== Templates ===
 +
 
 +
Templates are used to describe ''what'' to display. Here is a template from [{{Code Base}}data/templates/main_page.html data/templates/main_page.html]:
  
 
<pre>
 
<pre>
>>> p = self.dbstate.db.get_default_person()
+
{% extends "gramps-base.html" %}
>>> p.get_primary_name().get_surname()
+
 
u'Blank'
+
{% block title %}GRAMPS Connect - main page {% endblock %}
 +
{% block heading %}GRAMPS - main page {% endblock %}
 +
 
 +
{% block content %}
 +
 
 +
<p id="description">Welcome to GRAMPS Connect, a new web-based collaboration tool.
 +
 
 +
{% if user.is_authenticated %}
 +
  You are now logged in
 +
  as <a href="/user/{{user.username}}">{{user.username}}</a>.
 +
{% endif %}
 +
</p>
 +
 
 +
<p id="description">
 +
Database information:
 +
<ul>
 +
{% for view in views %}
 +
  <li><a href="/{{view|lower}}">{{view}}</a></li>
 +
{% endfor %}
 +
</ul>
 +
</p>
 +
{% endblock %}
 
</pre>
 
</pre>
  
Because you can't get back generators nor database objects, sometimes you need to do some processing on the server. To do this, you can use self.remote:
+
=== CSS ===
  
>>> self.remote("name = self.sdb.name(self.dbstate.db.get_default_person())")
+
Finally, here is a screen shot of the main_page.html (above) showing some initial testing of Gramps in Django using the Mainz CSS from the NarrWeb report:
>>> self.remote("name")
 
u'Blank, Living'
 
  
self.remote takes a string and sends it to the server to be evaluated (or executed). self contains:
+
[[Image:Gramps_in_django.gif]]
  
<pre>
+
=== Getting Started with Gramps in Django ===
>>> self.dir()
+
 
['__doc__', '__init__', '__module__', 'arghandler', 'climanager', 'dbstate', 'env', 'eval', 'reset', 'sdb']
+
A prototype of a Gramps Django webapp is now in trunk and gramps32. To run it, do the following:
</pre>
+
 
 +
# Download Django version 1.3 or greater
 +
## On yum-based systems, try "yum install Django"
 +
## On apt-based systems, try "sudo apt-get install python-django"
 +
## Other systems: get the sources from http://www.djangoproject.com/download/
 +
# clone the Git repository and checkout either the gramps32 or master branch
 +
# cd src/web/
 +
# Build the database, and load with default data:
 +
## make clean
 +
## make
 +
## This will ask for an id, email, and password for a superuser. You can add one later if you don't do it now.
 +
# Run the test webserver:
 +
## make run
 +
# Point your webbrowser to:
 +
## http://127.0.0.1:8000/
 +
 
 +
At this point, you can now export your Gramps data to Django (and back). In another terminal window:
 +
 
 +
# Start up gramps:
 +
## cd ../..
 +
## python src/gramps.py
 +
# Download the Django Import/Export Addon from [[3.3_Addons]]
 +
# Run the Django Exporter
 +
## Select Family Tree -> Export
 +
## Select Django
 +
 
 +
This will export your regular Gramps BSDDB data into whatever Django database you have defined in settings.py above. You now have your data in a sqlite SQL database, and can access it via the webbrowser.
 +
 
 +
To import data back from Django's SQL table back into Gramps from the website:
 +
 
 +
# Create a file named "import.django" somewhere (just needs to end in ".django").
 +
# Start up this version of Gramps
 +
## python src/gramps.py
 +
# Run the Django Importer
 +
## Select Family Tree -> Import  [[Image:DjangoImportExport.jpg|thumb|right|150px]]
 +
## Select the "import.django" (from above) as the file to import
 +
 
 +
To add a superuser (after the initialization):
 +
 
 +
# cd src/web
 +
# PYTHONPATH=../../src python manage.py createsuperuser
 +
 
 +
For more on Django, try their tutorial:
 +
 
 +
* Tutorial: http://docs.djangoproject.com/en/dev/intro/tutorial01/#intro-tutorial01
 +
 
 +
=== Webapp Files ===
 +
 
 +
There are two subdirectories and two files of interest to the Gramps webapp:
 +
 
 +
# {{Code Base}}data/templates/ - HTML templates
 +
# {{Code Base}}gramps/webapp/ - Webapp main directory
 +
## {{Code Base}}gramps/webapp/libdjango.py - library interface
 +
## {{Code Base}}gramps/webapp/grampsdb - gramps table models
 +
# http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ExportDjango.py?view=markup - Exporter
 +
# http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ImportDjango.py?view=markup - Importer
 +
 
 +
== Roadmap ==
  
* arghandler - the object that handles the command-line operations
+
Phase 1: get the basic Django skeleton in place, including the core HTML templates, models, views, and templatetags. Should be able to browse the 8 primary tables. Get translations in place. Goal for version 0.1 to be announced with Gramps 3.2 in March 2010.
** dbman - Functions related to database management
 
*** active
 
*** break_lock
 
*** create_new_db_cli
 
*** current_names
 
*** dbstate
 
*** empty
 
*** family_tree_list
 
*** family_tree_summary
 
*** get_dbdir_summary
 
*** get_family_tree_path
 
*** icon_values
 
*** import_new_db
 
*** is_locked
 
*** msg
 
*** needs_recovery
 
* climanager - low-level CLI management
 
** db_loader
 
** dbstate
 
** do_load_plugins
 
** file_loaded
 
** open_activate
 
* dbstate - holds the state of the database
 
** dbstate.db - the database
 
* sdb - simple database access
 
  
=== Web Client ===
+
Phase 2: Be able to run all of the reports directly from the web with an option interface. Be able to import/export from the web. This will largely depend on a gen/db/dbdjango library. Goal for version 0.5beta, May 2010.
  
Finally, here is an example for listing on the web the properties and surnames of a family tree:
+
Phase 3: add and edit data from the web. This would complete the functionality of the web interface. Goal July 2010.
  
<pre>
+
Phase 4: Refine and polish. Release with Gramps 3.3.
#!/usr/bin/python
 
  
import sys, os
+
If you would like to work on an area, please note it here:
sys.path.append("/home/dblank/gramps/trunk/src")
 
os.environ["HOME"] = "/home/dblank/html/gramps"
 
  
from cli.client import *
+
# Kathy - edits and adding new data
 +
# Doug - Integration with gramps core; browsing data
 +
# - Translation system
 +
# - Proxy interface to show Private data
 +
# - concurrent edits
 +
# - date widget
 +
# - running reports interface
 +
# - media files... where do they go?
 +
# - options interface, for editing options to run report
 +
# - import GEDCOM from web
 +
# - full djangodb.py to replicate all functions of bsddb
 +
# - user support (email, mailing lists, permissions, etc)
  
self = RemoteObject("localhost", 50001)
+
== Issues ==
  
print "Content-type: text/html"
+
=== Concurrent Edits ===
print
 
print "<html>"
 
print "<body>"
 
print "<h1>First Demo of GRAMPS --server</h1>"
 
  
dbfile = self.dbstate.db.full_name
+
Concurrent access for write and read imply several problems when people by accident change the same objects at the same time. Gramps itself has an elaborate signal handling for cases when dialogs are open with no longer current information. In a web environment, this becomes more difficult however. This is not built into Django.
people, dbversion = self.arghandler.dbman.get_dbdir_summary(dbfile)
 
summary_list = self.arghandler.dbman.family_tree_summary()
 
summary_dict = {}
 
  
for sdict in summary_list:
+
For discussion on this issue in Django, see:
    if sdict["Path"] == dbfile:
 
        summary_dict = sdict
 
  
print "<h2>Properties:</h2>"
+
* [http://groups.google.com/group/django-users/browse_thread/thread/c138ec11c6ad282e?hl=en# Django User Question]
for prop in summary_dict:
+
** [http://groups.google.com/group/django-developers/browse_thread/thread/fd5d45fc6cd6a760 Developer discussion on topic]  
    print "<b>%s</b>: %s <br/>" % (prop, summary_dict[prop])
 
  
print "<h2>Surnames:</h2>"
+
== Example GMS Web Sites ==
surnames = self.dbstate.db.surname_list[:10]
 
for name in surnames:
 
    if len(name) == 0:
 
        name = "[Missing Surname]"
 
    print "<li><a href=\"?surname=%s\">%s</a></li>" % (name, name)
 
  
print "<hr>"
+
We now have a example gramps webapp on the web:
print "</body>"
 
print "</html>"
 
</pre>
 
  
Save the above in a file ending with .cgi. I used index.cgi. I then configured apache to allow CGI, added index.cgi, turned off SELiux, and started up the httpd service.
+
* http://gramps-connect.org/
  
The resulting screen shot:
+
Genealogy Management Systems on the web:
  
[[Image:Screenshot-server.gif]]
+
* http://www.dertinger.de/Dertinger_database/en/en_index.htm (Oxy-gen)
 +
* http://www.admiraal.org (PhpGedView)
 +
: Note here: the intro page is a collection of gadgets/controls, which then link into the real data.
 +
* http://webtrees.net/demo/next (webtrees)
 +
* http://beck.org.il/humogen/ (HuMogen)
 +
* http://genealogies.geneamania.net/servin/ (Généamania)
 +
* http://www.geneotree.com/geneotree/index.php (Geneotree)
 +
* http://ancestorsnow.com/ancestors
 +
* http://www.phpmyfamily.net/demo/
 +
* http://www.frog.za.net/family/surname-list.php ([[Other_genealogy_tools#Gramps-php-exporter|gramps-php-exporter]])
  
== Discussion ==
+
Collaborative database (user/wizard/password):
  
* I had to make some other changes in GRAMPS that I haven't committed yet that allows GRAMPS to run without having X or a display. But you won't notice this issue when you run the code if you have a display---it only shows up when running without one.
+
* http://roglo.eu/roglo?lang=en (GeneWeb)
* It isn't clear yet from the prototype if this interface will be able to do everything. I have changed the default person in the dbstate, and I have changed a name. Can you get all (and edit all) of the data by just using handles?
+
* http://gennus.org ([http://beta.gennus.org/en/page/about.html beta][http://beta.gennus.org/en/page/releasenotes.html])
* The interface allows for creating variables on the server, but that could get overwritten by multiple clients. Do we need variables, and how can we keep them separate by client?
+
* http://brozer.fr (alpha[http://www.innovup.com/evenement/124/89-actualites-agenda.htm])
* A new interface may need to be created, similar to SimpleAccess, but for this type of communication.
 
* How could security be implemented to protect one's data from viewing or edits?
 
* Note that this proposal does not limit the webapp language to be Python. Any language that implements the Python pickle format can be used. In reality, the system uses a subset of pickle. In addition, there are raw_data only methods, so one restrict further the required subset of functionality of the pickle.
 
  
The next step would be to build a web app. The organisation of the web app could also be discussed here, although that would quickly become a bigger topic. In any event, some notes on this aspect of the proposal are below.
+
Source oriented:
  
== Web App Architecture ==
+
* http://solumslekt.org/forays/yggdrasil.php [http://code.google.com/p/yggdrasil-genealogy/][http://solumslekt.org/blog/]
  
* There is a rewritten GRAMPS html formatting library ./src/plugins/lib/libhtml.py which should be used
+
==See also==
* Consider the report system to generate the web app client
+
*[[Gramps-Connect: Introduction|gramps-connect]]
* Use CGI at first
 
* Use the new CSS tags and organization
 
* Use as much as possible from Narrated and Webcal reports
 
* This could be a new plugin report type, webapp
 
  
[[Category:GEPS|S]]
+
[[Category:GEPS|G]]

Revision as of 08:04, 5 January 2017

Many Gramps users would like to collaborate or share their genealogy data on the web. This GEP describes a webapp, a web-based application that runs in your browser, and requires a server.

A prototype is on-line at http://gramps-connect.org/ which is running trunk on a sample database. You can log into the site, as a:

  • superuser (id=admin, password=gramps) or a
  • regular user (id=admin1, password=gramps)

or just view as an anonymous user.

There are two additional pages on this project:

Motivation

The main focus of a Gramps-based webapp is collaboration. The Gramps webapp will allow users to easily move their genealogy data to the web to be seen, and edited with proper login and permissions, in a live, collaborative environment.

Here is a small list of goals:

  1. Create a fullscale Gramps web framework
  2. Allow multiple users via the standard web browser
    1. Users will log in and have various levels of permissions
  3. Build on Gramps codebase and wealth of resources
    1. Reports
    2. Tools
    3. Visualizations
    4. Date and calendar functions
    5. Translations
    6. Manual and documentation
  4. Use standards and well-known, well-tested frameworks where possible
    1. WSGI protocol for running code
    2. Django framework
    3. Underlying powerful database engines

FAQ

1. Aren't there already many fine, web-based genealogy programs? Why don't you just use one of those? Aren't you re-inventing the wheel?

There are indeed many fine, web-based genealogy programs, and some are even free software/open source. However, there are a few good reasons to develop a Gramps-based webapp:

  1. Gramps has hundreds of thousands of lines of code, some of which could be re-used directly in a webapp. For example, the reports could be run and downloaded directly from the webapp.
  2. Gramps has a very well-defined set of tables and relationships that could be re-implemented for on-line use.
  3. Users have grown to appreciate the design of Gramps, and we want to continue to build on this design.
  4. Many users want to collaborate. Currently, they would either have to move their data in and out of Gramps, or give up Gramps completely.
  5. We want to keep the developers and users that we have, and so not splinter our groups. By building the webapp on top of core gramps code, we continue to refine and make better our current code, and keep our current developers working on the parts that they know and love.

2. Why do you need a web framework like Django? Can't you just use the same Python code, and same database that Gramps already uses?

We can't use the same database (what is called a "backend") directly. Currently Gramps uses BSDDB, and it is not configured for use in a multiuser, client/server environment. But even if we could use the same backend, we would still want some type of web development framework. Django is one of the best in any language, and it just happens to be in Python.

3. How easy will this be for me to use on my website?

We have designed it to be as easy as it can be, given that we are using Python. Many web sites allow Python programs, and Django allows many different variations in running. We picked the protocol with the most availability (WSGI). Don't worry if you haven't heard of it. Your webserver can probably run it.

4. When is this going to be available?

We are hoping to have a fully functioning webapp ready for testing July 2010.

5. How can I help?

You can start by reading the rest of this page and sending ideas and comments to the Gramps-developers mailing list, and running the code if you can.

Overview

The Gramps webapp is written in Django. Django is a sophisticated, modern web development framework. Django is written in Python, albeit in a very different style from Gramps. However, part of the motivation of using Django is that it breaks up web development into very clearly defined parts following the Model-View-Controller paradigm. Two of these parts require no special programming knowledge, and thus will allow more people to be able to possibly customize and participate in the Gramps project.

The Gramps webapp (and Django in general) is broken into three well-defined parts:

  1. models/views
  2. templates
  3. CSS (Cascading Style Sheets)

The models define the tables and relationships, but this is done in Python (not SQL). The models also define the API to read/writing/editing the data. The views are also written in Python, and are closely tied to the models. The templates are written in HTML and a template language that is very easy for non-programmers to use and understand. Finally, CSS is just Cascading Style Sheets, where all of the graphical definitions are made. The webapp uses pre-existing CSS created for the "Narrated Web" report of Gramps which was used for created static web pages. Let's take a look at specific examples of each of these parts.

Models/Views

Here is the model that defines the Person table from gramps/webapp/grampsdb/models.py:

class Person(PrimaryObject):
    gender_type = models.ForeignKey('GenderType')
    families = models.ManyToManyField('Family', blank=True, null=True)
    parent_families = models.ManyToManyField('Family', 
                                             related_name="parent_families",
                                             blank=True, null=True)
    references = generic.GenericRelation('PersonRef', related_name="refs",
                                         content_type_field="object_type",
                                         object_id_field="object_id")

Here, you can see that Person only has 4 parts: gender_type, families, parent_families, and references. There are additional properties, but they are defined in the PrimaryObject class which is shared with other tables. Here is PrimaryObject:

class PrimaryObject(models.Model):
    class Meta: abstract = True
    id = models.AutoField(primary_key=True)
    handle = models.CharField(max_length=19, unique=True)
    gramps_id =  models.CharField('gramps id', max_length=25, blank=True)
    last_saved = models.DateTimeField('last changed', auto_now=True) 
    last_changed = models.DateTimeField('last changed', null=True,
                                        blank=True) # user edits
    private = models.BooleanField('private')
    marker_type = models.ForeignKey('MarkerType')

The big difference here between typical Python programming is that the Person class defines the Person table, and the interface to it. Most Python code would probably have Person be an instance of a class, but Django uses classes to represent many things.

Here are three examples using the Person class:

   % cd trunk
   % PYTHONPATH=src DJANGO_SETTINGS_MODEL=webapp.settings python 
   >>> from webapp.grampsdb.models import Person
   >>> Person.objects.all()
   [<Person>, <Person>, ...]
   >>> Person.objects.get(id=1)
   <Person>
   >>> Person.objects.get(handle='gh71234dhf3746347734')
   <Person>

The first retrieves all of the rows for the Person table; the second retrieves just the one record that has the unique, primary key 1, and the third retrieves the single record that has the unique handle of 'gh71234dhf3746347734'. Note that we never connected onto a database... Django is (currently) define to connect on to one database, and it does it on import. The database is set in gramps/webapp/settings.py.

An alternative method of interactively talking to the database is to use manage.py:

   % cd master/gramps/webapp 
   % PYTHONPATH=../../gramps python manage.py shell
   >>>

That will give you an ipython shell, if you have it installed. Very nice environment!

You can also use the Person interface to select a subset of people:

   >>> from webapp.grampsdb.models import *
   >>> Person.objects.filter(gender_type=1)
   [<Person>, <Person>, ...]

or even more clearly:

   >>> Person.objects.filter(gender_type__name="Male")
   [<Person>, <Person>, ...]

The double-underscore in the keyword "gender_type__name" of the filter method is a Django convention. It means "replace with the correct syntax". If Python allowed it, it would be written as Person.objects.filter(gender_type.name="Male") but that is not legal syntax.

Model overview

Here is an overview of all of the models and how they are related:

All-tables.gif

Gramps-notes.png

This article's content is incomplete or a placeholder stub.
Please update or expand this section.


Tango-Dialog-information.png
To update this (Gramps 3.x and earlier)

To see more graphical representations of the data, run "make docs" in the src/webapp/ directory, and then look in src/webapp/docs/.


Templates

Templates are used to describe what to display. Here is a template from data/templates/main_page.html:

{% extends "gramps-base.html" %}

{% block title %}GRAMPS Connect - main page {% endblock %}
{% block heading %}GRAMPS - main page {% endblock %}

{% block content %} 

<p id="description">Welcome to GRAMPS Connect, a new web-based collaboration tool.

{% if user.is_authenticated %}
  You are now logged in
  as <a href="/user/{{user.username}}">{{user.username}}</a>.
{% endif %}
</p>

<p id="description">
Database information:
<ul>
{% for view in views %}
   <li><a href="/{{view|lower}}">{{view}}</a></li>
{% endfor %}
</ul>
</p>
{% endblock %}

CSS

Finally, here is a screen shot of the main_page.html (above) showing some initial testing of Gramps in Django using the Mainz CSS from the NarrWeb report:

Gramps in django.gif

Getting Started with Gramps in Django

A prototype of a Gramps Django webapp is now in trunk and gramps32. To run it, do the following:

  1. Download Django version 1.3 or greater
    1. On yum-based systems, try "yum install Django"
    2. On apt-based systems, try "sudo apt-get install python-django"
    3. Other systems: get the sources from http://www.djangoproject.com/download/
  2. clone the Git repository and checkout either the gramps32 or master branch
  3. cd src/web/
  4. Build the database, and load with default data:
    1. make clean
    2. make
    3. This will ask for an id, email, and password for a superuser. You can add one later if you don't do it now.
  5. Run the test webserver:
    1. make run
  6. Point your webbrowser to:
    1. http://127.0.0.1:8000/

At this point, you can now export your Gramps data to Django (and back). In another terminal window:

  1. Start up gramps:
    1. cd ../..
    2. python src/gramps.py
  2. Download the Django Import/Export Addon from 3.3_Addons
  3. Run the Django Exporter
    1. Select Family Tree -> Export
    2. Select Django

This will export your regular Gramps BSDDB data into whatever Django database you have defined in settings.py above. You now have your data in a sqlite SQL database, and can access it via the webbrowser.

To import data back from Django's SQL table back into Gramps from the website:

  1. Create a file named "import.django" somewhere (just needs to end in ".django").
  2. Start up this version of Gramps
    1. python src/gramps.py
  3. Run the Django Importer
    1. Select Family Tree -> Import
      DjangoImportExport.jpg
    2. Select the "import.django" (from above) as the file to import

To add a superuser (after the initialization):

  1. cd src/web
  2. PYTHONPATH=../../src python manage.py createsuperuser

For more on Django, try their tutorial:

Webapp Files

There are two subdirectories and two files of interest to the Gramps webapp:

  1. https://github.com/gramps-project/gramps/blob/master/data/templates/ - HTML templates
  2. https://github.com/gramps-project/gramps/blob/master/gramps/webapp/ - Webapp main directory
    1. https://github.com/gramps-project/gramps/blob/master/gramps/webapp/libdjango.py - library interface
    2. https://github.com/gramps-project/gramps/blob/master/gramps/webapp/grampsdb - gramps table models
  3. http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ExportDjango.py?view=markup - Exporter
  4. http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Django/ImportDjango.py?view=markup - Importer

Roadmap

Phase 1: get the basic Django skeleton in place, including the core HTML templates, models, views, and templatetags. Should be able to browse the 8 primary tables. Get translations in place. Goal for version 0.1 to be announced with Gramps 3.2 in March 2010.

Phase 2: Be able to run all of the reports directly from the web with an option interface. Be able to import/export from the web. This will largely depend on a gen/db/dbdjango library. Goal for version 0.5beta, May 2010.

Phase 3: add and edit data from the web. This would complete the functionality of the web interface. Goal July 2010.

Phase 4: Refine and polish. Release with Gramps 3.3.

If you would like to work on an area, please note it here:

  1. Kathy - edits and adding new data
  2. Doug - Integration with gramps core; browsing data
  3. - Translation system
  4. - Proxy interface to show Private data
  5. - concurrent edits
  6. - date widget
  7. - running reports interface
  8. - media files... where do they go?
  9. - options interface, for editing options to run report
  10. - import GEDCOM from web
  11. - full djangodb.py to replicate all functions of bsddb
  12. - user support (email, mailing lists, permissions, etc)

Issues

Concurrent Edits

Concurrent access for write and read imply several problems when people by accident change the same objects at the same time. Gramps itself has an elaborate signal handling for cases when dialogs are open with no longer current information. In a web environment, this becomes more difficult however. This is not built into Django.

For discussion on this issue in Django, see:

Example GMS Web Sites

We now have a example gramps webapp on the web:

Genealogy Management Systems on the web:

Note here: the intro page is a collection of gadgets/controls, which then link into the real data.

Collaborative database (user/wizard/password):

Source oriented:

See also