GEPS 048: Format Negotiation Layer for Import/Export Plugins

GEPS 048: Format Negotiation Layer for Import-Export Plugins | 2026-03-20 | Draft | ? | ? | |} -->

Type

Standards Track

Authors

ChatGPT (OpenAI GPT-5.3) — initial proposal and specification
Claude (Anthropic, claude-sonnet-4-6) — editorial review, motivation

 expansion, workaround sections, collapsible formatting

Abstract

Gramps users routinely encounter incompatibility, silent data loss, and format confusion when importing or exporting files whose internal dialect does not match what the importer expects. This GEPS proposes a Format Negotiation Layer for import and export plugins: a dialect-aware detection, routing, and transformation architecture that works across GEDCOM, CSV, vCard, and Gramps XML. It introduces an extensible dialect registry, a sniffer interface, and a processing pipeline. Two interim workaround approaches are documented for use before core integration is complete.

Motivation

Gramps selects import/export plugins by file extension alone. This ignores dialect variation — the differences in structure, encoding, and vendor-specific tags that make two .ged files or two .csv files require completely different handling. The result is silent data loss, import failures, and no path to improve support without modifying core plugins.

Affected formats and consequences (click to expand)

GEDCOM is a family of dialects, not a single format. GEDCOM 5.5,

 5.5.1, and 7.0 differ structurally. Major platforms (Ancestry,
 FamilySearch, Legacy Family Tree, MacFamilyTree) add proprietary tags
 and non-standard structures. Records associated with unknown tags are
 silently dropped by the current importer.

CSV files vary in delimiter (, ;

 \t), quoting convention, encoding (UTF-8, UTF-8-BOM,
 Windows-1252), and column schema. A file exported by one application
 may be misread by an importer expecting different conventions.

vCard files exist in versions 2.1, 3.0, and 4.0, each with

 distinct encoding rules and property semantics.

Gramps XML evolves across schema versions. There is currently no

 mechanism to export a lower schema version for compatibility with older
 Gramps installations, even when the database contains no data requiring
 the newer schema features.

New dialects emerge during development cycles. Flaws in existing dialect handling are discovered after release. The system must be able to expand without requiring changes to core plugins.

Related work:

These remain format-specific. A generalized solution is needed to avoid repeated per-format effort and to establish a shared extensibility pattern.

Rationale

A Format Negotiation Layer decouples "file format" (the extension and container) from "format interpretation" (the dialect, version, and vendor conventions). New dialects can be supported by new addon plugins without touching core importers. Experimental or vendor-specific support can mature independently before integration.

Design capabilities provided by the layer (click to expand)

Automatic dialect detection — inspect file content before

 selecting a handler, rather than relying on extension alone

Dynamic routing — direct files to the handler best suited for

 their detected dialect

Graceful fallback — when a dialect is unrecognized, fall back to

 the closest known handler and preserve unknown data rather than
 discarding it

Export targeting — allow users to export to a specific schema

 version or vendor profile for compatibility

Specification

The Format Negotiation Layer consists of four components: a Dialect Registry, a Sniffer Interface, a Processing Pipeline, and Dialect Handlers. User configuration and edge-case handling (unknown dialects, schema downgrade) round out the design.

Dialect Registry — maps formats to versioned dialect handlers

Introduce a registry mapping formats to dialect handlers. Each entry is defined by a unique identifier (e.g., gedcom-5.5.1, gedcom-ancestry, csv-excel), a detection function (sniffer), and a processing handler class. Dialects may be registered by core or addon plugins. A new dialect means a new registration, not a core modification.

<source lang="python"> DIALECT_REGISTRY = {

   "gedcom": {
       "5.5.1":    Gedcom551Dialect,
       "7.0":      Gedcom7Dialect,
       "ancestry": AncestryDialect,
   },
   "csv": {
       "excel":   CsvExcelDialect,
       "rfc4180": CsvRFCDialect,
   },
   "vcard": {
       "2.1": VCard21Dialect,
       "3.0": VCard30Dialect,
       "4.0": VCard40Dialect,
   },
   "gramps_xml": {
       "1.7": GrampsXML17,
       "1.8": GrampsXML18,
   },

} </source>

Sniffer Interface — detects format and dialect from file content

Each format provides a sniffer that inspects the first 100–500 lines or bytes and returns a detection result including format, dialect, confidence score, and detected features.

<source lang="python"> class FormatSniffer:

   def detect(self, filename) -> DetectionResult:
       return {
           "format":     "gedcom",
           "dialect":    "5.5.1",
           "confidence": 0.92,
           "features":   {...},
       }

</source>

Detection signals by format:

GEDCOM: HEAD.GEDC.VERS for version;

 HEAD.SOUR for vendor; proprietary tags such as
 _APID, _UID, _FSFTID

CSV: delimiter, quoting style, encoding, header names
vCard: VERSION property, line folding, encoding

 declarations

Gramps XML: root namespace, schema version attribute

Processing Pipeline — wraps core importers/exporters without modifying them

Import pipeline:

File → Sniffer → Dialect Selection → Preprocess → Core Importer → Postprocess → Database

Export pipeline:

Database → Dialect Selection → Pre-export Transform → Core Exporter → Post-export Transform → File

Core importers and exporters are not modified. The negotiation layer wraps them.

Dialect Handlers — per-dialect pre/post processing hooks

Each dialect handler implements a standard four-method interface:

<source lang="python"> class Dialect:

   def preprocess(self, stream): ...
   def import_transform(self, data): ...
   def export_transform(self, data): ...
   def postprocess(self, data): ...

</source>

User Configuration — auto-detect, dialect override, export targeting

Users may configure:

Import mode: Auto-detect (default), preferred dialect, or strict

 mode (fail on unrecognized dialect)

Fallback dialect: used when confidence falls below a threshold
Export compatibility target: select a schema version or vendor

 profile

Backward Compatibility (Gramps XML) — optional schema downgrade on export

During export, if the database contains no records requiring features introduced after a given schema version, the exporter may offer export to that older version. The exporter strips unsupported elements, downgrades structures, and warns the user if any downgrade is lossy.

<source lang="python"> if not db_uses_features(requiring="1.8"):

   target_schema = "1.7"

</source>

Unknown Dialect Handling — preserve rather than discard

When a dialect is not recognized:

Fall back to the closest known dialect
Preserve unknown tags as object attributes or notes rather than

 discarding them (e.g.,
 person.add_attribute(Attribute("_RAW__APID", value)))

Log anomalies for user and developer review
Optionally generate a dialect report summarizing unrecognized

 constructs

Workaround / Interim Approaches

Two approaches are available while this GEPS is being formally implemented. Both require no changes to Gramps core. They are ordered from lowest to highest investment.

Workaround A: Fork-and-Hide (Low Investment — Proof of Concept)

This approach tests the viability of dialect-enhanced import/export without building a new plugin from scratch. The strategy is to fork (copy) an existing built-in plugin, extend it with dialect handling, and suppress the original to eliminate file-type ambiguity in the import dialog.

Why fork rather than replace? The Gramps plugin system does not support overriding a registered plugin. Registering a second importer for the same extension creates ambiguity — both appear in the dialog. Forking and then hiding the original resolves this cleanly.

The current plugin registration model selects importers by file extension and static label. There is no scoring or priority mechanism. If two importers claim .ged, the user must choose manually every time. Hiding the built-in removes the duplicate while keeping the fork as the sole handler.

Step 1: Fork the built-in plugin (copy, rename, extend)

Copy the target built-in plugin (e.g., gramps/plugins/importer/importgedcom.py and its .gpr.py registration file) into a new addon directory. Rename the plugin class and its registration label (e.g., "GEDCOM (Enhanced)"). Make dialect-handling changes directly in the fork — additional tag handling, pre/post processing, or a simple internal dialect branch — without touching the original.

This is the lowest-risk starting point: the fork starts as a working importer identical to the built-in, and changes are incremental.

Step 2: Hide the original — manual or automatic

Manual hiding (simplest): After installing the fork addon, the user opens Plugin Manager Enhanced, locates the original built-in importer, and uses the "Hide" function to suppress it. The fork becomes the sole handler for that file extension. This is reversible: unhiding the built-in and hiding the fork restores original behavior.

Automatic hiding (more capable): The fork's .gpr.py registration can include startup logic that hides the upstream built-in programmatically, removing the need for manual user action.

The fork knows the upstream plugin's identity (its registered name or ID) and can call into the plugin manager's hidden-plugin list on registration or first run:

In the fork's .gpr.py or plugin __init__:

from gramps.gen.plug import PluginRegister

UPSTREAM_PLUGIN_ID = "im_gedcom" # ID of the built-in GEDCOM importer

def _hide_upstream():

   pmgr = PluginRegister.get_instance()
   hidden = pmgr.get_hidden_plugin_ids()   # returns current hidden set
   if UPSTREAM_PLUGIN_ID not in hidden:
       hidden.add(UPSTREAM_PLUGIN_ID)
       pmgr.save_hidden_plugin_ids(hidden)

_hide_upstream() </source>

The fork should also expose a preference or menu option to reverse which plugin is hidden, so users can toggle back to the built-in if desired:

<source lang="python"> def toggle_active_importer(use_fork=True):

   pmgr = PluginRegister.get_instance()
   hidden = pmgr.get_hidden_plugin_ids()
   if use_fork:
       hidden.discard(FORK_PLUGIN_ID)
       hidden.add(UPSTREAM_PLUGIN_ID)
   else:
       hidden.discard(UPSTREAM_PLUGIN_ID)
       hidden.add(FORK_PLUGIN_ID)
   pmgr.save_hidden_plugin_ids(hidden)

</source>

Note: the exact API for reading and writing the hidden plugin list should be verified against the current Plugin Manager Enhanced implementation, as this capability may be accessed via its own API rather than directly through PluginRegister.

Limitations of Workaround A

The fork must be kept in sync with upstream bug fixes and changes to

 the built-in — this is the primary maintenance cost

Dialect logic lives inside the fork, not in a shared registry; a

 second format requires a second fork

No sniffing infrastructure is shared across formats
The automatic hide relies on Plugin Manager Enhanced being

 installed; it is not a core capability

This approach is best treated as a proof of concept and a way to validate specific dialect-handling needs before investing in Workaround B or the full GEPS implementation.

Workaround B: Addon-based Meta-Importer (Higher Investment — Full Interim Architecture)

This approach builds new addon plugins that stand alongside (rather than forking) the built-in importers, using the same hide mechanism for dialog disambiguation. It is more work upfront but produces reusable infrastructure aligned with the full GEPS design.

Step 1: Build a Meta-Importer addon per format

Register a new importer addon for each affected file extension with a clear user-facing name (e.g., "GEDCOM (Auto-detect)", "CSV (Auto-detect)"). This meta-importer inspects the file, selects a dialect handler from an internal registry, and delegates. The existing core importer is the default fallback.

<source lang="python"> class GedcomMetaImporter(ImportPlugin):

   def import_file(self, db, filename, user):
       meta = sniff(filename)
       handler = DIALECT_MAP.get(meta["dialect"], DefaultGedcomHandler)
       return handler().import_file(db, filename, user)

</source>

Step 2: Suppress competing built-in via Plugin Manager Enhanced

As with Workaround A, once the meta-importer is installed and verified, hide the built-in importer for the same extension. This may be done manually by the user or automatically by the addon on registration (see Workaround A, Step 2 for the hide/toggle pattern). The meta-importer becomes the sole entry point for that file type.

Step 3: User preferences for dialect override

Expose a preference via the Gramps config namespace or a plugin preferences dialog allowing the user to pin a specific dialect (e.g., "GEDCOM 5.5.1 strict", "Ancestry", "FamilySearch", "GEDCOM 7") or set a fallback when auto-detection confidence falls below a threshold.

Step 4: Preserve unknown tags

Store unrecognized tags as attributes with a _RAW_ prefix rather than discarding them. This preserves round-trip fidelity and allows reprocessing when a proper handler becomes available.

<source lang="python"> person.add_attribute(Attribute("_RAW__APID", value)) </source>

Step 5: Profile-driven export

Register an enhanced exporter addon per format. Expose profile selection (defined in bundled JSON or YAML files) in the export assistant. Profiles cover tag inclusion/exclusion, encoding rules, and structural quirks for target applications. For Gramps XML, offer "Export as 1.7 (maximum compatibility)" when the database content permits it.

Limitations of Workaround B

New dialects require addon updates; there is no dynamic registration

 shared across format meta-importers

No shared sniffer or registry infrastructure across formats (each

 meta-importer is self-contained)

The "Hide" mechanism is global, not per-file
User preferences are not portable across installations

These limitations are precisely what the full GEPS implementation addresses.

Backwards Compatibility

Existing plugins remain unchanged. The negotiation layer operates as an optional meta-layer, implementable entirely via addon plugins. Integration into core does not require removal or modification of existing importers and exporters.

Reference Implementation

Prototype scope

A prototype implementation would include:

Meta-importer addons per format (e.g., "GEDCOM Auto-detect")
A dialect registry module
Sniffer utilities for GEDCOM, CSV, vCard, and Gramps XML
Bundled dialect handler classes for common dialects
Integration with Plugin Manager Enhanced for visibility control

Drawbacks

Known drawbacks and risks

Increased architectural complexity: the negotiation layer adds

 indirection between file selection and parsing. Developers adding
 new format support must understand the registry and sniffer interface,
 not just the importer API.

Dialect definition coordination: dialect identifiers must be

 consistent across the registry, sniffers, and handlers. An informal
 process risks fragmentation.

Potential duplication: if only partially implemented (e.g., for

 GEDCOM only), logic may be duplicated rather than shared across
 formats.

Alternatives

Alternatives considered

Maintain current extension-based selection: simple, but does not

 address dialect variation or cross-format extensibility.

Format-specific improvements only (e.g., GEDCOM-only

 enhancements, as in GEPS 037 and GEPS 043): addresses immediate needs
 but requires repeated effort per format and does not establish a
 shared pattern.

Future Work

Planned follow-on work

In natural implementation sequence:

Core integration of dialect registry (enabling dynamic registration

 without addon workarounds)

Importer scoring and selection mechanism

 (score = importer.probe(file); highest score wins)

Shared dialect definition repository for community-maintained profiles
UI feedback displaying detected dialect at import time
"Re-import using different dialect" user action

Credits

This proposal was developed with AI assistance in accordance with the Gramps project policy on AI-generated contributions.

Initial proposal and specification: ChatGPT (OpenAI GPT-5.3). (click to expand)

 Prompt summary: The user requested a generalized, extensible framework
 for handling dialect variations across multiple file formats in Gramps,
 including dynamic detection, plugin augmentation, and backward
 compatibility support.

Editorial review, rationale expansion, workaround sections, and collapsible formatting: Claude (Anthropic, claude-sonnet-4-6).

 Prompt summary: The user requested a second AI review of the GEPS for
 readability using MediaWiki collapsible sections, addition of a
 low-investment fork-and-hide workaround, and updated AI credits
 compliant with Gramps contributor policy.

Human author and project coordinator: Brian McCullough (emyoulation), Gramps project.

GEPS 048: Format Negotiation Layer for Import-Export Plugins

Contents

GEPS 048: Format Negotiation Layer for Import/Export Plugins

Type

Authors

Abstract

Motivation

Rationale

Specification

Workaround / Interim Approaches

Workaround A: Fork-and-Hide (Low Investment — Proof of Concept)

Workaround B: Addon-based Meta-Importer (Higher Investment — Full Interim Architecture)

Backwards Compatibility

Reference Implementation

Drawbacks

Alternatives

Future Work

See Also

Credits

Navigation menu

Views

Personal tools

Navigation

Contributor help pages

wiki

Search

Tools