View source for GEPS 048: Format Negotiation Layer for Import-Export Plugins

= GEPS 048: Format Negotiation Layer for Import/Export Plugins =
<!--
{| class="wikitable sortable" {{Prettytable}}
|+ Gramps Enhancement Proposals (GEPS) - Summary
|-
! GEPS - Title
! Proposed
! Status
! Released
! Note
! Revision Control
|-
| <!-- GEPS - Title -->GEPS 048: Format Negotiation Layer for Import-Export Plugins
| <!-- Proposed     -->2026-03-20
| <!-- Status       -->Draft
| <!-- Released     -->?
| <!-- Note         -->?
| <!-- SVN          -->
|} 
-->
== Type == 
Standards Track

== Authors ==
* ChatGPT (OpenAI GPT-5.3) — initial proposal and specification
* Claude (Anthropic, claude-sonnet-4-6) — editorial review, motivation
  expansion, workaround sections, collapsible formatting

== Abstract ==

Gramps users routinely encounter incompatibility, silent data loss, and
format confusion when importing or exporting files whose internal dialect
does not match what the importer expects. This GEPS proposes a Format
Negotiation Layer for import and export plugins: a dialect-aware
detection, routing, and transformation architecture that works across
GEDCOM, CSV, vCard, and Gramps XML. It introduces an extensible dialect
registry, a sniffer interface, and a processing pipeline. Two interim
workaround approaches are documented for use before core integration is
complete.

== Motivation ==

Gramps selects import/export plugins by file extension alone. This
ignores dialect variation — the differences in structure, encoding, and
vendor-specific tags that make two <code>.ged</code> files or two
<code>.csv</code> files require completely different handling. The result
is silent data loss, import failures, and no path to improve support
without modifying core plugins.

<div class="mw-collapsible mw-collapsed">
'''Affected formats and consequences''' (click to expand)
<div class="mw-collapsible-content">

* '''GEDCOM''' is a family of dialects, not a single format. GEDCOM 5.5,
  5.5.1, and 7.0 differ structurally. Major platforms (Ancestry,
  FamilySearch, Legacy Family Tree, MacFamilyTree) add proprietary tags
  and non-standard structures. Records associated with unknown tags are
  '''silently dropped''' by the current importer.

* '''CSV''' files vary in delimiter (<code>,</code> <code>;</code>
  <code>\t</code>), quoting convention, encoding (UTF-8, UTF-8-BOM,
  Windows-1252), and column schema. A file exported by one application
  may be misread by an importer expecting different conventions.

* '''vCard''' files exist in versions 2.1, 3.0, and 4.0, each with
  distinct encoding rules and property semantics.

* '''Gramps XML''' evolves across schema versions. There is currently no
  mechanism to export a lower schema version for compatibility with older
  Gramps installations, even when the database contains no data requiring
  the newer schema features.

New dialects emerge during development cycles. Flaws in existing dialect
handling are discovered after release. The system must be able to expand
without requiring changes to core plugins.

Related work:
* [[GEPS 037: Support GEDCOM X]]
* [[GEPS 043: Improving GEDCOM support for Places]]

These remain format-specific. A generalized solution is needed to avoid
repeated per-format effort and to establish a shared extensibility
pattern.

</div>
</div>

== Rationale ==

A Format Negotiation Layer decouples "file format" (the extension and
container) from "format interpretation" (the dialect, version, and
vendor conventions). New dialects can be supported by new addon plugins
without touching core importers. Experimental or vendor-specific support
can mature independently before integration.

<div class="mw-collapsible mw-collapsed">
'''Design capabilities provided by the layer''' (click to expand)
<div class="mw-collapsible-content">

* '''Automatic dialect detection''' — inspect file content before
  selecting a handler, rather than relying on extension alone
* '''Dynamic routing''' — direct files to the handler best suited for
  their detected dialect
* '''Graceful fallback''' — when a dialect is unrecognized, fall back to
  the closest known handler and preserve unknown data rather than
  discarding it
* '''Export targeting''' — allow users to export to a specific schema
  version or vendor profile for compatibility

</div>
</div>

== Specification ==

The Format Negotiation Layer consists of four components: a Dialect
Registry, a Sniffer Interface, a Processing Pipeline, and Dialect
Handlers. User configuration and edge-case handling (unknown dialects,
schema downgrade) round out the design.

<div class="mw-collapsible mw-collapsed">
'''Dialect Registry''' — maps formats to versioned dialect handlers
<div class="mw-collapsible-content">

Introduce a registry mapping formats to dialect handlers. Each entry is
defined by a unique identifier (e.g., <code>gedcom-5.5.1</code>,
<code>gedcom-ancestry</code>, <code>csv-excel</code>), a detection
function (sniffer), and a processing handler class. Dialects may be
registered by core or addon plugins. A new dialect means a new
registration, not a core modification.

<source lang="python">
DIALECT_REGISTRY = {
    "gedcom": {
        "5.5.1":    Gedcom551Dialect,
        "7.0":      Gedcom7Dialect,
        "ancestry": AncestryDialect,
    },
    "csv": {
        "excel":   CsvExcelDialect,
        "rfc4180": CsvRFCDialect,
    },
    "vcard": {
        "2.1": VCard21Dialect,
        "3.0": VCard30Dialect,
        "4.0": VCard40Dialect,
    },
    "gramps_xml": {
        "1.7": GrampsXML17,
        "1.8": GrampsXML18,
    },
}
</source>

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Sniffer Interface''' — detects format and dialect from file content
<div class="mw-collapsible-content">

Each format provides a sniffer that inspects the first 100–500 lines or
bytes and returns a detection result including format, dialect,
confidence score, and detected features.

<source lang="python">
class FormatSniffer:
    def detect(self, filename) -> DetectionResult:
        return {
            "format":     "gedcom",
            "dialect":    "5.5.1",
            "confidence": 0.92,
            "features":   {...},
        }
</source>

Detection signals by format:

* '''GEDCOM''': <code>HEAD.GEDC.VERS</code> for version;
  <code>HEAD.SOUR</code> for vendor; proprietary tags such as
  <code>_APID</code>, <code>_UID</code>, <code>_FSFTID</code>
* '''CSV''': delimiter, quoting style, encoding, header names
* '''vCard''': <code>VERSION</code> property, line folding, encoding
  declarations
* '''Gramps XML''': root namespace, schema version attribute

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Processing Pipeline''' — wraps core importers/exporters without modifying them
<div class="mw-collapsible-content">

'''Import pipeline:'''
<pre>
File → Sniffer → Dialect Selection → Preprocess → Core Importer → Postprocess → Database
</pre>

'''Export pipeline:'''
<pre>
Database → Dialect Selection → Pre-export Transform → Core Exporter → Post-export Transform → File
</pre>

Core importers and exporters are not modified. The negotiation layer
wraps them.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Dialect Handlers''' — per-dialect pre/post processing hooks
<div class="mw-collapsible-content">

Each dialect handler implements a standard four-method interface:

<source lang="python">
class Dialect:
    def preprocess(self, stream): ...
    def import_transform(self, data): ...
    def export_transform(self, data): ...
    def postprocess(self, data): ...
</source>

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''User Configuration''' — auto-detect, dialect override, export targeting
<div class="mw-collapsible-content">

Users may configure:

* '''Import mode''': Auto-detect (default), preferred dialect, or strict
  mode (fail on unrecognized dialect)
* '''Fallback dialect''': used when confidence falls below a threshold
* '''Export compatibility target''': select a schema version or vendor
  profile

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Backward Compatibility (Gramps XML)''' — optional schema downgrade on export
<div class="mw-collapsible-content">

During export, if the database contains no records requiring features
introduced after a given schema version, the exporter may offer export
to that older version. The exporter strips unsupported elements,
downgrades structures, and warns the user if any downgrade is lossy.

<source lang="python">
if not db_uses_features(requiring="1.8"):
    target_schema = "1.7"
</source>

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Unknown Dialect Handling''' — preserve rather than discard
<div class="mw-collapsible-content">

When a dialect is not recognized:

* Fall back to the closest known dialect
* Preserve unknown tags as object attributes or notes rather than
  discarding them (e.g.,
  <code>person.add_attribute(Attribute("_RAW__APID", value))</code>)
* Log anomalies for user and developer review
* Optionally generate a dialect report summarizing unrecognized
  constructs

</div>
</div>

== Workaround / Interim Approaches ==

Two approaches are available while this GEPS is being formally
implemented. Both require no changes to Gramps core. They are ordered
from lowest to highest investment.

=== Workaround A: Fork-and-Hide (Low Investment — Proof of Concept) ===

This approach tests the viability of dialect-enhanced import/export
without building a new plugin from scratch. The strategy is to fork
(copy) an existing built-in plugin, extend it with dialect handling, and
suppress the original to eliminate file-type ambiguity in the import
dialog.

<div class="mw-collapsible mw-collapsed">
'''Why fork rather than replace?''' The Gramps plugin system does not
support overriding a registered plugin. Registering a second importer
for the same extension creates ambiguity — both appear in the dialog.
Forking and then hiding the original resolves this cleanly.
<div class="mw-collapsible-content">

The current plugin registration model selects importers by file
extension and static label. There is no scoring or priority mechanism.
If two importers claim <code>.ged</code>, the user must choose manually
every time. Hiding the built-in removes the duplicate while keeping the
fork as the sole handler.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 1: Fork the built-in plugin''' (copy, rename, extend)
<div class="mw-collapsible-content">

Copy the target built-in plugin (e.g.,
<code>gramps/plugins/importer/importgedcom.py</code> and its
<code>.gpr.py</code> registration file) into a new addon directory.
Rename the plugin class and its registration label (e.g.,
<code>"GEDCOM (Enhanced)"</code>). Make dialect-handling changes
directly in the fork — additional tag handling, pre/post processing, or
a simple internal dialect branch — without touching the original.

This is the lowest-risk starting point: the fork starts as a working
importer identical to the built-in, and changes are incremental.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 2: Hide the original''' — manual or automatic
<div class="mw-collapsible-content">

'''Manual hiding''' (simplest): After installing the fork addon, the
user opens [[Plugin Manager Enhanced]], locates the original built-in
importer, and uses the "Hide" function to suppress it. The fork becomes
the sole handler for that file extension. This is reversible: unhiding
the built-in and hiding the fork restores original behavior.

'''Automatic hiding''' (more capable): The fork's
<code>.gpr.py</code> registration can include startup logic that hides
the upstream built-in programmatically, removing the need for manual
user action.

The fork knows the upstream plugin's identity (its registered name or
ID) and can call into the plugin manager's hidden-plugin list on
registration or first run:

<source lang="python">
# In the fork's .gpr.py or plugin __init__:
from gramps.gen.plug import PluginRegister

UPSTREAM_PLUGIN_ID = "im_gedcom"   # ID of the built-in GEDCOM importer

def _hide_upstream():
    pmgr = PluginRegister.get_instance()
    hidden = pmgr.get_hidden_plugin_ids()   # returns current hidden set
    if UPSTREAM_PLUGIN_ID not in hidden:
        hidden.add(UPSTREAM_PLUGIN_ID)
        pmgr.save_hidden_plugin_ids(hidden)

_hide_upstream()
</source>

The fork should also expose a preference or menu option to reverse which
plugin is hidden, so users can toggle back to the built-in if desired:

<source lang="python">
def toggle_active_importer(use_fork=True):
    pmgr = PluginRegister.get_instance()
    hidden = pmgr.get_hidden_plugin_ids()
    if use_fork:
        hidden.discard(FORK_PLUGIN_ID)
        hidden.add(UPSTREAM_PLUGIN_ID)
    else:
        hidden.discard(UPSTREAM_PLUGIN_ID)
        hidden.add(FORK_PLUGIN_ID)
    pmgr.save_hidden_plugin_ids(hidden)
</source>

Note: the exact API for reading and writing the hidden plugin list
should be verified against the current [[Plugin Manager Enhanced]]
implementation, as this capability may be accessed via its own API
rather than directly through <code>PluginRegister</code>.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Limitations of Workaround A'''
<div class="mw-collapsible-content">

* The fork must be kept in sync with upstream bug fixes and changes to
  the built-in — this is the primary maintenance cost
* Dialect logic lives inside the fork, not in a shared registry; a
  second format requires a second fork
* No sniffing infrastructure is shared across formats
* The automatic hide relies on [[Plugin Manager Enhanced]] being
  installed; it is not a core capability

This approach is best treated as a proof of concept and a way to
validate specific dialect-handling needs before investing in Workaround B
or the full GEPS implementation.

</div>
</div>

----

=== Workaround B: Addon-based Meta-Importer (Higher Investment — Full Interim Architecture) ===

This approach builds new addon plugins that stand alongside (rather than
forking) the built-in importers, using the same hide mechanism for
dialog disambiguation. It is more work upfront but produces reusable
infrastructure aligned with the full GEPS design.

<div class="mw-collapsible mw-collapsed">
'''Step 1: Build a Meta-Importer addon per format'''
<div class="mw-collapsible-content">

Register a new importer addon for each affected file extension with a
clear user-facing name (e.g., "GEDCOM (Auto-detect)", "CSV
(Auto-detect)"). This meta-importer inspects the file, selects a dialect
handler from an internal registry, and delegates. The existing core
importer is the default fallback.

<source lang="python">
class GedcomMetaImporter(ImportPlugin):
    def import_file(self, db, filename, user):
        meta = sniff(filename)
        handler = DIALECT_MAP.get(meta["dialect"], DefaultGedcomHandler)
        return handler().import_file(db, filename, user)
</source>

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 2: Suppress competing built-in via Plugin Manager Enhanced'''
<div class="mw-collapsible-content">

As with Workaround A, once the meta-importer is installed and verified,
hide the built-in importer for the same extension. This may be done
manually by the user or automatically by the addon on registration (see
Workaround A, Step 2 for the hide/toggle pattern). The meta-importer
becomes the sole entry point for that file type.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 3: User preferences for dialect override'''
<div class="mw-collapsible-content">

Expose a preference via the Gramps config namespace or a plugin
preferences dialog allowing the user to pin a specific dialect (e.g.,
"GEDCOM 5.5.1 strict", "Ancestry", "FamilySearch", "GEDCOM 7") or set a
fallback when auto-detection confidence falls below a threshold.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 4: Preserve unknown tags'''
<div class="mw-collapsible-content">

Store unrecognized tags as attributes with a <code>_RAW_</code> prefix
rather than discarding them. This preserves round-trip fidelity and
allows reprocessing when a proper handler becomes available.

<source lang="python">
person.add_attribute(Attribute("_RAW__APID", value))
</source>

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Step 5: Profile-driven export'''
<div class="mw-collapsible-content">

Register an enhanced exporter addon per format. Expose profile selection
(defined in bundled JSON or YAML files) in the export assistant. Profiles
cover tag inclusion/exclusion, encoding rules, and structural quirks for
target applications. For Gramps XML, offer "Export as 1.7 (maximum
compatibility)" when the database content permits it.

</div>
</div>

<div class="mw-collapsible mw-collapsed">
'''Limitations of Workaround B'''
<div class="mw-collapsible-content">

* New dialects require addon updates; there is no dynamic registration
  shared across format meta-importers
* No shared sniffer or registry infrastructure across formats (each
  meta-importer is self-contained)
* The "Hide" mechanism is global, not per-file
* User preferences are not portable across installations

These limitations are precisely what the full GEPS implementation
addresses.

</div>
</div>

----

== Backwards Compatibility ==

Existing plugins remain unchanged. The negotiation layer operates as an
optional meta-layer, implementable entirely via addon plugins. Integration
into core does not require removal or modification of existing importers
and exporters.

----

== Reference Implementation ==

'''Prototype scope''' 

A prototype implementation would include:

* Meta-importer addons per format (e.g., "GEDCOM Auto-detect")
* A dialect registry module
* Sniffer utilities for GEDCOM, CSV, vCard, and Gramps XML
* Bundled dialect handler classes for common dialects
* Integration with [[Plugin Manager Enhanced]] for visibility control

== Drawbacks ==

'''Known drawbacks and risks''' 

* '''Increased architectural complexity''': the negotiation layer adds
  indirection between file selection and parsing. Developers adding
  new format support must understand the registry and sniffer interface,
  not just the importer API.
* '''Dialect definition coordination''': dialect identifiers must be
  consistent across the registry, sniffers, and handlers. An informal
  process risks fragmentation.
* '''Potential duplication''': if only partially implemented (e.g., for
  GEDCOM only), logic may be duplicated rather than shared across
  formats.

== Alternatives ==

'''Alternatives considered'''

* '''Maintain current extension-based selection''': simple, but does not
  address dialect variation or cross-format extensibility.
* '''Format-specific improvements only''' (e.g., GEDCOM-only
  enhancements, as in GEPS 037 and GEPS 043): addresses immediate needs
  but requires repeated effort per format and does not establish a
  shared pattern.

== Future Work ==

'''Planned follow-on work''' 

In natural implementation sequence:

# Core integration of dialect registry (enabling dynamic registration
  without addon workarounds)
# Importer scoring and selection mechanism
  (<code>score = importer.probe(file)</code>; highest score wins)
# Shared dialect definition repository for community-maintained profiles
# UI feedback displaying detected dialect at import time
# "Re-import using different dialect" user action

== See Also ==

* [[GEPS 037: Support GEDCOM X]]
* [[GEPS 043: Improving GEDCOM support for Places]]
* Addon:[[Addon:Plugin_Manager|Plugin Manager Enhanced]]

----

== Credits ==

This proposal was developed with AI assistance in accordance with the
Gramps project policy on AI-generated contributions.

<div class="mw-collapsible mw-collapsed">
'''Initial proposal and specification''': ChatGPT (OpenAI GPT-5.3). (click to expand)
<div class="mw-collapsible-content">
  Prompt summary: The user requested a generalized, extensible framework
  for handling dialect variations across multiple file formats in Gramps,
  including dynamic detection, plugin augmentation, and backward
  compatibility support.
</div>
</div>
<div class="mw-collapsible mw-collapsed">
'''Editorial review, rationale expansion, workaround sections, and collapsible formatting''': Claude (Anthropic, claude-sonnet-4-6).
<div class="mw-collapsible-content">  
  Prompt summary: The user requested a second AI review of the GEPS for
  readability using MediaWiki collapsible sections, addition of a
  low-investment fork-and-hide workaround, and updated AI credits
  compliant with Gramps contributor policy.
</div>
</div>
'''Human author and project coordinator''': Brian McCullough  (emyoulation), Gramps project.
  
[[Category:GEPS]]