Database Build

We read many sources by many means in order to build the database browsed with El Dorado UI. We typically extract with shell scripts from APIs, Files, or occasionally scraping web pages. Transform and load follows advice from the Neo4j community.

See Data in Context for sample data for four hypothetical sources: org chart, source repos, architectural diagrams, and performance data. Your sources will surely differ.

See Tips for Modeling complex metadata.

We merge sources with a single ruby script that mostly constructs foreign keys where associations are not obvious and then writes csv files to be loaded into Neo4j.

We deploy Neo4j in Docker starting from a Neo4j image. We load from csv files, mark the database as read-only, then deploy as many instances as appropriate.

Michael Hunger & Mark Needham offer good advice for importing data. post

Menome describes Breaking Down Silos in enterprise data with Neo4j.